summaryrefslogtreecommitdiff
path: root/utils
Commit message (Collapse)AuthorAge
* [fix] issues reported by shellcheck v0.9.0Markus Heiser2024-01-09
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] get rid of ./utils/brand.env and its workflowMarkus Heiser2024-01-09
| | | | | | | | All the environments defined in ./utils/brand.env are generated on the fly, so there is no longer a need to define the brand environment in this file and all the workflows to handle this file. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] ./utils/get_setting.py tiny YAML parser for settings.ymlAlexandre Flament2024-01-09
| | | | | | | | | | | | | This allow to read settings on the fly even without virtualenv. The ultimate goal of the commit is to remove utils/brand.env from the git repository. The code includes a tiny yaml parser that **should** be good enough. The code read searx/settings.yml directly (and ignore the environment variables). yq [1] is a more reliable alternative but this require to download a binary from github which is not great. [1] https://github.com/mikefarah/yq/#install
* Change the brand.env file to the original stateCoccocoa's Helper2023-11-28
|
* [feat] engine: implementation of bpbBnyro2023-11-27
|
* Docker: add UWSGI_WORKERS and UWSGI_THREAD environment variables (#2992)Alexandre Flament2023-11-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Docker: add UWSGI_WORKERS and UWSGI_THREAD. UWSGI_WORKERS specifies the number of process. UWSGI_THREADS specifies the number of threads. The Docker convention is to specify the whole configuration through environment variables. While not done in SearXNG, these two additional variables allows admins to skip uwsgi.ini In additional, https://github.com/searxng/preview-environments starts Docker without additional files through searxng-helm-chat. Each instance consumes 1Go of RAM which is a lot especially when there are a lot of instances / pull requests. * [scripts] add environments UWSGI_WORKERS and UWSGI_THREADS - UWSGI_WORKERS specifies the number of process. - UWSGI_THREADS specifies the number of threads. Templates for uwsgi scripts can be tested by:: UWSGI_WORKERS=8 UWSGI_THREADS=9 \ ./utils/searxng.sh --cmd\ eval "echo \"$(cat utils/templates/etc/uwsgi/*/searxng.ini*)\""\ | grep "workers\|threads" Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> --------- Signed-off-by: Markus Heiser <markus.heiser@darmarit.de> Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] settings.yml: remove plugin settings for plugins that don't exist anymoreBnyro2023-09-29
|
* [fix] harmonize default settings for caching of /static filesMarkus Heiser2023-09-25
| | | | | | | | | | | | | | | | | | | | | Caching files on the client side for more than a day can confuse the end user when updating static files[1]. Depending on the way of providing a SearXNG instance via HTTP, there are several ways to optimize the access to the /static files. However, since we don't know what optimization an admin has provided for his static files, we should have moderate settings in the defaults that run robustly in a wide variety of installations. In this sense, all caches on the client side should be cleared after one day at the latest. So far the files were cached for one year on client side; as soon as changes are made to the static files (with the option `static_use_hash: true`) the old static files are kept for one year on the CLient side / which can also be evaluated as unnecessary caching. [1] https://github.com/searxng/searxng/discussions/2821 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Revert "[mod] activate limiter & link_token method (aka CSS ping) by default"Emilien Devos2023-09-23
| | | | This reverts commit 3af629ec090d1ab135fbd7c9b45088c8ca4e541e.
* [mod] activate limiter & link_token method (aka CSS ping) by defaultMarkus Heiser2023-09-23
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] accidentally pushed utils/brand.envMarkus Heiser2023-09-18
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [feat] engine: implementation of internet archive scholarBnyro2023-09-18
|
* [fix] spellingjazzzooo2023-09-18
|
* [fix] typos in documentation & messagesToxyFlog16272023-08-13
|
* uwsgi.ini: remove unused cache2Alexandre Flament2023-08-13
| | | | cache2 was used before PR #1856
* [doc] improve documentation of make targets and ./manage scriptMarkus Heiser2023-08-06
| | | | | | | | | | | | BTW force modularization of the ./mange script into sub modules: - utils/lib_sxng_data.sh - utils/lib_sxng_node.sh - utils/lib_sxng_static.sh - utils/lib_sxng_test.sh - utils/lib_sxng_themes.sh Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [doc] update & fix documentation of the "SearXNG LXC suite"Markus Heiser2023-05-21
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] Google: reversed engineered & upgrade to data_type: traits_v1Markus Heiser2023-03-24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Partial reverse engineering of the Google engines including a improved language and region handling based on the engine.traits_v1 data. When ever possible the implementations of the Google engines try to make use of the async REST APIs. The get_lang_info() has been generalized to a get_google_info() function / especially the region handling has been improved by adding the cr parameter. searx/data/engine_traits.json Add data type "traits_v1" generated by the fetch_traits() functions from: - Google (WEB), - Google images, - Google news, - Google scholar and - Google videos and remove data from obsolete data type "supported_languages". A traits.custom type that maps region codes to *supported_domains* is fetched from https://www.google.com/supported_domains searx/autocomplete.py: Reversed engineered autocomplete from Google WEB. Supports Google's languages and subdomains. The old API suggestqueries.google.com/complete has been replaced by the async REST API: https://{subdomain}/complete/search?{args} searx/engines/google.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. - always use the async REST API (formally known as 'use_mobile_ui') - use *supported_domains* from traits - improved the result list by fetching './/div[@data-content-feature]' and parsing the type of the various *content features* --> thumbnails are added searx/engines/google_images.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - if exists, freshness_date is added to the result - issue 1864: result list has been improved a lot (due to the new cr parameter) searx/engines/google_news.py Reverse engineering and extensive testing .. - fetch_traits(): Fetch languages & regions from Google properties. *supported_domains* is not needed but a ceid list has been added. - different region handling compared to Google WEB - fixed for various languages & regions (due to the new ceid parameter) / avoid CONSENT page - Google News do no longer support time range - result list has been fixed: XPath of pub_date and pub_origin searx/engines/google_videos.py - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - add paging support - implement a async request ('asearch': 'arc' & 'async': 'use_ac:true,_fmt:html') - simplified code (thanks to '_fmt:html' request) - issue 1359: fixed xpath of video length data searx/engines/google_scholar.py - fetch_traits(): Fetch languages & regions from Google properties. - use *supported_domains* from traits - request(): include patents & citations - response(): fixed CAPTCHA detection (Scholar has its own CATCHA manager) - hardening XPath to iterate over results - fixed XPath of pub_type (has been change from gs_ct1 to gs_cgt2 class) - issue 1769 fixed: new request implementation is no longer incompatible Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [doc] slight improvements to the doc of the settings (base_url)Markus Heiser2023-02-17
| | | | | | Closes: https://github.com/searxng/searxng/issues/2190 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] make nvm.env exit with error when VERBOSE is unsetMarkus Heiser2023-02-11
| | | | | | | | | | | | | | | | | | | $ make nvm.install INFO: install (update) NVM at /800GBPCIex4/share/SearXNG/.nvm INFO: already cloned at: /800GBPCIex4/share/SearXNG/.nvm || Fetching origin INFO: checkout v0.39.1 || HEAD is now at 9600617 v0.39.1 make: *** [Makefile:96: nvm.install] Error 1 Without this fix we need to set VERBOSE environment to avoid the 'Error 1': $ VERBOSE=0 make nvm.install BTW: fix an issue if there are any leftovers in ${NVM_DIR} from previos installations Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [LXC] ubuntu 21.10 upgrade to ubuntu 22.04 LTS [1]Markus Heiser2023-02-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [1] https://wiki.ubuntu.com/Releases Tested by:: # build the container .. $ sudo -H ./utils/lxc.sh build searxng-ubu2204 # install a complete SearXNG suite .. $ sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh install all # install nginx to export the SearXNG instance by HTTP $ sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh install nginx # check instance $ sudo -H ./utils/lxc.sh cmd searxng-ubu2204 FORCE_TIMEOUT=0 ./utils/searxng.sh instance check INFO: [searxng-ubu2204] FORCE_TIMEOUT=0 ./utils/searxng.sh instance check INFO: wrapper: utils/searxng.sh instance _call searxng.check SearXNG checks -------------- ... INFO searx : merge the default settings ( /usr/local/searxng/searxng-src/searx/settings.yml ) and the user settings ( /etc/searxng/settings.yml ) INFO searx : max_request_timeout=None INFO searx.redisdb : connecting to Redis db=0 path='/usr/local/searxng-redis/run/redis.sock' INFO searx.redisdb : connected to Redis Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] typos in utils/searxng.shMilad-Laly2023-01-08
|
* Fix checksum virtualenv checks.llmII2023-01-03
| | | | | | | There's already precedence for not using GNUism sha256sum longopts as seen in searxng/utils/lib_go.sh so update lib.sh to not use them either. A nice side effect is now the sha256sum usage doesn't care if you're using BSD sha256sum or GNU sha256sum which makes this work under FreeBSD.
* move searx.shared.redisdb to searx.redisdbAlexandre Flament2022-12-10
|
* Initialize Redis in searx/webapp.pyAlexandre FLAMENT2022-11-05
| | | | | | | | | | | | | | settings.yml: * The default URL was unix:///usr/local/searxng-redis/run/redis.sock?db=0 * The default URL is now "false" The default URL makes the log difficult to deal with: if the admin didn't install a Redis instance, the logs record a false error. It worked before because SearXNG initialized the Redis connection when the limiter started. In this commit, SearXNG initializes Redis in searx/webapp.py so various components can use Redis without taking care of the initialization step.
* [fix] typos / reported by @kianmeng in searx PR-3366Markus Heiser2022-09-27
| | | | | | [PR-3366] https://github.com/searx/searx/pull/3366 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] uWSGI: increase buffer-sizeLéon Tiekötter2022-07-31
| | | | | | | Increase max size of a request, by default it is 4k [1]. 4096 as buffer-size is too small and will result in the preference urls not working. [1] https://uwsgi-docs.readthedocs.io/en/latest/Options.html#buffer-size
* [mod] fix minor leftovers from PR #1332Markus Heiser2022-07-30
| | | | | Related: https://github.com/searxng/searxng/pull/1332 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [clean up] drop obsolete searx, filtron and morty install scriptsMarkus Heiser2022-07-30
| | | | | | | | | | Since ./utils/searxng.sh is implemented, the old installation procedures from filtron, morty and searx can be removed. For users who want to upgrade, the procedures for removing old installations have still been retained. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [doc] update documentation of the installation proceduresMarkus Heiser2022-07-30
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [utils/searxng.sh] implement new script to install SearXNGMarkus Heiser2022-07-30
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [install scripts] rename SEARX_<name> variables to SEARXNG_<name>Markus Heiser2022-07-30
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* theme: remove __common__Alexandre Flament2022-05-07
|
* [theme] peel out oscar from SearXNG developmentMarkus Heiser2022-04-30
| | | | | | This is the first step of removing oscar theme Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] since git v2.35.2 root can't read from unsafe repositoryMarkus Heiser2022-04-22
| | | | | | | | | | | | Git v2.35.2 closes an security issue, it is no longer possible that root uses a git repo that is owned by someone else, the error message is:: fatal: unsafe repository ('/share/darmarit.org/cache/searxng' is owned by someone else) The fix is to run the `git diff --name-only` not as root in a sudo command. [1] https://github.blog/2022-04-12-git-security-vulnerability-announced/ Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* typo: wanr_msg -> warn_msgHZH2022-04-13
|
* don't reinstall existing packages on archJabster282022-04-11
|
* [LXC] ubuntu 20.10 EOL, upgrade to ubunut 21.10 [1]Markus Heiser2022-03-12
| | | | | | | | | | | | | | | | | | | | | | | | [1] https://wiki.ubuntu.com/Releases Tested by:: # build the container .. $ sudo -H ./utils/lxc.sh build searx-ubu2110 # open a shell in the container $ sudo -H ./utils/lxc.sh cmd searx-ubu2110 bash [root@searx-ubu2110 SearXNG]# # install a complete SearXNG suite .. [root@searx-ubu2110 SearXNG]# ./utils/searx.sh install all ... # install apache to export the SearXNG instance by HTTP [root@searx-ubu2110 SearXNG]# ./utils/searx.sh apache install ... INFO: got 200 from http://10.174.184.94/searx Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [LXC] fedora33 EOL [1], upgrade to fedora35 [2]Markus Heiser2022-03-12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | [1] https://docs.fedoraproject.org/en-US/releases/eol/ [2] https://docs.fedoraproject.org/en-US/releases/f35/ Tested by:: # build the container .. $ sudo -H ./utils/lxc.sh build searx-fedora35 # open a shell in the container $ sudo -H ./utils/lxc.sh cmd searx-fedora35 bash [root@searx-fedora35 SearXNG]# # install a complete SearXNG suite .. [root@searx-fedora35 SearXNG]# ./utils/searx.sh install all ... # install apache to export the SearXNG instance by HTTP [root@searx-fedora35 SearXNG]# ./utils/searx.sh apache install ... INFO: got 200 from http://10.174.184.94/searx To build wheel `python3-devel` needs to be added to SEARX_PACKAGES_fedora:: |searx| × Building wheel for setproctitle (pyproject.toml) did not run successfully. |searx| │ exit code: 1 ... |searx| In file included from src/spt.h:15, |searx| from src/setproctitle.c:14: |searx| src/spt_python.h:16:10: fatal error: Python.h: No such file or directory |searx| 16 | #include <Python.h> |searx| | ^~~~~~~~~~ |searx| compilation terminated. |searx| error: command '/usr/bin/gcc' failed with exit code 1 |searx| [end of output] ... |searx| ERROR: Failed building wheel for setproctitle Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] Serving static files with uWSGI (searxng.ini)Markus Heiser2022-03-08
| | | | | | | | | | 1. Serving static files with uWSGI by using static file mount points [1]. 2. Expires set to one year since there are hashes [2] [1] https://uwsgi-docs.readthedocs.io/en/latest/StaticFiles.html#mode-3-using-static-file-mount-points [2] https://github.com/searxng/searxng/pull/932 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] socket in SearXNG's uWSGI app (searxng.ini)Markus Heiser2022-03-08
| | | | | | Use SEARX_UWSGI_SOCKET in uWSGI systemd service Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] APACHE_SEARX_SITE=searxng.conf renamed in 2b125214Markus Heiser2022-03-08
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Merge pull request #804 from return42/minor-fixMartin Fischer2022-01-24
|\ | | | | [mod] lib_nvm.sh: minor improvements / no functional change
| * [mod] lib_nvm.sh: minor improvements / no functional changeMarkus Heiser2022-01-22
| | | | | | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* | [mod] introduce node.env.devtools functionMarkus Heiser2022-01-24
| |
* | [mod] remove .nvm_packages, add eslint to package.jsonMartin Fischer2022-01-23
|/
* Merge pull request #686 from return42/lib_redisMarkus Heiser2022-01-11
|\ | | | | Add redis DB and connector
| * [mod] script to build & install a redis instanceMarkus Heiser2022-01-03
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A script to build & install a simple & isolated redis service, dedicated to SearXNG and connected via Unix socket. $ ./manage redis.help redis.: devpkg : install essential packages to compile redis build : build redis binaries at /800GBPCIex4/share/SearXNG/dist/redis/6.2.6/amd64 install : create user (searxng-redis) and install systemd service (searxng-redis) remove : delete user (searxng-redis) and remove service (searxng-redis) shell : start bash interpreter from user searxng-redis src : clone redis source code to <path> and checkput 6.2.6 useradd : create user (searxng-redis) at /usr/local/searxng-redis userdel : delete user (searxng-redis) addgrp : add <user> to group (searxng-redis) rmgrp : remove <user> from group (searxng-redis) Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* | reference docs.searxng.orgAlexandre Flament2022-01-02
|/
* [mod] make static.build.commit more robustMarkus Heiser2021-12-01
| | | | | | | - use single quote in the STATIC_BUILT_PATHS to avoid bash globbing - don't try to commit if no files have been changed Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-02 23:04:14 +0000