summaryrefslogtreecommitdiff
path: root/tests
Commit message (Collapse)AuthorAge
* [mod] isolation of botdetection from the limiterMarkus Heiser2023-11-01
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch was inspired by the discussion around PR-2882 [2]. The goals of this patch are: 1. Convert plugin searx.plugin.limiter to normal code [1] 2. isolation of botdetection from the limiter [2] 3. searx/{tools => botdetection}/config.py and drop searx.tools 4. in URL /config, 'limiter.enabled' is true only if the limiter is really enabled (Redis is available). This patch moves all the code that belongs to botdetection into namespace searx.botdetection and code that belongs to limiter is placed in namespace searx.limiter. Tthe limiter used to be a plugin at some point botdetection was added, it was not a plugin. The modularization of these two components was long overdue. With the clear modularization, the documentation could then also be organized according to the architecture. [1] https://github.com/searxng/searxng/pull/2882 [2] https://github.com/searxng/searxng/pull/2882#issuecomment-1741716891 To test: - check the app works without the limiter, check `/config` - check the app works with the limiter and with the token, check `/config` - make docs.live .. and read - http://0.0.0.0:8000/admin/searx.limiter.html - http://0.0.0.0:8000/src/searx.botdetection.html#botdetection Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] HTMLParser: undocumented not implemented methodMarkus Heiser2023-10-22
| | | | | | | | | | | | | | | In python versions <py3.10 there is an issue with an undocumented method HTMLParser.error() [1][2] that was deprecated in Python 3.4 and removed in Python 3.5. To be compatible to higher versions (>=py3.10) an error method is implemented which throws an AssertionError exception like the higher Python versions do [3]. [1] https://github.com/python/cpython/issues/76025 [2] https://bugs.python.org/issue31844 [3] https://github.com/python/cpython/pull/8562 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* better error message when no results foundEmilien Devos2023-09-19
|
* [feat] implement feeling lucky featureBnyro2023-09-19
|
* [fix] spellingjazzzooo2023-09-18
|
* [fix] make flask_babel.gettext() work in engine modules (L10n & threads)Markus Heiser2023-08-09
| | | | | | | | | | | | | | | | | | | | | | | | incident: flask_babel.gettext() does not work in the engine modules. cause: the request() and response() functions of the engine modules run in the processor, whose search() method runs in a thread and in the threads the context of the Flask app does not exist. The context of the Flask app is needed by the gettext() function for the L10n. Solution: copy context of the Flask app into the threads. [1] special case: We cannot equip the search() method of the processors with the decorator [1], because the decorator requires a context (Flask app) that does not yet exist at the time of the initialization of the processors (the initialization of the processors is part of the initialization of the Flask app). [1] https://flask.palletsprojects.com/en/2.3.x/api/#flask.copy_current_request_context Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] move some code from webapp module to webutils module (no functional ↵Markus Heiser2023-06-19
| | | | | | | | | | | | change) Over the years the webapp module became more and more a mess. To improve the modulaization a little this patch moves some implementations from the webapp module to webutils module. HINT: this patch brings non functional change Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Merge pull request #2357 / limiter -> botdetectionMarkus Heiser2023-06-03
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The monolithic implementation of the limiter was divided into methods and implemented in the Python package searx.botdetection. Detailed documentation on the methods has been added. The methods are divided into two groups: 1. Probe HTTP headers - Method http_accept - Method http_accept_encoding - Method http_accept_language - Method http_connection - Method http_user_agent 2. Rate limit: - Method ip_limit - Method link_token (new) The (reduced) implementation of the limiter is now in the module searx.botdetection.limiter. The first group was transferred unchanged to this module. The ip_limit contains the sliding windows implemented by the limiter so far. This merge also fixes some long outstandig issue: - limiter does not evaluate the Accept-Language correct [1] - limiter needs a IPv6 prefix to block networks instead of IPs [2] Without additional configuration the limiter works as before (apart from the bugfixes). For the commissioning of additional methods (link_toke), a configuration must be made in an additional configuration file. Without this configuration, the limiter runs as before (zero configuration). The ip_limit Method implements the sliding windows of the vanilla limiter, additionally the link_token method can be used in this method. The link_token method can be used to investigate whether a request is suspicious. To activate the link_token method in the ip_limit method add the following to your /etc/searxng/limiter.toml:: [botdetection.ip_limit] link_token = true [1] https://github.com/searxng/searxng/issues/2455 [2] https://github.com/searxng/searxng/issues/2477
| * [fix] correct determination of the IP for the requestMarkus Heiser2023-06-01
| | | | | | | | | | | | | | | | | | | | | | | | | | For correct determination of the IP to the request the function botdetection.get_real_ip() is implemented. This fonction is used in the ip_limit and link_token method of the botdetection and it is used in the self_info plugin. A documentation about the X-Forwarded-For header has been added. [1] https://github.com/searxng/searxng/pull/2357#issuecomment-1566211059 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* | [mod] template preferences: split into elements (no functional change)Markus Heiser2023-06-02
|/ | | | | | | | | | | | | | | | | | HINT: this patch has no functional change / it is the preparation for following changes and bugfixes Over the years, the preferences template became an unmanageable beast. To make the source code more readable the monolith is splitted into elements. The splitting into elements also has the advantage that a new template can make use of them. The reversed checkbox is a quirk that is only used in the prefereces and must be eliminated in the long term. For this the macro 'checkbox_onoff_reversed' was added to the preferences.html template. The 'checkbox' macro is also a quirk of the preferences.html we don't want to use in other templates (it is an input-checkbox in a HTML form that was misused for status display). Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] replace utils.match_language by locales.match_localeMarkus Heiser2023-03-24
| | | | | | | | | | | | | | | | | | | | | | | | This patch replaces the *full of magic* ``utils.match_language`` function by a ``locales.match_locale``. The ``locales.match_locale`` function is based on the ``locales.build_engine_locales`` introduced in 9ae409a0 [1]. In the past SearXNG did only support a search by a language but not in a region. This has been changed a long time ago and regions have been added to SearXNG core but not to the engines. The ``utils.match_language`` was the function to handle the different aspects of language/regions in SearXNG core and the supported *languages* in the engine. The ``utils.match_language`` did it with some magic and works good for most use cases but fails in some edge case. To replace the concurrence of languages and regions in the SearXNG core the ``locales.build_engine_locales`` was introduced in 9ae409a0 [1]. With the last patches all engines has been migrated to a ``fetch_traits`` and a language/region concept that is based on ``locales.build_engine_locales``. To summarize: there is no longer a need for the ``locales.match_language``. [1] https://github.com/searxng/searxng/pull/1652 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Add "Auto-detected" as a language.Alexandre Flament2023-02-17
| | | | | | | | | | | When the user choose "Auto-detected", the choice remains on the following queries. The detected language is displayed. For example "Auto-detected (en)": * the next query language is going to be auto detected * for the current query, the detected language is English. This replace the autodetect_search_language plugin.
* Merge pull request #2109 from ahmad-alkadri/fix/highlight-full-wordAlexandre Flament2023-01-17
|\ | | | | Standalone words highlighting for query result in non-CJK characters
| * A little fix and modified the testing for content highlightahmad-alkadri2023-01-15
| |
* | Add search.suspended_times settingsLéon Tiekötter2023-01-15
|/ | | | | | | Make suspended_time changeable in settings.yml Allow different values to be set for different exceptions. Co-authored-by: Alexandre Flament <alex@al-f.net>
* Replace langdetect with fasttextArtikusHG2022-12-16
|
* Fix: don't crash when engine or name is missing in settings.ymlAlexandre Flament2022-12-04
| | | | | | | SearXNG crashes if the engine or name fields are missing. With this commit, the app displays an error in the log and keeps loading. Close #1951
* Merge branch 'searxng:master' into masterMohamed Elashri2022-09-30
|\
| * [fix] typos / reported by @kianmeng in searx PR-3366Markus Heiser2022-09-27
| | | | | | | | | | | | [PR-3366] https://github.com/searx/searx/pull/3366 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* | correct sci-hub links/ add `.ru` and remove other 3rd party domains.Mohamed Elashri2022-09-24
|/
* Remove usage of SEARX environment variablesAlexandre FLAMENT2022-08-28
|
* [fix] improve OpenSearch descriptionMarkus Heiser2022-08-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some HTTP-Clients do have issues with the ``opensearch.xml`` from SearXNG (related [1][2]) while other OpenSearch descriptions[3] (e.g. from qwant) work flawles. Inspired by the OpenSearch description from qwant and with informations from the specification[4] the ``opensearch.xml`` has been *improved*. - convert `<Url>` methods from lower case to upper case (`POST`|`GET`) - add `<moz:SearchForm>` and `xmlns:moz="http://www.mozilla.org/2006/browser/search/"` - add `<Query role="example" searchTerms="SearXNG" />` [4] OpenSearch description documents should include at least one Query element of `role="example"` that is expected to return search results. Search clients may use this example query to validate that the search engine is working properly. - modified `<LongName>` to SearXNG - modified `<Description>` the word 'hackable' scares uninitiated users and was removed - add the `type="image/png"` to `<Image>` Test can be done by:: make run Visit http://127.0.0.1:8888/ and add the search engine to your WEB-Browser / test with different WEB-Browser from desktop and Smartphones (are there any iOS user here, please test on Safari and Chrome). [1] https://app.element.io/#/room/#searxng:matrix.org/$xN_abdKhNqUlgXRBrb_9F3pqOxnSzGQ1TG0s0G9hQVw [2] https://github.com/searxng/searxng/issues/431 [3] https://developer.mozilla.org/en-US/docs/Web/OpenSearch [4] https://github.com/dewitt/opensearch/blob/master/opensearch-1-1-draft-6.md#the-query-element Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* bing.py: resolve bing.com/ck/a redirectionsAlexandre Flament2022-07-08
| | | | add a new function searx.network.multi_requests to send multiple HTTP requests at once
* Bump splinter from 0.17.0 to 0.18.0dependabot[bot]2022-06-18
| | | | | | | | | | | | | | | | | Bumps [splinter](https://github.com/cobrateam/splinter) from 0.17.0 to 0.18.0. - [Release notes](https://github.com/cobrateam/splinter/releases) - [Changelog](https://github.com/cobrateam/splinter/blob/master/docs/news.rst) - [Commits](https://github.com/cobrateam/splinter/compare/0.17.0...0.18.0) --- updated-dependencies: - dependency-name: splinter dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Alexandre Flament <alex@al-f.net>
* [fix] move locale code from webapp.py to locales.py and fix #1303Markus Heiser2022-06-12
| | | | | | | | | | | | | | | | | | | | To improve modularization this patch: - moves *locale* related implementation from the webapp.py application to the locale.py module. - The initialization of the locales is now done in the application (webapp) and is no longer done while importing searx.locales. In the searx.locales module a new dictionary named `LOCALE_BEST_MATCH` has been added. In this dictionary we can map languages without a translation to languages we have a translation for. To fix #1303 zh-HK has been mapped to zh-Hant-TW (we do not need additional translations of traditional Chinese) Closes: https://github.com/searxng/searxng/issues/1303 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* theme: remove __common__Alexandre Flament2022-05-07
|
* [theme] peel out oscar from SearXNG developmentMarkus Heiser2022-04-30
| | | | | | This is the first step of removing oscar theme Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] dailymotion engine: filter by language & countryMarkus Heiser2022-04-16
| | | | | | | | | | | | | | | | | | - fix the issue of fetching more the 7000 *languages* - improve the request function and filter by language & country - implement time_range_support & safesearch - add more fields to the response from dailymotion (allow_embed, length) - better clean up of HTML tags in the 'content' field. This is more or less a complete rework based on the '/videos' API from [1]. This patch cleans up the language list in SearXNG that has been polluted by the ISO-639-3 2 and 3 letter codes from dailymotion languages which have never been used. [1] https://developers.dailymotion.com/tools/ Closes: https://github.com/searxng/searxng/issues/1065 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] add missing territory (country) nameMarkus Heiser2022-04-05
| | | | | Related-to: https://github.com/searxng/searxng/pull/1029#issuecomment-1086824911 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [fix] issue when upgrading from werkzeug v2.0.3 to v2.1.0Markus Heiser2022-04-01
| | | | | | | | | | | | | | | | | | | | | | | In v2.1.0 werkzeug [1] fixed an issue [2] to keep relative redirect locations by default [3]. Since relative locations are returned, we need to fix out test cases to avoid AssertionErrors like this one:: ====================================================================== FAIL: test_index_html_get (tests.unit.test_webapp.ViewsTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/runner/work/searxng/searxng/tests/unit/test_webapp.py", line 105, in test_index_html_get self.assertEqual(result.location, 'http://localhost/search?q=test') AssertionError: '/search?q=test' != 'http://localhost/search?q=test' - /search?q=test + http://localhost/search?q=test [1] https://werkzeug.palletsprojects.com/ [2] https://github.com/pallets/werkzeug/issues/2352 fixed in [3] https://github.com/pallets/werkzeug/pull/2354 Related-to: https://github.com/searxng/searxng/pull/1039#issuecomment-1085538288 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Merge pull request #894 from dalf/simple-aria-1Alexandre Flament2022-03-19
|\ | | | | [enh] simple: basic ARIA fixes
| * [enh] simple: basic ARIA fixesAlexandre Flament2022-03-08
| |
* | [mod] replace /help by /info pages and include pages in project docsMarkus Heiser2022-03-12
|/ | | | | | | | | | | | | | | | | This patch implements a bolierplate to share content from info-pages of the SearXNG instance (URL /info) with the project documentation (path /docs/user). The info pages are using Markdown (CommonMark), to include them in the project documentation (reST) the myst-parser [1] is used in the Sphinx-doc build chain. If base_url is known (defined in settings.yml) links to the instance are also inserted into the project documentation:: searxng_extra/docs_prebuild [1] https://www.sphinx-doc.org/en/master/usage/markdown.html Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* [mod] URL for the static file contains the sha1Alexandre Flament2022-03-05
| | | | | * allow to cache the static file forever * avoid bugs when the static files are updated but not reloaded
* [simple theme] replace Image_layout.js with flexbox CS impl.mrpaulblack2022-02-26
| | | | | | | | | | | | | | * drop image_layout.js from simple theme * move image_layout.js to oscar theme and delete common js dir (since its empty now) * align top position of image detail modal with bottom position of search header * use flexbox to display images; row height can be set via @results-image-row-height in defenitions.less * display span title underneath each image with a max width of 12rem * increase margin and padding around image article on desktop and tablet * make article height smaller on phone layout (height of 6rem) to display more content on current view * remove content from result, if the title and content matches * use a group that cotains the flex image article, if images are mixed with other categories * fix pylint issues in webapp.py * use the default.html result template in unit tests (thanks @return42)
* [fix] changeover of the unit tests to the simple themeMarkus Heiser2022-02-20
| | | | Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Merge pull request #848 from not-my-profile/help-routeAlexandre Flament2022-02-05
|\ | | | | Introduce `/help` route
| * [enh] introduce /help routeMartin Fischer2022-02-01
| | | | | | | | | | Translation will be implemented in the future. For now the "en" in /help/en/<pagename> is hardcoded.
| * [help] remove link from about.md titleMartin Fischer2022-02-01
| | | | | | | | | | | | | | Now that about.html extends page_with_header.html it already has a link to the start page and removing the link makes it easier to extract the page title from the Markdown for the following commit.
* | Merge pull request #839 from dalf/docs_searx_utilsMarkus Heiser2022-02-04
|\ \ | | | | | | [mod] add documentation about searx.utils
| * | [mod] add documentation about searx.utilsAlexandre Flament2022-01-29
| |/ | | | | | | | | | | | | This module is a toolbox for the engines. Is should be documented. In addition, searx/utils.py is checked by pylint.
* / preferences: Set autocomplete=off for formMartin Fischer2022-02-04
|/ | | | | | Otherwise you can change the value of a select, refresh the page and the preferences stay changed, leaving the wrong impression that they were saved.
* [typing] add results.TimingMartin Fischer2022-01-17
|
* [fix] external bangs: don't overwrite Bangs in data trieMarkus Heiser2022-01-12
| | | | | | | | | | Bangs with a `*` suffix (e.g. `!!d*`) overwrite Bangs with the same prefix (e.g. `!!d`) [1]. This can be avoid when a non printable character is used to tag a LEAF_KEY. [1] https://github.com/searxng/searxng/pull/740#issuecomment-1010411888 Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* Merge pull request #708 from not-my-profile/pref-refactorMartin Fischer2022-01-07
|\ | | | | Refactor `preferences`
| * [refactor] refactor SwitchableSettingMartin Fischer2022-01-06
| | | | | | | | | | | | | | | | The previous implementation used two hash sets and a list. ... that's not necessary ... a single hash map suffices. And it's also less error prone ... because the previous data structure allowed a setting to be enabled and disabled at the same time.
| * [refactor] add type hints & remove Setting._post_initMartin Fischer2022-01-06
| | | | | | | | | | Previously the Setting classes used a horrible _post_init hack that prevented proper type checking.
* | [fix] remove broken ? search operatorMartin Fischer2022-01-06
|/ | | | | | | | | | | | | | | | | The ? search operator has been broken for some time and currently only raises the question why it's still there. ## Context ## The query "Paris !images" searches for "Paris" in the "images" category. Once upon a time Searx supported "Paris ?images" to search for "Paris" in the currently enabled categories and the "images" category. The feature makes sense ... the ? syntax does not. We will hopefully introduce a +!images syntax in the future. Fixes #702.
* Merge pull request #634 from not-my-profile/powered-byAlexandre Flament2022-01-06
|\ | | | | Introduce `categories_as_tabs` & group engines in tabs
| * [enh] change categories_as_tabs from a list to a dictMartin Fischer2022-01-05
| | | | | | | | | | | | | | | | | | The tab icon names are currently hard coded in the templates. This commit lets us introduce an icon property in the future, e.g: categories_as_tabs: general: icon: search-outline
generated by cgit v1.2.3 (git 2.39.1) at 2024-06-01 04:14:22 +0000