[doc] improve documentation of make targets and ./manage script

BTW force modularization of the ./mange script into sub modules:

- utils/lib_sxng_data.sh
- utils/lib_sxng_node.sh
- utils/lib_sxng_static.sh
- utils/lib_sxng_test.sh
- utils/lib_sxng_themes.sh

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

7 files changed, 236 insertions, 181 deletions

diff --git a/docs/admin/buildhosts.rst b/docs/admin/buildhosts.rst
index d31e4b79..b7f317f7 100644
--- a/docs/admin/buildhosts.rst
+++ b/docs/admin/buildhosts.rst
@@ -4,22 +4,30 @@
 Buildhosts
 ==========
 
-.. sidebar:: This article needs some work
-
-   If you have any contribution send us your :pull:`PR <../pulls>`, see
-   :ref:`how to contribute`.
-
 .. contents::
    :depth: 2
    :local:
    :backlinks: entry
 
-To get best results from build, it's recommend to install additional packages
-on build hosts (see :ref:`searxng.sh`).::
+To get best results from build, it's recommend to install additional packages on
+build hosts (see :ref:`searxng.sh`).
+
+.. _searxng.sh install buildhost:
+
+Build and Development tools
+===========================
+
+To Install tools used by build and development tasks in once:
+
+.. tabs::
+
+  .. group-tab:: SearXNG's development tools
+
+     .. code:: sh
 
-  sudo -H ./utils/searxng.sh install buildhost
+        $ sudo -H ./utils/searxng.sh install buildhost
 
-This will install packages needed by searx:
+This will install packages needed by SearXNG:
 
 .. kernel-include:: $DOCS_BUILD/includes/searxng.rst
    :start-after: START distro-packages
@@ -73,7 +81,7 @@ If your docs build (``make docs.html``) shows warnings like this::
             display), check the imgmath_latex setting
 
 you need to install additional packages on your build host, to get better HTML
-output.
+output (:ref:`install buildhost <searxng.sh install buildhost>`).
 
 .. tabs::
 
@@ -93,7 +101,7 @@ output.
 
       .. code-block:: sh
 
-         $ sudo dnf install graphviz graphviz-gd texlive-xetex-bin librsvg2-tools
+         $ sudo dnf install graphviz graphviz-gd ImageMagick texlive-xetex-bin librsvg2-tools
 
 
 For PDF output you also need:
@@ -117,9 +125,8 @@ For PDF output you also need:
       .. code:: sh
 
       	 $ sudo dnf install \
-	        texlive-collection-fontsrecommended texlive-collection-latex \
-		dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts \
-		ImageMagick
+             texlive-collection-fontsrecommended texlive-collection-latex \
+             dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts
 
 .. _sh lint:
 
@@ -128,7 +135,8 @@ Lint shell scripts
 
 .. _ShellCheck: https://github.com/koalaman/shellcheck
 
-To lint shell scripts, we use ShellCheck_ - a shell script static analysis tool.
+To lint shell scripts we use ShellCheck_ - a shell script static analysis tool
+(:ref:`install buildhost <searxng.sh install buildhost>`).
 
 .. SNIP sh lint requirements
 
diff --git a/docs/admin/settings/settings_redis.rst b/docs/admin/settings/settings_redis.rst
index 49640f31..df5dc743 100644
--- a/docs/admin/settings/settings_redis.rst
+++ b/docs/admin/settings/settings_redis.rst
@@ -24,20 +24,26 @@ developer) account needs to be added to the *searxng-redis* group.
     rediss://[[username]:[password]]@localhost:6379/0
     unix://[[username]:[password]]@/path/to/socket.sock?db=0
 
-.. admonition:: Tip for developers
+.. _Redis Developer Notes:
 
-   To set up a local redis instance, first set the socket path of the Redis DB
-   in your YAML setting:
+Redis Developer Notes
+=====================
 
-   .. code:: yaml
+To set up a local redis instance, first set the socket path of the Redis DB
+in your YAML setting:
 
-      redis:
-        url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
+.. code:: yaml
 
-   Then use the following commands to install the redis instance ::
+   redis:
+     url: unix:///usr/local/searxng-redis/run/redis.sock?db=0
 
-     $ ./manage redis.build
-     $ sudo -H ./manage redis.install
-     $ sudo -H ./manage redis.addgrp "${USER}"
-     # don't forget to logout & login to get member of group
+Then use the following commands to install the redis instance (:ref:`manage
+redis.help`):
+
+.. code:: sh
+
+   $ ./manage redis.build
+   $ sudo -H ./manage redis.install
+   $ sudo -H ./manage redis.addgrp "${USER}"
+   # don't forget to logout & login to get member of group
 
diff --git a/docs/build-templates/filtron.rst b/docs/build-templates/filtron.rst
deleted file mode 100644
index a858f7dd..00000000
--- a/docs/build-templates/filtron.rst
+++ /dev/null
@@ -1,53 +0,0 @@
-.. START create user
-
-.. tabs::
-
-  .. group-tab:: bash
-
-    .. code-block:: sh
-
-      $ sudo -H useradd --shell /bin/bash --system \\
-          --home-dir \"$SERVICE_HOME\" \\
-          --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
-
-      $ sudo -H mkdir \"$SERVICE_HOME\"
-      $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
-
-.. END create user
-
-.. START install go
-
-.. tabs::
-
-  .. group-tab:: os: linux / arch: amd64
-
-
-    .. code-block:: bash
-
-       $ cat > \"$GO_ENV\" <<EOF
-       export GOPATH=${SERVICE_HOME}/go-apps
-       export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
-       EOF
-       $ sudo -i -u \"${SERVICE_USER}\"
-       (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
-       (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
-       (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
-                   \"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
-       (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
-       (${SERVICE_USER}) $ which go
-       ${SERVICE_HOME}/local/go/bin/go
-
-.. END install go
-
-.. START install filtron
-
-.. tabs::
-
-  .. group-tab:: bash
-
-    .. code-block:: bash
-
-       $ sudo -i -u \"${SERVICE_USER}\"
-       (${SERVICE_USER}) $ go get -v -u github.com/searxng/filtron
-
-.. END install filtron
diff --git a/docs/build-templates/morty.rst b/docs/build-templates/morty.rst
deleted file mode 100644
index b03a54f9..00000000
--- a/docs/build-templates/morty.rst
+++ /dev/null
@@ -1,53 +0,0 @@
-.. START create user
-
-.. tabs::
-
-  .. group-tab:: bash
-
-    .. code-block:: sh
-
-      $ sudo -H useradd --shell /bin/bash --system \\
-          --home-dir \"$SERVICE_HOME\" \\
-          --comment \"Privacy-respecting metasearch engine\" $SERVICE_USER
-
-      $ sudo -H mkdir \"$SERVICE_HOME\"
-      $ sudo -H chown -R \"$SERVICE_GROUP:$SERVICE_GROUP\" \"$SERVICE_HOME\"
-
-.. END create user
-
-.. START install go
-
-.. tabs::
-
-  .. group-tab:: os: linux / arch: amd64
-
-
-    .. code-block:: bash
-
-       $ cat > \"$GO_ENV\" <<EOF
-       export GOPATH=${SERVICE_HOME}/go-apps
-       export PATH=\$PATH:${SERVICE_HOME}/local/go/bin:\$GOPATH/bin
-       EOF
-       $ sudo -i -u \"${SERVICE_USER}\"
-       (${SERVICE_USER}) $ echo 'source $GO_ENV' >> ~/.profile
-       (${SERVICE_USER}) $ mkdir ${SERVICE_HOME}/local
-       (${SERVICE_USER}) $ wget --progress=bar -O \"${GO_VERSION}.linux-amd64.tar.gz\" \\
-                   \"${GO_DL_URL}/${GO_VERSION}.linux-amd64.tar.gz\"
-       (${SERVICE_USER}) $ tar -C ${SERVICE_HOME}/local -xzf \"${GO_VERSION}.linux-amd64.tar.gz\"
-       (${SERVICE_USER}) $ which go
-       ${SERVICE_HOME}/local/go/bin/go
-
-.. END install go
-
-.. START install morty
-
-.. tabs::
-
-  .. group-tab:: bash
-
-    .. code-block:: bash
-
-       $ sudo -i -u \"${SERVICE_USER}\"
-       (${SERVICE_USER}) $ go get -v -u github.com/asciimoo/morty
-
-.. END install morty
diff --git a/docs/dev/contribution_guide.rst b/docs/dev/contribution_guide.rst
index 67133b6b..bc4f6514 100644
--- a/docs/dev/contribution_guide.rst
+++ b/docs/dev/contribution_guide.rst
@@ -148,7 +148,7 @@ live build
 Live build is like WYSIWYG.  If you want to edit the documentation, its
 recommended to use.  The Makefile target ``docs.live`` builds the docs, opens
 URL in your favorite browser and rebuilds every time a reST file has been
-changed.
+changed (:ref:`make docs.clean`).
 
 .. code:: sh
 
@@ -183,3 +183,8 @@ commit and push:
 .. code:: sh
 
    $ make docs.clean docs.gh-pages
+
+.. attention::
+
+   If you are working in your own brand, don't forgett to adjust your
+   :ref:`settings brand`.
diff --git a/docs/dev/makefile.rst b/docs/dev/makefile.rst
index ad5665de..62ac2e90 100644
--- a/docs/dev/makefile.rst
+++ b/docs/dev/makefile.rst
@@ -1,32 +1,49 @@
 .. _makefile:
 
-========
-Makefile
-========
+=======================
+Makefile & ``./manage``
+=======================
 
 .. _gnu-make: https://www.gnu.org/software/make/manual/make.html#Introduction
 
+All relevant build and development tasks are implemented in the
+:origin:`./manage <manage>` script and for CI or IDE integration a small
+:origin:`Makefile` wrapper is available.  If you are not familiar with
+Makefiles, we recommend to read gnu-make_ introduction.
+
 .. sidebar:: build environment
 
    Before looking deeper at the targets, first read about :ref:`make
    install`.
 
-   To install system requirements follow :ref:`buildhosts`.
-
-All relevant build tasks are implemented in :origin:`manage` and for CI or
-IDE integration a small ``Makefile`` wrapper is available.  If you are not
-familiar with Makefiles, we recommend to read gnu-make_ introduction.
-
-The usage is simple, just type ``make {target-name}`` to *build* a target.
-Calling the ``help`` target gives a first overview (``make help``):
+   To install developer requirements follow :ref:`buildhosts`.
 
-.. program-output:: bash -c "cd ..; make --no-print-directory help"
 
 .. contents::
    :depth: 2
    :local:
    :backlinks: entry
 
+The usage is simple, just type ``make {target-name}`` to *build* a target.
+Calling the ``help`` target gives a first overview (``make help``):
+
+.. tabs::
+
+  .. group-tab:: ``make``
+
+     .. program-output:: bash -c "cd ..; make --no-print-directory help"
+
+
+  .. group-tab:: ``./manage``
+
+     The Makefile targets are implemented for comfort, if you can do without
+     tab-completion and need to have a more granular control, use
+     :origin:`manage` without the Makefile wrappers.
+
+     .. code:: sh
+
+        $ ./manage help
+
 .. _make install:
 
 Python environment (``make install``)
@@ -158,29 +175,49 @@ Node.js environment (``make node.env``)
    Manager) to install latest LTS of Node.js_ locally: there is no need to
    install nvm_ or npm_ on your system.
 
-Use ``make nvm.status`` to get the current status of you Node.js_ and nvm_ setup.
+To install NVM_ and Node.js_ in once you can use :ref:`make nvm.nodejs`.
+
+.. _make nvm:
+
+NVM ``make nvm.install nvm.status``
+-----------------------------------
 
-Here is the output you will typically get on a Ubuntu 20.04 system which serves
-only a `no longer active <https://nodejs.org/en/about/releases/>`_ Release
-`Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
+Use ``make nvm.status`` to get the current status of your Node.js_ and nvm_
+setup.
 
-::
+.. tabs::
 
-  $ make nvm.status
-  INFO:  Node.js is installed at /usr/bin/node
-  INFO:  Node.js is version v10.19.0
-  WARN:  minimal Node.js version is 16.13.0
-  INFO:  npm is installed at /usr/bin/npm
-  INFO:  npm is version 6.14.4
-  WARN:  NVM is not installed
-  INFO:  to install NVM and Node.js (LTS) use: manage nvm install --lts
+  .. group-tab:: nvm.install
 
-To install you can also use :ref:`make nvm.nodejs`
+     .. code:: sh
+
+        $ LANG=C make nvm.install
+        INFO:  install (update) NVM at ./searxng/.nvm
+        INFO:  clone: https://github.com/nvm-sh/nvm.git
+          || Cloning into './searxng/.nvm'...
+        INFO:  checkout v0.39.4
+          || HEAD is now at 8fbf8ab v0.39.4
+
+  .. group-tab:: nvm.status (ubu2004)
+
+     Here is the output you will typically get on a Ubuntu 20.04 system which
+     serves only a `no longer active <https://nodejs.org/en/about/releases/>`_
+     Release `Node.js v10.19.0 <https://packages.ubuntu.com/focal/nodejs>`_.
+
+     .. code:: sh
+
+        $ make nvm.status
+        INFO:  Node.js is installed at /usr/bin/node
+        INFO:  Node.js is version v10.19.0
+        WARN:  minimal Node.js version is 16.13.0
+        INFO:  npm is installed at /usr/bin/npm
+        INFO:  npm is version 6.14.4
+        WARN:  NVM is not installed
 
 .. _make nvm.nodejs:
 
 ``make nvm.nodejs``
-===================
+-------------------
 
 Install latest Node.js_ LTS locally (uses nvm_)::
 
@@ -213,10 +250,29 @@ sources of the theme need to be rebuild.  You can do that by running::
   $ make themes.all
 
 Alternatively to ``themes.all`` you can run *live builds* of the theme you are
-modify::
+modify (:ref:`make themes`)::
 
   $ LIVE_THEME=simple make run
 
+.. _make format.python:
+
+``make format.python``
+======================
+
+Format Python sourcee code using `Black code style`_.  See ``$BLACK_OPTIONS``
+and ``$BLACK_TARGETS`` in :origin:`Makefile`.
+
+.. attention::
+
+   We stuck at Black 22.12.0, please read comment in PR `Bump black from 22.12.0
+   to 23.1.0`_
+
+.. _Bump black from 22.12.0 to 23.1.0:
+   https://github.com/searxng/searxng/pull/2159#pullrequestreview-1284094735
+
+.. _Black code style:
+   https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
+
 .. _make clean:
 
 ``make clean``
@@ -237,18 +293,34 @@ calling ``make clean`` stop all processes using the :ref:`make install` or
 
 .. _make docs:
 
-``make docs docs.autobuild docs.clean``
-=======================================
+``make docs``
+=============
+
+Target ``docs`` builds the documentation:
+
+.. code:: bash
+
+   $ make docs
+   HTML ./docs --> file://
+   DOCS      build build/docs/includes
+   ...
+   The HTML pages are in dist/docs.
+
+.. _make docs.clean:
+
+``make docs.clean docs.live``
+----------------------------------
 
 We describe the usage of the ``doc.*`` targets in the :ref:`How to contribute /
 Documentation <contrib docs>` section.  If you want to edit the documentation
 read our :ref:`make docs.live` section.  If you are working in your own brand,
 adjust your :ref:`settings brand`.
 
+
 .. _make docs.gh-pages:
 
 ``make docs.gh-pages``
-======================
+----------------------
 
 To deploy on github.io first adjust your :ref:`settings brand`.  For any
 further read :ref:`deploy on github.io`.
@@ -261,17 +333,17 @@ further read :ref:`deploy on github.io`.
 Runs a series of tests: :ref:`make test.pylint`, ``test.pep8``, ``test.unit``
 and ``test.robot``.  You can run tests selective, e.g.::
 
-  $ make test.pep8 test.unit test.sh
+  $ make test.pep8 test.unit test.shell
   TEST      test.pep8 OK
   ...
   TEST      test.unit OK
   ...
-  TEST      test.sh OK
+  TEST      test.shell OK
 
 .. _make test.shell:
 
 ``make test.shell``
-===================
+-------------------
 
 :ref:`sh lint` / if you have changed some bash scripting run this test before
 commit.
@@ -279,7 +351,7 @@ commit.
 .. _make test.pylint:
 
 ``make test.pylint``
-====================
+--------------------
 
 .. _Pylint: https://www.pylint.org/
 
@@ -289,8 +361,8 @@ found in project's root folder :origin:`.pylintrc`.
 
 .. _make search.checker:
 
-``search.checker.{engine name}``
-================================
+``make search.checker.{engine name}``
+=====================================
 
 To check all engines::
 
@@ -318,3 +390,63 @@ To filter out HTTP redirects (3xx_)::
     https://news.google.com:443 "GET /search?q=computer&hl=en&lr=lang_en&ie=utf8&oe=utf8&ceid=US%3Aen&gl=US HTTP/1.1" 302 0
     https://news.google.com:443 "GET /search?q=computer&hl=en-US&lr=lang_en&ie=utf8&oe=utf8&ceid=US:en&gl=US HTTP/1.1" 200 None
     --
+
+.. _make themes:
+
+``make themes.*``
+=================
+
+.. sidebar:: further read
+
+   - :ref:`devquickstart`
+
+The :origin:`Makefile` targets ``make theme.*`` cover common tasks to build the
+theme(s).  The ``./manage themes.*`` command line can be used to convenient run
+common theme build tasks.
+
+.. program-output:: bash -c "cd ..; ./manage themes.help"
+
+To get live builds while modifying CSS & JS use (:ref:`make run`):
+
+.. code:: sh
+
+   $ LIVE_THEME=simple make run
+
+.. _make static.build:
+
+``make static.build.*``
+=======================
+
+.. sidebar:: further read
+
+   - :ref:`devquickstart`
+
+The :origin:`Makefile` targets ``static.build.*`` cover common tasks to build (a
+commit of) the static files.  The ``./manage static.build..*`` command line
+can be used to convenient run common build tasks of the satic files.
+
+.. program-output:: bash -c "cd ..; ./manage static.help"
+
+
+.. _manage redis.help:
+
+``./manage redis.help``
+=======================
+
+The ``./manage redis.*`` command line can be used to convenient run common Redis
+tasks (:ref:`Redis developer notes`).
+
+.. program-output:: bash -c "cd ..; ./manage redis.help"
+
+
+.. _manage go.help:
+
+``./manage go.help``
+====================
+
+The ``./manage go.*`` command line can be used to convenient run common `go
+(wiki)`_ tasks.
+
+.. _go (wiki): https://en.wikipedia.org/wiki/Go_(programming_language)
+
+.. program-output:: bash -c "cd ..; ./manage go.help"
diff --git a/docs/dev/quickstart.rst b/docs/dev/quickstart.rst
index 921384aa..9bee8f60 100644
--- a/docs/dev/quickstart.rst
+++ b/docs/dev/quickstart.rst
@@ -7,8 +7,16 @@ Development Quickstart
 .. _npm: https://www.npmjs.com/
 .. _Node.js: https://nodejs.org/
 
-SearXNG loves developers, just clone and start hacking.  All the rest is done for
-you simply by using :ref:`make <makefile>`.
+
+.. sidebar:: further read
+
+   - :ref:`makefile`
+   - :ref:`buildhosts`
+
+SearXNG loves developers; Developers do not need to worry about tool chains, the
+usual developer tasks can be comfortably executed via :ref:`make <makefile>`.
+
+Don't hesitate, just clone SearXNG's sources and start hacking right now ..
 
 .. code:: bash
 
@@ -18,25 +26,23 @@ Here is how a minimal workflow looks like:
 
 1. *start* hacking
 2. *run* your code: :ref:`make run`
-3. *test* your code: :ref:`make test`
+3. *format & test* your code: :ref:`make format.python` and :ref:`make test`
 
 If you think at some point something fails, go back to *start*.  Otherwise,
 choose a meaningful commit message and we are happy to receive your pull
 request. To not end in *wild west* we have some directives, please pay attention
 to our ":ref:`how to contribute`" guideline.
 
-If you implement themes, you will need to setup a :ref:`make node.env` once:
+.. sidebar:: further read
 
-.. code:: bash
+   - :ref:`make nvm`
+   - :ref:`make themes`
 
-   make node.env
+If you implement themes, you will need to setup a :ref:`Node.js environment
+<make node.env>`: ``make node.env``
 
 Before you call *make run* (2.), you need to compile the modified styles and
-JavaScript:
-
-.. code:: bash
-
-   make themes.all
+JavaScript: ``make themes.all``
 
 Alternatively you can also compile selective the theme you have modified,
 e.g. the *simple* theme.
@@ -49,6 +55,10 @@ e.g. the *simple* theme.
 
    To get live builds while modifying CSS & JS use: ``LIVE_THEME=simple make run``
 
+.. sidebar:: further read
+
+   - :ref:`make static.build`
+
 If you finished your *tests* you can start to commit your changes.  To separate
 the modified source code from the build products first run: