aboutsummaryrefslogtreecommitdiff
path: root/test/test-lib.sh
Commit message (Collapse)AuthorAge
* test: Uniformly canonicalize actual and expected JSONAustin Clements2012-08-03
| | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we used a variety of ad-hoc canonicalizations for JSON output in the test suite, but were ultimately very sensitive to JSON irrelevancies such as whitespace. This introduces a new test comparison function, test_expect_equal_json, that first pretty-prints *both* the actual and expected JSON and the compares the result. The current implementation of this simply uses Python's json.tool to perform pretty-printing (with a fallback to the identity function if parsing fails). However, since the interface it introduces is semantically high-level, we could swap in other mechanisms in the future, such as another pretty-printer or something that does not re-order object keys (if we decide that we care about that). In general, this patch does not remove the existing ad-hoc canonicalization because it does no harm. We do have to remove the newline-after-comma rule from notmuch_json_show_sanitize and filter_show_json because it results in invalid JSON that cannot be pretty-printed. Most of this patch simply replaces test_expect_equal and test_expect_equal_file with test_expect_equal_json. It changes the expected JSON in a few places where sanitizers had placed newlines after commas inside strings.
* test: use subtest name for generated message subject by defaultDmitry Kurochkin2012-03-18
| | | | | | | | | | | | | | | | | | Before the change, messages generated by generate_message() used "Test message #N" for default subject where N is the generated messages counter. Since message subject is commonly present in expected results, there is a chance of breaking other tests when a new generate_message() call is added. The patch changes default subject value for generated messages to subtest name if it is available. If subtest name is not available (i.e. message is generated during test initialization), the old default value is used (in this case it is fine to have the counter in the subject). Another benefit of this change is a sane default value for subject in generated messages, which would allow to simplify code like: test_begin_subtest "test for a cool feature" add_message [subject]="message for test for a cool feature"
* test: Add test for searching of uncommonly encoded messagesMichal Sojka2012-02-29
| | | | | | Emails that are encoded differently than as ASCII or UTF-8 are not indexed properly by notmuch. It is not possible to search for non-ASCII words within those messages.
* test: always report missing prereqs, independent of `--verbose' optionPieter Praet2012-02-20
| | | | | | When tests are skipped due to missing prereqs, those prereqs are only displayed when running with the `--verbose' option. This is essential information when troubleshooting, so always send to stdout.
* test: auto load elisp tests file in test_emacs if availableDmitry Kurochkin2012-02-12
| | | | This allows us to simplify shell part of tests written in elisp.
* test: Add `test_emacs_expect_t'.David Edmondson2012-01-25
| | | | | | | | | Add a new test function to allow simpler testing of emacs functionality. `test_emacs_expect_t' takes one argument - a lisp expression to evaluate. The test passes if the expression returns `t', otherwise it fails and the output is reported to the tester.
* test: Don't return the result of checking for running emacs to the tester.David Edmondson2012-01-25
| | | | | | | | When checking for a running emacs, test_emacs evaluates the empty list '()'. This returns 'nil' when emacs is running, which is then prepended to the actual test result. Given that it is not part of the actual test output the test harness can incorrectly report test failure (or success).
* test: whitespace-cleanup for most test/* filesTomi Ollila2012-01-22
| | | | | | Used emacs (whitespace-cleanup) function to "cleanup blank problems" in test files where that could be done without breaking tests; test/emacs was partially, and test/multipart was fully reverted.
* test: don't bail out of `run_emacs' too early when missing prereqsPieter Praet2012-01-15
| | | | | | | | | | | | | | | | | | | | | | | | | | When running the Emacs tests in verbose mode, only the first missing prereq is reported because the `run_emacs' function is short-circuited early: #+begin_example emacs: Testing emacs interface missing prerequisites: [0] emacs(1) skipping test: [0] Basic notmuch-hello view in emacs SKIP [0] Basic notmuch-hello view in emacs #+end_example This can lead to situations reminiscent of "dependency hell", so instead of returning based on each individual `test_require_external_prereq's exit status, we now do so only after checking all the prereqs: #+begin_example emacs: Testing emacs interface missing prerequisites: [0] dtach(1) emacs(1) emacsclient(1) skipping test: [0] Basic notmuch-hello view in emacs SKIP [0] Basic notmuch-hello view in emacs #+end_example Also added missing prereq for dtach(1).
* test: Remove #! line from test-lib.shAustin Clements2011-12-29
| | | | | | | | It makes no sense to run test-lib.sh, so it makes no sense to give it an interpreter. This is particularly annoying for Emacs users who have executable-insert set, since the presence of the #! line will cause Emacs to mark test-lib.sh executable when saving it, which will in turn case the 'basic' test to fail.
* test: Make generated message date a real dateAustin Clements2011-12-29
| | | | | | | | | January 5, 2001 was a Tuesday, not a Friday. Jameson fixed this exact problem for the multipart test in ec2b0a98cc, but not for generate_message itself. As Jameson pointed out in ec2b0a98cc, if we want to test date parsing, we should do it separately.
* test: allow user to choose which emacs to run tests with.David Bremner2011-12-28
| | | | | | | | | | | | As we start to pay more attention to emacs24, it helps to be able to select a different version of emacs to run the tests with to verify version specific bugs. A separate variable TEST_EMACS is needed to avoid being overwritten by the make variable EMACS in Makefile.config For what it's worth, the value of emacs is chosen at the time tmp.emacs/run_emacs is created, so is fixed for all subtests.
* test: optionally print subtest numberDavid Bremner2011-12-18
| | | | | | | The idea is that $test_count could be used in tests to label intermediate files. The output enabled by this patch (and --debug) helps figure out which OUTPUT.nn file belongs to which test in case several subtests write to OUTPUT.$test_count
* test: use python2 instead of python if availableThomas Jost2011-12-11
| | | | | | Some distros (Arch Linux) ship Python as python2 and Python 3 as python. Checking for python2 is necessary for the Python tests to work on these platforms.
* test: add a function to run Python testsThomas Jost2011-12-11
| | | | | | | | | | | | | The new test_python() function makes writing Python tests a little easier: - it sets the environment variables as needed - it redirects stdout to the OUTPUT file (like test_emacs()). This commit also declares python as an external prereq. The stdout redirection is required to avoid trouble when running commands like "python 'script' | sort > OUTPUT": in such a case, any error due to a missing external prereq would be "swallowed" by sort, resulting to a failed test instead of a skipped one.
* test: add functions to count how much times notmuch was calledDmitry Kurochkin2011-12-07
| | | | | | | | | | | | | | | | The patch adds two auxiliary functions and a variable: notmuch_counter_reset $notmuch_counter_command notmuch_counter_value They allow to count how many times notmuch binary is called. notmuch_counter_reset() function generates a script that counts how many times it is called and resets the counter to zero. The function sets $notmuch_counter_command variable to the path to the generated script that should be called instead of notmuch to do the counting. The notmuch_counter_value() function returns the current counter value.
* test: fix error messages for missing binary dependenciesDmitry Kurochkin2011-11-30
| | | | | | | The fake missing binary functions check if the binary has already be added to the diagnostic message to avoid duplicates. Unfortunately, this check was buggy because the message string does not have the trailing space.
* test: fix spurious output from missing external binaries functionsDmitry Kurochkin2011-11-30
| | | | | The grep(1) command used in the fake binary functions was missing the quiet option.
* test: add missing escape backslash in test_declare_external_prereq()Dmitry Kurochkin2011-11-30
|
* test: fix test_require_external_prereq()Dmitry Kurochkin2011-11-30
| | | | | | test_missing_external_prereq_${binary}_ variable indicates that the binary is missing. It must be set in test_declare_external_prereq() outside of the fake $binary() function.
* test: check if emacs is available in the beginning of test_emacsDmitry Kurochkin2011-11-27
| | | | Unfortunately, this is needed to avoid the emacs waiting loop.
* test: add function to explicitly check for external dependenciesDmitry Kurochkin2011-11-27
| | | | Useful when binary is called indirectly (e.g. from emacs).
* test: declare external dependencies for the testsDmitry Kurochkin2011-11-27
| | | | That are: dtach(1), emacs(1), emacsclient(1), gdb(1) and gpg(1).
* test: skip all subtests if external dependencies are missing during initDmitry Kurochkin2011-11-27
| | | | | | | | | | Some tests (e.g. crypto) do a common initialization required for all subtests. The patch adds a check for missing external dependencies during this initialization. If any prerequisites are missing, all subtests are skipped. The check is run on the first call of test_reset_state_ function, so no changes for the tests are needed.
* test: fix "skipping test" verbose outputDmitry Kurochkin2011-11-27
|
* test: add support for external executable dependenciesDmitry Kurochkin2011-11-27
| | | | | | | | | | | | | | | There is existing support for general prerequisites in the test suite. But it is not very convenient to use: every test case has to keep track for it's dependencies and they have to be explicitly listed. The patch aims to add better support for a particular type of external dependencies: external executables. The main idea is to replace missing external binaries with shell functions that have the same name. These functions always fail and keep track of missing dependencies for a subtest. The result reporting functions later can check that an external binaries are missing and correctly report SKIP result instead of FAIL. The primary benefit is that the test cases do not need to declare their dependencies or be changed in any way.
* test: add test state reset to test_expect_* functions that did not have itDmitry Kurochkin2011-11-27
|
* test: set EMACS_SERVER variable only after dtach(1) was successfully startedDmitry Kurochkin2011-11-27
| | | | | Otherwise, we can set the EMACS_SERVER and return with an error. And subsequent calls to test_emacs would assume that emacs server is running.
* test: move subtest variables reset into a dedicated functionDmitry Kurochkin2011-11-27
| | | | | Currently, there is only one such variable test_subtest_known_broken_. But more will be added in the future.
* test: remove executable permissions from test-lib.shDmitry Kurochkin2011-11-26
| | | | It is not supposed to be run, only sourced.
* test: attempt to send QUIT to smtp-dummy in case mail send failedTomi Ollila2011-11-18
| | | | | | If mail sending from emacs fails before it has chance to connect to the smtp-dummy mail server, the opportunistic QUIT message sending makes smtp-dummy to exit.
* test: create dtach's session socket in $TEST_TMPDIRTomi Ollila2011-11-17
| | | | | | | | Due to 108-character limit in unix domain socket path this change is required; it is more probable that length of ${TMPDIR:-/tmp} is shorter than length of path to the current directory of notmuch test source directory. One can expect to create reasonable-length unix domain sockets wherever $TMPDIR points to.
* test: create TEST_TMPDIR for holding temporary filesTomi Ollila2011-11-17
| | | | | | The TEST_TMPDIR if first needed to hold dtach's socket (due to 108-character limit in socket file names). Later it can be used to hold other temporary files; directory deleted at exit.
* test: do not hide test_emacs errorsDmitry Kurochkin2011-11-13
| | | | | | | | Do not redirect test_emacs stderr to /dev/null. Test_emacs uses emacsclient(1) now and it does not print unwanted messages (like those from `message') to stderr. But it does print useful errors, e.g. when emacs server connection fails, given expression is not valid or undefined function is called.
* exec emacs at the end of run_emacs scriptTomi Ollila2011-11-12
| | | | | In the last line of run_emacs, exec the emacs process. With one fork less the process list is (also) neater.
* test: use dtach(1) instead of screen(1) in emacs testsTomi Ollila2011-11-11
| | | | | dtach is simpler than screen and is not setuid/setgid program so TMPDIR does not get cleared by dynamic loader when executed
* test: avoid using screen(1) configuration filesDmitry Kurochkin2011-11-05
| | | | | | Set SCREENRC and SYSSCREENRC environment variables to "/dev/null" as suggested by Jim Paris to avoid potential problems with screen(1) configuration files.
* test: run emacs inside screenDmitry Kurochkin2011-11-05
| | | | | | | | | | | | | Before the change, emacs run in daemon mode without any visible buffers. Turns out that this affects emacs behavior in some cases. In particular, `window-end' function returns `point-max' instead of the last visible position. That makes it hard or impossible to implement some tests. The patch runs emacs in a detached screen(1) session. So that it works exactly as if it has a visible window. Note: screen terminates when emacs exits. So the patch does not introduce new "running processes left behind" issues.
* test/test-lib.sh: update commentsDavid Bremner2011-10-22
| | | | | | | - explain test_expect_equal_file - remove mention of test_expect_failure, since that function was removed. Based on id:"1317317811-29540-1-git-send-email-thomas@schwinge.name"
* test: Report test failures from test_expect_*Austin Clements2011-09-12
| | | | | | This makes test_expect_* return non-zero if the test fails, so the caller can make decisions based on this, such as setting test prerequisites.
* test: Fix message when skipping test_expect_equal* testsAustin Clements2011-09-12
| | | | | | For these types of tests, the test name is previously recorded in a variable, not passed to the test function, so pass this variable to test_skip.
* test: reset test_subtest_known_broken_ after each success/failure.David Bremner2011-09-12
| | | | | | | | | | | This means that test_subtest_known_broken needs to be called before every known broken subtest, which is no different than what is documented for the test_begin_subtest case. The assumption is that every test ends up calling either skipping, calling test_ok_ or test_failure_ and and the latter in turn delegate to the known_broken versions in the case where test_subtest_known_broken_ is set.
* test: improve known broken tests supportDmitry Kurochkin2011-09-10
| | | | | | | | | | | | | | | | | There is existing support for broken tests. But it is not convenient to use. The primary issue is that we have to maintain a set of test_expect_*_failure functions which are equivalent to the normal test_expect_* counterparts except for what functions are called for result reporting. The patch adds test_subtest_known_broken function which marks a subset as broken, making the normal test_expect_* functions behave as test_expect_*_failure. All test_expect_*_failure functions are removed. Test_known_broken_failure_ is changed to format details the same way as test_failure_ does. Another benefit of this change is that the diff when a broken test is fixed would be small and nice. Documentation is updated accordingly.
* test: Nix increment_mtime.Austin Clements2011-06-29
| | | | | With the fix for the mtime race, this workaround is no longer necessary.
* test: use emacsclient(1) for Emacs testsDmitry Kurochkin2011-06-28
| | | | | | | | | | | | | | | | | Before the change, every Emacs test ran in a separate Emacs instance. Starting Emacs many times wastes considerable time and it gets worse as the test suite grows. The patch solves this by using a single Emacs server and emacsclient(1) to run multiple tests. Emacs server is started on the first test_emacs call and stopped when test_done is called. We take care not to leave orphan Emacs processes behind when test is terminated by whatever reason: Emacs server runs a watchdog that periodically checks that the test is still running. Some tests need to provide user input. Before the change, this was done using echo(1) to Emacs stdin. This no longer works and instead `standard-input' variable is set accordingly to make `read' return the appropriate string.
* test: Increment mtime when delivering a message with emacs_deliver_messageCarl Worth2011-06-28
| | | | | | Without this, mail messages delivered by emacs_deliver_message might not be seen by the next invocation of "notmuch new", (which can lead to test-suite failures if emacs_deliver_message is fast enough).
* test:Improve test behaviors when --root is usedMark Anderson2011-06-28
| | | | | | | | | | | Change add_email_corpus, emacs_deliver_message and tests to use $TEST_DIRECTORY instead of '..'. This improves the behavior of the usage of --root=<dir>, as the assumption of what '..' means will usually be incorrect. Document -root option in README and update valgrind to work with -root.
* test: generate run_emacs script once on test startupDmitry Kurochkin2011-06-28
| | | | | | | Instead of generating auxiliary run_emacs script every time test_emacs is run, do it once in the beginning of the test. Also, use absolute paths in the script to make it more robust.
* test: set variables using `let' instead of `setq' in Emacs testsDmitry Kurochkin2011-06-28
| | | | | | | | | Using `setq' for setting variables in Emacs tests affect other tests that may run in the same Emacs environment. Currently it works because each test is run in a separate Emacs instance. But in the future multiple tests will run in a single Emacs instance. The patch changes all variables to use `let', so the scope of the change is limited to a single test.
* test: wrap and indent test_emacs callsDmitry Kurochkin2011-06-28
| | | | | | | Most test_emacs calls have long arguments that consist of many expressions. Putting them on a single line makes it hard to read and produces poor diff when they are changed. The patch puts every expression in test_emacs calls on a separate line.
generated by cgit v1.2.3 (git 2.39.1) at 2024-05-18 15:52:17 +0000