summaryrefslogtreecommitdiff
path: root/lib/sbin/RunTestUtils.pl
Commit message (Collapse)AuthorAge
* Capture stderr as well when issuing a commandschnetter2008-01-25
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@4449 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Change layout of test suite results:schnetter2007-01-30
| | | | | | | | | | - Print all file names beginning at the same column, followed by a colon - Be consistent about inserting empty lines - Ensure that a final newline character is added git-svn-id: http://svn.cactuscode.org/flesh/trunk@4405 17b73243-c579-4c4c-a9d2-2d5706c11dac
* bugfix in treatment of output files which are expected but weren't createdtradke2007-01-26
| | | | | | | | | | | | in a testsuite: * report the total number of such missing files * don't list them anymore in the interactive file comparison mode This fixes http://www.cactuscode.org/old/pipermail/developers/2007-January/005249.html. git-svn-id: http://svn.cactuscode.org/flesh/trunk@4404 17b73243-c579-4c4c-a9d2-2d5706c11dac
* fixed a couple bugs:tradke2007-01-22
| | | | | | | | | | | | | | | | - replace the excessive use of perl hash arrays to store ordinary perl array information about absolute/relative differences found in a file comparison; store a reference to that array instead and use it later in the testsuite report output; this should speed up the comparison runtime by an order of magnitude - really output all absolute/relative differences found in a file comparison (this was missing for files with special characters ('[',']') in their names because the filename was used unprotected as a hash key) - properly sort listing of absolute/relative differences by column number in a datafile git-svn-id: http://svn.cactuscode.org/flesh/trunk@4402 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Implement and document the new possibility to specify the number of processorstradke2007-01-18
| | | | | | | | | | required to run a testsuite in a thorn's test.ccl configuration file. This closes PR 2078/Cactus: "specify the number of processors required to run a testsuite". git-svn-id: http://svn.cactuscode.org/flesh/trunk@4401 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Speed up output file comparison.tradke2006-12-22
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@4398 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Make the testsuite mechanism usable for automatic Cactus integration teststradke2006-12-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (run nightly as non-interactive cron jobs): 1) Via shell environment variables 'CCTK_TESTSUITE_RUN_PROCESSORS' and 'CCTK_TESTSUITE_RUN_COMMAND', the user can override the default number of processors to use, and the default run command. 2) When running the testsuite scripts, the user is now prompted with a template for the command to be executed: now an arbitrary command line can be specified with the number of processors, the executable, and the name of the parameter file as template parameters. 3) A summary.log file is created in TEST/<configuration>/ subdir containing the run details of a given configuration and a summary on the number of tests of tests available/run/passed/failed. This information is the same as printed to stdout and can be parsed appropriately by an automatic integration test script. 4) For each testsuite that was run, a <testsuite>.diffs file containing the differences detected while comparing output files. This information is the same as printed to stdout and can be parsed appropriately by an automatic integration test script. This closes PR Cactus/2072: "enhance testsuite scripts to be used within automatic Cactus integration tests". git-svn-id: http://svn.cactuscode.org/flesh/trunk@4391 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fix parsing of testsuite output filename extenstions in multiple test.ccl files.tradke2006-03-24
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@4273 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Ignore duplicate entries in the ThornList file when querying thorns withtradke2005-12-08
| | | | | | | testsuites. git-svn-id: http://svn.cactuscode.org/flesh/trunk@4203 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Patch from Yaakoub for PRs 1480 and 1690. Now have workinggoodale2004-07-15
| | | | | | | code to allow tests to set individual absolute and relative tolerances. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3808 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Removing some stuff flagged by perl -cw .goodale2004-06-06
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@3758 17b73243-c579-4c4c-a9d2-2d5706c11dac
* A better hack to get tests to run under the latest cygwin.goodale2004-05-19
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@3745 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Finish error message about missing executable with a newline charactertradke2004-05-19
| | | | | | | so that the perl script stops correctly. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3743 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Changed to using the cygwin version of Perl under cygwin. This hasgoodale2004-05-19
| | | | | | | | | | | | | | | | | | | | | | | | | | the translation of CCTK_HOME to a windows format from the master Makefile, and forced the addition of a new Make macro TRANSLATE_DIRS which is used in various places in Make to translate things like /cygwin/f/... into f:\... This is a 'call'able macro in Make, and defaults to $(1). There is a new subroutine in RunTestUtils which serves the same purpose. This change should not affect anyone not using Cactus on Windows. If you are using Windows you will need to reconfigure, making sure that the Perl you use is the cygwin one and not the ActiveState one. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3742 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed bug when searching for thorn config files.tradke2004-05-06
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@3694 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Added debugging code which allows simultaneous debugging of RunTest.plrideout2004-01-13
| | | | | | | | | | | | | | | and RunTestUtils.pl. Commented reference to $loop in RunTest.pl's main loop, since it is not referenced anywhere else. Fixed ViewResults to handle the case when a file is missing from the test output or test archive. Some minor cleanup and performance improvements to CompareTestFiles. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3504 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Offer to display test case results with gnuplot as well.schnetter2003-10-13
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@3430 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Traverse all hashes alphabetically, so that all output files of theschnetter2003-10-07
| | | | | | | | CST stage will be identical. This is necessary at least since perl 5.8.1 to avoid redundant recompilation. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3425 17b73243-c579-4c4c-a9d2-2d5706c11dac
* For the 'Choose test from [T]horn or [A]rrangement' menu,tradke2003-09-05
| | | | | | | sort the list of thorns to test by alphabetical order. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3397 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Parse a config file in a test directory to set a different relative orallen2003-07-23
| | | | | | | | absolute tolerance for the whole testsuite. The notation will change in the future so I don't think this feature will be documented properly yet. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3349 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Add name of test to comment before comparing files.allen2003-07-06
| | | | | | | Think this fixes Cactus/1066 git-svn-id: http://svn.cactuscode.org/flesh/trunk@3254 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Do not compare data files with .# (CVS back up files)allen2003-07-04
| | | | | | | | | (Also check for exclusions first, instead of after checking extensions) Fixes Cactus/1547 git-svn-id: http://svn.cactuscode.org/flesh/trunk@3246 17b73243-c579-4c4c-a9d2-2d5706c11dac
* When finding the active thorns of a parameter file, concatenate all ↵hawke2003-06-24
| | | | | | | | | ActiveThorns lines (as more than one may exist). Fixes PR1493. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3243 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed parsing of EXE and EXEDIR from make.config.defn.rideout2003-04-28
| | | | | | | | | | | | | | Reject broken default argument of " " for some (all?) prompts. Fix 'select tests by arrangement' functionality. Make 'select tests by arrangement' prompts mirror those of 'select tests by thorn'. Fix parsing for RUNNABLEARRANGEMENTS database entry. git-svn-id: http://svn.cactuscode.org/flesh/trunk@3201 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Get the configuration name right in the test outputallen2002-05-31
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2881 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed bug in creating the TEST directory initiallyallen2002-05-31
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2877 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Added support for a config file for testsuites, where IO thorns can provide ↵allen2002-05-29
| | | | | | | | | | | | | | | information about the data files which they produce, and thorns providing testsuites can provide information about the tests they are supplying. The syntax for the config file is not finalised yet, and will be subject to change depending on the feedback we get. The config file is called "config" and must be located in the "test" directory of a thorn. Right now, the only line recognized is EXTENSIONS dat xl yl zl which should be added by any thorn providing output files eg with extensions dat xl yl zl which should be tested. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2864 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Initialising some database entries in a different placeallen2002-05-23
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2855 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Only print out the "differences below tolerance" buffer onceallen2002-05-23
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2853 17b73243-c579-4c4c-a9d2-2d5706c11dac
* If there are no output files created just say so and don't list everyallen2002-05-22
| | | | | | | single one twice ... git-svn-id: http://svn.cactuscode.org/flesh/trunk@2848 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Formattingallen2002-05-21
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2839 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Add support for new file namesallen2002-05-21
| | | | | | | Removed some blank lines git-svn-id: http://svn.cactuscode.org/flesh/trunk@2837 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixing the logic for testing if output files are created and if they are ↵allen2002-05-04
| | | | | | empty etc. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2782 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Added file extensions gauss, alm and ulallen2002-05-01
| | | | | | | | | | | | All file extensions will be added by testsuite configuration files soon, once another perl script is written. (thorns can either add global extensions for all tests, or local extensions just for their own thorn) Catch the case when the original file has a nan or an inf but the new file has a real number (this was caught but there was no message to tell you about the nans, and the reported differences were out by a lot :) ) git-svn-id: http://svn.cactuscode.org/flesh/trunk@2777 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Check for empty files in both test and archive and do the appropriateallen2002-04-30
| | | | | | | | | | thing Check for divide by zeros even in the cases where you think they could never ever happen and ask how you could ever get this. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2776 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Added an option to rerun a test but show the Cactus output on screenallen2002-04-27
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2765 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed my dodgy matching of known datafile extensions, also now reportallen2002-04-27
| | | | | | | on which files aren't being tested (ignoring . .. CVS *.par *~) git-svn-id: http://svn.cactuscode.org/flesh/trunk@2760 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Now works for all the ascii data from IOBasic and IOASCIIallen2002-04-27
| | | | | | | | | | | | | | | - recognize extensions xl yl zl tl dl asc gnuplot (later this will be configurable) - check all columns - ignore comments lines " and # (not sure if really want this) Please look out for problems since a lot of the logic in the comparison section changed. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2759 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Added back prompt command to scripts this means thatallen2002-04-26
| | | | | | | | | | gmake <config>-testsuite PROMPT=no will run through with the defaults for everything, including running all the tests, and then stop. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2757 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed bug where par files which core dumped would passallen2002-04-24
| | | | git-svn-id: http://svn.cactuscode.org/flesh/trunk@2751 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Reworked the comparing part of the routineallen2002-04-22
| | | | | | | | | | | | | | | | | | | | | | | - now use absolute and relative errors test is |v1 - v2| < max ( abs_tol, rel_tol*max(|v1|,|v2|) where for now abs_tol = 1e-12 and rel_tol = 0 which should be what we had before (new implementation now means that a difference of 1000 doesn't pass!) - check values in first column (coordinates) as well as second column - give more information about the comparisons At the moment you can't set the rel_tol yourself but I do print what you would need for a test to pass to screen. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2741 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Splitting up database into configuration, static and dynamic stuff.allen2002-04-21
| | | | | | | | | | | Added an initial try at interactively looking at the differences between files, now there are options to list both files, diff them or xgraph them. Any comments on the new testsuites are very welcome, in particular on the interfaces, please mail flesh@cactuscode.org with any suggestions. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2736 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixing bug in detection of MPI configurations.goodale2002-04-16
| | | | | | | Tom git-svn-id: http://svn.cactuscode.org/flesh/trunk@2722 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Mainly reorganising of the database and removing unneeded local variables.allen2002-04-16
| | | | | | | | | | Broke up the runtest subroutine into running the test and reporting on the test. Includes fix for Cactus/451 (don't report NaNs/Infs which should be there) git-svn-id: http://svn.cactuscode.org/flesh/trunk@2720 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Fixed bug in last commit, and added ability to run all testsuites for aallen2002-04-14
| | | | | | | chosen thorn (not for an arrangement yet). git-svn-id: http://svn.cactuscode.org/flesh/trunk@2718 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Change to user interface. Unfortunately now I replaced the choiceallen2002-04-14
| | | | | | | | | | | | "All" with "Entire" so as not to clash with "Arrangement", but the default is still to run all tests so it shouldn't be too bad. Will add soon an option to run all the tests for a thorn/arrangement. Closes Cactus/802 git-svn-id: http://svn.cactuscode.org/flesh/trunk@2717 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Remove all pwds to fix Cactus/480allen2002-04-14
| | | | | | | | The test directory can now be set to any location by setting an environment variable TESTS_DIR git-svn-id: http://svn.cactuscode.org/flesh/trunk@2716 17b73243-c579-4c4c-a9d2-2d5706c11dac
* Somewhat tidied up version of Runtest for doing testsuite checking. Thisallen2002-04-13
is in preparation for getting together a complete spec for and doing it properly. A lot of the internals have been changed around, so in case of problems I've left the original version there for the moment, to run the original version just edit Makefile and replace RunTest.pl with Runtest.pl The main differences though should be - a lot more output at the end, probably too much, would welcome any ideas on what information to display and how - better menu interface for choosing a test to run, also started adding a couple of other features. Hopefully removed all the infinite loops from the menu. - now works properly when a test with the same name is in two different thorns ... note that this meant adding an extra layer of directories in the TEST directory so if anyone uses a path to the output files in the TEST directories (like in the testsuites for IOHDF5/IOFlexIO) these will need to be changed. Should now be able to go on to fix outstanding requests in gnats for the testsuites. git-svn-id: http://svn.cactuscode.org/flesh/trunk@2712 17b73243-c579-4c4c-a9d2-2d5706c11dac
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-26 10:58:10 +0000