This page describes the process of software testing

CTest, included in CMake, is used in automatic testing, and all tests are executed with one command after the library is built.

Notice that tests create files, which remain on disk and may take a lot of space.

Tests for the library functions

C source codes for testing are included in the source codes of the library, and compiled as the project executables are compiled, and executed as explained in README.md.

Running these tests do not require any special settings on the platform, but the directory of the compiled executables should be in the search path.

Tests for executables

Since executables (binaries) are based on library functions, which have already been tested (in v2), the tests for executables are not very detailed.

Very basic testing is first performed using CMake commands, verifying that executables can be found and run, and that they provide usage information with appropriate command-line options.

Running these tests do not require any special settings on the platform. However, library applications must reside in a folder included in the systems search path for the application test scripts to work.

In the crucial tests the executables are tested to perform as expected in normal use, and therefore tests are run using multi-platform scripts that mimic the real use cases. Note that these tests are not intended to test whether the applied method is suitable to retrieve meaningful outcome from any specific data set; that is considered as science, and out of the scope of software development.

These tests are based on bash scripts, located in test folder under the source directories. CMake is used to execute these test scripts automatically. In Windows platform this requires that the MinGW/MSYS bin folders (containing for example bash.exe, and other bash command executables) are included in Windows path. When correctly installed, you should be able to open Windows command prompt window (cmd.exe), and run there for example the commands

gcc --version
pwd
ls -al

If test script is not ready at the time when application itself is added, then create at least a dummy script which returns a non-zero value and prints to stdout 'test not yet made' or something like that.

Running tests

After successful make or make install, you can run the tests. Running tests may take a long time. Recommended command to run the tests in quad-core computer is

ctest --output-on-failure --parallel 4

because then CTest uses 4 threads for testing (you can change the number according to your computer), and prints to screen all output from those tests only that failed.

Alternatively, you can run

ctest -V

to get more verbose output from (the same) tests; ctest is included in CMake installation. Do not use --parallel option here, because it would mess up the verbose output.

If you are in the root build folder, for instance ~/tpcclib/build, then the tests are run for all applications and libraries. If you are in an application sub-folder, for instance in ~/tpcclib/build/v2/bfm, then only applications of that folder are tested. To run the tests for a single application, go to the application test folder, for instance ~/tpcclib/build/v2/bfm/test/bfmsrtm, and execute the test script, for instance ./test_bfmsrtm.sh.

The most recently built version of the application that resides in the build folder is tested. But application tests are dependent on several other applications of the tpcclib. After make install the tests use applications that you just compiled, and with make the previous versions of tpcclib applications in your computers search path are being used. For the tests to work, you must have in your computers search path the binary folder where the applications are installed, for instance ~/tpcclib/build/bin.

If you modify the test script or data in the source folder, the versions in the build folder that actually are used in the tests are updated when you rebuild the application. Thus, you need to run

make

or

make clean; make

before testing the application(s), even if the application source code was not changed, just to update the test script, or to update the test script and rebuild the test data, accordingly. To remove all built files, including test scripts and data, go to the root build directory and enter

cmake --build . --target clean

and then build and test the project as usual:

make -j 4 install
ctest --output-on-failure --parallel 4

Warnings

Note that when running tests in parallel the computer may run out of memory.

The default test sets consume a lot of memory and disk space, and take a long time to run. Very large large files may remain on disk if tests fail. To exclude the slow and memory consuming tests, use option -LE slow, for example

ctest --output-on-failure --parallel 6 -LE slow