Testing¶
Why set-up a testing policy ?¶
Code_TYMPAN is quite a big application and one can not expect that some fix or new feature will just work : a fix somewhere can - and will - trigger a bug or a mere unexpected behaviour later and elsewhere… In order to circumvent this major source of complexity it is crucial to cover new bug and features with automated tests.
Automated tests are testing procedures :
which can be run automatically, without requiring for a user to perform a series of actions in the GUI typically
which are implemented a piece of code running them
which will either succeed in case what they are intended to check is asserted or fail which does not mean crashing on an error but instead reporting this was expected but it got that instead.
Moreover, it is much easier to debug a small automated test than a complete application like Code_TYMPAN. So coding a test should be done at the same time - or even before - coding the actual feature so that the developer can draw the most benefit from it while devising the feature.
What is a testing framework ?¶
Now, writing a stand alone program to test each new feature or bug
newly reported can be a tedious task. That’s why automated testing
framework exists : to make this most important task lighter. They allow
to concentrate on asserting the feature or bug, not the plumbing of
the reporting (see tests/test_examples.cpp
).
In Code_TYMPAN we use the testing framework GTest aka Google Test : it was chosen for it simplicity and the quality of its documentation. You are strongly encouraged to read the primer now.
GTest (like most unit testing frameworks) builds upon the notion of
test case : a test case represent one simple setting in which you
want to assert some behaviour of your feature, or lack of some bug.
You create a test case (which actually is a class under the hood) with
the TEST
macro:
TEST(TestSuite, testname)
{
// Set up the objects and their state
...
EXPECT_EQUAL(expected, value_tested);
...
}
When you run the resulting executable, the body of each test case in
your .cpp
file will be run separately, just like the main
function of as many small programs, and reported as pass, fail or
error. In case of failure each failing EXPECT
or ASSERT
will
be detailed in the report.
It is a good practice to organise automated tests in a directory tree by the components or interactions they test. Ideally each class and major interaction should be tested in its own test file. For a complex application like Code_TYMPAN there will thus be a whole tree a tests to run, that’s why a test driver is most useful.
How to launch automated tests ?¶
Launching by hand all the automated tests of the tests
tree
and aggregating their reports would be a daunting task. Here the
CTest test driver comes into play. CTest is a companion to CMake
which uses information already available in CMake to automate the
execution and reporting of a whole set of tests.
In order to be able to run tests with CTest, one must set the environement before. Section Running Tests gives the way to do so using command lines.
Another way to use CTest is to launch the tests with Visual Studio. In this case, environement must also be set. The script 0_Launch_Visual_Studio_with_env.bat allows to do so.
CTest accepts numerous options.
-C
indicates the configuration which can be Debug or Release in our case.
With --output-on-failure
option, the test will be quiet if successfull
and verbose if it fails.
The single -V
option make CTest a bit more verbose and a double
-VV
makes it a lot more verbose. You can also filter (out) by test
executable name (with reg-exp if needed) and many advanced features not
covered here.
Test coverage¶
In order to be representative, the tests must execute a maximum of code when running.
At this time, the study and the results of test coverage are documented in this issue : https://gitlab.com/tympan/code_tympan/-/issues/324
Writing automated tests¶
Todo
Best practices. How can you do.
Data for Testing¶
The directory tests/data
stores various pieces of data used by
some of the automated tests.
For example the tests/data/projects-panel
provides a set of
XML files dedicated to test some features. These files deal with very
simple Code_TYMPAN project: single site, one building, one pair of
source/receptor, etc. which are very useful for testing. Moreover, for
each of them, a typical screenshot is provided in
doc/_static/images/tests/
By hand validation¶
Because the automated tests coverage is yet unsufficient, and because some high level, global validation procedure is always useful a by hand validation procedure is proposed.
For now it this procedure is documented in this external
PDF document
and the
resources it refers to are stored under
tests/data/manual_validation/
.
Tips and Tricks¶
Launch Unit tests¶
cd c:/projects/code_tympan_build_d
SetEnvTympanTests.bat
## It is necessary to set the TYMPAN_PYTHON_INTERP variable for the test_m_b_altimetrie.cpp test because it calls the altimetry module
set TYMPAN_PYTHON_INTERP=C:\dists\python\venv310tympan\Scripts\python.exe
set TYMPAN_SOLVERDIR=C:\projects\code_tympan_install\pluginsd
ctest -C Debug
To run a specific test:
ctest -C Debug -R test_altimetry_builder -V
If a test blocks at number %N%, it can be necessary to kill Python in the task manager and restart with:
ctest -C Debug -I %N%
Launch Python tests¶
cd C:\projects\code_tympan_build_d\python\tests
python -m unittest discover
Debug PYTHON test :
C:\dists\python\venv310tympan\Scripts\python.exe -m pdb "C:/projects/code_tympan_build_d/python/tests/test_altimetry_roads.py" "-v"
or add in the python file :
breakpoint()
Debug¶
To keep temporary files set the environment variable:
set TYMPAN_DEBUG=keep_tmp_files;monothread
To debug by running a Python script please set:
set TYMPAN_DEBUG=keep_tmp_files;interactive;monothread
To Activate the virtual environment:
C:\dists\python\venv310tympan\Scripts\activate.bat
To debug a calculation launched by Python with UseVolumesLandtake set to False:
python C:/projects/code_tympan_install_d/bin/solve_tympan_project.py
C:\projects\tympan\path_to_file\input_uvl_false.xml
C:\projects\tympan\path_to_file\result_uvl_false.xml
C:\projects\tympan\path_to_file\mesh_uvl_false.ply
C:/projects/code_tympan_install_d/pluginsd 0.0 True False
With UseVolumesLandatke to True :
python C:/projects/code_tympan_install_d/bin/solve_tympan_project.py
C:\projects\tympan\path_to_file\input_uvl_true.xml
C:\projects\tympan\path_to_file\result_uvl_true.xml
C:\projects\tympan\path_to_file\mesh_uvl_true.ply
C:/projects/code_tympan_install_d/pluginsd 0.0 True True