.. _devel-tools: ======= Tools ======= .. :authors: SEGURET Aymeric - externe RIPOLLES Jean - EDF This section provides some introduction to using third-party tools or libraries on which Code_TYMPAN relies. Those dependencies are introduced, with installation instructions in the section :ref:`devel-installation`. In this section we give some entry-points regarding use of the tools and libraries. We also explain some *not so common* C++ idoms used in some portion of the code. Accessing the source-code ========================= The source code of the Code_TYMPAN project is versionned using the Git_ Version Control System. On GNU/Linux the simplest way to install is through your distribution package manager (sudo apt-get install git). The software is also available on Windows and a good alternative is to use the Tortoise application which integrates well with the Explorer (also available on GNU/Linux). Git_Extensions_ and SourceTree_ are interesting software allowing navigation and visualization of the repository as well as the visualization of the corresponding trees. The following *named branches* are currently used: master The default branch name in Git is master. As you start making commits, you're given a master branch that points to the last commit you made. develop Is the current branch used to keep in sync the feature branches. Indeed, it is critical to avoid any significant divergence regarding the build system for example. Moreover some changes (many bug fixes, documentation improvement) can be put in common without interfering. release/x.x.x is the branch for the production version x.x.x Last version : 4.2.1 issue_xxx This type of branch allows the developer to work on different issues. Each branch corresponds to an issue which can be a bug fix, a function addition, an optimization etc... Once the developer has completed his development and validated, he delivers on the branch before creating a merge request which will be reviewed by a second person. Once this review is done, the branch can be merged into develop. Example of operation: * Alice is working on issue 14 which corresponds to a bug reported by a user. * She places itself on the develop branch and does an update of it. * She creates the Issue_14 branch from develop and works on it locally. * Once satisfied with its development and this one having been validated by the TU and possibly manual tests. She decides to deliver her modifications and therefore the new branch. * Once the pipeline is validated, Alice creates a merge request corresponding to her issue and her branch and requests a review from TOM. * Tom does the code review and eventually retrieves the branch locally for testing (if necessary) * Tom can possibly provide feedback to Alice to improve the work. But once this is ok, Tom validates the merge request. * The Issue_14 branch can then be merged into develop and the corresponding issue can be closed (after demo to the rest of team) Other existing branch : * hotfix_branch : used for hotfixes, mainly on new releases * fix_ci_appveyor : used for pipeline maintenance and contains modification for pipeline debug Of course other feature branches can be added by need. .. _Git: https://git-scm.com/downloads .. _Git_Extensions: https://sourceforge.net/projects/gitextensions/ .. _SourceTree: https://www.sourcetreeapp.com/ .. _devel-tools-cmake: CMake ===== `CMake`_ is an open-source build system. It is used to control the software compilation process using a simple platform and compiler independent configuration files. CMake generates native makefiles and workspaces that can be used in the compiler environment of your choice. On GNU/Linux, CMake generates Makefiles and on Windows, it can generate Visual Studio solutions. Using CMake ----------- First, create a directory where you'll put the building of Code_TYMPAN. **Command line** Do:: cmake path/to/Code_TYMPAN or:: ccmake path/to/Code_TYMPAN cmake-gui path/to/Code_TYMPAN to launch the *ncurses* or *Qt* graphical interfaces respectively. The command line is very helpful to get some info about a command, a variable, etc.:: # List of CMake commands cmake --help-command-list # List of CMake variables cmake --help-variable-list # Get doc for 'add_library' command cmake --help-command add_library # Get doc for 'CMAKE_BUILD_TYPE' variable cmake --help-variable CMAKE_BUILD_TYPE .. note:: A CMake 'command' is like a function. Documentation about CMake commands gives you indication about their different arguments etc. **GUI** You can use either ``ccmake`` or ``cmake-gui``. Note that ``ccmake`` is only available on GNU/Linux. On Windows, you should use ``cmake-gui`` (see the page `Running CMake on Windows `_). **Documentation** Getting help on CMake thanks to ``--help`` command lines as above. Moreover, there are other resources: - The reference book `Mastering CMake `_ - The `online documentation `_ with the `cmake tutorial `_, a `Wiki `_ and a `FAQ `_. .. _CMake: https://cmake.org/ Add a library to the project ---------------------------- Suppose you create a new directory with C++ headers and compilation units. The best way is to create a ``CMakeLists.txt`` file in this current directory and write: .. code-block:: cmake # The name of your new module (i.e. library) set(MODULE_NAME new_tympan_module_name) # Group all source files in a CMake list variable. file(GLOB ${MODULE_NAME}_SRCS *.cpp *.h) # Create the library as ('library_name' 'list of source files') add_library(${MODULE_NAME} ${${MODULE_NAME}_SRCS}) You can further refer to your module with the variable ``MODULE_NAME``. You can also use (if needed): .. code-block:: cmake # If your module depends on other Tympan modules (i.e. other Tympan libraries). add_dependencies(${MODULE_NAME} tympan_common tympan_lib) # If you have to link your library with one or a few libraries such as Boost, # Qt or OpenGL. target_link_libraries(${MODULE_NAME} ${QT_LIBRARIES}) # If you would like to add a specific directory to the include path. A # 'include_dir' is missing when you have a compilation error such as 'header.h # no such file or directory'. include_directory() Best Practices -------------- * Do not build a project in the source directory. * Do not add a manual path to look for a dependency library or to find a source file : instead edit the said path in the GUI (or code *reliable* auto-detection in case you are a seasoned CMakeLists' developer). * CMake language is not case-sensitive for identifiers, but is for some optional arguments names within commands. ``VARIABLE_NAME`` or ``variable_name`` are the same variable but ``FILE(EXISTS ...)`` will be ok whereas ``FILE(exists ...)`` will **not**. Please try to follow the existed convention in reading some ``CmakeLists.txt``. * Create a ``CMakeLists.txt`` file for each directory. Use ``add_subdirectory`` command to execute the CMake file in this directory. * How to clean the CMake cache: remove the ``CMakeCache.txt`` file. * Display the value of a variable --- useful for debugging:: message (STATUS "Your message: " ${VARIABLE}) * ``ccmake .`` or ``cmake-gui .`` in the building directory to display and edit all CMake variables of the current building project. * ``cmake --build `` to launch compilation, whatever the environment. Very useful when you would like to compile on Windows without launching Visual Studio. Documentation ============= The C++ source code is annotated and API documentation can be extracted with Doxygen_. The main documentation is generated from reStructuredText_ sources in the ``doc`` directory thanks to Sphinx_. Build the documentation ----------------------- On Windows ;;;;;;;;;; #. Go to the root source documentation directory and:: make.bat html #. See the results in the ``_build/html/`` directory and opening the file ``index.html`` with your favorite Web browser. #. Check the external links of the documentation with:: make.bat linkcheck On Linux ;;;;;;;; #. Go to the root source documentation directory and:: make html #. See the results in the ``_build/html/`` directory and opening the file ``index.html`` with your favorite Web browser. #. Check the external links of the documentation with:: make linkcheck .. note:: The tools used to display maths equations properly is `MathJax`_, a JavaScript library which prettily renders equations written in LaTeX. Useful directives ----------------- Take a look at the `Sphinx`_ and `reStructuredText`_ websites. * ``.. note::`` write a note. * ``.. code-block:: c++`` piece of code with the C++ syntax color. * ``:ref:`section_name``` cross reference with a different location such as a section of an other file. * ``.. todo::`` add a todo task * ``.. doxygenclass:: `` show the Doxygen documentation of the class ````. * ``.. doxygenfunction:: `` show the Doxygen documentation of the class ````. .. note:: See other Doxygen directives in the official `Doxygen`_ website. * ``:math:`x^2 -2x +1=0``` or ``.. math::`` to write LaTeX equations. .. warning:: A Web connection is needed to be able to see nice equations. .. _reStructuredText: https://docutils.sourceforge.io/rst.html .. _Sphinx: https://www.sphinx-doc.org/en/master/ .. _Doxygen: https://www.doxygen.nl/index.html .. _MathJax: https://www.mathjax.org/ .. _formatting-and-linting: Formatting and Linting ======================= We use formatters to automatically ensure consistency across the code base and thus reduce the amount of stylistic choices the developers must make. Linters are used to detect easily discoverable code smells. The following tools should be used to ensure that a minimal level of quality is preserved. On Gitlab, new commits trigger a pipeline which runs all these tools and fails when any of them reports an error. When creating a Merge Request, make sure that all these checks pass. Linting python files with Flake8_ --------------------------------- Flake8_ is a python linter which both checks for pep8_ compliance and find code smells. If it is not already installed you can install it using pip:: $ pip install flake8 Assuming that ``flake8`` is in your ``PATH`` (which should be the case if you installed it in your active virtualenv), it can be run from command line:: $ flake8 :2:1: F401 'this' imported but unused Formatting python files with black_ ----------------------------------- black_ is used to format python files. It can be installed using pip:: $ pip install black When calling black with a file or directory, all the passed files are formatted:: $ black config_gui.py reformatted config_gui.py All done! 1 file reformatted. Linting cython files with cython-lint_ -------------------------------------- cython-lint_ is a cython linter which both checks for pep8_ compliance and find code smells. If it is not already installed you can install it using pip:: It is an equivalent to the flake8 library for cython files. $ pip install cython-lint Assuming that ``cython-lint`` is in your ``PATH`` (which should be the case if you installed it in your active virtualenv), it can be run from command line:: $ cython-lint --max-line-length 120 :2:1: F401 'this' imported but unused Formatting C++ files with clang-format_ --------------------------------------- For C++ source and header files, the formatting is done using clang-format_. On Debian based linux distributions it can be installed with apt:: $ sudo apt install clang-format-15 On Windows clang-format is distributed as part of LLVM_. You can download the installer here: https://github.com/llvm/llvm-project/releases The current version to install is 15.0.6. When running installation, make sure that one of ``Add LLVM to the system PATH for all users`` and ``Add LLVM to the system PATH for current user`` is checked. Once the installation is finished, Check that it is correctly installed by running ``clang-format --version``. It should output something like ``clang-format version 15.0.6``. .. warning:: Some formatting rules need a version of clang-format greater than 11. On some linux distributions, the ``clang-format`` command is bound to an old version of clang-format_ (version 7 on debian buster). You have to ensure that your editor can find a version greater than or equal to 11 of clang-format. You can easily check that the whole C++ code base is correctly formatted by running the script ``tools/check_format.py``. Experimenting clang-tidy_ ------------------------- clang-tidy can be used to enforce coding rules like C++ core guidelines https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines. It is included in the LLVM distribution installed before. A first experimentation is done on Code_TYMPAN by implementing the rule ``cppcoreguidelines-init-variables``. The ``.clang-tidy`` configuration file is used at the root of the source tree. In order to run clang-tidy, we must generate a compilation database following this guide : https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html#setup-clang-tooling-using-cmake-on-windows To run clang-tidy, we can use cmder and run this command : .. code-block:: winbatch find ..\code_tympan\Tympan\core -iname "*.hpp" -o -iname "*.h" -o -iname "*.cpp" -type f -print0 | xargs -0 clang-tidy -p C:\projects\code_tympan_build_ninja_d\compile_commands.json --fix Launch linters from cmake ------------------------- You can launch all the linters with cmake:: $ cmake --build . --target lint Integration with IDEs --------------------- Usage with Vim_ ;;;;;;;;;;;;;;; In Vim_, all these linters and formatters can be automatically used by installing ALE. Assuming you are using version 7 or greater of Vim, setup is made as follow (on linux or using git-bash):: $ mkdir -p ~/.vim/pack/lint/start $ git clone --depth 1 https://github.com/dense-analysis/ale.git ~/.vim/pack/lint/start/ale Lines with warnings should now be highlighted. The warnings are displayed at the bottom of the screen when the cursor is on these lines. black and clang-format can be activated by adding the following lines to your ``~/.vimrc`` file:: let g:ale_fixers = { \ 'python': ['black'], \ 'cpp': ['clang-format'], \} " Set this variable to 1 to fix files when you save them. let g:ale_fix_on_save = 1 Running clang-format on Visual Studio ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; clang-format_ is already available in Visual Studio. When opening the project, the clang configuration file should be automatically detected and Visual Studio should ask you if you want to enable clang-format support. If that's not the case you can enable it by checking ``Tools->Options->Text Editor->C/C++->Formatting->General->Enable ClangFormat support``. For the moment being, Visual Studio Community 2019 integrates the version 12.0.0 of clang-format. You must specify the version of clang-format you wish to use (15.y.z) by selecting clang-format exe in ``Tools->Options->Text Editor->C/C++->Code Style->Formatting->General->Use custom clang-format exe file``. You should now be able to format your code using the ``Ctrl-K, Ctrl-D`` keyboard shortcut. Install the extension named ``Format on Save`` if you want to run clang-format on save. Running flake8 in PyCharm ;;;;;;;;;;;;;;;;;;;;;;;;; Flake8 is not natively supported by PyCharm. You can add it as an external tool. Open ``Settings`` (Ctrl-Alt-S) and go to ``Tools->External Tools``. Then click the ``+`` button and fill in the form this way: .. figure:: _static/images/PyCharm_Flake8.png :align: center You can then run this external tool from ``Tools->linters->Flake8``. Running black in PyCharm ;;;;;;;;;;;;;;;;;;;;;;;; Black can be configured the same way as flake8 as an external tool: .. figure:: _static/images/PyCharm_Black.png :align: center The BlackConnect_ plugin uses a black server to avoid restarting a python interpreter each time the files are formatted. You can also configure it so that black run automatically when files are saved. You can install it from ``Settings->Plugins`` and configure it from ``Settings->Tools->BlackConnect``. You will have to install blackd (``pip install black[d]``) and run ``blackd`` in order for this plugin to work. .. _clang-format: https://clang.llvm.org/docs/ClangFormat.html .. _clang-tidy: https://clang.llvm.org/extra/clang-tidy/ .. _flake8: https://flake8.pycqa.org/en/latest/ .. _pep8: https://www.python.org/dev/peps/pep-0008/ .. _black: https://black.readthedocs.io/en/stable/ .. _cython-lint: https://github.com/MarcoGorelli/cython-lint .. _llvm: https://llvm.org .. _BlackConnect: https://plugins.jetbrains.com/plugin/14321-blackconnect .. _Vim: https://www.vim.org/ C++ idioms ========== C++ is a very complex language, full of subtleties and pitfalls... but some good practices, nice (template) libraries and *well identified idioms* can improve development efficiency and code base maintainability. In this section, we highlight some of such idioms and tools which are used (or to be used) in Code_TYMPAN. Anyhow we strongly recommend C++ developers to have a look at some the following books: Practical and motivated idioms Scott Meyers, Addison-Wesley, *Effective C++* and *Effective STL* Complete introduction of the language Bjarne Stroustrup, Addison-Wesley, *C++, the language* Some powerful techniques are exposed to the developer through *relatively* simple interfaces by Boost_ or the STL_, such as the standard smart pointers or the ranges concept and the *foreach loops* for example. .. note:: Further resources Here are some useful inline resources regarding C++ - `Introductory online C++ course` OpenClassRoomsCpp_ with corresponding printed book, in french - cplusplus.com `tutorial` cplusplus_tutorial_ or `reference` cplusplus_reference_ - Google hands-on, practical online course https://developers.google.com/edu/c++/ .. _Boost: https://www.boost.org/ .. _STL: https://en.cppreference.com/w/ .. _OpenClassRoomsCpp: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-096-introduction-to-c-january-iap-2011/ .. _cplusplus_tutorial: https://www.cplusplus.com/doc/tutorial/ .. _cplusplus_reference: https://www.cplusplus.com/reference/ Smart Pointers -------------- https://www.dreamincode.net/forums/topic/86846-tr1-intro-smart-pointers/ A very good, nice and accessible, introduction to smart pointers and modern C++ memory management. https://www.boost.org/doc/libs/1_53_0/libs/smart_ptr/smart_ptr.htm Official documentation for Boost smart pointers, among which ``shared_ptr`` and ``weak_ptr`` have been included in the TR1 and C++'11. BOOST_FOREACH ------------- Before C++'11 the language lacked a *foreach loop* construct to iterate other data structure in a convenient and readable way. Boost_ tries to workaround this limitation with the BOOST_FOREACH macro (which is some kind of black magic we should get rid of as soon as C++'11 is widespread and fully supported). For example, say `c` is some standard STL_ container of type ``C`` (i.e. with a ``begin()`` and ``end()`` method providing iterators to elements of type ``T``) then you can write: .. code-block:: c++ BOOST_FOREACH( T& e, c ) { // do something with the element e } instead of the usual and horribly verbose when ``C`` and/or ``T`` are complex: .. code-block:: c++ for( C::iterator it = c.begin() ; it != c.end() ; ++it ) { T& e = *it; // do something with the element e } https://www.boost.org/doc/libs/1_53_0/doc/html/foreach.html Official documentation for Boost *foreach loops*. The range concept ----------------- To be right to ``BOOST_FOREACH``, this construct is far more powerful than *just* iterating in a simple way through a container. It relies on the *Range* concept. The *Range* concept is a generalisation of the *input sequences* introduced by Stroustrup in *C++, the language* 18.3.1 : basically a *range* is something that can be seen as a pair of begin/end iterators. The whole point of ranges resides in the fact that they do not have to be STL container, or any data-structure, they can pretty-well represent a simple counter as well as an on-fly processing over an other range ! For example : .. code-block:: c++ /* ... */ struct double_int { typedef int result_type; int operator()(int x) const { return x * 2; } }; int main(int argc, const char* argv[]) { std::vector input; // An STL container is a range /* Fill `input` with 1, 2, 3, ... */ BOOST_FOREACH( int i, input | transformed(double_int()) ) { cout << i << ", "; // Will display 2, 4, 6, ... } return 0; } https://www.boost.org/doc/libs/1_70_0/libs/range/doc/html/index.html Official documentation for Boost *range concept* which goes along so well with *foreach loops*. **Beware** : *Ranges* are a very powerful but quite advanced - and thus pitfall prone - C++ technique. Please resort to it only if you know what you are doing, otherwise just transforming the data by-hand and storing the result in some container is likely to be a far easier and cleaner way...