Skip to content

Refactoring task for the modularization of OTB

New modular architecture

otb_9_archi

1. Simplify CMake options

In OTB there is almost 40 cmake options when you call ccmake which is a bit confusing for the average user wanting to compile OTB for his project. A lot of option are about the dependencies of OTB, where you can mix system and superbuild dependencies, which can lead to eratic behaviors which the OTB team can't give a lot of time to debug.

  • Propose a simplified approach for the dependencies:

The user who wants to compile OTB will be given two options:

  • Use all dependencies from the Superbuild which will be built before OTB
  • Use all system dependencies, which will be listed in the cookbook for a one line simple installation

This approach permits to hide all USE_SYSTEM_DEPNAME flags which complicates the cmake process. The user will have to enable or disable the only flag USE_SUPERBUILD_DEPS which will activate/deactivate the flags USE_SYSTEM_XXX in the background

  • Review of all other cmake variables to determine which ones are mandatory, and which ones can be hidden/removed

  • Create OTB_BUILD_CORE, OTB_BUILD_LEARNING ... cmake targets to compile thematic modules

Instead of the current OTBGroup_XXX which is not thematic and can lead to misunderstandings for beginners, we propose to add options which will correspond to the future modules of OTB, which will be packaged with the same name.

2. New source organisation

The main idea is to reorganize the modules of OTB in coherent groups, which implies a reorganization of the source code. Here is an example of a module (Core for example):

  • Applications
  • Functional_Tests : which will call one or a bunch of applications related to real treatments in production chains
  • SubModule1... (For example IO)
    • src
    • include
    • unit_tests
  • SubModule2... (For example Common)
    • src
    • include
    • unit_tests
  • ...

With this organization, we have the applications related to this module, the tests calling those applications, and a list of submodules which contains filters/C++ classes grouped by theme. These filters are unit tested in each sub module. To do this reorganization, the following guideline has been done:

  • Create a folder for each module (Learning, Stereo, SAR...)
  • Move src, include, tests and the related CMakeLists of each submodule to its related module
  • Connect CMAKE variables (OTB_BUILD_CORE etc) to the right module folder
  • Adapt the declared dependencies of each module in the CMakeLists
  • Reactivate the unit tests on the CI for each module
  • Reactivate the app tests ont the CI for each module
  • Package OTB Core and OTB Full with the current packaging method
  • Test the compilation of remote modules with otb core
  • move "to be deprecated" apps to a miscellaneous folder

3. Test refactoring

All the tests (TU + app) for each modules have been reactivated on the CI. Now it is important to see if all those tests are relevant.

  • Unit test + app tests

A Lot of baselines and input has been added since the beginning of the project, but some tests have been deprecated. In order to make this refactoring relevant, we will follow this guideline :

  • Make a listing of currently used baselines/input files, to see what file is not needed anymore, and if some files are not too old to be relevant.

  • See if there are any non relevant tests

  • Determine which largeInput tests can be transformed into functional tests

  • Use cases for validation (fonctional tests)

The current LargeInput tests are kind of outdated and not really relevant of real production chains. The idea is to replace thoses tests with functional tests, containint one or a sequence of applications, which is the case in production chains. We have to determine real use cases with production teams, to create those functional tests. They will be present in each thematic module.

While the unit test + app tests will be run on the CI, the new functional tests will be run on the HAL cluster

4. New packaging

The current packaging of OTB contains everything, all dependencies, all apps... Makeself is currently used to deliver a .run. At the moment, there is an OTB Core package, and a full OTB package produced by the CI. The idea is to use CPack which is integrated with cmake, to deliver packages of each OTB modules, that can install along the otb-core package.

For example, a user needs to do some machine learning with OTB, he should then install otb-core package which contains the core modules / apps of OTB and its dependencies, and then he will install otb-ML module, which will be installed in the same install tree.

This modularity permits each user to customize its installation, instead of having a big package while using 50% of it.

Edited by Thibaut ROMAIN