Commit cf480361 authored by Victor Poughon's avatar Victor Poughon
Browse files

DOC: migrate developer's guide to the cookbook

parent 735a69d9
Pipeline #319 passed with stage
in 27 minutes and 34 seconds
import argparse
import re
import os
import os.path
from os.path import join
import subprocess
def sed(content, regex, repl):
return re.sub(regex, repl, content, flags = re.MULTILINE | re.DOTALL)
if __name__ == "__main__":
parser = argparse.ArgumentParser(usage="migrate sg tex file")
parser.add_argument("filename", help="")
parser.add_argument("output_dir", help="")
args = parser.parse_args()
input = args.filename
output = join(args.output_dir, os.path.basename(args.filename).replace(".tex", ".rst"))
content = open(input).read()
content = sed(content,
r"\\doxygen\{otb\}\{(.*?)\}",
r":doxygen:`\1`")
content = sed(content,
r"\\doxygen\{itk\}\{(.*?)\}",
r":doxygen-itk:`\1`")
content = sed(content,
r"\\code\{(.*?)\}",
r"\\texttt{\1}")
content = sed(content,
r"cmakecode",
r"verbatim")
content = sed(content,
r"cppcode",
r"verbatim")
content = sed(content,
r"\\input\{(.*?)\}",
r"See example :ref:`\1`")
content = sed(content,
r"\\input (\w+)\n",
r"See example \1\n")
content = sed(content,
r"\\begin\{figure\}",
r"\\begin{verbatim}\\begin{figure}")
content = sed(content,
r"\\end\{figure\}",
r"\\end{figure}\\end{verbatim}")
open(output, "w").write(content)
subprocess.check_call("pandoc -f latex -t rst -o {} {}".format(output, output), shell=True)
subprocess.check_call(['sed', '-i', "s ‘ ` g", output])
print(output)
......@@ -95,9 +95,10 @@ def render_example(filename, otb_root):
rst_description = ""
# Render the template
name = os.path.basename(filename)
template_example = open("templates/example.rst").read()
output_rst = template_example.format(
label="example-" + root,
label=name,
heading=rst_section(name, "="),
description=rst_description,
usage=example_usage,
......@@ -108,7 +109,7 @@ def render_example(filename, otb_root):
return output_rst
if __name__ == "__main__":
def main():
parser = argparse.ArgumentParser(usage="Export examples to rst")
parser.add_argument("rst_dir", help="Directory where rst files are generated")
parser.add_argument("otb_root", help="OTB repository root")
......@@ -130,3 +131,6 @@ if __name__ == "__main__":
os.makedirs(join(args.rst_dir, "C++", "Examples", tag), exist_ok=True)
with open(join(args.rst_dir, "C++", "Examples", tag, root + ".rst"), "w") as output_file:
output_file.write(render_example(filename, args.otb_root))
if __name__ == "__main__":
main()
......@@ -360,7 +360,7 @@ def multireplace(string, replacements):
def make_links(text, allapps):
"Replace name of applications by internal rst links"
rep = {appname: ":ref:`{}`".format("app-" + appname) for appname in allapps}
rep = {appname: ":ref:`{}`".format(appname) for appname in allapps}
return multireplace(text, rep)
def render_application(appname, allapps):
......@@ -374,7 +374,7 @@ def render_application(appname, allapps):
application_documentation_warnings(app)
output = template_application.format(
label="app-" + appname,
label=appname,
heading=rst_section(app.GetName(), '='),
description=app.GetDescription(),
longdescription=make_links(app.GetDocLongDescription(), allapps),
......
......@@ -4,5 +4,13 @@ C++ API
=======
.. toctree::
:maxdepth: 2
C++/SystemOverview.rst
C++/Iterators.rst
C++/Filters.rst
C++/StreamingAndThreading.rst
C++/PersistentFilters.rst
C++/WriteAnApplication.rst
C++/AddingNewModules.rst
C++/Examples.rst
Adding New Modules
==================
This chapter is concerned with the creation of new modules. The
following sections give precise instructions about:
- the organization of directories
- the included files
- what they must contain
- ...
How to Write a Module
---------------------
There is a template of OTB remote module which help you start developing
a remote module: `External Module
Template <https://gitlab.orfeo-toolbox.org/remote_modules/remote-module-template>`__.
Each module is made of different components, described in the
following sections.
The otb-module.cmake file
-------------------------
This file is mandatory. It follows the CMake syntax, and has two
purposes:
- Declare dependencies to other modules,
- Provide a short description of the module purpose.
These purposes are fulfilled by a single CMake Macro call:
.. code-block:: cmake
otb_module(TheModuleName DEPENDS OTBModule1 OTBModule2 ... OTBModuleN DESCRIPTION "A description string")
**Note**: You can use the keyword ``TESTDEPENDS`` to declare module
dependencies that only applies to the tests.
The CMakeLists.txt file
-----------------------
The ``CMakeLists.txt`` file is mandatory. It contains only a few things.
First, it declares a new CMake project with the name of the module:
.. code-block:: cmake
project(TheModuleName)
Second, if the module contain a library (see src folder section below),
it initializes the TheModuleNameLIBRARIES CMake variable (if your module
only contains headers or template code, skip this line):
.. code-block:: cmake
set(TheModuleName_LIBRARIES OTBTheModuleName)
You can build your remote modules inside the OTB source tree by copying
your source inside the directory ``Module/Remote`` or against an existing
OTB build tree (note that it does not work with an install version of
OTB).
The configuration below will handle both cases and take care of all the
CMake plumbing of the module:
.. code-block:: cmake
if(NOT OTB_SOURCE_DIR)
find_package(OTB REQUIRED)
list(APPEND CMAKE_MODULE_PATH ${OTB_CMAKE_DIR})
include(OTBModuleExternal)
else()
otb_module_impl()
endif()
The overall file should look like this:
.. code-block:: cmake
cmake_minimum_required(VERSION 2.8.9)
project(TheModuleName)
set(ExternalTemplate_LIBRARIES OTBTheModuleName)
if(NOT OTB_SOURCE_DIR)
find_package(OTB REQUIRED)
list(APPEND CMAKE_MODULE_PATH ${OTB_CMAKE_DIR})
include(OTBModuleExternal)
else()
otb_module_impl()
endif()
The include folder
------------------
The include folder will contain all your headers (``*.h`` files) and
template method files (``*.hxx`` or ``*.hxx``). It does not require any
additional file (in particular, no CMakeLists.txt file is required).
The src folder
--------------
The src folder contains the internal implementation of your module:
- It typically contains cxx source files that will be compiled into a
library.
- It can contain header files for classes used only within the
implementation files of your module. Any header file present in the
src folder will not be installed, and will not be available to other
modules depending on your module.
If your modules is made of template only code, you do not need a src
folder at all.
If present, the src folder requires a CMakeLists.txt file.
The first part of the CMakeLists.txt file is classical, as it builds the
library and links it:
.. code-block:: cmake
set(OTBTheModuleName_SRC
sourceFile1.cxx
sourceFile2.cxx
sourceFile3.cxx
...
sourceFileN.cxx)
add_library(OTBTheModuleName ${OTBTheModuleName_SRC})
target_link_libraries(OTBTheModuleName ${OTBModule1_LIBRARIES} ${OTBModule2_LIBRARIES} ... ${OTBModuleN_LIBRARIES})
**Notes**:
- Library name should match the one declared in the root CMakeLists.txt
when setting CMake variable TheModuleNameLIBRARIES,
- Linked libraries should match the dependencies of your module
declared in the root otb-module.cmake file.
The last line of CMake code takes care of installation instructions:
.. code-block:: cmake
otb_module_target(TBTheModuleName)
The overall CMakeLists.txt file should look like:
.. code-block:: cmake
set(OTBTheModuleName_SRC
sourceFile1.cxx
sourceFile2.cxx
sourceFile3.cxx
...
sourceFileN.cxx)
add_library(OTBTheModuleName ${OTBTheModuleName_SRC})
target_link_libraries(OTBTheModuleName ${OTBModule1_LIBRARIES} ${OTBModule2_LIBRARIES} ... ${OTBModuleN_LIBRARIES})
otb_module_target(TBTheModuleName)
The app folder
--------------
The app folder contains the code of applications shipped with your
module. If your module has no application, you do not need the app
folder.
**Notes**: If your module contains application (and an app folder), do
not forget to add the ApplicationEngine in the dependencies listed in
the otb-module.cmake file.
In addition to the applications source code, the app folder should
contain a CMakeLists.txt file as follows.
For each application, a single call otbcreateapplication is required:
.. code-block:: cmake
otb_create_application(
NAME TheModuleApplication1
SOURCES TheModuleApplication1.cxx
LINK_LIBRARIES ${OTBModule1_LIBRARIES} ${OTBModule2_LIBRARIES} ... ${OTBModuleN_LIBRARIES})
The test folder
---------------
This folder contains tests of the module. If your module has no test in
it (which is not recommended, you do not need it).
The test folder should contain the source files of tests, as well as a
CMakeLists.txt file. This file will contain the following.
First, indicate that this folder contains tests.
.. code-block:: cmake
otb_module_test()
Then, build the test driver:
.. code-block:: cmake
set(OTBTheModuleNameTests
testFile1.cxx
testFile2.cxx
...
testFileN.cxx)
add_executable(otbTheModuleNameTestDriver ${OTBTheModuleNameTests})
target_link_libraries(otbTheModuleNameTestDriver ${OTBTheModuleName-Test_LIBRARIES})
otb_module_target_label(otbTheModuleNameTestDriver)
Finally, you can add your tests:
.. code-block:: cmake
otb_add_test(NAME nameOfTheTest COMMAND otbTheModuleNameTestDriver
--compare-image ${EPSILON_8} ... # baseline comparison if needed
nameOfTheTestFunction
testParameters)
If your module contains one or more application in the app folder, you
should also write tests for them, in the test folder. Running an
application test is easily done with the helper macro
otbtestapplication:
.. code-block:: cmake
otb_test_application(NAME nameofApplication1Test1
APP TheModuleApplication1
OPTIONS -in1 ${INPUTDATA}/input1.tif
-in2 ${INPUTDATA}/input2.tif
-out ${TEMP}/nameofApplication1Test1_result.tif
VALID --compare-image ${EPSILON_8}
${BASELINE}/nameofApplication1Test1_result.tif
${TEMP}/nameofApplication1Test1_result.tif)
Overall CMakeLists.txt should look like:
.. code-block:: cmake
otb_module_test()
set(OTBTheModuleNameTests
testFile1.cxx
testFile2.cxx
...
testFileN.cxx)
add_executable(otbTheModuleNameTestDriver ${OTBTheModuleNameTests})
target_link_libraries(otbTheModuleNameTestDriver ${OTBTheModuleName-Test_LIBRARIES})
otb_module_target_label(otbTheModuleNameTestDriver)
otb_add_test(NAME nameOfTheTest COMMAND otbTheModuleNameTestDriver
--compare-image ${EPSILON_8} ... # baseline comparison if needed
nameOfTheTestFunction
testParameters)
Including a remote module in OTB
--------------------------------
* Local build of a remote module
Your remote module can be build inside the OTB source tree or outside as
a external CMake project with an existing OTB. Please note in that case
that you’ll have to set OTBDIR CMake option.
If OTBDIR is an OTB build tree, there are two ways of compiling:
- Build as a module, in which case build files will be written to the
OTB build tree as other modules. Main benefit is that this will
enrich the current OTB build with your new module, but you need to
have write access to the build directory.
- Build as a standalone CMake project, in which case build files will
remain in the module build folder. This build is fully independent
from the build (or install) directory, but the module will not be
recognized as an OTB module (still you will be able to use its
binaries and libraries).
This behaviour is controlled by the ``OTB_BUILD_MODULE_AS_STANDALONE``, which is
OFF by default (hence first behaviour).
Note that when dealing with an installed OTB, only the second behaviour
(build as standalone) is available.
Optionally, you can build your new remote module inside the OTB source
tree by simply copy the folder containing the module component to
Modules/Remote, then run CMake configuration. you should see a new CMake
option named MODULETheModuleName. Simply turn this option to ON, and
finish CMake configuration. Your module will be built with the rest of
the OTB project.
* Sharing your remote module
To make your remote module available to others when building OTB, you
should provide a CMake file named TheModuleName.remote.cmake file for
inclusion in the Modules/Remote folder in OTB source tree.
This file should contain the following:
.. code-block:: cmake
# Contact: Author name <author email address>
otb_fetch_module(TheModuleName
"A description of the module, to appear during CMake configuration step"
GIT_REPOSITORY http_link_to_a_git_repository_hosting the module
GIT TAG the git revision to checkout
)
This file should be provided along with your remote module inclusion
proposal email to the otb community list. Please refer to the
contributors guidelines for more information.
This diff is collapsed.
This diff is collapsed.
Persistent filters
==================
Introduction
------------
As presented in chapter :ref:`StreamingAndThreading`, OTB has two main mechanisms
to handle large data: streaming allows to process image piece-wise, and
multi-threading allows to process concurrently several pieces of one streaming
block. Using these concepts, one can easily write pixel-wise or
neighborhood-based filters and insert them into a pipeline which will be
scalable with respect to the input image size.
Yet, sometimes we need to compute global features on the whole image.
One example is to determine image mean and variance of the input image
in order to produce a centered and reduced image. The operation of
centering and reducing each pixel is fully compliant with streaming and
threading, but one has to first estimate the mean and variance of the
image. This first step requires to walk the whole image once, and
traditional streaming and multi-threading based filter architecture is
of no help here.
This is because there is a fundamental difference between these two
operations: one supports streaming, and the other needs to perform
streaming. In fact we would like to stream the whole image piece by
piece through some filter that will collect and keep mean and variance
cumulants, and then synthetize theses cumulants to compute the final
mean and variance once the full image as been streamed. Each stream
would also benefit from parallel processing. This is exactly what
persistent filters are for.
Architecture
------------
There are two main objects in the persistent filters framework. The
first is the :doxygen:`PersistentImageFilter`, the second is the
:doxygen:`PersistentFilterStreamingDecorator`.
The persistent filter class
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The :doxygen:`PersistentImageFilter` class is a regular
:doxygen-itk:`ImageToImageFilter`, with two additional pure virtual
methods: the ``Synthetize()`` and the ``Reset()`` methods.
Imagine that the ``GenerateData()`` or ``ThreadedGenerateData()``
progressively computes some global feature of the whole image, using
some member of the class to store intermediate results. The
``Synthetize()`` is an additional method which is designed to be called
one the whole image has been processed, in order to compute the final
results from the intermediate results. The ``Reset()`` method is
designed to allow the reset of the intermediate results members so as to
start a fresh processing.
Any sub-class of the :doxygen:`PersistentImageFilter` can be used as a
regular :doxygen-itk:`ImageToImageFilter` (provided that both
``Synthetize()`` and ``Reset()`` have been implemented, but the real
interest of these filters is to be used with the streaming decorator
class presented in the next section.
The streaming decorator class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The :doxygen:`PersistentFilterStreamingDecorator` is a class designed to
be templated with subclasses of the :doxygen:`PersistentImageFilter`. It
provides the mechanism to stream the whole image through the templated
filter, using a third class called
:doxygen:`StreamingImageVirtualWriter`. When the ``Update()`` method is
called on a :doxygen:`PersistentFilterStreamingDecorator`, a pipeline
plugging the templated subclass of the :doxygen:`PersistentImageFilter`
to an instance of :doxygen:`StreamingImageVirtualWriter` is created. The
latter is then updated, and acts like a regular
:doxygen:`ImageFileWriter` but it does not actually write anything to
the disk : streaming pieces are requested and immediately discarded. The
:doxygen:`PersistentFilterStreamingDecorator` also calls the ``Reset()``
method at the beginning and the ``Synthetize()`` method at the end of
the streaming process. Therefore, it packages the whole mechanism for
the use of a :doxygen:`PersistentImageFilter`:
#. Call the ``Reset()`` method on the filter so as to reset any
temporary results members,
#. Stream the image piece-wise through the filter,
#. Call the ``Synthetize()`` method on the filter so as to compute the
final results.
There are some methods that allows to tune the behavior of the
:doxygen:`StreamingImageVirtualWriter`, allowing to change the image
splitting methods (tiles or strips) or the size of the streams with
respect to some target available amount of memory. Please see the class
documentation for details. The instance of the
:doxygen:`StreamingImageVirtualWriter` can be retrieved from the
:doxygen:`PersistentFilterStreamingDecorator` through the
``GetStreamer()`` method.
Though the internal filter of the
:doxygen:`PersistentFilterStreamingDecorator` can be accessed through
the ``GetFilter()`` method, the class is often derived to package the
streaming-decorated filter and wrap the parameters setters and getters.
An end-to-end example
---------------------
This is an end-to-end example to compute the mean over a full image,
using a streaming and threading-enabled filter. Please note that only
specific details are explained here. For more general information on how
to write a filter, please refer to section :ref:`Filters`.
First step: writing a persistent filter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The first step is to write a persistent mean image filter. We need to
include the appropriate header :
.. code-block:: cpp
#include "otbPersistentImageFilter.h"
Then, we declare the class prototype as follows:
.. code-block:: cpp
template<class TInputImage>
class ITK_EXPORT PersistentMeanImageFilter :
public PersistentImageFilter<TInputImage, TInputImage>
Since the output image will only be used for streaming purpose, we do
not need to declare different input and output template types.
In the *private* section of the class, we will declare a member which
will be used to store temporary results, and a member which will be used
to store the final result.
.. code-block:: cpp
private:
// Temporary results container
std::vector<PixelType> m_TemporarySums;
// Final result member
double m_Mean;