otb issueshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues2022-01-07T11:47:24Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1885Use a test framework for unit tests (Follow-up from "Large refactoring of rad...2022-01-07T11:47:24ZJulien MichelUse a test framework for unit tests (Follow-up from "Large refactoring of radiometric indices")The following discussion from !464 should be addressed:
- [ ] @lhermitte started a [discussion](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/merge_requests/464#note_77961): (+3 comments)
> As this is a new test, what about us...The following discussion from !464 should be addressed:
- [ ] @lhermitte started a [discussion](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/merge_requests/464#note_77961): (+3 comments)
> As this is a new test, what about using a real unit test framework that'll display the bogus line?
In the discussion, `boost.test`, `catch2` and `doctest` are proposed. We should decide which one to pick, and identify current tests that could be refactored using a dedicated test framework.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1882Standard verbose in applications2022-01-06T13:56:37ZLuc HermitteStandard verbose in applications# TL;DR:
Have all OTB applications have a `-v` option.
# Rationale
It'd be, very, very nice to have `-v|-verbose` option in all OTB applications in standard. That's what we have everywhere in *nix world. I'm not picky to require that...# TL;DR:
Have all OTB applications have a `-v` option.
# Rationale
It'd be, very, very nice to have `-v|-verbose` option in all OTB applications in standard. That's what we have everywhere in *nix world. I'm not picky to require that long option names should start with double dashes, but really, a `-v` would change life.
I know, there is a (let me search...) `OTB_LOGGER_LEVEL=DEBUG` that we can export. But the trouble is... searching, every time, while `-v` is a _de facto_ standard.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1876No debug build on CI or dashboard2022-01-07T07:21:02ZVictor PoughonNo debug build on CI or dashboard### Description
There's no debug build in the CI, so asserts are not tested. Example of problems that should be caught by testing in debug mode: #1874, !457.
### Configuration information
debug### Description
There's no debug build in the CI, so asserts are not tested. Example of problems that should be caught by testing in debug mode: #1874, !457.
### Configuration information
debughttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1865The Big Examples Review2019-06-03T12:39:41ZVictor PoughonThe Big Examples ReviewThis is a planning issue for the review and modernization of OTB C++ examples, now rendered in the cookbook documentation (!395).
### How examples work
Examples are made of two files: a `.cxx` and an optional `.rst` file with the same ...This is a planning issue for the review and modernization of OTB C++ examples, now rendered in the cookbook documentation (!395).
### How examples work
Examples are made of two files: a `.cxx` and an optional `.rst` file with the same basename, in a directory indicating their category, for example:
```bash
Examples/FeatureExtraction/TextureExample.cxx
Examples/FeatureExtraction/TextureExample.rst (optional file)
```
##### The cxx file
The C++ code should be normal C++14 code with two additional things:
* The license header at the top (available in `Documentation/Cookbook/rst/templates/examples_license_header.txt`)
* An example usage, i.e a comment that looks like this:
```cpp
/* Example usage:
./TextureExample Input/ADS40RoiSmall.png Output/TextureOutput.tif Output/pretty_TextureOutput.png 2 1 1
*/
```
The directories `Input/` and `Output/` are a reference to `Data/Input` and `Data/Output` in OTB main repository. Input and output data are stored with git LFS.
The license header and example usage are not shown in the code in the html rendering.
##### The rst file
In the description `.rst` file, you can write [normal Sphinx rst](http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) and make automatic references to a doxygen class, an application or another example with special roles:
```rst
the class :doxygen:`Image` is used for...
the ITK class :doxygen-itk:`ImageToImageFilter` is a...
the application :ref:`BundleToPerfectSensor` is...
see example :ref:`ApplicationExample.cxx` for more...
```
In the description `.rst` file, you can include input or output images from `Data/Input` or `Data/Output` using the absolute path syntax, for example:
```rst
.. figure:: /Input/QB_Toulouse.png
.. figure:: /Output/qb_BandMath-pretty.jpg
:scale: 50%
Thresholded NDVI of a Quickbird image computed with a BandMathFilter
```
##### How to run an example
From your OTB build directory, use the script:
```
$ ~/otb/Documentation/Cookbook/Scripts/RunExamples.py ~/otb --name BandMathFilterExample
```
This will run the given example in the working directory `~/otb/Data`.
### How to review an example
*Before you begin:* Have a working OTB build of develop with the CookBook and examples activated:
```
cmake -DBUILD_COOKBOOK=ON -DBUILD_EXAMPLES=ON .
make
firefox Documentation/Cookbook/html/index.html &
```
Note: currently it is advised that you delete `Documentation/CookBook/` manually from your build directory between each rebuild of the cookbook.
Pick an example to review from the list below, then:
1. Check that there is an "example usage", if not add it and check it with `RunExamples.py`
2. Make a description `.rst` file from:
* the content of the C++ comments (usually the first big block of text)
* latex content from [the old software guide](https://www.orfeo-toolbox.org/packages/nightly/latest/SoftwareGuide-6.7/) where the example appears
* the included figures
3. Review C++ comments, make sure they don't contain embedded latex
4. Review C++ code, make sure it's clean and modern C++14
5. Run the example, add output images that are necessary to the build to the MR (don't add other output, e.g. `.tif`, `.shp`)
6. Re-build the cookbook and check the output
When you are done, make a MR and list the example you reviewed. Edit the list with your progress.
You can also propose some examples for merge, deletion or refactoring.
### List of examples
##### BasicFilters
* [x] `BasicFilters/BandMathFilterExample.cxx`
* [ ] `BasicFilters/BandMathXImageFilterExample.cxx`
* [x] `BasicFilters/DEMToRainbowExample.cxx`
* [x] `BasicFilters/FrostImageFilter.cxx`
* [x] `BasicFilters/HillShadingExample.cxx`
* [x] `BasicFilters/IndexedToRGBExample.cxx`
* [x] `BasicFilters/LeeImageFilter.cxx`
* [x] `BasicFilters/MeanShiftSegmentationFilterExample.cxx`
* [x] `BasicFilters/PrintableImageFilterExample.cxx`
* [x] `BasicFilters/ScalingFilterExample.cxx`
##### ChangeDetection
* [x] `ChangeDetection/CorrelChDet.cxx`
* [x] `ChangeDetection/DiffChDet.cxx`
* [x] `ChangeDetection/KullbackLeiblerDistanceChDet.cxx`
* [x] `ChangeDetection/KullbackLeiblerProfileChDet.cxx`
* [x] `ChangeDetection/MultivariateAlterationDetector.cxx`
* [x] `ChangeDetection/RatioChDet.cxx`
##### Classification
* [ ] `Classification/ClassificationMapRegularizationExample.cxx` (doesn't run, no usage)
* [ ] `Classification/DempsterShaferFusionOfClassificationMapsExample.cxx` (doesn't run, no usage)
* [ ] `Classification/SOMImageClassificationExample.cxx` (doesn't run, no usage)
* [ ] `Classification/SupervisedImageClassificationExample.cxx` (doesn't run, no usage)
##### DimensionReduction
(all can't run because of missing file: `Input/wv2_cannes_8bands.tif`)
* [ ] `DimensionReduction/ICAExample.cxx`
* [ ] `DimensionReduction/MaximumAutocorrelationFactor.cxx`
* [ ] `DimensionReduction/MNFExample.cxx`
* [ ] `DimensionReduction/NAPCAExample.cxx`
* [ ] `DimensionReduction/PCAExample.cxx`
##### DisparityMap
* [ ] `DisparityMap/FineRegistrationImageFilterExample.cxx`
* [ ] `DisparityMap/NCCRegistrationFilterExample.cxx`
* [ ] `DisparityMap/StereoReconstructionExample.cxx`
##### FeatureExtraction
* [ ] `FeatureExtraction/AsymmetricFusionOfLineDetectorExample.cxx`
* [ ] `FeatureExtraction/CloudDetectionExample.cxx`
* [ ] `FeatureExtraction/ComplexMomentPathExample.cxx`
* [ ] `FeatureExtraction/ComplexMomentsImageFunctionExample.cxx`
* [ ] `FeatureExtraction/CorrelationLineDetectorExample.cxx`
* [ ] `FeatureExtraction/EdgeDensityExample.cxx`
* [ ] `FeatureExtraction/ExtractRoadByStepsExample.cxx`
* [ ] `FeatureExtraction/ExtractRoadExample.cxx`
* [ ] `FeatureExtraction/FlusserMomentsImageFunctionExample.cxx`
* [ ] `FeatureExtraction/HarrisExample.cxx`
* [ ] `FeatureExtraction/HuMomentsImageFunctionExample.cxx`
* [ ] `FeatureExtraction/LineSegmentDetectorExample.cxx`
* [ ] `FeatureExtraction/PanTexExample.cxx`
* [ ] `FeatureExtraction/ParallelLineDetectionExample.cxx`
* [ ] `FeatureExtraction/RatioLineDetectorExample.cxx`
* [ ] `FeatureExtraction/RightAngleDetectionExample.cxx`
* [ ] `FeatureExtraction/SeamCarvingExample.cxx`
* [ ] `FeatureExtraction/SeamCarvingOtherExample.cxx`
* [ ] `FeatureExtraction/SFSExample.cxx`
* [ ] `FeatureExtraction/SURFExample.cxx`
* [x] `FeatureExtraction/TextureExample.cxx`
* [ ] `FeatureExtraction/ThresholdToPointSetExample.cxx`
* [ ] `FeatureExtraction/TouziEdgeDetectorExample.cxx`
##### Filtering
* [ ] `Filtering/CompositeFilterExample.cxx`
* [ ] `Filtering/DanielssonDistanceMapImageFilter.cxx`
* [ ] `Filtering/SecondDerivativeRecursiveGaussianImageFilter.cxx`
##### Fusion
* [ ] `Fusion/BayesianFusionImageFilter.cxx`
* [ ] `Fusion/PanSharpeningExample.cxx`
##### Hyperspectral
* [ ] `Hyperspectral/HyperspectralUnmixingExample.cxx`
##### Image
* [ ] `Image/Image1.cxx`
* [ ] `Image/Image2.cxx`
* [ ] `Image/Image3.cxx`
* [ ] `Image/Image4.cxx`
* [ ] `Image/Image5.cxx`
* [ ] `Image/ImageListExample.cxx`
* [ ] `Image/VectorImage.cxx`
##### Installation
* [ ] `Installation/HelloWorld.cxx`
##### IO
* [ ] `IO/ComplexImageReadWrite.cxx`
* [ ] `IO/DEMHandlerExample.cxx`
* [ ] `IO/DEMToImageGenerator.cxx`
* [ ] `IO/ExtractROI.cxx`
* [ ] `IO/ImageReadCastWrite.cxx`
* [ ] `IO/ImageReadRegionOfInterestWrite.cxx`
* [ ] `IO/ImageReadWrite.cxx`
* [ ] `IO/ImageSeriesIOExample.cxx`
* [ ] `IO/MetadataExample.cxx`
* [ ] `IO/MultibandImageReadWrite.cxx`
* [ ] `IO/VectorDataIOExample.cxx`
##### Iterators
* [ ] `Iterators/ImageLinearIteratorWithIndex2.cxx`
* [ ] `Iterators/ImageLinearIteratorWithIndex.cxx`
* [ ] `Iterators/ImageRandomConstIteratorWithIndex.cxx`
* [ ] `Iterators/ImageRegionIterator.cxx`
* [ ] `Iterators/ImageRegionIteratorWithIndex.cxx`
* [ ] `Iterators/ImageSliceIteratorWithIndex.cxx`
* [ ] `Iterators/IteratorsExamples.cxx`
* [ ] `Iterators/NeighborhoodIterators1.cxx`
* [ ] `Iterators/NeighborhoodIterators2.cxx`
* [ ] `Iterators/NeighborhoodIterators3.cxx`
* [ ] `Iterators/NeighborhoodIterators4.cxx`
* [ ] `Iterators/NeighborhoodIterators5.cxx`
* [ ] `Iterators/NeighborhoodIterators6.cxx`
* [ ] `Iterators/ShapedNeighborhoodIterators1.cxx`
* [ ] `Iterators/ShapedNeighborhoodIterators2.cxx`
##### Learning
* [ ] `Learning/GenerateTrainingImageExample.cxx`
* [ ] `Learning/SEMModelEstimatorExample.cxx`
* [ ] `Learning/SOMClassifierExample.cxx`
* [ ] `Learning/SOMExample.cxx`
* [ ] `Learning/SVMImageEstimatorClassificationMultiExample.cxx`
* [ ] `Learning/TrainMachineLearningModelFromImagesExample.cxx`
* [ ] `Learning/TrainMachineLearningModelFromSamplesExample.cxx`
##### Markov
* [ ] `Markov/MarkovClassification1Example.cxx`
* [ ] `Markov/MarkovClassification2Example.cxx`
* [ ] `Markov/MarkovClassification3Example.cxx`
* [ ] `Markov/MarkovRegularizationExample.cxx`
* [ ] `Markov/MarkovRestorationExample.cxx`
##### OBIA
* [ ] `OBIA/HooverMetricsEstimation.cxx`
* [ ] `OBIA/LabelMapToVectorData.cxx`
* [ ] `OBIA/RadiometricAttributesLabelMapFilterExample.cxx`
##### Patented
* [ ] `Patented/SIFTDensityExample.cxx`
* [ ] `Patented/SIFTDisparityMapEstimation.cxx`
* [ ] `Patented/SIFTExample.cxx`
* [ ] `Patented/SIFTFastExample.cxx`
##### Projections
* [ ] `Projections/EstimateRPCSensorModelExample.cxx`
* [ ] `Projections/GeometriesProjectionExample.cxx`
* [ ] `Projections/OrthoRectificationExample.cxx`
* [ ] `Projections/PlaceNameToLonLatExample.cxx`
* [ ] `Projections/VectorDataExtractROIExample.cxx`
* [ ] `Projections/VectorDataProjectionExample.cxx`
##### Radiometry
* [ ] `Radiometry/ARVIMultiChannelRAndBAndNIRVegetationIndexImageFilter.cxx` (doesn't run, missing data)
* [ ] `Radiometry/AtmosphericCorrectionSequencement.cxx`
* [ ] `Radiometry/AVIMultiChannelRAndGAndNIRVegetationIndexImageFilter.cxx`
* [ ] `Radiometry/NDVIRAndNIRVegetationIndexImageFilter.cxx`
##### Simulation
* [ ] `Simulation/LAIAndPROSAILToSensorResponse.cxx`
* [ ] `Simulation/LAIFromNDVIImageTransform.cxx`
* [ ] `Simulation/ProsailModel.cxx`
##### Tutorials
* [ ] `Tutorials/FilteringPipeline.cxx`
* [ ] `Tutorials/HelloWorldOTB.cxx`
* [ ] `Tutorials/Multispectral.cxx`
* [ ] `Tutorials/OrthoFusion.cxx`
* [ ] `Tutorials/Pipeline.cxx`
* [ ] `Tutorials/ScalingPipeline.cxx`https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1858QGIS plugin, test in OTB2019-02-26T09:33:21ZAntoine RegimbeauQGIS plugin, test in OTB## QGIS
OTB has been added to QGIS as a provider. [See here.](https://github.com/qgis/QGIS/pull/8814)
We need to add the different tests used in QGIS, in OTB so that we will be able to prevent changes in OTB and report it to QGIS.## QGIS
OTB has been added to QGIS as a provider. [See here.](https://github.com/qgis/QGIS/pull/8814)
We need to add the different tests used in QGIS, in OTB so that we will be able to prevent changes in OTB and report it to QGIS.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1849otbcli_DimensionalityReduction: Add options to output eigenvalues, apply inve...2022-01-07T07:22:27Zaloboaotbcli_DimensionalityReduction: Add options to output eigenvalues, apply inverse transform for PCA and correctely deal with no-dataAdd options to otbcli_DimensionalityReduction
1. ~~Output eigenvalues along eigenvectors for PCA~~
2. ~~Apply the inverse transform~~ (I understand this is already done using the "-outinv" parameter.
3. Deal with nodata
4. Option to ...Add options to otbcli_DimensionalityReduction
1. ~~Output eigenvalues along eigenvectors for PCA~~
2. ~~Apply the inverse transform~~ (I understand this is already done using the "-outinv" parameter.
3. Deal with nodata
4. Option to estimate stats from a certain % sampled from the image
5. ~~Option to output eigenvalues and eigenvectors only, avoiding calcalating and writing the components (useful to decide how many components to keep in a second run that would perform the actual calculation of the components)~~https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1817Huge Memory-leaks when in-memory chaining OTB-applications from Python2024-03-26T14:52:03ZStéphane AlbertHuge Memory-leaks when in-memory chaining OTB-applications from Python### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ co...### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ code:
1. create_water_mask:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->SuperImpose;
ExtractROI_XS-->WaterMaskMaker;
WaterMaskMaker-->SuperImpose;
```
2. create_color_image:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->BundleToPerfectSensor;
ExtractROI_XS-->BundleToPerfectSensor;
BundleToPerfectSensor-->Convert;
```
A main processing function runs alternatively those two pipeline on a set of (PAN ; XS) 2-uplets in a (Python) loop (see [test_wwm_cim.py](/uploads/73d2c9bc174f7d53cb5b4ed431fe7ae0/test_wwm_cim.py))
Memory-leak instrumentation done with [vg-mem.sh](/uploads/fb31fe9ded1f9868cd832a8132fc7961/vg-mem.sh).
#### Instrumentation of the `WaterMaskMaker` C++ OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine WaterMaskMaker -progress true -in /home/salbert/data/bench-2/QB_TLSE_MUL_512x512_1024x1024.TIF -out /home/salbert/tmp/QB_TLSE_WMM.tif`
Valgrind log: \[otbApplicationLauncherCommandLine-20181213-105906.log\](/uploads/7beb36ede3bca565a38fd58dc6d19c18/otbApplicationLauncherCommandLine-20181213-105906.log
Small memory-leaks in _OSSIM_ projection and _RTTI_ systems. Same for TIF and JP2000
#### Instrumentation of the `BundleToPerfectSensor` OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine BundleToPerfectSensor -progress true -inp /home/salbert/data/bench-2/PRIMARY_BUNDLE_P_1024x1024.tif -inxs /home/salbert/data/bench-2/PRIMARY_BUNDLE_MUL_1024x1024.tif -out /home/salbert/tmp/PRIMARY_BUNDLE_BTPS.tif uint16`
Valgrind log: [otbApplicationLauncherCommandLine-20181220-130819.log](/uploads/19f7dfbdda614771744711de8a1c0028/otbApplicationLauncherCommandLine-20181220-130819.log)
L924, L910: Memory-leaks in pipeline `::UpdateOutputInformation()` from https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkImportImageContainer.hxx#L188
L896: Also inquire about leak of process-object instance of https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkInPlaceImageFilter.hxx#L38.
#### Instrumentation of _SWIG_ OTB-Application wrapper
_SWIG_ wrapper of OTB-applications have been instrumented using the [otb-app-swig-mem-log.patch](/uploads/2f8c6b39793aadb2e3b2f35c4612b3b1/otb-app-swig-mem-log.patch) trace patch and all instances of OTB-Applications (be they composite or not) have been manually counted).
Command: `vg-mem.sh ./test_OTB_application.py` (see [test_OTB_application.py](/uploads/d578087dea8c209fb841166256a0c50e/test_OTB_application.py))
Valgrind log: [test_OTB_application_memory.py-20181214-113339.log](/uploads/becd9109e348354e8001c583a47df1c8/test_OTB_application_memory.py-20181214-113339.log)
Command: `vg-mem.sh ./test_OTB_application_memory_JP2.py` ([test_OTB_application_memory_JP2.py](/uploads/e5003321db00cb3091ed9320c04621af/test_OTB_application_memory_JP2.py))
Valgrind log: [test_OTB_application_memory_JP2.py-20181217-104219.log](/uploads/6bc5de2ae3275759af70c798b17703ae/test_OTB_application_memory_JP2.py-20181217-104219.log)
There's no memory leak due to incorrect reference-counting of the SWIG wrapper. However:
1. the reference-counter wrapper implementation \[[1](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/release-6.6/Modules/Wrappers/SWIG/src/Python.i#L45)\] is out of date and could be replaced by direct _SWIG_ 3 handling \[[2](http://www.swig.org/Doc3.0/SWIGPlus.html#SWIGPlus_smart_pointers)\]
2. `ImageBaseType` (which is a reference-counted instance) is not wrapped, but OTB-Application wrapper provides a `::GetParameterImage()` returning a C++ raw-pointer without reference-counting handling, which may cause some memory leaks (see next section about ) in the _Python_ environment, e.g.:
```python
def main( self ):
app = ...
img = app.GetParameterImage( ... ) # img is not reference-counted
# If not giving img to a wrapped C++ function putting it into an itk::SmartPointer<> will cause a memory-leak which of all image-data.
```
#### Instrumentation of full _Python_ sample code
Same memory-leaks as described above are traces. Moreover, the metadata-dictionary is lost when running multiple-times this instrumentation test without deleting the temporary output file.
Command: `vg-mem.sh ./test_wmm_cim.py` (2 iterations without removing `${HOME}/tmp` files)
Valgrind log: [test_wwm_cim.py-20181219-161831.log](/uploads/f4322d42a25e01cfdd71a26c99ded4f7/test_wwm_cim.py-20181219-161831.log) (see also: [test_wwm_cim.py-20181217-174610.log](/uploads/ff1b413d2c2ca2efcf96937f2dd5d90c/test_wwm_cim.py-20181217-174610.log) L3666 which is related to a suspicious call to `::GetParameterImage()` from `otb::ImageFileReader<>` when initializing pipeline on existing output-data file.)
### Configuration information
OTB-6.6 release and ITK-4.12 on Ubuntu 18.04 LTS.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1798Handle no data in BandMath and BandMathX2024-03-27T16:19:19ZVictor PoughonHandle no data in BandMath and BandMathXRequested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is s...Requested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is set to the nodata value reported by gdal
* support `nan()` function in BandMathX #1801https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1793Workflow improvement for OGR::DataSet2022-01-07T11:47:25ZAntoine RegimbeauWorkflow improvement for OGR::DataSet### What changes will be made and why they would make a better Orfeo ToolBox?
The changes made would be :
* improving base filter for `otb::ogr::DataSource` and for `otb::GeometriesSet`
* refactoring filters using `otb::VectorData`
* re...### What changes will be made and why they would make a better Orfeo ToolBox?
The changes made would be :
* improving base filter for `otb::ogr::DataSource` and for `otb::GeometriesSet`
* refactoring filters using `otb::VectorData`
* refactoring VectorData parameter in application
#### High level description
Assiociated tasks:
1. [ ] Streaming capabilities on `otb::ogr::DataSource` (*13pts*)
2. [ ] Improving `otb::ogr::DataSource` filters (*34pts*)
3. [ ] Migrating filter using `otb::VectorData` (*13pts*)
4. [ ] Adding an OGR parameter in application (*13pts*)
##### Streaming capabilities on `otb::ogr::DataSource`
Different types of streaming can be implemented on `otb::ogr::DataSource` :
* on feature id (Range region)
* on spatial extend (Spatial region)
We need to be careful on the way those two streaming types may work with each other in the same pipeline.
##### Improving filter base for `otb::ogr::DataSource` and for `otb::GeometriesSet`
We do have some base filters for processing vector data structure, however they are rarely used.
With some improvement, those filters could be the equivalents of `itk::ImageToImageFilter` and `itk::ImageSource`. One major improvement will be the streaming on `ogr::DataSet`.
Once developed, we will be able to make already existing filters using `ogr::DataSet` to derive from those filters and thus improve the processing of this data structure.
##### Refactoring filters using otb::VectorData
After those changes we will be able to migrate the different functionalities using VectorData. At this point if a filter seems irrelevant or useless we can remove it if the community agrees on it.
##### Adding an OGR parameter in application
This modification will benefit applications using filename parameters to load `otb::ogr::DataSource`. We might also be able to refactor applications so that they will use OGR parameter rather than VectorData one.
#### Risks and benefits
API might be broken.
This work will benefit to lots of users and developers, as it introduces new features for `ogr::DataSet` manipulation.
We need to decide if `otb::VectorData` will be removed, and if so this can be done during this story.
#### Alternatives for implementationshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1786Incorrect estimated memory and number of blocks in DynamicConvert with OTB_MA...2018-12-11T08:19:20ZVictor PoughonIncorrect estimated memory and number of blocks in DynamicConvert with OTB_MAX_RAM_HINT=4096### Description
With the following example in DynamicConvert, setting OTB_MAX_RAM_HINT=2250 results in:
```
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 bl...### Description
With the following example in DynamicConvert, setting OTB_MAX_RAM_HINT=2250 results in:
```
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 blocks
2018-12-04 14:31:59 (INFO): Estimation will be performed in 4 blocks of 7168x6144 pixels
```
But increasing to OTB_MAX_RAM_HINT=4096 results in:
```
018-12-04 14:32:36 (INFO): Estimated memory for full processing: 1.109e+07MB (avail.: 4096 MB), optimal image partitioning: 2708 blocks
2018-12-04 14:32:36 (INFO): File out_test.tif will be written in 2916 blocks of 205x205 pixels
```
See full log below.
### Steps to reproduce
```
poughov@pc-victor ~ OTB_MAX_RAM_HINT=2250 time otbcli_DynamicConvert -in T44NNN_20181129T050551_B08.jp2 -out out_test.tif
2018-12-04 14:31:59 (INFO): No kwl metadata found in file T44NNN_20181129T050551_B08.jp2
2018-12-04 14:31:59 (INFO): Default RAM limit for OTB is 2250 MB
2018-12-04 14:31:59 (INFO): GDAL maximum cache size is 383 MB
2018-12-04 14:31:59 (INFO): OTB will use at most 8 threads
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 blocks
2018-12-04 14:31:59 (INFO): Estimation will be performed in 4 blocks of 7168x6144 pixels
Computing shrink Image for min/max estimation...: 100% [**************************************************] (5 seconds)
2018-12-04 14:32:05 (INFO): Estimated memory for full processing: 1724.62MB (avail.: 2250 MB), optimal image partitioning: 1 blocks
2018-12-04 14:32:05 (INFO): File out_test.tif will be written in 1 blocks of 10980x10980 pixels
Writing out_test.tif...: 100% [**************************************************] (4 seconds)
56.85user 1.54system 0:10.38elapsed 562%CPU (0avgtext+0avgdata 2136324maxresident)k
0inputs+235656outputs (0major+933435minor)pagefaults 0swaps
poughov@pc-victor ~ OTB_MAX_RAM_HINT=4096 time otbcli_DynamicConvert -in T44NNN_20181129T050551_B08.jp2 -out out_test.tif
2018-12-04 14:32:30 (INFO): No kwl metadata found in file T44NNN_20181129T050551_B08.jp2
2018-12-04 14:32:30 (INFO): Default RAM limit for OTB is 4096 MB
2018-12-04 14:32:30 (INFO): GDAL maximum cache size is 383 MB
2018-12-04 14:32:30 (INFO): OTB will use at most 8 threads
2018-12-04 14:32:30 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 4096 MB), optimal image partitioning: 1 blocks
2018-12-04 14:32:30 (INFO): Estimation will be performed in 1 blocks of 10980x10980 pixels
Computing shrink Image for min/max estimation...: 100% [**************************************************] (5 seconds)
2018-12-04 14:32:36 (INFO): Estimated memory for full processing: 1.109e+07MB (avail.: 4096 MB), optimal image partitioning: 2708 blocks
2018-12-04 14:32:36 (INFO): File out_test.tif will be written in 2916 blocks of 205x205 pixels
Writing out_test.tif...: 100% [**************************************************] (10 seconds)
54.74user 6.27system 0:16.18elapsed 377%CPU (0avgtext+0avgdata 1889620maxresident)k
0inputs+235656outputs (0major+776377minor)pagefaults 0swaps
```
### Configuration information
Linux standalone package OTB 6.6.1https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1778Test refactoring2022-01-07T11:47:24ZGuillaume PaseroTest refactoring### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better...### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better balance between over-tested and under-tested classes.
#### High level description
3 steps are proposed for this refactoring:
* [ ] Refactor the longest tests. 41 tests have been identified as they take more than 10s to run.
* [ ] Design a test framework without IO, only in-memory images
* [ ] Refactor filters `Tv` to use the new test framework
Prior to these steps, 2 other taks should simplify this work:
* #1765 : the modules to be moved out of OTB should reduce the number of tests
* !299 : refactor all functor based filters, their test will be much simpler
#### Risks and benefits
The main risk is on the refactoring time, that could be significant as there are about 2300 tests.
On the bright side, it will be possible to greatly reduce the total testing time, as well as the need for large test data repository.
#### Alternatives for implementations
##### Step 1
The list of 41 pre-selected tests to refactor is here (with associated timings in seconds):
```
10.1 ioTvDEMToImageGeneratorFromImageTest_SensorModel
10.38 bfTuMeanShiftSmoothingImageFilterROIQBMul4
10.49 coTvCoordinateToNameMultithreadTest
10.93 obTuMeanShiftStreamingConnectedComponentSegmentationOBIAToVectorDataFilter
11.1 ioTvKmzProductWriterWithGCP
11.27 apTvSeLargeScaleMeanShiftTest
11.6 leTvSVMMachineLearningModelReg
13.18 bfTvMeanShiftSmoothingImageFilterThreadingNonOpt
13.28 apTvSeSegmentationWatershedVector
14.37 dmTvDisparityMapEstimationMethod
14.72 leTeSEMModelEstimatorExampleTest
14.92 apTvSeSegmentationCCVector_ULOVW
15.2 leTeSOMClassifierExampleTest
15.29 apTvSeSegmentationMeanshiftVector
15.84 obTuMeanShiftConnectedComponentSegmentationFilter
16.47 feTvLocalHoughDraw2
16.63 apTvRaOpticalCalibration_UnknownSensor
16.79 apTvSeSegmentationCCVector_ULU
18.25 apTvClKMeansImageClassification_composite
20.58 apTvSeSegmentationCCVector
22.13 raTvReflectanceToSurfaceReflectanceImageFilter2
23.11 bfTuMeanShiftSmoothingImageFilterQBRoad
23.75 apTvCdbDSFuzzyModelEstimation_LI
26.45 apTvCdbDSFuzzyModelEstimation_LI_autoInit
27.77 apTvClSampleAugmentationReplicate
28.12 apTvFeLineSegmentDetectionNoRescale
29.79 apTvClSampleAugmentationSmote
29.96 leTvGradientBoostedTreeMachineLearningModel
31.1 maTeMarkovClassification3ExampleTest
32.73 leTeSVMImageEstimatorClassificationMultiExampleTest
33.31 apTvClSampleAugmentationJitter
34.57 hyTvMDMDNMFImageFilterTest2
36.32 apTvFeLineSegmentDetection
38.8 leTeSOMExampleTest
51.22 ioTeTileMapImageIOExampleTest
52.66 reTeImageRegistration2ExampleTest
60.84 ioTvSHPVectorDataFileReader3
60.98 ioTvTileMapImageIOWeb
88.13 dmTvFineRegistrationImageFilterTestWithMeanSquare
112.69 owTvQtWidgetShow
135.51 apTvClMethodGBTImageClassifierQB1
```
##### Step 2
In module OTBTestKernel:
* a set of functions to produce hardcoded images:
* template functions working on both otb::Image and otb::VectorImage
* start with image allocation (buffered region equals to largest possible region)
* fills the image with a formula, an array, ...
* a set of functions to set hardcoded image metadata
* fills any image metadata (projection ref, origin, spacing, OSSIM keywordlist, ...)
* a set of functions to produce hardcoded vector data:
* two families of functions : for otb::VectorData or otb::OGRDataSource
* fills the dataset with hardcoded geometries and fields
* hardcoded metadata (projection ref, ...)
* new generic functions to check the result of a boolean condition (similar to
an `assert` statement). There are blocking and non-blocking flavours (a
blocking check will stop the test execution if it fails).
##### Step 3
`Tv` tests can be re-written to use the in-memory test data. They are composed
of the following steps:
* Instanciate the filter to be tested.
* Instanciate a given fake image
* Set the fake image as input of the filter
* Set the filter parameters
* Call Update() on the filter
* Check the differences with the baseline (if any)
* Check other outputs of the filter (if any)
* If all checks passed: returns EXIT_SUCCESS
* If one check failed:
* write PNG file for the difference image (if any)
* return EXIT_FAILURE
With this strategy, the comparison of test and baseline images in CDash is
preserved. The baselines don't have to be images on disk, they can also be
hardcoded in the test (suited for tiny images, like 10 x 10 pixels).
### Who will be developing the proposed changes?
@salbert & OTB Dev Teamhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1754GDAL VSI support2024-03-27T14:47:33ZLaurențiu NicolaGDAL VSI supportI've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage dr...I've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage drivers. Unfortunately, OTB doesn't seem to.
I tried a couple of tests:
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
2018-10-24 13:26:34 (INFO): Default RAM limit for OTB is 128 MB
2018-10-24 13:26:34 (INFO): GDAL maximum cache size is 393 MB
2018-10-24 13:26:34 (INFO): OTB will use at most 4 threads
2018-10-24 13:26:35 (INFO):
Image general information:
Number of bands : 3
No data flags : Not found
Start index : [0,0]
Size : [65536,65536]
Origin : [0.5,0.5]
Spacing : [1,1]
Estimated ground spacing (in meters): [547.072,296.427]
Image acquisition information:
Sensor :
Image identification number:
Image default RGB composition:
[R, G, B] = [0,1,2]
Ground control points information:
Number of GCPs = 0
GCPs projection =
Output parameters value:
indexx: 0
indexy: 0
sizex: 65536
sizey: 65536
spacingx: 1
spacingy: 1
originx: 0.5
originy: 0.5
estimatedgroundspacingx: 547.0715332
estimatedgroundspacingy: 296.4269409
numberbands: 3
sensor:
id:
time:
town:
country:
rgb.r: 0
rgb.g: 1
rgb.b: 2
projectionref:
keyword:
gcp.count: 0
gcp.proj:
gcp.ids:
gcp.info:
gcp.imcoord:
gcp.geocoord
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt gdalinfo http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
Driver: GTiff/GeoTIFF
Files: none associated
Size is 514, 515
Coordinate System is:
PROJCS["unnamed",
GEOGCS["NAD27",
DATUM["North_American_Datum_1927",
SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
AUTHORITY["EPSG","7008"]],
AUTHORITY["EPSG","6267"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4267"]],
PROJECTION["Cylindrical_Equal_Area"],
PARAMETER["standard_parallel_1",33.75],
PARAMETER["central_meridian",-117.333333333333],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (-28493.166784412522247,4255884.543802191503346)
Pixel Size = (60.022136983193739,-60.022136983193739)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( -28493.167, 4255884.544) (117d38'27.05"W, 33d56'37.74"N)
Lower Left ( -28493.167, 4224973.143) (117d38'27.05"W, 33d39'53.81"N)
Upper Right ( 2358.212, 4255884.544) (117d18'28.38"W, 33d56'37.74"N)
Lower Right ( 2358.212, 4224973.143) (117d18'28.38"W, 33d39'53.81"N)
Center ( -13067.478, 4240428.844) (117d28'27.71"W, 33d48'15.38"N)
Band 1 Block=514x15 Type=Byte, ColorInterp=Gray
```
Notice that OTB reports wrong origin, ground spacing and size values.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -e trace=file -f otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif"
[snip]
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid 6069] getcwd("/home/user", 4096) = 16
[pid 6069] lstat("/home/user/http:", 0x7ffed725d380) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.til", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] stat("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x7ffed725dc50) = -1 ENOENT (No such file or directory)
[pid 6069] readlink("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x245a730, 2048) = -1 ENOENT (No such file or directory)
2018-10-24 13:26:57 (INFO): No kwl metadata found in file http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
```
OTB tries to open HTTP URLs as files.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_ComputeImagesStatistics -il "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
[snip]
[pid 6110] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqrr HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
OTB gets into an infinite loop trying to request the wrong URL (notice the `qqqqqqrr` suffix).
```
CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_Convert -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true" -out foo.tif
[snip]
[pid 6171] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqqq HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
Same issue, different suffix. Uninitialized memory?
Of course, I expect performance to be pretty poor when doing network access, but sometimes it's useful, and the alternative is to use a FUSE emulation driver that might have its own problems.
Related: #1746
Related: #1506
Related: #1642https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1731Improve rasterization2022-01-07T11:47:25ZRémi CressonImprove rasterizationIn 7.0 `VectorData` might be replaced with `ogr::DataSource` and also in application parameter input/output (#1716).
Some filters for `VectorData` and `ogr::DataSource` have some overlap...
> - `VectorDataToLabelImageFilter` : `OGRDataS...In 7.0 `VectorData` might be replaced with `ogr::DataSource` and also in application parameter input/output (#1716).
Some filters for `VectorData` and `ogr::DataSource` have some overlap...
> - `VectorDataToLabelImageFilter` : `OGRDataSourceToLabelImageFilter`
> - `LabelImageToVectorDataFilter` : `LabelImageToOGRDataSourceFilter`
> - `VectorDataIntoImageProjectionFilter` : `GeometriesProjectionFilter`
(from !222)
...but some features were added to `VectorDataToLabelImageFilter` and not propagated to `OGRDataSourceToLabelImageFilter`
I propose to discuss a possible improvement for `OGRDataSourceToLabelImageFilter` (or `VectorDataToLabelImageFilter` is `VectorData` would be kept, but I don't feel so)
In my opinion, the following burning mode could be useful:
1. Binary (Burn the same value for each geometry. Params: Foreground value (ImagePixelValueType))
2. Field (Burn the value of the given field. Param: Field name (string))
3. FID (Burn the value corresponding to the (cast of the) geometry ID into ImagePixelValueType. (Param: Starting value (size_t) default could be background value))
4. Container (Burn the N-th value of the given container for the N-th geometry. Param: TContainer (e.g. std::unordered_map...))
For `Mosaic` and `ZonalStatistics` this would be useful, and I think we could improve also some other stuff with that.
What do you think?https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1698Missing test for SparseWvltToAngleMapperListFilter2019-01-21T15:12:21ZManuel GrizonnetMissing test for SparseWvltToAngleMapperListFilterThe class SparseWvltToAngleMapperListFilter has a test named otbSparseWvltToAngleMapperListFilterTest (previously named otbSparseWvltToAngleMapperListFilterNewTest but renamed in MR !173) but there is no corresponding otb_add_test in the...The class SparseWvltToAngleMapperListFilter has a test named otbSparseWvltToAngleMapperListFilterTest (previously named otbSparseWvltToAngleMapperListFilterNewTest but renamed in MR !173) but there is no corresponding otb_add_test in the CMakeLists (module DimensionnalityReduction).
This bug was spotted when we remove all new test (this one was wrongly prefixed with the keywork New).
A test with proper input and baseline should be added here.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1679Documentation improvements2022-01-06T14:43:11ZVictor PoughonDocumentation improvementsVarious documentation improvements (this is an evolving list):
##### CookBook
* [ ] Write MPI and parallel processing
* [X] Mention existing remote modules and add link to the list
##### Examples
* [ ] Finish reviewing examples after m...Various documentation improvements (this is an evolving list):
##### CookBook
* [ ] Write MPI and parallel processing
* [X] Mention existing remote modules and add link to the list
##### Examples
* [ ] Finish reviewing examples after migration to sphinx #1865
##### Tutorials
* [ ] Rewrite Segmentation exercises to use all in one app for LSMS, change images...
* [ ] Add a 30 minutes introduction with illustrations about SAR images (find Creative Commons support look at SAREDU project for instance https://saredu.dlr.de/)
##### Build system
* [ ] Remove examples tests?
* [ ] Remove examples tests baselines?
* [ ] Add RunExamples.py script in a new test
* [ ] Fix missing example usages
* [ ] Improve license header filtering in rst render (itk copyright)
* [ ] Handle `ApplicationExample`
* [ ] Move `Documentation/Cookbook/Art/` to Data/ after the LFS migration is complete (!387)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1678Application improvements2018-08-20T12:03:28ZYannick TANGUYApplication improvementsThis story gathers list of improvements for OTB Application from the old wiki.
# General improvements
* Improve otbgui look and feel (big white spaces, default window size, alignment, parameters ordering, etc.)
* Add a "same as input" ou...This story gathers list of improvements for OTB Application from the old wiki.
# General improvements
* Improve otbgui look and feel (big white spaces, default window size, alignment, parameters ordering, etc.)
* Add a "same as input" output type
* Internationalization of otb applications (challenge: without making otbcli depend on qt)
* Improve application Engine API to allow to mark some parameters as "Advanced". It will allow to provide solution to simplify some of the application GUI (like fold all advanced parameters in a specific widget)
* Allow in-memory connection for OGRDataSource between applications (see TrainImagesClassifier)
* In otb::Wrapper::Application, provide a "GenerateBufferOutput()" function that calls Update() on an output image parameter without writing it.
* Interpolation on complex data
* Save N-dimensional images (where N>2) (see SOM maps)
* Display internal information of trained Machine Learning models (for instance, check the variables mostly used in a random forest model,...)
# Existing applications
* Implement IMORM approach in LSMSSmallRegionsMerging
* ComputeConfusionMatrix: Also compute precision, recall and F1 score and optionally write to a csv file
* Radiometric index: document which bands need to be filled for different indices
* Improve performance of sampling applications :
1. * In SampleExtraction : allow to compute features only on selected samples instead of computing them on the full image.
2. * When using a mask during training, CPU usage is not at 100% but rather 70% : why ?
3. * How to skip the processing of tiles without any sample ?
4. * Better estimation of RAM usage for sampling applications (at the moment, the memory cost of OGRData is not taken into account).
* (from M. Planells) In Orthorectification application: output the projected incidence angle (useful in case of SAR image orthorectification). See what is done in S1 toolbox Range Doppler Correction processing. It should output an optionnal image (1 local incidence angle per angle)
*TrainImageClassifierBetter*: explain what happens when no validation vector is provided in the TrainImageClassifier application-> in this case the rating is used and -> idea: display the number of extracted pixels in the log
*ComputeConfusionMatrix*: Improve the log of the confusion matrix (should be aligned properly)
*OpticalCalibration*: Improve OpticalCalibration doc to explain that in case of TOA To Image the input is not a DN image
# New applications
* Apply a polynomial correction
* Bundle block adjustment
* Morphological profiles and profiles classification
* The part of the object detection framework that can know be plugged in the new classification framework
* DSM shading and other stuff like this
* Proper denoising (the smoothing applications is quite poor and there are other filters in ITK)
* Sharpening (there are filters in ITK)
* HDR compression
* Haze correction
* Histogram application, lots of parameters (see numpy.histogram for inspiration) and which output format?
# C++ API
* Add topographic correction of reflectance. OTB filters can already take into account environment effects but not topo effects. It was requested on Mantis (https://bugs.orfeo-toolbox.org/view.php?id=1146)
* Re-write a decent 'compare-ogr' method for the test driverhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1671Remove hard-coded muParser expression library-wise2024-03-20T14:05:37ZJulien MichelRemove hard-coded muParser expression library-wise### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muP...### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muParser` based filters are not as fast as their functor based counter-parts
* `muParser` is an optional dependency. Using it heavily results in many applications and filters disabled if `muParser` is disabled (see #1670 for instance)
#### Risks and benefits
Only risk might be small changing with floating point baselines.
Benefit is a faster OTB less dependent on muParser.
#### Alternatives for implementations
We need to spot places where `muParser` is mis-used:
```
$ grep -R -l BandMath Modules/ | grep -v BandMath | grep -v Parser
Modules/Applications/AppProjection/app/otbGridBasedImageResampling.cxx
Modules/Applications/AppImageUtils/app/otbCompareImages.cxx
Modules/Applications/AppStereo/app/otbStereoFramework.cxx
Modules/Applications/AppStereo/app/otbBlockMatching.cxx
```
### Who will be developing the proposed changes?
TDB.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1662mandatory parameters: why and how?2022-01-07T07:46:22ZRashad Kanavathmandatory parameters: why and how?### Description
Current state of mandatory parameter is really confusing.
Parameter class has a bool m_Mandatory and is true by default. Unless a parameter is set to off with `MandatoryOff(p)` then they are really required.
This make...### Description
Current state of mandatory parameter is really confusing.
Parameter class has a bool m_Mandatory and is true by default. Unless a parameter is set to off with `MandatoryOff(p)` then they are really required.
This make me tip toe around in finding mandatory flag in [otbQgisDescriptor.cxx](Modules/Wrappers/QGIS/src/otbQgisDescriptor.cxx#L302). Now that doesn't solve issue and ends up in error when using otbExtractROI. Error message from QGIS is `mode.fit.vect` is not set. QGIS sees this parameter as mandatory probably because otb description for ExtractROI said so!
This parameter is only required if `mode = fit`. so it must be optional. But not according to `param->GetMandatory()`. It says parameter is mandatory no matter what value is set for `mode` parameter. This is wrong and is problematic for apps like QGIS and others who rely on descriptor files.
In QGIS description file, there is a csv field to identify that parameter as optional or not. (mandatory or not in OTB terms).
Here is the list of paramters reported by otbcli_ExtractROI
```
in (mandatory)
out (mandatory)
mode.fit.im (mandatory)
mode.fit.vect (mandatory)
mode.extent.ulx (mandatory, default value is 0)
mode.extent.uly (mandatory, default value is 0)
mode.extent.lrx (mandatory, default value is 0)
mode.extent.lry (mandatory, default value is 0)
mode.extent.unit (mandatory, default value is pxl)
mode.radius.r (mandatory, default value is 0)
mode.radius.unitr (mandatory, default value is pxl)
mode.radius.cx (mandatory, default value is 0)
mode.radius.cy (mandatory, default value is 0)
mode.radius.unitc (mandatory, default value is pxl)
startx (mandatory, default value is 0)
starty (mandatory, default value is 0)
sizex (mandatory, default value is 0)
sizey (mandatory, default value is 0)
cl (mandatory, no default value)
elev.dem (optional, off by default)
elev.geoid (optional, off by default)
elev.default (mandatory, default value is 0)
ram (optional, off by default, default value is 128)
inxml (optional, off by default)
```
my suggestion is to use below rule to find out if parameter is mandatory.
A parameter is mandatory if statisfies all of the conditions:
1. Must have parent parameter set. `mode.fit.vect` is mandatory only if `mode = fit`
2. Must not have default a value.
Below code can cover first case in this rule. Before going on to add this in otbQgisDescriptor, I would like to find out if there is a better way to deal with mandatory flag in OTBApplicationEngine so other can rely on Parameter::GetMandatory() and stick to API rather than spaghetti forever.
```
+ std::string optional = "false";
+ ParameterKey pName(name);
+ auto splitName = pName.Split();
+ if (splitName.size() > 2)
+ optional = "true";
+ else
+ optional = param->HasValue() ? "true" : "false";
+
```
we can add a check for if-default-value-exists.
### Steps to reproduce
try ExtractROI application using QGIS 3.2 and OTB 6.6
### Configuration information
OS, OTB version or tag, information related to build (binaries, superbuild, system libs ...)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1640[DOC] Document alternate Application API Python syntax2024-03-06T14:17:38ZManuel Grizonnet[DOC] Document alternate Application API Python syntax### Target documentation ressources
bandmath.IN = image.tif
bandmath.EXP = "im1b1"
This type of syntax should be described in the otb app section of the Cookbook (explain why upper case parameter is mandatory for instance).
### Chang...### Target documentation ressources
bandmath.IN = image.tif
bandmath.EXP = "im1b1"
This type of syntax should be described in the otb app section of the Cookbook (explain why upper case parameter is mandatory for instance).
### Change requested
Describe precisely the changes that are required.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1577Potential issue in 6S ground reflectance equation2018-04-30T18:44:45ZVictor PoughonPotential issue in 6S ground reflectance equation### Description
A user reported a potential bug in OTB implementation of Vermote’s 1997 6S paper. Here is the full text of their message:
> Greetings,
>
> I see a discrepancy between the ground reflectance equation 12.10 and Vermote’s...### Description
A user reported a potential bug in OTB implementation of Vermote’s 1997 6S paper. Here is the full text of their message:
> Greetings,
>
> I see a discrepancy between the ground reflectance equation 12.10 and Vermote’s 1997 6S paper equation 1. Should the t(g)allgass term be the total gaseous transmission and not the spherical albedo of the atmosphere? Should your variable Sx be the spherical albedo of the atmosphere.
>
> Also, Vermote’s equations appears to have Tg (gaseous transmission) multiplied by the atmospheric reflectance term. This appears to be missing in your equation. Below are both equations solved for the ground reflectance.
>
> OTB (12.10): rho(s) = rho(TOA) - rho(ATM) / (Tup*Tdown*Tg + Sx*rho(TOA) – Sx*rho(ATM))
> Vermote (1): rho(s) = rho(TOA) - Tg*rho(R+A) / (Tup*Tdown*Tg + S*rho(TOA) – S*Tg*rho(R+A) )
> http://6s.ltdri.org/files/publication/Vermote_et_al_1997.pdf
>
> rho(ATM) = Tg*rho(R+A)? Or did something change with 6S between Vermote’s paper and OTB’s implementation.
> Sx = S?
>
> Thanks,
### Steps to reproduce
To be confirmed.
### Configuration information
NA