otb merge requestshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests2024-03-28T15:01:28Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1017DOC: add missing parameters doc about QGis plugin2024-03-28T15:01:28ZTristan LaurentDOC: add missing parameters doc about QGis pluginInformation comes from https://docs.qgis.org/testing/en/docs/user_manual/processing/3rdParty.html#documentation-of-otb-settings-available-in-qgis-processingInformation comes from https://docs.qgis.org/testing/en/docs/user_manual/processing/3rdParty.html#documentation-of-otb-settings-available-in-qgis-processingTristan LaurentTristan Laurenthttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1016Draft: BUILD: change otbTarget to name with module2024-03-28T15:09:11ZTristan LaurentDraft: BUILD: change otbTarget to name with moduleTristan LaurentTristan Laurenthttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1015Add Spot5 SWH support to otb92024-03-28T15:09:01ZThibaut ROMAINAdd Spot5 SWH support to otb9Spot 5 support was not provided when OSSIM was gone from OTB because it does not have an RPC model. It is a specific sensor model that we had to implementSpot 5 support was not provided when OSSIM was gone from OTB because it does not have an RPC model. It is a specific sensor model that we had to implementThibaut ROMAINThibaut ROMAINhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1012ENH: Add diapOTB in Release 9.02024-03-21T09:38:00ZThibaut ROMAINENH: Add diapOTB in Release 9.0DiapOTB was ported to OTB v8, this MR intends to reactivate it in OTB 9DiapOTB was ported to OTB v8, this MR intends to reactivate it in OTB 9Thibaut ROMAINThibaut ROMAINhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1008Support sensor doc update2024-03-12T16:25:23ZTristan LaurentSupport sensor doc updateRelated to #1648Related to #1648Tristan LaurentTristan Laurenthttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/1007Draft: Miscellaneous as external2024-03-26T17:37:39ZTristan LaurentDraft: Miscellaneous as externalTristan LaurentTristan Laurenthttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/937Draft: Reduce memory footprint when ImageFileWriter can't stream its input pi...2022-10-18T08:37:20ZRémi CressonDraft: Reduce memory footprint when ImageFileWriter can't stream its input pipeline to the output fileCloses #2310
This MR enables to reduce memory footprint to write output images into file formats that does not support streaming write.
# Benchmarks
All benchmarks have been performed on a laptop with 32Gb RAM, SSD, Ubuntu 20.04 + a l...Closes #2310
This MR enables to reduce memory footprint to write output images into file formats that does not support streaming write.
# Benchmarks
All benchmarks have been performed on a laptop with 32Gb RAM, SSD, Ubuntu 20.04 + a lot of tabs opened in web browser (that makes roughly **20Gb** RAM available). We let OTB with the default `OTB_MAX_RAM_HINT` which I believe is 256Mb.
Our goal is to perform the pansharpening of a Spot-7 (or Pléiades, or PNeo) image, in an output image format for which the GDAL driver only supports the `GDALDriver::CreateCopy()`, which is intended to create a copy of an existing dataset (hence the whole dataset must already exist, either in-memory or in another raster file).
In the following, we use the [GDAL Cloud Optimized Geotiff driver](https://gdal.org/drivers/raster/cog.html) to write the output image.
We use the following command to perform the processing:
```
otbcli_BundleToPerfectSensor -inp $dim_pan -inxs $dim_xs -out "/data/pxs.tif" int16
```
We did use some Spot-7 image but of course we expect the same kind of behavior for Pléiades and PNeo.
# Measurements
We have measured the processing time on the `BundleToPerfectSensor`.
## 10k x 10k subset:
When everything is fine, and the memory budget is enough.
* BundleToPerfectSensor (cog): **52s**
- BundleToPerfectSensor (cog, OTB_FORCE_STREAMING=1): **53s**
## 20k x 20k subset:
When the image is big and the processing without streaming requires extra memory.
### Original approach (trigger the entire pipeline, everything is in-memory)
![Fail](/uploads/2d18343dd79e5a5162f6eeb00539674e/image140.jpg)
### Proposed approach (when OTB_FORCE_STREAMING is set to 1)
![Force_streaming](/uploads/d5d88d1354feb5e56fb49964af251500/toto.jpg)
### Comparison with original approach + gdal_translate
- BundleToPerfectSensor (cog, `OTB_FORCE_STREAMING=1`): **3m37s**
- BundleToPerfectSensor (gtiff) + gdal_translate (cog): 1m24s + 2m09s = **3m33s**
# Conclusion
## PROS
- The MR enables to save memory when writing output image format for which the GDAL driver only supports the `GDALDriver::CreateCopy()`
## CONS
- Saving the output image in a streamable-capable raster format + using gdal_translate to achieve the final conversion is **as fast**, and **use a ridiculous smaller memory footprint** (because the whole output image is not stored in memory).
- Ultimately, there will always be an image size, for which the memory budget won't be enough, even with the `OTB_FORCE_STREAMING` approach. In this case, the user will fall back to the otb + gdal_translate approach.
# Discussion
Maybe this MR should be generalized to python API in order to get the output as numpy array with `OTB_FORCE_STREAMING` (for now, the entire pipeline is triggered over the largest possible image region).Rémi CressonRémi Cressonhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/816Fix the neural network classifier and the TrainImagesClassifier tests2022-10-05T07:18:32ZCédric TraizetFix the neural network classifier and the TrainImagesClassifier tests#### Summary
Closes #2191
##### Current behavior
The `TrainImagesClassifier` tests are broken (the `apTvClTrainMethodXXXImagesClassifierQB1` family of test) : they are always successful on CI but, as mentioned by an user on the forum...#### Summary
Closes #2191
##### Current behavior
The `TrainImagesClassifier` tests are broken (the `apTvClTrainMethodXXXImagesClassifierQB1` family of test) : they are always successful on CI but, as mentioned by an user on the forum, the output are incoherent.
On the CI, these tests are based on a ascii comparison of the computed confusion matrix, with a tolerance of 2 for numerical values in the tested files. The tolerance has been added in !224 to take into account the fact that Shark classifier use randomness, and that the random seed cannot be set for these tests, which is fixed by allowing an error of two misclassified samples. But the tolerance in --compare-ascii is relative ... which means that a tolerance of two is very large will always be verified, this is why the tests are always successful on CI.
Summary, there are two initial errors :
- the value of the tolerance
- the incoherent output of shark classifier
For example the ANN test produces the confusion matrix:
```
#Reference labels (rows):1,2,3,4
#Produced labels (columns):1,2,3,4
42,0,0,0
42,0,0,0
42,0,0,0
42,0,0,0
```
while the baseline is:
```
#Reference labels (rows):1,2,3,4
#Produced labels (columns):1,2,3,4
42,0,0,0
0,42,0,0
0,0,42,0
0,0,0,42
```
but the test is successful on CI. This hides bug #2191.
##### Correction of intial errors :
- [x] Tolerance value error : In this merge request the tolerance has been removed on this test, the produced confusion matrix should be the same as the one in the baseline.
- [x] Incoherent output of shark classifier : The baseline checks for the Shark classifier tests in `TrainImagesClassifier` have been removed, as it was the case before !224.
##### Other corrections :
Once the tolerance value error has been fixed, new bugs appears:
- [x] Fix bug in OpenCV neural network : This merge request also fix #2191. When fetching values from an openCV matrix (cv::Mat) with the `template<typename T> at<T>` method, the template parameter should be the type of data actually stored in the matrix, not the desired output type. Using the wrong type results in a reinterpretation of the stored value. In `NeuralNetworkMachineLearningModel`, this was resulting in a reinterpretation of float values into int values when writing the output labels.
- [ ] Confusion matrix differences produced by libSVM : the results produced by the SVM classfier are different on Windows and Linux builds. This require further investigations.
- [ ] Confusion matrix differences produced by OpenCV's random forest : The difference are caused by the update of the OpenCV version in https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/822.
| Parameter | before !224 | !224 | Regression #2191 | !816 tolerance set to zero | !816 Disable Shark tests | !816 Fix ann bug |
|-----------------------------------|-------------|--------------------------------------------------------|-----------------------------------------------------|----------------------------|--------------------------|------------------|
| Tolerance | 0 | 2 (relative tolerance, the absolute tolerance is high) | 2 (relatif donc tolérance très grande en absolu) | 0 | 0 | 0 |
| Shark RF (sharkrf) | Not tested | OK | OK | Test failing | Not tested | Not tested |
| Shark Kmeans (sharkkm) | Not tested | OK | OK | Test failing | Not tested | Not tested |
| OpenCV Random Forest (rf) | OK | OK | OK (regression hidden by the tolerance on Windows) | Test failing | Test failing | Test failing |
| OpenCV Decision Tree (dt) | OK | OK | OK | OK | OK | OK |
| OpenCV Neural network (ann) | OK | OK | OK (regression hidden by the tolerance) | Test failing | Test failing | OK |
| OpenCV Normal Bayes (bayes) | OK | OK | OK | OK | OK | OK |
| OpenCV Boosting (boost) | OK | OK | OK | OK | OK | OK |
| OpenCV K nearest neighbors (knn) | OK | OK | OK | OK | OK | OK |
| LibSVM | OK | OK | OK (regression hidden by the tolerance on Windows) | Test failing | Test failing | Test failing |
#### Copyright
The copyright owner is *CNES* and has signed the ORFEO ToolBox Contributor License Agreement.
<hr>
***Check before merging:***
- All discussions are resolved
- At least 2 :thumbsup: votes from core developers, no :thumbsdown: vote.
- The feature branch is (reasonably) up-to-date with the base branch
- Dashboard is green
- Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
- Optionally, run `git diff develop... -U0 --no-color | clang-format-diff.py -p1 -i` on latest changes and commithttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/805ENH: add 3d image support to StreamingManager and PipelineMemoryPrintCalculator2021-03-29T18:25:06ZJulien OsmanENH: add 3d image support to StreamingManager and PipelineMemoryPrintCalculatorThis Merge Request [was open on GitHub](https://github.com/orfeotoolbox/OTB/pull/28) and manually imported to GitLab.
…Calculator
#### Summary
Add nD image support to `otbStreamingManager` and 3D image support to `PipelineMemoryPrintC...This Merge Request [was open on GitHub](https://github.com/orfeotoolbox/OTB/pull/28) and manually imported to GitLab.
…Calculator
#### Summary
Add nD image support to `otbStreamingManager` and 3D image support to `PipelineMemoryPrintCalculator`
#### Rationale
The `StreamingManager` class defines a small, i.e. 100 pixel, region around the image centre that is used to estimate the memory footprint of the processing pipeline. The current implementation only explicitly sets the index for dimensions `0` and `1` respectively. Since the index value assignment does not account for the actual number of dimensions of the image, a `1D` image would produce a segmentation fault when the value of `index[1]` is assigned.
The PipelineMemoryPrintCalculator currently only supports 2D images. The number of image dimensions is currently hard coded in ::EvaluateDataObjectPrint in macro OTB_IMAGE_SIZE_BLOCK.
#### Implementation Details
##### otbStreamingManager.hxx
- The assignment of index and size values of the test region is done considering the actual number of image dimensions.
- `smallRegion.Crop(region)` is removed because it is superfluous with the new implementation.
##### otbPipelineMemoryPrintCalculator.cxx
- I added a set of image memory print calculation statements for 3-dimensional images to the `OTB_IMAGE_SIZE_BLOCK` macro
- *Note*: This revision will be performance neutral for 2D images but enables use of this class for 3D images. A future improvement of this class would be to template it over the image type to enable nD support.
#### Copyright
The copyright owner is Manaaki Whenua - Landcare Research (MWLR). MWLR has signed the ORFEO ToolBox Contributor License Agreement.
<hr>
***Check before merging:***
- All discussions are resolved
- At least 2 :thumbsup: votes from core developers, no :thumbsdown: vote.
- The feature branch is (reasonably) up-to-date with the base branch
- Dashboard is green
- Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
- Optionally, run `git diff develop... -U0 --no-color | clang-format-diff.py -p1 -i` on latest changes and commithttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/799Switch to c++172023-04-03T06:21:10ZJulien OsmanSwitch to c++17#### Summary
Upgrade to the C++ 2017 standard
#### Rationale
OTB is currently compiled with c++14. Now that the c++17 standard is well implemented by the compilers, we could upgrad to c++17. This would come with a lot of advantages:
...#### Summary
Upgrade to the C++ 2017 standard
#### Rationale
OTB is currently compiled with c++14. Now that the c++17 standard is well implemented by the compilers, we could upgrad to c++17. This would come with a lot of advantages:
- There are a lot of TODOs in the code telling that when we switch to c++17 we will be able to simplify that part of code.
- The new `std::clamp` is useful (and could simplify !797).
- We will be able to remove the homemade emulation of the string_view, and use the standard one.
- A bunch of new algorithms and mathematical functions.
- Some new features concerning the templates
- The new `if constexpr` will be handful to simplify the code.
- Many other goodies.
This update means we drop support for older compilers (like VisualC++14).
#### Implementation Details
Modify CMAKE_CXX_STANDARD to 17 in main CMakeFiles.txt.
Remove the REGISTER keyword from 6S code.
#### Copyright
The copyright owner is *CNES* and has signed the ORFEO ToolBox Contributor License Agreement.
<hr>
***Check before merging:***
- All discussions are resolved
- At least 2 :thumbsup: votes from core developers, no :thumbsdown: vote.
- The feature branch is (reasonably) up-to-date with the base branch
- Dashboard is green
- Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
- Optionally, run `git diff develop... -U0 --no-color | clang-format-diff.py -p1 -i` on latest changes and commithttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/merge_requests/528Support of ITK 5.0 in OTB (wip)2023-06-23T04:54:21ZCédric TraizetSupport of ITK 5.0 in OTB (wip)#### Summary
ITK 5.0 has been [released](https://discourse.itk.org/t/itk-5-0-0-has-been-released/1931) !
This merge request is based on the work done in !194, the original branch ( `cs-si:itk5_preperation`) has been rebased on top of ...#### Summary
ITK 5.0 has been [released](https://discourse.itk.org/t/itk-5-0-0-has-been-released/1931) !
This merge request is based on the work done in !194, the original branch ( `cs-si:itk5_preperation`) has been rebased on top of develop, and additional changes have been made to make the branch compatible with the release.
#### Rationale
See the [ITK 5 migration guidelines](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Documentation/ITK5MigrationGuide.md)
Prior to ITK 5.0, only a subset of C+11 functionality was used, when available, through backports and macros. ITK now requires a C+11 standard compiler. And this get rid of lot of macros. Move to C++11 does not affect OTB since it has already moved to a higher standard (C++14).
Another important API change in ITK is its threading model. This affects OTB , Remote Modules and projects depending on OTB (both official and unofficial) in large scale. This merge request approaches threading model issues between ITK 5.0 and OTB. We had observed a 99% of passing test with our changeset while not breaking compatibility with ITK 4.x.
This MR does not fully cover compatibility yet. There can be classes of filters that are not covered by testing and hence remain un-patched.
Until debian-testing, OpenSUSE, ubuntu and many others pick up the yet-to-release ITK 5.0, it is important to keep support of ITK 4.x. Even though binary packages and superbuild can use alpha version but still a compatibility layer is necessary.
There are several levels of compatibility:
- `ITKV4_COMPATIBILITY=ON` and `ITK_LEGACY_REMOVE=OFF`
- `ITKV4_COMPATIBILITY=OFF` and `ITK_LEGACY_REMOVE=OFF`
- `ITKV4_COMPATIBILITY=OFF` and `ITK_LEGACY_REMOVE=ON`
For now the code compiles with the first level of compatibility, but there are still issues with classes using spatial objects (most tests are failing, and the `SpatialObjectToImageDrawing` test has been commented for now because it doesn't compile). According to the guidelines, I think we should aim for the second level of compatibility, but this implies some additional API changes.
#### Implementation Details
TODO
#### Copyright
The copyright owner is *CNES* and has signed the ORFEO ToolBox Contributor License Agreement.
<hr>
***Check before merging:***
- All discussions are resolved
- At least 2 :thumbsup: votes from core developers, no :thumbsdown: vote.
- The feature branch is (reasonably) up-to-date with the base branch
- Dashboard is green
- Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
- Optionally, run `git diff develop... -U0 --no-color | clang-format-diff.py -p1 -i` on latest changes and commitCédric TraizetCédric Traizet