otb issueshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues2024-03-06T14:05:30Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1514Improvements for ApplicationEngine2024-03-06T14:05:30ZGuillaume PaseroImprovements for ApplicationEngine### What changes will be made and why they would make a better Orfeo ToolBox?
The OTB Application are now used very often, so it is important to keep the ApplicationEngine code clean. This RFC proposes small enhancements that will simpl...### What changes will be made and why they would make a better Orfeo ToolBox?
The OTB Application are now used very often, so it is important to keep the ApplicationEngine code clean. This RFC proposes small enhancements that will simplify the code base, and also new services available from the C++ ApplicationEngine API but also wrapped into the Python API.
#### High level description
The following are small improvement to the ApplicationEngine module:
* [x] __Handling of "UserValue" flag__: the optional boolean in 3rd argument of the functions SetParameterString/SetParameterInt/... is not very convenient. I think it is possible to get rid of it using a simple state flag "IsInPrivateDo" in the Application class. This flag would be set to ON during calls to DoInit(), DoUpdateParameters() and DoExecute(). During these calls, the values set on parameters are always "Automatic" values. Outside these calls, we can assume that the values set on parameters come from the user.
* [x] __Merging "UserValue" and "AutomaticValue" flags__: why do we have both ? I believe we could choose the convention : UserValue = not(AutomaticValue). The initial default value of a parameter would be considered "automatic". We could remove 1 of these 2 flags, there are already enough in the Parameter class.
* [ ] __Handling of the m_Active flag__: this flag should be handled mostly by the Parameters themselves in the ApplicationEngine module. For instance the call to EnableParameter() in otb::Wrapper::CommandLineLauncher::LoadParameters() should not be here, but rather in each "param->SetValue(string)". In a similar way, in otb::Wrapper::Application::SetParameterUserValue(), the call to EnableParameter() should not be here (in addition, it will always switch ON an EmptyParameter). The benefit is also to guarantee the same behaviour on CLI / GUI /Python.
* [x] __Refactor the ParameterType_Empty__: there are a lot of special cases for this parameter because its value (a boolean flag) and its Active flag are actually the same. For instance, in the current state of ApplicationEngine, it is impossible to have an Empty parameter with Role_Output. This parameter should have its value stored in a specific flag, proper functions Application::Set/GetParameterBool(), and maybe easy conversions to/from string and integer values. Maybe a specific ParameterType_Boolean can be added...
* [ ] __String getters and setters for all parameters__: it would be nice if all the parameters could be set using a common Application::SetParameterString(), or SetParameterStringList(). Same remark for getters. It would remove some long sections of if/elseif/elseif/... in the CommandLineLauncher but also in the Input/OutputXML parameters.
* [x] __Move m_IsChecked to QtWrappers__: in the Parameter base class, the flag m_IsChecked is only used by QtWrapper, so it should be moved outside ApplicationEngine.
* [ ] __Improve architecture of Parameters__: there is also room for improvement in the architecture of Parameters. There is a lot of code that is duplicated between similar parameter types, we should consider a better class architecture to gather common features between parameters with similar behaviour.
* [x] __Remove enum m_DefaultValueMode__: in the Parameter base class, the enum m_DefaultValueMode is never used in OTB. We could check if other external projects use it. We could get rid of it.
If you see other potential improvements to the core of this module, feel free to give your idea.
The following are a list of new services for ApplicationEngine framework:
* [x] __Streaming and output information__: this is a set of service that let you have insights on what output the
application will generate, and what pieces of the input you need to generate a piece of the output.
* [x] __Get the output information__: given a parameter key corresponding to an OutputImageParameter, retrieve the output size, spacing, origin, projection, geom file and so on
* [x] __Get the requested input regions__: given a parameter key corresponding to an OutputImageParameter, and a region corresponding to a piece of this output, get for each InputImageParameter the region required to produce that piece of output. The application could also return the estimated amount of RAM needed to compute that piece.
* [x] __Attach metadata to numpy array interface__: the numpy array interface to application is useless in many cases because it only allows to pass the pixel buffer, and no metadata. Many OTB filters need additional metadata (such as origin, spacing, size, projection and geom file) to be able to process the data.
* [x] __Get rid of ComplexParameterInput/OutputImage__: there is no reason to have a separate complex image parameter. This makes most application incompatible with complex images (such as ExtractROI for instance). It would be better to be able to detect complex images with InputImageParameter, and to add new output types for OutputImageParameter (cint, cfloat, ...). Check Request_for_changes-125:_Merge_Complex_and_standard_Image_Parameter
* [x] __Set/get all parameters at once through a dictionnary in Pyton API__: this is a small helper for large application with many parameters to set.
* [x] __Get proper logging / progress reporting in Python__: self explanatory.
* [x] __Register/free filter resources__: provide a way to register filters and free them at the end of the application processing to avoid retaining memory. Check improvements to fix bug #1475
* [ ] __Interface with OGR Python wrappers__: GDAL/OGR provides Python wrappers for vector data. It would be useful to import them in OTB Applications (into ParameterType_VectorData). Solution for vectors defined as ParameterType_InputFilename ?
* [x] __Stop button for Graphic Interface__: Be able to stop a running application.
* [ ] __Document new features in CookBook__: new recipe ?
#### Risks and benefits
The main risk is to break existing applications, so we need to focus on compatibility. In the end, the ApplicationEngine will have a cleaner code base and new useful features.
#### Alternatives for implementations
### Who will be developing the proposed changes?
Work can be split, different contributors expected.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1519Add spatialite to Superbuild2019-04-16T12:57:37ZJulien MichelAdd spatialite to SuperbuildSpatialite allows to use vector db with spatial index, resulting in faster spatial filtering operations.Spatialite allows to use vector db with spatial index, resulting in faster spatial filtering operations.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1528Filename VS VectorData2022-01-07T11:47:24ZRémi CressonFilename VS VectorData### What changes will be made and why they would make a better Orfeo ToolBox?
Hello guys,
This is an open question.
Should/could we enforce the following rule of thumb for vector data I/O?
* Use `ParameterType_InputVectorData` rather ...### What changes will be made and why they would make a better Orfeo ToolBox?
Hello guys,
This is an open question.
Should/could we enforce the following rule of thumb for vector data I/O?
* Use `ParameterType_InputVectorData` rather than `ParameterType_InputFilename` for input vector layers.
* Use `ParameterType_OutputVectorData` rather than `ParameterType_OutputFilename` for output vector layers.
Currently we use to mix both in applications, but in my opinion we should use the correct parameter type for a better integration of OTB applications in external projects.
#### High level description
Refactor some applications parameters.
#### Risks and benefits
Benefits: I/O data standardization in applications.
Risks: side effects ? (ParameterType_XXputFilename is traditionally used with ogr datasources)
#### Alternatives for implementations
### Who will be developing the proposed changes?https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1542Make SampleAugmentationFilter able to work inplace2018-03-28T09:04:17ZJulien MichelMake SampleAugmentationFilter able to work inplace`SampleAugmentationFilter` currently copies sample to a new output. It would be great if it could work inplace (see !25)`SampleAugmentationFilter` currently copies sample to a new output. It would be great if it could work inplace (see !25)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1577Potential issue in 6S ground reflectance equation2018-04-30T18:44:45ZVictor PoughonPotential issue in 6S ground reflectance equation### Description
A user reported a potential bug in OTB implementation of Vermote’s 1997 6S paper. Here is the full text of their message:
> Greetings,
>
> I see a discrepancy between the ground reflectance equation 12.10 and Vermote’s...### Description
A user reported a potential bug in OTB implementation of Vermote’s 1997 6S paper. Here is the full text of their message:
> Greetings,
>
> I see a discrepancy between the ground reflectance equation 12.10 and Vermote’s 1997 6S paper equation 1. Should the t(g)allgass term be the total gaseous transmission and not the spherical albedo of the atmosphere? Should your variable Sx be the spherical albedo of the atmosphere.
>
> Also, Vermote’s equations appears to have Tg (gaseous transmission) multiplied by the atmospheric reflectance term. This appears to be missing in your equation. Below are both equations solved for the ground reflectance.
>
> OTB (12.10): rho(s) = rho(TOA) - rho(ATM) / (Tup*Tdown*Tg + Sx*rho(TOA) – Sx*rho(ATM))
> Vermote (1): rho(s) = rho(TOA) - Tg*rho(R+A) / (Tup*Tdown*Tg + S*rho(TOA) – S*Tg*rho(R+A) )
> http://6s.ltdri.org/files/publication/Vermote_et_al_1997.pdf
>
> rho(ATM) = Tg*rho(R+A)? Or did something change with 6S between Vermote’s paper and OTB’s implementation.
> Sx = S?
>
> Thanks,
### Steps to reproduce
To be confirmed.
### Configuration information
NAhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1640[DOC] Document alternate Application API Python syntax2024-03-06T14:17:38ZManuel Grizonnet[DOC] Document alternate Application API Python syntax### Target documentation ressources
bandmath.IN = image.tif
bandmath.EXP = "im1b1"
This type of syntax should be described in the otb app section of the Cookbook (explain why upper case parameter is mandatory for instance).
### Chang...### Target documentation ressources
bandmath.IN = image.tif
bandmath.EXP = "im1b1"
This type of syntax should be described in the otb app section of the Cookbook (explain why upper case parameter is mandatory for instance).
### Change requested
Describe precisely the changes that are required.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1662mandatory parameters: why and how?2022-01-07T07:46:22ZRashad Kanavathmandatory parameters: why and how?### Description
Current state of mandatory parameter is really confusing.
Parameter class has a bool m_Mandatory and is true by default. Unless a parameter is set to off with `MandatoryOff(p)` then they are really required.
This make...### Description
Current state of mandatory parameter is really confusing.
Parameter class has a bool m_Mandatory and is true by default. Unless a parameter is set to off with `MandatoryOff(p)` then they are really required.
This make me tip toe around in finding mandatory flag in [otbQgisDescriptor.cxx](Modules/Wrappers/QGIS/src/otbQgisDescriptor.cxx#L302). Now that doesn't solve issue and ends up in error when using otbExtractROI. Error message from QGIS is `mode.fit.vect` is not set. QGIS sees this parameter as mandatory probably because otb description for ExtractROI said so!
This parameter is only required if `mode = fit`. so it must be optional. But not according to `param->GetMandatory()`. It says parameter is mandatory no matter what value is set for `mode` parameter. This is wrong and is problematic for apps like QGIS and others who rely on descriptor files.
In QGIS description file, there is a csv field to identify that parameter as optional or not. (mandatory or not in OTB terms).
Here is the list of paramters reported by otbcli_ExtractROI
```
in (mandatory)
out (mandatory)
mode.fit.im (mandatory)
mode.fit.vect (mandatory)
mode.extent.ulx (mandatory, default value is 0)
mode.extent.uly (mandatory, default value is 0)
mode.extent.lrx (mandatory, default value is 0)
mode.extent.lry (mandatory, default value is 0)
mode.extent.unit (mandatory, default value is pxl)
mode.radius.r (mandatory, default value is 0)
mode.radius.unitr (mandatory, default value is pxl)
mode.radius.cx (mandatory, default value is 0)
mode.radius.cy (mandatory, default value is 0)
mode.radius.unitc (mandatory, default value is pxl)
startx (mandatory, default value is 0)
starty (mandatory, default value is 0)
sizex (mandatory, default value is 0)
sizey (mandatory, default value is 0)
cl (mandatory, no default value)
elev.dem (optional, off by default)
elev.geoid (optional, off by default)
elev.default (mandatory, default value is 0)
ram (optional, off by default, default value is 128)
inxml (optional, off by default)
```
my suggestion is to use below rule to find out if parameter is mandatory.
A parameter is mandatory if statisfies all of the conditions:
1. Must have parent parameter set. `mode.fit.vect` is mandatory only if `mode = fit`
2. Must not have default a value.
Below code can cover first case in this rule. Before going on to add this in otbQgisDescriptor, I would like to find out if there is a better way to deal with mandatory flag in OTBApplicationEngine so other can rely on Parameter::GetMandatory() and stick to API rather than spaghetti forever.
```
+ std::string optional = "false";
+ ParameterKey pName(name);
+ auto splitName = pName.Split();
+ if (splitName.size() > 2)
+ optional = "true";
+ else
+ optional = param->HasValue() ? "true" : "false";
+
```
we can add a check for if-default-value-exists.
### Steps to reproduce
try ExtractROI application using QGIS 3.2 and OTB 6.6
### Configuration information
OS, OTB version or tag, information related to build (binaries, superbuild, system libs ...)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1671Remove hard-coded muParser expression library-wise2024-03-20T14:05:37ZJulien MichelRemove hard-coded muParser expression library-wise### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muP...### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muParser` based filters are not as fast as their functor based counter-parts
* `muParser` is an optional dependency. Using it heavily results in many applications and filters disabled if `muParser` is disabled (see #1670 for instance)
#### Risks and benefits
Only risk might be small changing with floating point baselines.
Benefit is a faster OTB less dependent on muParser.
#### Alternatives for implementations
We need to spot places where `muParser` is mis-used:
```
$ grep -R -l BandMath Modules/ | grep -v BandMath | grep -v Parser
Modules/Applications/AppProjection/app/otbGridBasedImageResampling.cxx
Modules/Applications/AppImageUtils/app/otbCompareImages.cxx
Modules/Applications/AppStereo/app/otbStereoFramework.cxx
Modules/Applications/AppStereo/app/otbBlockMatching.cxx
```
### Who will be developing the proposed changes?
TDB.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1678Application improvements2018-08-20T12:03:28ZYannick TANGUYApplication improvementsThis story gathers list of improvements for OTB Application from the old wiki.
# General improvements
* Improve otbgui look and feel (big white spaces, default window size, alignment, parameters ordering, etc.)
* Add a "same as input" ou...This story gathers list of improvements for OTB Application from the old wiki.
# General improvements
* Improve otbgui look and feel (big white spaces, default window size, alignment, parameters ordering, etc.)
* Add a "same as input" output type
* Internationalization of otb applications (challenge: without making otbcli depend on qt)
* Improve application Engine API to allow to mark some parameters as "Advanced". It will allow to provide solution to simplify some of the application GUI (like fold all advanced parameters in a specific widget)
* Allow in-memory connection for OGRDataSource between applications (see TrainImagesClassifier)
* In otb::Wrapper::Application, provide a "GenerateBufferOutput()" function that calls Update() on an output image parameter without writing it.
* Interpolation on complex data
* Save N-dimensional images (where N>2) (see SOM maps)
* Display internal information of trained Machine Learning models (for instance, check the variables mostly used in a random forest model,...)
# Existing applications
* Implement IMORM approach in LSMSSmallRegionsMerging
* ComputeConfusionMatrix: Also compute precision, recall and F1 score and optionally write to a csv file
* Radiometric index: document which bands need to be filled for different indices
* Improve performance of sampling applications :
1. * In SampleExtraction : allow to compute features only on selected samples instead of computing them on the full image.
2. * When using a mask during training, CPU usage is not at 100% but rather 70% : why ?
3. * How to skip the processing of tiles without any sample ?
4. * Better estimation of RAM usage for sampling applications (at the moment, the memory cost of OGRData is not taken into account).
* (from M. Planells) In Orthorectification application: output the projected incidence angle (useful in case of SAR image orthorectification). See what is done in S1 toolbox Range Doppler Correction processing. It should output an optionnal image (1 local incidence angle per angle)
*TrainImageClassifierBetter*: explain what happens when no validation vector is provided in the TrainImageClassifier application-> in this case the rating is used and -> idea: display the number of extracted pixels in the log
*ComputeConfusionMatrix*: Improve the log of the confusion matrix (should be aligned properly)
*OpticalCalibration*: Improve OpticalCalibration doc to explain that in case of TOA To Image the input is not a DN image
# New applications
* Apply a polynomial correction
* Bundle block adjustment
* Morphological profiles and profiles classification
* The part of the object detection framework that can know be plugged in the new classification framework
* DSM shading and other stuff like this
* Proper denoising (the smoothing applications is quite poor and there are other filters in ITK)
* Sharpening (there are filters in ITK)
* HDR compression
* Haze correction
* Histogram application, lots of parameters (see numpy.histogram for inspiration) and which output format?
# C++ API
* Add topographic correction of reflectance. OTB filters can already take into account environment effects but not topo effects. It was requested on Mantis (https://bugs.orfeo-toolbox.org/view.php?id=1146)
* Re-write a decent 'compare-ogr' method for the test driverhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1679Documentation improvements2022-01-06T14:43:11ZVictor PoughonDocumentation improvementsVarious documentation improvements (this is an evolving list):
##### CookBook
* [ ] Write MPI and parallel processing
* [X] Mention existing remote modules and add link to the list
##### Examples
* [ ] Finish reviewing examples after m...Various documentation improvements (this is an evolving list):
##### CookBook
* [ ] Write MPI and parallel processing
* [X] Mention existing remote modules and add link to the list
##### Examples
* [ ] Finish reviewing examples after migration to sphinx #1865
##### Tutorials
* [ ] Rewrite Segmentation exercises to use all in one app for LSMS, change images...
* [ ] Add a 30 minutes introduction with illustrations about SAR images (find Creative Commons support look at SAREDU project for instance https://saredu.dlr.de/)
##### Build system
* [ ] Remove examples tests?
* [ ] Remove examples tests baselines?
* [ ] Add RunExamples.py script in a new test
* [ ] Fix missing example usages
* [ ] Improve license header filtering in rst render (itk copyright)
* [ ] Handle `ApplicationExample`
* [ ] Move `Documentation/Cookbook/Art/` to Data/ after the LFS migration is complete (!387)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1698Missing test for SparseWvltToAngleMapperListFilter2019-01-21T15:12:21ZManuel GrizonnetMissing test for SparseWvltToAngleMapperListFilterThe class SparseWvltToAngleMapperListFilter has a test named otbSparseWvltToAngleMapperListFilterTest (previously named otbSparseWvltToAngleMapperListFilterNewTest but renamed in MR !173) but there is no corresponding otb_add_test in the...The class SparseWvltToAngleMapperListFilter has a test named otbSparseWvltToAngleMapperListFilterTest (previously named otbSparseWvltToAngleMapperListFilterNewTest but renamed in MR !173) but there is no corresponding otb_add_test in the CMakeLists (module DimensionnalityReduction).
This bug was spotted when we remove all new test (this one was wrongly prefixed with the keywork New).
A test with proper input and baseline should be added here.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1731Improve rasterization2022-01-07T11:47:25ZRémi CressonImprove rasterizationIn 7.0 `VectorData` might be replaced with `ogr::DataSource` and also in application parameter input/output (#1716).
Some filters for `VectorData` and `ogr::DataSource` have some overlap...
> - `VectorDataToLabelImageFilter` : `OGRDataS...In 7.0 `VectorData` might be replaced with `ogr::DataSource` and also in application parameter input/output (#1716).
Some filters for `VectorData` and `ogr::DataSource` have some overlap...
> - `VectorDataToLabelImageFilter` : `OGRDataSourceToLabelImageFilter`
> - `LabelImageToVectorDataFilter` : `LabelImageToOGRDataSourceFilter`
> - `VectorDataIntoImageProjectionFilter` : `GeometriesProjectionFilter`
(from !222)
...but some features were added to `VectorDataToLabelImageFilter` and not propagated to `OGRDataSourceToLabelImageFilter`
I propose to discuss a possible improvement for `OGRDataSourceToLabelImageFilter` (or `VectorDataToLabelImageFilter` is `VectorData` would be kept, but I don't feel so)
In my opinion, the following burning mode could be useful:
1. Binary (Burn the same value for each geometry. Params: Foreground value (ImagePixelValueType))
2. Field (Burn the value of the given field. Param: Field name (string))
3. FID (Burn the value corresponding to the (cast of the) geometry ID into ImagePixelValueType. (Param: Starting value (size_t) default could be background value))
4. Container (Burn the N-th value of the given container for the N-th geometry. Param: TContainer (e.g. std::unordered_map...))
For `Mosaic` and `ZonalStatistics` this would be useful, and I think we could improve also some other stuff with that.
What do you think?https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1754GDAL VSI support2024-03-27T14:47:33ZLaurențiu NicolaGDAL VSI supportI've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage dr...I've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage drivers. Unfortunately, OTB doesn't seem to.
I tried a couple of tests:
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
2018-10-24 13:26:34 (INFO): Default RAM limit for OTB is 128 MB
2018-10-24 13:26:34 (INFO): GDAL maximum cache size is 393 MB
2018-10-24 13:26:34 (INFO): OTB will use at most 4 threads
2018-10-24 13:26:35 (INFO):
Image general information:
Number of bands : 3
No data flags : Not found
Start index : [0,0]
Size : [65536,65536]
Origin : [0.5,0.5]
Spacing : [1,1]
Estimated ground spacing (in meters): [547.072,296.427]
Image acquisition information:
Sensor :
Image identification number:
Image default RGB composition:
[R, G, B] = [0,1,2]
Ground control points information:
Number of GCPs = 0
GCPs projection =
Output parameters value:
indexx: 0
indexy: 0
sizex: 65536
sizey: 65536
spacingx: 1
spacingy: 1
originx: 0.5
originy: 0.5
estimatedgroundspacingx: 547.0715332
estimatedgroundspacingy: 296.4269409
numberbands: 3
sensor:
id:
time:
town:
country:
rgb.r: 0
rgb.g: 1
rgb.b: 2
projectionref:
keyword:
gcp.count: 0
gcp.proj:
gcp.ids:
gcp.info:
gcp.imcoord:
gcp.geocoord
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt gdalinfo http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
Driver: GTiff/GeoTIFF
Files: none associated
Size is 514, 515
Coordinate System is:
PROJCS["unnamed",
GEOGCS["NAD27",
DATUM["North_American_Datum_1927",
SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
AUTHORITY["EPSG","7008"]],
AUTHORITY["EPSG","6267"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4267"]],
PROJECTION["Cylindrical_Equal_Area"],
PARAMETER["standard_parallel_1",33.75],
PARAMETER["central_meridian",-117.333333333333],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (-28493.166784412522247,4255884.543802191503346)
Pixel Size = (60.022136983193739,-60.022136983193739)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( -28493.167, 4255884.544) (117d38'27.05"W, 33d56'37.74"N)
Lower Left ( -28493.167, 4224973.143) (117d38'27.05"W, 33d39'53.81"N)
Upper Right ( 2358.212, 4255884.544) (117d18'28.38"W, 33d56'37.74"N)
Lower Right ( 2358.212, 4224973.143) (117d18'28.38"W, 33d39'53.81"N)
Center ( -13067.478, 4240428.844) (117d28'27.71"W, 33d48'15.38"N)
Band 1 Block=514x15 Type=Byte, ColorInterp=Gray
```
Notice that OTB reports wrong origin, ground spacing and size values.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -e trace=file -f otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif"
[snip]
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid 6069] getcwd("/home/user", 4096) = 16
[pid 6069] lstat("/home/user/http:", 0x7ffed725d380) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.til", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] stat("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x7ffed725dc50) = -1 ENOENT (No such file or directory)
[pid 6069] readlink("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x245a730, 2048) = -1 ENOENT (No such file or directory)
2018-10-24 13:26:57 (INFO): No kwl metadata found in file http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
```
OTB tries to open HTTP URLs as files.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_ComputeImagesStatistics -il "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
[snip]
[pid 6110] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqrr HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
OTB gets into an infinite loop trying to request the wrong URL (notice the `qqqqqqrr` suffix).
```
CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_Convert -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true" -out foo.tif
[snip]
[pid 6171] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqqq HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
Same issue, different suffix. Uninitialized memory?
Of course, I expect performance to be pretty poor when doing network access, but sometimes it's useful, and the alternative is to use a FUSE emulation driver that might have its own problems.
Related: #1746
Related: #1506
Related: #1642https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1778Test refactoring2022-01-07T11:47:24ZGuillaume PaseroTest refactoring### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better...### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better balance between over-tested and under-tested classes.
#### High level description
3 steps are proposed for this refactoring:
* [ ] Refactor the longest tests. 41 tests have been identified as they take more than 10s to run.
* [ ] Design a test framework without IO, only in-memory images
* [ ] Refactor filters `Tv` to use the new test framework
Prior to these steps, 2 other taks should simplify this work:
* #1765 : the modules to be moved out of OTB should reduce the number of tests
* !299 : refactor all functor based filters, their test will be much simpler
#### Risks and benefits
The main risk is on the refactoring time, that could be significant as there are about 2300 tests.
On the bright side, it will be possible to greatly reduce the total testing time, as well as the need for large test data repository.
#### Alternatives for implementations
##### Step 1
The list of 41 pre-selected tests to refactor is here (with associated timings in seconds):
```
10.1 ioTvDEMToImageGeneratorFromImageTest_SensorModel
10.38 bfTuMeanShiftSmoothingImageFilterROIQBMul4
10.49 coTvCoordinateToNameMultithreadTest
10.93 obTuMeanShiftStreamingConnectedComponentSegmentationOBIAToVectorDataFilter
11.1 ioTvKmzProductWriterWithGCP
11.27 apTvSeLargeScaleMeanShiftTest
11.6 leTvSVMMachineLearningModelReg
13.18 bfTvMeanShiftSmoothingImageFilterThreadingNonOpt
13.28 apTvSeSegmentationWatershedVector
14.37 dmTvDisparityMapEstimationMethod
14.72 leTeSEMModelEstimatorExampleTest
14.92 apTvSeSegmentationCCVector_ULOVW
15.2 leTeSOMClassifierExampleTest
15.29 apTvSeSegmentationMeanshiftVector
15.84 obTuMeanShiftConnectedComponentSegmentationFilter
16.47 feTvLocalHoughDraw2
16.63 apTvRaOpticalCalibration_UnknownSensor
16.79 apTvSeSegmentationCCVector_ULU
18.25 apTvClKMeansImageClassification_composite
20.58 apTvSeSegmentationCCVector
22.13 raTvReflectanceToSurfaceReflectanceImageFilter2
23.11 bfTuMeanShiftSmoothingImageFilterQBRoad
23.75 apTvCdbDSFuzzyModelEstimation_LI
26.45 apTvCdbDSFuzzyModelEstimation_LI_autoInit
27.77 apTvClSampleAugmentationReplicate
28.12 apTvFeLineSegmentDetectionNoRescale
29.79 apTvClSampleAugmentationSmote
29.96 leTvGradientBoostedTreeMachineLearningModel
31.1 maTeMarkovClassification3ExampleTest
32.73 leTeSVMImageEstimatorClassificationMultiExampleTest
33.31 apTvClSampleAugmentationJitter
34.57 hyTvMDMDNMFImageFilterTest2
36.32 apTvFeLineSegmentDetection
38.8 leTeSOMExampleTest
51.22 ioTeTileMapImageIOExampleTest
52.66 reTeImageRegistration2ExampleTest
60.84 ioTvSHPVectorDataFileReader3
60.98 ioTvTileMapImageIOWeb
88.13 dmTvFineRegistrationImageFilterTestWithMeanSquare
112.69 owTvQtWidgetShow
135.51 apTvClMethodGBTImageClassifierQB1
```
##### Step 2
In module OTBTestKernel:
* a set of functions to produce hardcoded images:
* template functions working on both otb::Image and otb::VectorImage
* start with image allocation (buffered region equals to largest possible region)
* fills the image with a formula, an array, ...
* a set of functions to set hardcoded image metadata
* fills any image metadata (projection ref, origin, spacing, OSSIM keywordlist, ...)
* a set of functions to produce hardcoded vector data:
* two families of functions : for otb::VectorData or otb::OGRDataSource
* fills the dataset with hardcoded geometries and fields
* hardcoded metadata (projection ref, ...)
* new generic functions to check the result of a boolean condition (similar to
an `assert` statement). There are blocking and non-blocking flavours (a
blocking check will stop the test execution if it fails).
##### Step 3
`Tv` tests can be re-written to use the in-memory test data. They are composed
of the following steps:
* Instanciate the filter to be tested.
* Instanciate a given fake image
* Set the fake image as input of the filter
* Set the filter parameters
* Call Update() on the filter
* Check the differences with the baseline (if any)
* Check other outputs of the filter (if any)
* If all checks passed: returns EXIT_SUCCESS
* If one check failed:
* write PNG file for the difference image (if any)
* return EXIT_FAILURE
With this strategy, the comparison of test and baseline images in CDash is
preserved. The baselines don't have to be images on disk, they can also be
hardcoded in the test (suited for tiny images, like 10 x 10 pixels).
### Who will be developing the proposed changes?
@salbert & OTB Dev Teamhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1786Incorrect estimated memory and number of blocks in DynamicConvert with OTB_MA...2018-12-11T08:19:20ZVictor PoughonIncorrect estimated memory and number of blocks in DynamicConvert with OTB_MAX_RAM_HINT=4096### Description
With the following example in DynamicConvert, setting OTB_MAX_RAM_HINT=2250 results in:
```
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 bl...### Description
With the following example in DynamicConvert, setting OTB_MAX_RAM_HINT=2250 results in:
```
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 blocks
2018-12-04 14:31:59 (INFO): Estimation will be performed in 4 blocks of 7168x6144 pixels
```
But increasing to OTB_MAX_RAM_HINT=4096 results in:
```
018-12-04 14:32:36 (INFO): Estimated memory for full processing: 1.109e+07MB (avail.: 4096 MB), optimal image partitioning: 2708 blocks
2018-12-04 14:32:36 (INFO): File out_test.tif will be written in 2916 blocks of 205x205 pixels
```
See full log below.
### Steps to reproduce
```
poughov@pc-victor ~ OTB_MAX_RAM_HINT=2250 time otbcli_DynamicConvert -in T44NNN_20181129T050551_B08.jp2 -out out_test.tif
2018-12-04 14:31:59 (INFO): No kwl metadata found in file T44NNN_20181129T050551_B08.jp2
2018-12-04 14:31:59 (INFO): Default RAM limit for OTB is 2250 MB
2018-12-04 14:31:59 (INFO): GDAL maximum cache size is 383 MB
2018-12-04 14:31:59 (INFO): OTB will use at most 8 threads
2018-12-04 14:31:59 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 2250 MB), optimal image partitioning: 2 blocks
2018-12-04 14:31:59 (INFO): Estimation will be performed in 4 blocks of 7168x6144 pixels
Computing shrink Image for min/max estimation...: 100% [**************************************************] (5 seconds)
2018-12-04 14:32:05 (INFO): Estimated memory for full processing: 1724.62MB (avail.: 2250 MB), optimal image partitioning: 1 blocks
2018-12-04 14:32:05 (INFO): File out_test.tif will be written in 1 blocks of 10980x10980 pixels
Writing out_test.tif...: 100% [**************************************************] (4 seconds)
56.85user 1.54system 0:10.38elapsed 562%CPU (0avgtext+0avgdata 2136324maxresident)k
0inputs+235656outputs (0major+933435minor)pagefaults 0swaps
poughov@pc-victor ~ OTB_MAX_RAM_HINT=4096 time otbcli_DynamicConvert -in T44NNN_20181129T050551_B08.jp2 -out out_test.tif
2018-12-04 14:32:30 (INFO): No kwl metadata found in file T44NNN_20181129T050551_B08.jp2
2018-12-04 14:32:30 (INFO): Default RAM limit for OTB is 4096 MB
2018-12-04 14:32:30 (INFO): GDAL maximum cache size is 383 MB
2018-12-04 14:32:30 (INFO): OTB will use at most 8 threads
2018-12-04 14:32:30 (INFO): Estimated memory for full processing: 2299.47MB (avail.: 4096 MB), optimal image partitioning: 1 blocks
2018-12-04 14:32:30 (INFO): Estimation will be performed in 1 blocks of 10980x10980 pixels
Computing shrink Image for min/max estimation...: 100% [**************************************************] (5 seconds)
2018-12-04 14:32:36 (INFO): Estimated memory for full processing: 1.109e+07MB (avail.: 4096 MB), optimal image partitioning: 2708 blocks
2018-12-04 14:32:36 (INFO): File out_test.tif will be written in 2916 blocks of 205x205 pixels
Writing out_test.tif...: 100% [**************************************************] (10 seconds)
54.74user 6.27system 0:16.18elapsed 377%CPU (0avgtext+0avgdata 1889620maxresident)k
0inputs+235656outputs (0major+776377minor)pagefaults 0swaps
```
### Configuration information
Linux standalone package OTB 6.6.1https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1793Workflow improvement for OGR::DataSet2022-01-07T11:47:25ZAntoine RegimbeauWorkflow improvement for OGR::DataSet### What changes will be made and why they would make a better Orfeo ToolBox?
The changes made would be :
* improving base filter for `otb::ogr::DataSource` and for `otb::GeometriesSet`
* refactoring filters using `otb::VectorData`
* re...### What changes will be made and why they would make a better Orfeo ToolBox?
The changes made would be :
* improving base filter for `otb::ogr::DataSource` and for `otb::GeometriesSet`
* refactoring filters using `otb::VectorData`
* refactoring VectorData parameter in application
#### High level description
Assiociated tasks:
1. [ ] Streaming capabilities on `otb::ogr::DataSource` (*13pts*)
2. [ ] Improving `otb::ogr::DataSource` filters (*34pts*)
3. [ ] Migrating filter using `otb::VectorData` (*13pts*)
4. [ ] Adding an OGR parameter in application (*13pts*)
##### Streaming capabilities on `otb::ogr::DataSource`
Different types of streaming can be implemented on `otb::ogr::DataSource` :
* on feature id (Range region)
* on spatial extend (Spatial region)
We need to be careful on the way those two streaming types may work with each other in the same pipeline.
##### Improving filter base for `otb::ogr::DataSource` and for `otb::GeometriesSet`
We do have some base filters for processing vector data structure, however they are rarely used.
With some improvement, those filters could be the equivalents of `itk::ImageToImageFilter` and `itk::ImageSource`. One major improvement will be the streaming on `ogr::DataSet`.
Once developed, we will be able to make already existing filters using `ogr::DataSet` to derive from those filters and thus improve the processing of this data structure.
##### Refactoring filters using otb::VectorData
After those changes we will be able to migrate the different functionalities using VectorData. At this point if a filter seems irrelevant or useless we can remove it if the community agrees on it.
##### Adding an OGR parameter in application
This modification will benefit applications using filename parameters to load `otb::ogr::DataSource`. We might also be able to refactor applications so that they will use OGR parameter rather than VectorData one.
#### Risks and benefits
API might be broken.
This work will benefit to lots of users and developers, as it introduces new features for `ogr::DataSet` manipulation.
We need to decide if `otb::VectorData` will be removed, and if so this can be done during this story.
#### Alternatives for implementationshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1798Handle no data in BandMath and BandMathX2024-03-27T16:19:19ZVictor PoughonHandle no data in BandMath and BandMathXRequested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is s...Requested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is set to the nodata value reported by gdal
* support `nan()` function in BandMathX #1801https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1817Huge Memory-leaks when in-memory chaining OTB-applications from Python2024-03-26T14:52:03ZStéphane AlbertHuge Memory-leaks when in-memory chaining OTB-applications from Python### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ co...### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ code:
1. create_water_mask:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->SuperImpose;
ExtractROI_XS-->WaterMaskMaker;
WaterMaskMaker-->SuperImpose;
```
2. create_color_image:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->BundleToPerfectSensor;
ExtractROI_XS-->BundleToPerfectSensor;
BundleToPerfectSensor-->Convert;
```
A main processing function runs alternatively those two pipeline on a set of (PAN ; XS) 2-uplets in a (Python) loop (see [test_wwm_cim.py](/uploads/73d2c9bc174f7d53cb5b4ed431fe7ae0/test_wwm_cim.py))
Memory-leak instrumentation done with [vg-mem.sh](/uploads/fb31fe9ded1f9868cd832a8132fc7961/vg-mem.sh).
#### Instrumentation of the `WaterMaskMaker` C++ OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine WaterMaskMaker -progress true -in /home/salbert/data/bench-2/QB_TLSE_MUL_512x512_1024x1024.TIF -out /home/salbert/tmp/QB_TLSE_WMM.tif`
Valgrind log: \[otbApplicationLauncherCommandLine-20181213-105906.log\](/uploads/7beb36ede3bca565a38fd58dc6d19c18/otbApplicationLauncherCommandLine-20181213-105906.log
Small memory-leaks in _OSSIM_ projection and _RTTI_ systems. Same for TIF and JP2000
#### Instrumentation of the `BundleToPerfectSensor` OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine BundleToPerfectSensor -progress true -inp /home/salbert/data/bench-2/PRIMARY_BUNDLE_P_1024x1024.tif -inxs /home/salbert/data/bench-2/PRIMARY_BUNDLE_MUL_1024x1024.tif -out /home/salbert/tmp/PRIMARY_BUNDLE_BTPS.tif uint16`
Valgrind log: [otbApplicationLauncherCommandLine-20181220-130819.log](/uploads/19f7dfbdda614771744711de8a1c0028/otbApplicationLauncherCommandLine-20181220-130819.log)
L924, L910: Memory-leaks in pipeline `::UpdateOutputInformation()` from https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkImportImageContainer.hxx#L188
L896: Also inquire about leak of process-object instance of https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkInPlaceImageFilter.hxx#L38.
#### Instrumentation of _SWIG_ OTB-Application wrapper
_SWIG_ wrapper of OTB-applications have been instrumented using the [otb-app-swig-mem-log.patch](/uploads/2f8c6b39793aadb2e3b2f35c4612b3b1/otb-app-swig-mem-log.patch) trace patch and all instances of OTB-Applications (be they composite or not) have been manually counted).
Command: `vg-mem.sh ./test_OTB_application.py` (see [test_OTB_application.py](/uploads/d578087dea8c209fb841166256a0c50e/test_OTB_application.py))
Valgrind log: [test_OTB_application_memory.py-20181214-113339.log](/uploads/becd9109e348354e8001c583a47df1c8/test_OTB_application_memory.py-20181214-113339.log)
Command: `vg-mem.sh ./test_OTB_application_memory_JP2.py` ([test_OTB_application_memory_JP2.py](/uploads/e5003321db00cb3091ed9320c04621af/test_OTB_application_memory_JP2.py))
Valgrind log: [test_OTB_application_memory_JP2.py-20181217-104219.log](/uploads/6bc5de2ae3275759af70c798b17703ae/test_OTB_application_memory_JP2.py-20181217-104219.log)
There's no memory leak due to incorrect reference-counting of the SWIG wrapper. However:
1. the reference-counter wrapper implementation \[[1](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/release-6.6/Modules/Wrappers/SWIG/src/Python.i#L45)\] is out of date and could be replaced by direct _SWIG_ 3 handling \[[2](http://www.swig.org/Doc3.0/SWIGPlus.html#SWIGPlus_smart_pointers)\]
2. `ImageBaseType` (which is a reference-counted instance) is not wrapped, but OTB-Application wrapper provides a `::GetParameterImage()` returning a C++ raw-pointer without reference-counting handling, which may cause some memory leaks (see next section about ) in the _Python_ environment, e.g.:
```python
def main( self ):
app = ...
img = app.GetParameterImage( ... ) # img is not reference-counted
# If not giving img to a wrapped C++ function putting it into an itk::SmartPointer<> will cause a memory-leak which of all image-data.
```
#### Instrumentation of full _Python_ sample code
Same memory-leaks as described above are traces. Moreover, the metadata-dictionary is lost when running multiple-times this instrumentation test without deleting the temporary output file.
Command: `vg-mem.sh ./test_wmm_cim.py` (2 iterations without removing `${HOME}/tmp` files)
Valgrind log: [test_wwm_cim.py-20181219-161831.log](/uploads/f4322d42a25e01cfdd71a26c99ded4f7/test_wwm_cim.py-20181219-161831.log) (see also: [test_wwm_cim.py-20181217-174610.log](/uploads/ff1b413d2c2ca2efcf96937f2dd5d90c/test_wwm_cim.py-20181217-174610.log) L3666 which is related to a suspicious call to `::GetParameterImage()` from `otb::ImageFileReader<>` when initializing pipeline on existing output-data file.)
### Configuration information
OTB-6.6 release and ITK-4.12 on Ubuntu 18.04 LTS.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1849otbcli_DimensionalityReduction: Add options to output eigenvalues, apply inve...2022-01-07T07:22:27Zaloboaotbcli_DimensionalityReduction: Add options to output eigenvalues, apply inverse transform for PCA and correctely deal with no-dataAdd options to otbcli_DimensionalityReduction
1. ~~Output eigenvalues along eigenvectors for PCA~~
2. ~~Apply the inverse transform~~ (I understand this is already done using the "-outinv" parameter.
3. Deal with nodata
4. Option to ...Add options to otbcli_DimensionalityReduction
1. ~~Output eigenvalues along eigenvectors for PCA~~
2. ~~Apply the inverse transform~~ (I understand this is already done using the "-outinv" parameter.
3. Deal with nodata
4. Option to estimate stats from a certain % sampled from the image
5. ~~Option to output eigenvalues and eigenvectors only, avoiding calcalating and writing the components (useful to decide how many components to keep in a second run that would perform the actual calculation of the components)~~https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1858QGIS plugin, test in OTB2019-02-26T09:33:21ZAntoine RegimbeauQGIS plugin, test in OTB## QGIS
OTB has been added to QGIS as a provider. [See here.](https://github.com/qgis/QGIS/pull/8814)
We need to add the different tests used in QGIS, in OTB so that we will be able to prevent changes in OTB and report it to QGIS.## QGIS
OTB has been added to QGIS as a provider. [See here.](https://github.com/qgis/QGIS/pull/8814)
We need to add the different tests used in QGIS, in OTB so that we will be able to prevent changes in OTB and report it to QGIS.