otb issueshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues2024-03-25T14:43:17Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2383Package module as "IMPORTED" cmake targets2024-03-25T14:43:17ZTristan LaurentPackage module as "IMPORTED" cmake targetsThe current cmake targets of OTB are packaged in Core. This is a problem if we want to separate OTB module.
Some modules depends of other OTB modules, thus when we want to build them, we need OTB Core and the package of these modules in...The current cmake targets of OTB are packaged in Core. This is a problem if we want to separate OTB module.
Some modules depends of other OTB modules, thus when we want to build them, we need OTB Core and the package of these modules installed. But if OTB Core is build without any OTBGroup, the cmake targets of theses dependencies are not declared.
One solution could be to create an OTB Core package with options -DOTBGroup corresponding to dependencies, but it lead to one specific OTB build per module. This is not a durable solution as it complexify CI, increase build time, is source of errors etc...
CMake propose a solution: IMPORTED_TARGETS: https://cmake.org/cmake/help/latest/guide/importing-exporting/index.html .
This can help to have cmake target of each package/module in the module package itself instead of Core package.10.0.0Tristan LaurentTristan Laurenthttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1817Huge Memory-leaks when in-memory chaining OTB-applications from Python2024-03-26T14:52:03ZStéphane AlbertHuge Memory-leaks when in-memory chaining OTB-applications from Python### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ co...### Description
Huge memory consumption when in-memory chaining OTB-Applications within loop from _Python_ code.
### Steps to reproduce
There are two processing pipelines defined by chaining OTB-applications in memory from _Python_ code:
1. create_water_mask:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->SuperImpose;
ExtractROI_XS-->WaterMaskMaker;
WaterMaskMaker-->SuperImpose;
```
2. create_color_image:
```mermaid
graph TD;
PAN-->ExtractROI_PAN;
XS-->ExtractROI_XS;
ExtractROI_PAN-->BundleToPerfectSensor;
ExtractROI_XS-->BundleToPerfectSensor;
BundleToPerfectSensor-->Convert;
```
A main processing function runs alternatively those two pipeline on a set of (PAN ; XS) 2-uplets in a (Python) loop (see [test_wwm_cim.py](/uploads/73d2c9bc174f7d53cb5b4ed431fe7ae0/test_wwm_cim.py))
Memory-leak instrumentation done with [vg-mem.sh](/uploads/fb31fe9ded1f9868cd832a8132fc7961/vg-mem.sh).
#### Instrumentation of the `WaterMaskMaker` C++ OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine WaterMaskMaker -progress true -in /home/salbert/data/bench-2/QB_TLSE_MUL_512x512_1024x1024.TIF -out /home/salbert/tmp/QB_TLSE_WMM.tif`
Valgrind log: \[otbApplicationLauncherCommandLine-20181213-105906.log\](/uploads/7beb36ede3bca565a38fd58dc6d19c18/otbApplicationLauncherCommandLine-20181213-105906.log
Small memory-leaks in _OSSIM_ projection and _RTTI_ systems. Same for TIF and JP2000
#### Instrumentation of the `BundleToPerfectSensor` OTB-Application
Command: `vg-mem.sh ./bin/otbApplicationLauncherCommandLine BundleToPerfectSensor -progress true -inp /home/salbert/data/bench-2/PRIMARY_BUNDLE_P_1024x1024.tif -inxs /home/salbert/data/bench-2/PRIMARY_BUNDLE_MUL_1024x1024.tif -out /home/salbert/tmp/PRIMARY_BUNDLE_BTPS.tif uint16`
Valgrind log: [otbApplicationLauncherCommandLine-20181220-130819.log](/uploads/19f7dfbdda614771744711de8a1c0028/otbApplicationLauncherCommandLine-20181220-130819.log)
L924, L910: Memory-leaks in pipeline `::UpdateOutputInformation()` from https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkImportImageContainer.hxx#L188
L896: Also inquire about leak of process-object instance of https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkInPlaceImageFilter.hxx#L38.
#### Instrumentation of _SWIG_ OTB-Application wrapper
_SWIG_ wrapper of OTB-applications have been instrumented using the [otb-app-swig-mem-log.patch](/uploads/2f8c6b39793aadb2e3b2f35c4612b3b1/otb-app-swig-mem-log.patch) trace patch and all instances of OTB-Applications (be they composite or not) have been manually counted).
Command: `vg-mem.sh ./test_OTB_application.py` (see [test_OTB_application.py](/uploads/d578087dea8c209fb841166256a0c50e/test_OTB_application.py))
Valgrind log: [test_OTB_application_memory.py-20181214-113339.log](/uploads/becd9109e348354e8001c583a47df1c8/test_OTB_application_memory.py-20181214-113339.log)
Command: `vg-mem.sh ./test_OTB_application_memory_JP2.py` ([test_OTB_application_memory_JP2.py](/uploads/e5003321db00cb3091ed9320c04621af/test_OTB_application_memory_JP2.py))
Valgrind log: [test_OTB_application_memory_JP2.py-20181217-104219.log](/uploads/6bc5de2ae3275759af70c798b17703ae/test_OTB_application_memory_JP2.py-20181217-104219.log)
There's no memory leak due to incorrect reference-counting of the SWIG wrapper. However:
1. the reference-counter wrapper implementation \[[1](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/release-6.6/Modules/Wrappers/SWIG/src/Python.i#L45)\] is out of date and could be replaced by direct _SWIG_ 3 handling \[[2](http://www.swig.org/Doc3.0/SWIGPlus.html#SWIGPlus_smart_pointers)\]
2. `ImageBaseType` (which is a reference-counted instance) is not wrapped, but OTB-Application wrapper provides a `::GetParameterImage()` returning a C++ raw-pointer without reference-counting handling, which may cause some memory leaks (see next section about ) in the _Python_ environment, e.g.:
```python
def main( self ):
app = ...
img = app.GetParameterImage( ... ) # img is not reference-counted
# If not giving img to a wrapped C++ function putting it into an itk::SmartPointer<> will cause a memory-leak which of all image-data.
```
#### Instrumentation of full _Python_ sample code
Same memory-leaks as described above are traces. Moreover, the metadata-dictionary is lost when running multiple-times this instrumentation test without deleting the temporary output file.
Command: `vg-mem.sh ./test_wmm_cim.py` (2 iterations without removing `${HOME}/tmp` files)
Valgrind log: [test_wwm_cim.py-20181219-161831.log](/uploads/f4322d42a25e01cfdd71a26c99ded4f7/test_wwm_cim.py-20181219-161831.log) (see also: [test_wwm_cim.py-20181217-174610.log](/uploads/ff1b413d2c2ca2efcf96937f2dd5d90c/test_wwm_cim.py-20181217-174610.log) L3666 which is related to a suspicious call to `::GetParameterImage()` from `otb::ImageFileReader<>` when initializing pipeline on existing output-data file.)
### Configuration information
OTB-6.6 release and ITK-4.12 on Ubuntu 18.04 LTS.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1754GDAL VSI support2024-03-27T14:47:33ZLaurențiu NicolaGDAL VSI supportI've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage dr...I've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage drivers. Unfortunately, OTB doesn't seem to.
I tried a couple of tests:
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
2018-10-24 13:26:34 (INFO): Default RAM limit for OTB is 128 MB
2018-10-24 13:26:34 (INFO): GDAL maximum cache size is 393 MB
2018-10-24 13:26:34 (INFO): OTB will use at most 4 threads
2018-10-24 13:26:35 (INFO):
Image general information:
Number of bands : 3
No data flags : Not found
Start index : [0,0]
Size : [65536,65536]
Origin : [0.5,0.5]
Spacing : [1,1]
Estimated ground spacing (in meters): [547.072,296.427]
Image acquisition information:
Sensor :
Image identification number:
Image default RGB composition:
[R, G, B] = [0,1,2]
Ground control points information:
Number of GCPs = 0
GCPs projection =
Output parameters value:
indexx: 0
indexy: 0
sizex: 65536
sizey: 65536
spacingx: 1
spacingy: 1
originx: 0.5
originy: 0.5
estimatedgroundspacingx: 547.0715332
estimatedgroundspacingy: 296.4269409
numberbands: 3
sensor:
id:
time:
town:
country:
rgb.r: 0
rgb.g: 1
rgb.b: 2
projectionref:
keyword:
gcp.count: 0
gcp.proj:
gcp.ids:
gcp.info:
gcp.imcoord:
gcp.geocoord
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt gdalinfo http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
Driver: GTiff/GeoTIFF
Files: none associated
Size is 514, 515
Coordinate System is:
PROJCS["unnamed",
GEOGCS["NAD27",
DATUM["North_American_Datum_1927",
SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
AUTHORITY["EPSG","7008"]],
AUTHORITY["EPSG","6267"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4267"]],
PROJECTION["Cylindrical_Equal_Area"],
PARAMETER["standard_parallel_1",33.75],
PARAMETER["central_meridian",-117.333333333333],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (-28493.166784412522247,4255884.543802191503346)
Pixel Size = (60.022136983193739,-60.022136983193739)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( -28493.167, 4255884.544) (117d38'27.05"W, 33d56'37.74"N)
Lower Left ( -28493.167, 4224973.143) (117d38'27.05"W, 33d39'53.81"N)
Upper Right ( 2358.212, 4255884.544) (117d18'28.38"W, 33d56'37.74"N)
Lower Right ( 2358.212, 4224973.143) (117d18'28.38"W, 33d39'53.81"N)
Center ( -13067.478, 4240428.844) (117d28'27.71"W, 33d48'15.38"N)
Band 1 Block=514x15 Type=Byte, ColorInterp=Gray
```
Notice that OTB reports wrong origin, ground spacing and size values.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -e trace=file -f otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif"
[snip]
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid 6069] getcwd("/home/user", 4096) = 16
[pid 6069] lstat("/home/user/http:", 0x7ffed725d380) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.til", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] stat("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x7ffed725dc50) = -1 ENOENT (No such file or directory)
[pid 6069] readlink("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x245a730, 2048) = -1 ENOENT (No such file or directory)
2018-10-24 13:26:57 (INFO): No kwl metadata found in file http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
```
OTB tries to open HTTP URLs as files.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_ComputeImagesStatistics -il "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
[snip]
[pid 6110] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqrr HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
OTB gets into an infinite loop trying to request the wrong URL (notice the `qqqqqqrr` suffix).
```
CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_Convert -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true" -out foo.tif
[snip]
[pid 6171] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqqq HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
Same issue, different suffix. Uninitialized memory?
Of course, I expect performance to be pretty poor when doing network access, but sometimes it's useful, and the alternative is to use a FUSE emulation driver that might have its own problems.
Related: #1746
Related: #1506
Related: #1642https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1798Handle no data in BandMath and BandMathX2024-03-27T16:19:19ZVictor PoughonHandle no data in BandMath and BandMathXRequested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is s...Requested by multiple users: #1511, #1801, otb-users
Better handling of images with no-data in BandMath and BandMathX:
* A flag to ignore no-data values (maybe even on by default?)
* A nodata token available in the expression that is set to the nodata value reported by gdal
* support `nan()` function in BandMathX #1801