otb issueshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues2021-02-10T07:39:09Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2136RFC: Enable `zstd` support in SuperBuild2021-02-10T07:39:09ZLaurențiu NicolaRFC: Enable `zstd` support in SuperBuild### What changes will be made and why they would make a better Orfeo ToolBox?
ZSTD is a modern compression format that offers compression ratios comparable with `zlib`, but faster compression and decompression speeds: https://engineerin...### What changes will be made and why they would make a better Orfeo ToolBox?
ZSTD is a modern compression format that offers compression ratios comparable with `zlib`, but faster compression and decompression speeds: https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/. GDAL supports ZSTD since 2.3.
This is a proposal to include `libzstd` in the SuperBuild package, and enable GDAL's support of it. OTB applications will be able to use ZSTD compression seamlessly, by virtue of GDAL.
Results for a noisy `Float32` 10980x10980 TIFF, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
| ------ | ------ | ------ | ------ | ------ |
| Uncompressed | 482 330 028 b | N/A | N/A | N/A |
| DEFLATE | 419 729 334 b | 4.030s | 0.687s | 1.647s |
| ZSTD-9 | 415 737 011 b | 8.514s | 3.787s | 0.903s |
Results for a 15-bit `Int16` 10980x10980 TIFF, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
| ------ | ------ | ------ | ------ | ------ |
| Uncompressed | 226 122 884 b | N/A | N/A | N/A |
| DEFLATE | 201 822 038 b | 9.292s | 7.615s | 8.064s |
| ZSTD-9 | 207 553 979 b | 13.921s | 7.801s | 7.696s |
Results for a the file above, converted to 16-bit, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
| ------ | ------ | ------ | ------ | ------ |
| Uncompressed | 241 187 444 b | N/A | N/A | N/A |
| DEFLATE | 177 040 501 b | 1.736s | 0.396s | 0.869s |
| ZSTD-9 | 184 242 872 b | 12.715s | 3.077s | 0.558s |
Results for a the file above, converted to 16-bit, horizontal differencing predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
| ------ | ------ | ------ | ------ | ------ |
| Uncompressed | 241 187 444 b | N/A | N/A | N/A |
| DEFLATE | 159 937 804 b | 2.053s | 0.392s | 1.722s |
| ZSTD-9 | 153 871 845 b | 13.487s | 3.058s | 0.735s |
| ZSTD-3 | 154 626 632 b | 1.643s | 0.378s | 0.565s |
Times are best of three runs, taken on my computer, with GDAL from `osgeo/gdal:ubuntu-full-3.2.0`.
It seems that users would need to be very careful selecting the compression level, because the default is less efficient than DEFLATE-6.
#### Risks and benefits
- it makes SuperBuild more complex (one more library to keep track of)
- the SuperBuild package will be slightly larger (`libzstd.so` is about 678 KB)
- files saved with ZSTD will probably be unreadable by most software, but it's not used by default
- it allows users to take advantage of the faster compression and decompression speeds
#### Alternatives for implementations
Do nothing.
### Who will be developing the proposed changes?
Not sure.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1778Test refactoring2022-01-07T11:47:24ZGuillaume PaseroTest refactoring### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better...### What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better balance between over-tested and under-tested classes.
#### High level description
3 steps are proposed for this refactoring:
* [ ] Refactor the longest tests. 41 tests have been identified as they take more than 10s to run.
* [ ] Design a test framework without IO, only in-memory images
* [ ] Refactor filters `Tv` to use the new test framework
Prior to these steps, 2 other taks should simplify this work:
* #1765 : the modules to be moved out of OTB should reduce the number of tests
* !299 : refactor all functor based filters, their test will be much simpler
#### Risks and benefits
The main risk is on the refactoring time, that could be significant as there are about 2300 tests.
On the bright side, it will be possible to greatly reduce the total testing time, as well as the need for large test data repository.
#### Alternatives for implementations
##### Step 1
The list of 41 pre-selected tests to refactor is here (with associated timings in seconds):
```
10.1 ioTvDEMToImageGeneratorFromImageTest_SensorModel
10.38 bfTuMeanShiftSmoothingImageFilterROIQBMul4
10.49 coTvCoordinateToNameMultithreadTest
10.93 obTuMeanShiftStreamingConnectedComponentSegmentationOBIAToVectorDataFilter
11.1 ioTvKmzProductWriterWithGCP
11.27 apTvSeLargeScaleMeanShiftTest
11.6 leTvSVMMachineLearningModelReg
13.18 bfTvMeanShiftSmoothingImageFilterThreadingNonOpt
13.28 apTvSeSegmentationWatershedVector
14.37 dmTvDisparityMapEstimationMethod
14.72 leTeSEMModelEstimatorExampleTest
14.92 apTvSeSegmentationCCVector_ULOVW
15.2 leTeSOMClassifierExampleTest
15.29 apTvSeSegmentationMeanshiftVector
15.84 obTuMeanShiftConnectedComponentSegmentationFilter
16.47 feTvLocalHoughDraw2
16.63 apTvRaOpticalCalibration_UnknownSensor
16.79 apTvSeSegmentationCCVector_ULU
18.25 apTvClKMeansImageClassification_composite
20.58 apTvSeSegmentationCCVector
22.13 raTvReflectanceToSurfaceReflectanceImageFilter2
23.11 bfTuMeanShiftSmoothingImageFilterQBRoad
23.75 apTvCdbDSFuzzyModelEstimation_LI
26.45 apTvCdbDSFuzzyModelEstimation_LI_autoInit
27.77 apTvClSampleAugmentationReplicate
28.12 apTvFeLineSegmentDetectionNoRescale
29.79 apTvClSampleAugmentationSmote
29.96 leTvGradientBoostedTreeMachineLearningModel
31.1 maTeMarkovClassification3ExampleTest
32.73 leTeSVMImageEstimatorClassificationMultiExampleTest
33.31 apTvClSampleAugmentationJitter
34.57 hyTvMDMDNMFImageFilterTest2
36.32 apTvFeLineSegmentDetection
38.8 leTeSOMExampleTest
51.22 ioTeTileMapImageIOExampleTest
52.66 reTeImageRegistration2ExampleTest
60.84 ioTvSHPVectorDataFileReader3
60.98 ioTvTileMapImageIOWeb
88.13 dmTvFineRegistrationImageFilterTestWithMeanSquare
112.69 owTvQtWidgetShow
135.51 apTvClMethodGBTImageClassifierQB1
```
##### Step 2
In module OTBTestKernel:
* a set of functions to produce hardcoded images:
* template functions working on both otb::Image and otb::VectorImage
* start with image allocation (buffered region equals to largest possible region)
* fills the image with a formula, an array, ...
* a set of functions to set hardcoded image metadata
* fills any image metadata (projection ref, origin, spacing, OSSIM keywordlist, ...)
* a set of functions to produce hardcoded vector data:
* two families of functions : for otb::VectorData or otb::OGRDataSource
* fills the dataset with hardcoded geometries and fields
* hardcoded metadata (projection ref, ...)
* new generic functions to check the result of a boolean condition (similar to
an `assert` statement). There are blocking and non-blocking flavours (a
blocking check will stop the test execution if it fails).
##### Step 3
`Tv` tests can be re-written to use the in-memory test data. They are composed
of the following steps:
* Instanciate the filter to be tested.
* Instanciate a given fake image
* Set the fake image as input of the filter
* Set the filter parameters
* Call Update() on the filter
* Check the differences with the baseline (if any)
* Check other outputs of the filter (if any)
* If all checks passed: returns EXIT_SUCCESS
* If one check failed:
* write PNG file for the difference image (if any)
* return EXIT_FAILURE
With this strategy, the comparison of test and baseline images in CDash is
preserved. The baselines don't have to be images on disk, they can also be
hardcoded in the test (suited for tiny images, like 10 x 10 pixels).
### Who will be developing the proposed changes?
@salbert & OTB Dev Teamhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2365QGIS Plugin being removed from core2024-03-25T13:21:26ZJulien CabiecesQGIS Plugin being removed from core### Remove OTB QGIS Plugin from core
#### High level description
OTB provider plugin is currently being part of the QGIS core, and some [discussions](https://lists.osgeo.org/pipermail/qgis-developer/2023-November/066221.html) has been ...### Remove OTB QGIS Plugin from core
#### High level description
OTB provider plugin is currently being part of the QGIS core, and some [discussions](https://lists.osgeo.org/pipermail/qgis-developer/2023-November/066221.html) has been raised to remove it so it becomes a fully 3rd party plugin, maintained by the OTB community.
I have both contributed to QGIS and OTB and I'm in favor of this proposal.
#### Risks and benefits
I think it would ease development of the plugin and would lower the pain of managing compatibility between QGIS and OTB.
#### Alternatives for implementations
That would require to move the OTB QGIS Plugin to the [QGIS Plugin platform](https://plugins.qgis.org/) or even embed it in the OTB installer.
### Who will be developing the proposed changes?
At the moment, no one is clearly identified to make the modifications.
OTB is a great tool and we would like to see it still integrated into QGIS.
If you have any questions or concerns, please let us know.
CC @ytanguyhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2248Roadmap for OTB 9.02023-06-23T04:54:23ZJulien OsmanRoadmap for OTB 9.0### Introduction
During the [OTB User Days](https://www.orfeo-toolbox.org/otb-user-days-2021-plenary-talks-workshops-and-more/), we had the opportunity to talk about the future for OTB. This helped us make some plan regarding the coming...### Introduction
During the [OTB User Days](https://www.orfeo-toolbox.org/otb-user-days-2021-plenary-talks-workshops-and-more/), we had the opportunity to talk about the future for OTB. This helped us make some plan regarding the coming months. This issue is a story aiming at providing a single place to discuss everything that we (as a community) want to do in the scope of 9.0. Please, feel free to comment this issue to enhance the following propositions.
### Scope of OTB 9.0
The purpose of OTB 9.0 will be to improve packaging and maintainability.
#### Remove Qt, the GUI app and Monteverdi
The dependency to Qt makes the OTB package quite heavy, and requires much maintenance. The purpose of this dependency is to provide some nice GUI so user don't have to use the command line. QGIS with [the OTB plugin](https://www.orfeo-toolbox.org/CookBook/QGISInterface.html) allows one to run OTB applications without command line, and even more (display images, save result in memory, etc). So we wondered if the GUI provided with OTB were still relevant. The [survey](https://forum.orfeo-toolbox.org/t/please-participate-to-our-survey-how-do-you-use-otb/872) organized on November 2020 pointed out that very few people used it anyway.
Qt is also required to build Monteverdi. Monteverdi is a tool used for very specific applications, and does it's job very well. However, it hasn't evolved for years. So we decided to stop it's development. It will still be available for download with older version of OTB, but will be removed from OTB's main repository.
Of course, if anyone wants to take over the maintenance of the GUI or Monteverdi, they will be welcome to create a remote module.
#### Remove MacOS support
As MacOs is used by very few people, and Apple is moving to an ARM architecture, it is becoming more difficult to maintain OTB on this platform. Docker is fuly supported on MacOs, so we decided to concentrate on a Linux docker image that people can use on any MacOs.
#### Enhance the DEMHandler to manage huge directory
With OTB 8, when one provides a directory to the DEMHandler, it loads all the content of the directory into memory. If this directory contains a lot of DEM tiles, it can take a long time or even crash. We propose to improve the DEMHandler so it only loads the tiles needed by the process.
### Postponed to OTB 10
### Re-organize the packaging in modules
OTB should be modular, installable via light packages
#### Re-organize the tests
Many unit test relies on heavy data, and should be made a lot simpler.
#### Upgrade to ITK 5.x
ITK is one of OTB's the main dependencies. It's last major version, 5.0, was release on May 2019, and the latest version is 5.2.1 released on August 2021. But OTB still relies on ITK 4.x. The time has come for OTB to catch up. It requires [some adjustments](https://github.com/InsightSoftwareConsortium/ITK/blob/master/Documentation/ITK5MigrationGuide.md), but some preliminary work was already done (!194, !528).
~story9.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2143Crashes in DEMToImageGenerator2022-01-07T11:47:25ZCédric TraizetCrashes in DEMToImageGeneratorDEMToImageGenerator crashes randomly, this has been observed on `ioTvDEMToImageGeneratorFromImageTest` and `ioTvDEMToImageGeneratorFromImageTest2` tests on [CI](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/pipelines/6781). This mi...DEMToImageGenerator crashes randomly, this has been observed on `ioTvDEMToImageGeneratorFromImageTest` and `ioTvDEMToImageGeneratorFromImageTest2` tests on [CI](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/pipelines/6781). This might also be the cause of [this issue](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2054) (DEM example)
I think it is caused by the fact that OGRCoordinateTransformation is [not thread safe](http://osgeo-org.1560.x6.nabble.com/gdal-dev-OGRCoordinateTransformation-Thread-Safety-td5421166.html), but the same OGRCoordinateTransformation is used by different threads in `DEMToImageGenerator`.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2069Conda package broken2022-08-30T13:26:40ZCédric TraizetConda package brokenThe conda package is broken. On Ubuntu 18.04, conda 4.8.3 and Python 3.7 and 3.8 environments, the command `conda install -c orfeotoolbox otb` leads :
```
Collecting package metadata (current_repodata.json): done
Solving environment: fa...The conda package is broken. On Ubuntu 18.04, conda 4.8.3 and Python 3.7 and 3.8 environments, the command `conda install -c orfeotoolbox otb` leads :
```
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
failed
UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:
Specifications:
- otb -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0|>=3.5,<3.6.0a0']
Your python: python=3.7
If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.
```
Also reproduced on Ubuntu Virtual Machine. See this [forum post](https://forum.orfeo-toolbox.org/t/otb-conda-installation-on-wsl/708)8.2.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2062Option to remove the progress bar in the Python wrapper2022-12-16T10:38:19ZCédric TraizetOption to remove the progress bar in the Python wrapperIn the command line interface (`otbcli_XXX`), there is an option `-progress` available to hide the progress of an application. For example :
* With progress :
``` cpp
otbcli_EdgeExtraction -in S2A_ROI.tif -out /tmp/out.tif -progress...In the command line interface (`otbcli_XXX`), there is an option `-progress` available to hide the progress of an application. For example :
* With progress :
``` cpp
otbcli_EdgeExtraction -in S2A_ROI.tif -out /tmp/out.tif -progress true
2020-06-17 16:28:59 (INFO) EdgeExtraction: Default RAM limit for OTB is 256 MB
2020-06-17 16:28:59 (INFO) EdgeExtraction: GDAL maximum cache size is 794 MB
2020-06-17 16:29:00 (INFO) EdgeExtraction: OTB will use at most 12 threads
2020-06-17 16:29:00 (INFO): Estimated memory for full processing: 38.8794MB (avail.: 256 MB), optimal image partitioning: 1 blocks
2020-06-17 16:29:00 (INFO): File /tmp/out.tif will be written in 1 blocks of 1000x1000 pixels
Writing /tmp/out.tif...: 100% [**************************************************] (0s)
```
* Without progress :
``` cpp
otbcli_EdgeExtraction -in S2A_ROI.tif -out /tmp/out.tif -progress false
2020-06-17 16:29:05 (INFO) EdgeExtraction: Default RAM limit for OTB is 256 MB
2020-06-17 16:29:05 (INFO) EdgeExtraction: GDAL maximum cache size is 794 MB
2020-06-17 16:29:05 (INFO) EdgeExtraction: OTB will use at most 12 threads
2020-06-17 16:29:05 (INFO): Estimated memory for full processing: 38.8794MB (avail.: 256 MB), optimal image partitioning: 1 blocks
2020-06-17 16:29:05 (INFO): File /tmp/out.tif will be written in 1 blocks of 1000x1000 pixels
```
The Python wrapper does not have the same functionality and it could be interesting to add it.
When creating an application in Python an observer object is added on the pipeline, the progress is then redirected to the python console. Calling `RemoveAllObserver()` on the application will remove the progress reporting. However this API is not very explicit, and it can have side effects as all observers are removed.
A solution would be to store in the application Python object the tag associated to the observer return by ITK when creating the observer, to be able to remove the observer directly from Python with `RemoveObserver(tag)`. This could be encapsulated in a `Application.ReportProgress(bool)` method that removes the observer if there is one, or create another one if it has been previously destroyed.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2061Available drivers in Datasource and VectorData2022-04-12T09:42:06ZCédric TraizetAvailable drivers in Datasource and VectorDataThe list of drivers available to open and write vector data is not the same in `otb::VectorData` and `otb::ogr::DataSource`. The lists of (file extensions / driver) are :
* For `otb::ogr::DataSource` :
``` cpp
const ExtensionDriverAss...The list of drivers available to open and write vector data is not the same in `otb::VectorData` and `otb::ogr::DataSource`. The lists of (file extensions / driver) are :
* For `otb::ogr::DataSource` :
``` cpp
const ExtensionDriverAssociation k_ExtensionDriverMap[] = {
{".SHP", "ESRI Shapefile"}, {".TAB", "MapInfo File"}, {".GML", "GML"}, {".GMT", "OGR_GMT"}, {".GPX", "GPX"},
{".SQLITE", "SQLite"}, {".KML", "KML"}, {".CSV", "CSV"}, {".GPKG", "GPKG"}};
```
* For `otb::VectorData` (OGRVectorDataIO)
``` cpp
const std::map<std::string, std::string> m_OGRExtensionsToDrivers = {
{".SHP", "ESRI Shapefile"}, {".TAB", "MapInfo File"}, {".GML", "GML"}, {".GPX", "GPX"}, {".SQLITE", "SQLite"}, {".KML", "KML"},
{".GMT", "OGR_GMT"}, {".GPKG", "GPKG"}, {".JSON", "GeoJSON"}, {".GEOJSON", "GeoJSON"}};
```
This is confusing and limiting for OTB users. Some application are based on `VectorData` and some are base on `otb::ogr::Datasource`.
I think the same list should be used for both classes (the union of the two list). The reading/writing is done by ogr so I don't think there is any limitation on this side.
See this [forum post](https://forum.orfeo-toolbox.org/t/problem-when-saving-geojson-file-with-sampleselection-app/700/2)https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1754GDAL VSI support2024-03-27T14:47:33ZLaurențiu NicolaGDAL VSI supportI've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage dr...I've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage drivers. Unfortunately, OTB doesn't seem to.
I tried a couple of tests:
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
2018-10-24 13:26:34 (INFO): Default RAM limit for OTB is 128 MB
2018-10-24 13:26:34 (INFO): GDAL maximum cache size is 393 MB
2018-10-24 13:26:34 (INFO): OTB will use at most 4 threads
2018-10-24 13:26:35 (INFO):
Image general information:
Number of bands : 3
No data flags : Not found
Start index : [0,0]
Size : [65536,65536]
Origin : [0.5,0.5]
Spacing : [1,1]
Estimated ground spacing (in meters): [547.072,296.427]
Image acquisition information:
Sensor :
Image identification number:
Image default RGB composition:
[R, G, B] = [0,1,2]
Ground control points information:
Number of GCPs = 0
GCPs projection =
Output parameters value:
indexx: 0
indexy: 0
sizex: 65536
sizey: 65536
spacingx: 1
spacingy: 1
originx: 0.5
originy: 0.5
estimatedgroundspacingx: 547.0715332
estimatedgroundspacingy: 296.4269409
numberbands: 3
sensor:
id:
time:
town:
country:
rgb.r: 0
rgb.g: 1
rgb.b: 2
projectionref:
keyword:
gcp.count: 0
gcp.proj:
gcp.ids:
gcp.info:
gcp.imcoord:
gcp.geocoord
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt gdalinfo http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
Driver: GTiff/GeoTIFF
Files: none associated
Size is 514, 515
Coordinate System is:
PROJCS["unnamed",
GEOGCS["NAD27",
DATUM["North_American_Datum_1927",
SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
AUTHORITY["EPSG","7008"]],
AUTHORITY["EPSG","6267"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4267"]],
PROJECTION["Cylindrical_Equal_Area"],
PARAMETER["standard_parallel_1",33.75],
PARAMETER["central_meridian",-117.333333333333],
PARAMETER["false_easting",0],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]]]
Origin = (-28493.166784412522247,4255884.543802191503346)
Pixel Size = (60.022136983193739,-60.022136983193739)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( -28493.167, 4255884.544) (117d38'27.05"W, 33d56'37.74"N)
Lower Left ( -28493.167, 4224973.143) (117d38'27.05"W, 33d39'53.81"N)
Upper Right ( 2358.212, 4255884.544) (117d18'28.38"W, 33d56'37.74"N)
Lower Right ( 2358.212, 4224973.143) (117d18'28.38"W, 33d39'53.81"N)
Center ( -13067.478, 4240428.844) (117d28'27.71"W, 33d48'15.38"N)
Band 1 Block=514x15 Type=Byte, ColorInterp=Gray
```
Notice that OTB reports wrong origin, ground spacing and size values.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -e trace=file -f otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif"
[snip]
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid 6069] getcwd("/home/user", 4096) = 16
[pid 6069] lstat("/home/user/http:", 0x7ffed725d380) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.til", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", F_OK) = -1 ENOENT (No such file or directory)
[pid 6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 6069] stat("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x7ffed725dc50) = -1 ENOENT (No such file or directory)
[pid 6069] readlink("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x245a730, 2048) = -1 ENOENT (No such file or directory)
2018-10-24 13:26:57 (INFO): No kwl metadata found in file http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif
```
OTB tries to open HTTP URLs as files.
```
$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_ComputeImagesStatistics -il "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"
[snip]
[pid 6110] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqrr HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
OTB gets into an infinite loop trying to request the wrong URL (notice the `qqqqqqrr` suffix).
```
CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_Convert -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true" -out foo.tif
[snip]
[pid 6171] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqqq HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96
```
Same issue, different suffix. Uninitialized memory?
Of course, I expect performance to be pretty poor when doing network access, but sometimes it's useful, and the alternative is to use a FUSE emulation driver that might have its own problems.
Related: #1746
Related: #1506
Related: #1642https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1671Remove hard-coded muParser expression library-wise2024-03-20T14:05:37ZJulien MichelRemove hard-coded muParser expression library-wise### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muP...### What changes will be made and why they would make a better Orfeo ToolBox?
#### High level description
Some filters and classes use the `muParser` based filters with hard-coded expressions. This is bad for at least 2 reasons:
* `muParser` based filters are not as fast as their functor based counter-parts
* `muParser` is an optional dependency. Using it heavily results in many applications and filters disabled if `muParser` is disabled (see #1670 for instance)
#### Risks and benefits
Only risk might be small changing with floating point baselines.
Benefit is a faster OTB less dependent on muParser.
#### Alternatives for implementations
We need to spot places where `muParser` is mis-used:
```
$ grep -R -l BandMath Modules/ | grep -v BandMath | grep -v Parser
Modules/Applications/AppProjection/app/otbGridBasedImageResampling.cxx
Modules/Applications/AppImageUtils/app/otbCompareImages.cxx
Modules/Applications/AppStereo/app/otbStereoFramework.cxx
Modules/Applications/AppStereo/app/otbBlockMatching.cxx
```
### Who will be developing the proposed changes?
TDB.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2332Improve and update docs for a working Python installation2023-10-13T13:33:37ZVincent DelbarImprove and update docs for a working Python installationToday I had to help a colleague who was having trouble running `import otbApplication`.
This is something I see a lot here in the Maison de la Télédétection with linux newbies / interns that needs OTB and are confused by the docs.
Th...Today I had to help a colleague who was having trouble running `import otbApplication`.
This is something I see a lot here in the Maison de la Télédétection with linux newbies / interns that needs OTB and are confused by the docs.
The CookBook is well updated regarding Python related caveats.
Nonetheless there are still missing recommendations for some specific cases and versions.
Especially regarding conda, since the latest package version updated there is 7.1. And it seems everyone is using conda (I believe this is hell).
We need a section about how to import the python API of a fresh binary installation, within a conda env.
May be I can create a MR - I still need to learn the the docs contribution workflow.
Here is what I suggest :
- May be update the FAQ section about the libpython3.5 symlink with current python versions, or both
- Warn that python-otb is missing in any recent ubuntu distributions (even in ubuntugis - is it fixable ?)
- Extend the conda section, add command line examples to create a working conda env (py37 or py38 + symlinks for libpython on Linux)
- Add a section about Jupyter / Google Colab environments
- I sometimes had to downgrade numpy in order to recompile bindings, I don't know about the state of this in v8.0, is there a max version number that we can provide ?
- Explain this "recompile python bindings" thing will not work every time
(this only work when using the python exe found in /usr - I was unable to use it for a conda env - using python > 3.8)
Regarding this last point, it would be really great to ease the process and also create a dedicated shell script, that support any PYTHON_EXE.
I really have a hard time to understand this ctest trick, and was unable to adapt it to my env needs.
And what about Windows ? For example if I want to run OTB with python 3.10, is this even possible ?https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2227OTB develop version name2022-08-30T13:26:19ZCédric TraizetOTB develop version nameThe command line applications display the current OTB version. For example in OTB 7.4.0 the following message appears when calling an application:
```
This is the EdgeExtraction application, version 7.4.0
```
In between version the sam...The command line applications display the current OTB version. For example in OTB 7.4.0 the following message appears when calling an application:
```
This is the EdgeExtraction application, version 7.4.0
```
In between version the same message is displayed, for example the current develop branch also return version 7.4.0. It would be interesting to print the next minor or major version, as well as the current branch name and commit sha for these versions. For example the current develop version would return
```
This is the EdgeExtraction application, version 8.0.0-develop-de58687e
```8.2.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2198Elevation management tricky bugs2022-01-07T11:47:23ZEmmanuel DuboisElevation management tricky bugs### Description
Discussing with @ytanguy , I open a bug happening in [cars](https://github.com/CNES/CARS) 3D software.
We are using several OTB applications and it occurs that there are tricky behaviors when using elevation management ...### Description
Discussing with @ytanguy , I open a bug happening in [cars](https://github.com/CNES/CARS) 3D software.
We are using several OTB applications and it occurs that there are tricky behaviors when using elevation management API.
When using only one time or with the same dem, geoid or default_alt, the problem doesn't appear
But using several apps (different or not) with elevation interface seems to not reproduce the same results.
It appears that the first setting is not reset and we cannot put another one in next apps which takes.
This behavior is difficult to detect. For instance, the default_alt is not changing the result if elev.dem is set before.
The default_alt is not stable when already used in another app, ...
Is there a way to reset the elevation management information between two apps usage ?
Thanks by advance. First Bug in OTB so don't hesitate to react, change this content if not clear.
Emmanuel
### Steps to reproduce
Take an OTB application with elevation management (Example ImageEnvelop) and launch it two times with two dem different and check values with Python interface in the same code.
Take two OTB application with elevation management (Example ImageEnvelop and ConvertSensorToGeoPoint) and see standalone results and when the two apps are sequenced in the same python file.
Reproducibility: always
### Configuration information
Ubuntu, Centos, OTB 7.x (7.2), installation from binary version.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2174Synchronize the writing of several products from Python API.2022-01-27T09:27:05ZLuc HermitteSynchronize the writing of several products from Python API.## Feature description
It would be nice to have a way of using multiwriter from Python API without having to define a new application like `AppImageListWriter` in MAJA.
If we consider a flow like the following:
```
+----+ /-------...## Feature description
It would be nice to have a way of using multiwriter from Python API without having to define a new application like `AppImageListWriter` in MAJA.
If we consider a flow like the following:
```
+----+ /--------\ +-----+ /----------------\ +---------+ /------------\ +----------+ /----------\ +-------+
| S1 | --> | to_xyz | --> | XYZ | --> | extractNormals | --> | Normals | --> | computeLIA | --> | LIA | --> | OrthoLIA | --> | S2LIA |
+----+ \--------/ +-----+ \----------------/ +---------+ \------------/ +----------+ \----------/ +-------+
|
|
v
+------------+ /----------\ +----------+
| Sin | --> | OrthoSin | --> | S2Sin |
+------------+ \----------/ +----------+
```
We don't have any way (as of v7.2) to plug extra applications on separate paths after an application that produces several images.
In an ideal world, we could write something like
```python
import otbApplication as otb
to_xyz = otb.Registry.CreateApplication("to_xyz")
to_xyz.SetParameterString("in", "S1")
extract_normals = otb.Registry.CreateApplication("extractNormals")
extract_normals.SetParameterInputImage("in", to_xyz.GetParameterOutputImage("out"))
compute_lia = otb.Registry.CreateApplication("computeLIA")
compute_lia.SetParameterInputImage("in", extract_normals.GetParameterOutputImage("out"))
ortho_lia = otb.Registry.CreateApplication("OrthoLIA")
ortho_lia.SetParameterInputImage("in", compute_lia.GetParameterOutputImage("out.lia"))
ortho_lia.SetParameterString("out", "S2LIA")
ortho_sin = otb.Registry.CreateApplication("OrthoSIN")
ortho_sin.SetParameterInputImage("in", compute_lia.GetParameterOutputImage("out.sin"))
ortho_sin.SetParameterString("out", "S2SIN")
otb.WriteOutputs([ortho_lia, ortho_sin])
```
## Implementation
Add the static function `WriteOutputs` to `otb::Wrapper::Application`.
```c++
void WriteOutputs(std::vector<otb::Wrapper::Application> apps);
```
This function does the same work than MAJA's application `AppImageListWriter`: instantiate an otbMultiImageFileWriter filter, set its parameters and update it.
Add the signature of this function to `Application.i`https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2171Support reading products from virtual file system2022-01-07T09:48:59ZMickael SavinaudSupport reading products from virtual file systemAs described by GDAL [here](https://gdal.org/user/virtual_file_systems.html), EO products are delivered as archive and in parallel more and more stored on network (in object storage for example).
Currently (with 8.0.0-alpha1) even if a...As described by GDAL [here](https://gdal.org/user/virtual_file_systems.html), EO products are delivered as archive and in parallel more and more stored on network (in object storage for example).
Currently (with 8.0.0-alpha1) even if a part of metadata are read with GDAL, we use specific code to retrieve the missing metadata. This code don't support virtual file system although the GDAL part yes. For example, you can made otbcli_ReadImage on a Sentinel-1 data in an archive and retrieve some metadata thanks to GDAL but not from the OTB part.
It would be nice to support this virtual file system in the OTB metadata. With that we can run an ortho on the Pleiades data without unarchive the product or run SARCalibration on a Sentinel1 data store in an object storage.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2146ImageClassifier to handle a list of images/masks2021-02-11T08:37:32ZThomas TilakImageClassifier to handle a list of images/masksClassifying an image with [ImageClassifier](https://www.orfeo-toolbox.org/CookBook/Applications/app_ImageClassifier.html) could be (at least in my use case) a long operation.
When one need to classify several images with a model, the l...Classifying an image with [ImageClassifier](https://www.orfeo-toolbox.org/CookBook/Applications/app_ImageClassifier.html) could be (at least in my use case) a long operation.
When one need to classify several images with a model, the loading of the model is repeated for each image.
One improvement would be to accept a list of images/masks as input of `otbcli_ImageClassifier`.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2112Use symlink in main package page2022-01-07T07:30:30ZCédric TraizetUse symlink in main package pageThere are two pages on the website containing release packages:
* `https://www.orfeo-toolbox.org/packages/`: packages corresponding to the current release
* `https://www.orfeo-toolbox.org/packages/archives/OTB/`: all otb packages.
Pac...There are two pages on the website containing release packages:
* `https://www.orfeo-toolbox.org/packages/`: packages corresponding to the current release
* `https://www.orfeo-toolbox.org/packages/archives/OTB/`: all otb packages.
Packages for the current release are duplicated on two different pages. We could simplify it by creating symlinks from the main package page to the archive page.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2110ConvertSensorToGeoPoint application with Elevation management2020-10-07T08:33:30ZJulien OsmanConvertSensorToGeoPoint application with Elevation managementIt would be [interesting](https://forum.orfeo-toolbox.org/t/convertsensortogeopoint-elevation-management/811) to be able to use a DEM as input of the ConvertSensorToGeoPoint application. The new RPC mechanism that will come with OTB 8.0 ...It would be [interesting](https://forum.orfeo-toolbox.org/t/convertsensortogeopoint-elevation-management/811) to be able to use a DEM as input of the ConvertSensorToGeoPoint application. The new RPC mechanism that will come with OTB 8.0 should make it easy to implement.https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2108Add setup.py to install python Bindings2020-10-23T07:48:26ZEsquis BenjaminAdd setup.py to install python BindingsFor now the python binding is only installed in the install folder but not officially in the pythons libraries
It would be interesting to provide a setup.py to install the python bindings in the python workspace:
For example:
```
#!/us...For now the python binding is only installed in the install folder but not officially in the pythons libraries
It would be interesting to provide a setup.py to install the python bindings in the python workspace:
For example:
```
#!/usr/bin/env python
from distutils.core import setup
setup(name='otbApplication',
version='7.0',
description='OTB Python bindings',
author='OTB Team',
author_email='https://forum.orfeo-toolbox.org/',
url='https://www.orfeo-toolbox.org/',
packages=['otbApplication'],
package_data={'otbApplication': ['_otbApplication.so']},
)
```https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2104Remove 6S2020-09-22T08:12:38ZMickael SavinaudRemove 6S### What changes will be made and why they would make a better Orfeo ToolBox?
6S code is outdated and very difficult to maintain. Other version are available but it could be complicated to update the code in OTB due to manual correction ...### What changes will be made and why they would make a better Orfeo ToolBox?
6S code is outdated and very difficult to maintain. Other version are available but it could be complicated to update the code in OTB due to manual correction done after f2c.
Therefore it could be interesting to remove 6S and keep outside the radiative transfer code from OTB.
The idea is to delegate the functionalities to an external code and managed file interface. For example we can use the [SOS](https://github.com/CNES/RadiativeTransferCode-SOS) code to generate the LUT as for MAJA software. Moreover the MAJA software will be released in open source, so we can reuse the code to read the data.
#### High level description
#### Risks and benefits
Mutualize code with MAJA
#### Alternatives for implementations
### Who will be developing the proposed changes?