otb issueshttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues2022-04-12T05:45:56Zhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2235Release 8.0.02022-04-12T05:45:56ZJulien OsmanRelease 8.0.0[[_TOC_]]
We are ready to release OTB version 8.0.0. The following steps need to be done:
## Blocking issues/MR
* [x] !865: Update the version of several OTB dependencies in the Superbuild
* [x] !868: Use std::chrono and the date libr...[[_TOC_]]
We are ready to release OTB version 8.0.0. The following steps need to be done:
## Blocking issues/MR
* [x] !865: Update the version of several OTB dependencies in the Superbuild
* [x] !868: Use std::chrono and the date library instead of boost date time as internal types
* [x] !870: Provide access to the metadata from the python API
* [x] !874: Fix segfault in Monteverdi
* [ ] !876
* [ ] !883
## Release Candidate 1
### 1. Branches
* [X] Feature freeze: [create the new release branch](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#create-release-branch)
* [X] Make sure the version number in `CMakeLists.txt` is 8.0.0
### 2. Housekeeping
* [x] In this story, make a list of blocking issues for the release (if any)
* [x] Update release notes (walk the GitLab MR merged history and log all improvements)
* [x] Update the date in RELEASE_NOTES.txt
* [x] Check [SonarQube](https://sonar.orfeo-toolbox.org/dashboard?id=orfeotoolbox-otb)
* [x] Run Debian [spelling](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#spelling-check) checker
* [x] Run shellcheck script from [OTB-Devutils/Scripts/](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb-devutils/blob/master/Scripts/run_shellcheck.sh)
* [x] [Update translation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#translation-for-monteverdi-mapla) for Monteverdi and Mapla
* [x] [Sanity check the binary packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#standalone-packages-sanity-check)
* [x] Windows
* [x] Linux
* [x] Mac
* [x] Test QGIS on qgis docker image
### 3. Actual release
Once all blocking issues are closed, and the previous steps are done:
* [x] [Tag the release candidate](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#release-tag)
* [x] Update GIT_TAG for all official remote modules (if needed)
### 4. Publish and plan next release
* [x] [Prepare and upload source packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#prepare-and-upload-source-packages)
* [x] [Promote staging packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#promote-staging-packages)
* [x] [Update documentation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#update-documentation)
* [x] Cookbook
* [x] Doxygen
* [x] [Update the SuperBuild archive](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#superbuild-archive)
* [x] Release Candidate announcement on the forum
## Release Candidate 2
### 1. Branches
* [X] Feature freeze: [create the new release branch](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#create-release-branch)
* [X] Make sure the version number in `CMakeLists.txt` is 8.0.0
### 2. Housekeeping
* [x] In this story, make a list of blocking issues for the release (if any)
* [x] Update release notes (walk the GitLab MR merged history and log all improvements)
* [x] Update the date in RELEASE_NOTES.txt
* [ ] Check [SonarQube](https://sonar.orfeo-toolbox.org/dashboard?id=orfeotoolbox-otb)
* [x] Run Debian [spelling](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#spelling-check) checker
* [x] Run shellcheck script from [OTB-Devutils/Scripts/](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb-devutils/blob/master/Scripts/run_shellcheck.sh)
* [x] [Update translation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#translation-for-monteverdi-mapla) for Monteverdi and Mapla
* [x] [Sanity check the binary packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#standalone-packages-sanity-check)
* [x] Windows
* [x] Linux
* [x] Mac
* [x] Test QGIS on qgis docker image
### 3. Actual release
Once all blocking issues are closed, and the previous steps are done:
* [x] [Tag the release candidate](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#release-tag)
* [x] Update GIT_TAG for all official remote modules (if needed)
### 4. Publish and plan next release
* [x] [Prepare and upload source packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#prepare-and-upload-source-packages)
* [x] [Promote staging packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#promote-staging-packages)
* [x] [Update documentation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#update-documentation)
* [x] Cookbook
* [x] Doxygen
* [x] [Update the SuperBuild archive](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#superbuild-archive)
* [x] Release Candidate announcement on the forum
## Release Candidate 3
### 1. Branches
* [x] Feature freeze: [create the new release branch](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#create-release-branch)
* [x] Make sure the version number in `CMakeLists.txt` is 8.0.0
### 2. Housekeeping
* [x] In this story, make a list of blocking issues for the release (if any)
* [x] Update release notes (walk the GitLab MR merged history and log all improvements)
* [x] Update the date in RELEASE_NOTES.txt
* [ ] Check [SonarQube](https://sonar.orfeo-toolbox.org/dashboard?id=orfeotoolbox-otb)
* [ ] Run Debian [spelling](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#spelling-check) checker
* [ ] Run shellcheck script from [OTB-Devutils/Scripts/](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb-devutils/blob/master/Scripts/run_shellcheck.sh)
* [x] [Update translation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#translation-for-monteverdi-mapla) for Monteverdi and Mapla
* [ ] [Sanity check the binary packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#standalone-packages-sanity-check)
* [x] Windows
* [x] Linux
* [x] Mac
* [x] Test QGIS on qgis docker image
### 3. Actual release
Once all blocking issues are closed, and the previous steps are done:
* [x] [Tag the release candidate](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#release-tag)
* [x] Update GIT_TAG for all official remote modules (if needed)
### 4. Publish and plan next release
* [x] [Prepare and upload source packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#prepare-and-upload-source-packages)
* [x] [Promote staging packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#promote-staging-packages)
* [x] [Update documentation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#update-documentation)
* [x] Cookbook
* [x] Doxygen
* [x] [Update the SuperBuild archive](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#superbuild-archive)
* [x] Release Candidate announcement on the forum
## Release
### 1. Branches
* [x] Make sure the version number in `CMakeLists.txt` is 8.0.0
### 2. Housekeeping
* [x] In this story, make a list of blocking issues for the release (if any)
* [ ] Fix compilation warnings on CI
* [x] Update release notes (walk the GitLab MR merged history and log all improvements)
* [x] Update the date in RELEASE_NOTES.txt
* [ ] Check [SonarQube](https://sonar.orfeo-toolbox.org/dashboard?id=orfeotoolbox-otb)
* [x] Run Debian [spelling](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#spelling-check) checker
* [ ] Run shellcheck script from [OTB-Devutils/Scripts/](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb-devutils/blob/master/Scripts/run_shellcheck.sh)
* [x] [Update translation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#translation-for-monteverdi-mapla) for Monteverdi and Mapla
* [ ] [Sanity check the binary packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#standalone-packages-sanity-check)
* [ ] Windows
* [x] Linux
* [ ] Mac
* [ ] Test QGIS on qgis docker image
### 3. Actual release
Once all blocking issues are closed, and the previous steps are done:
* [x] [Tag the release](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#release-tag)
* [x] Merge the release into develop
* [x] Merge the release into master
* [x] Update GIT_TAG for all official remote modules (if needed)
### 4. Publish and plan next release
* [x] [Prepare and upload source packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#prepare-and-upload-source-packages)
* [x] [Promote staging packages](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#promote-staging-packages)
* [x] [Update documentation](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#update-documentation)
* [x] Cookbook
* [x] Doxygen
* [x] WordPress page "Home" and "Download" pages
* [x] Upload OTB source archive to [Zenodo](https://zenodo.org/) to create a unique Digital Object Identifier (DOI)
* [ ] [Update the SuperBuild archive](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/wikis/Help-for-release-actions#superbuild-archive)
* [x] Release announcement
* [x] On the [forum](https://forum.orfeo-toolbox.org/)
* [x] On the [blog](https://www.orfeo-toolbox.org/blog/)
* [x] On [Twitter](https://twitter.com/orfeotoolbox)
* [x] Forward announcement to news_item@osgeo.org ([OSGeo news](https://www.osgeo.org/foundation-news/))
* [x] Remove public branches related to MR or bugfix merged before the release8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2142Refactor SAR sensor models2021-11-08T09:56:32ZCédric TraizetRefactor SAR sensor modelsPart of the [sensor model refactoring](#2040).
The Sar functionalites should be refactored to become independent of Ossim.
### SAR model
##### Metadata parsing
SARImageMetadataInterface : use of `ossimSarSensorModel` to parse metadat...Part of the [sensor model refactoring](#2040).
The Sar functionalites should be refactored to become independent of Ossim.
### SAR model
##### Metadata parsing
SARImageMetadataInterface : use of `ossimSarSensorModel` to parse metadata, it has already been refactored to rely on GDAL instead of Ossim for metadata parsing, metadata parsing from geom file is still to be implemented.
While other sensors models are implemented in QGIS plugin, OTB currently uses 4 SAR sensor models:
* Sentinel 1
* RadarSat 2
* TerraSar-X
* CosmoSkyMed
The metadata parsing has been implemented in !761
Support for geom files should still be added. TODO: can this be done for all sensors at once ? Or do we need to implement geom sensor by sensor ? (this was required for Optical Sensor Models, because each sensor had specific keys and needed post-processing)
##### ossimSARSensorModel
SAR processing in OTB is based on Ossim. Most SAR classes are implemented in the OssimPlugins module. The `ossimSarSensorModel`, deriving from `ossimSensorModel` from Ossim is the main virtual class used for SAR functionalities and SAR metadata reading.
SARSensorModelAdapters : Adapter class for ossimSarSensorModel processing functions, it defines the following methods :
``` cpp
bool LoadState(const ImageKeywordlist& image_kwl);
bool SaveState(ImageKeywordlist& image_kwl);
bool IsValidSensorModel() const;
bool Deburst(std::vector<std::pair<unsigned long, unsigned long>>& lines, std::pair<unsigned long, unsigned long>& samples, bool onlyValidSample = false);
bool BurstExtraction(const unsigned int burst_index, std::pair<unsigned long, unsigned long>& lines, std::pair<unsigned long, unsigned long>& samples, bool allPixels = false);
bool DeburstAndConcatenate(std::vector<std::pair<unsigned long, unsigned long>>& linesBursts, std::vector<std::pair<unsigned long, unsigned long>>& samplesBursts, unsigned int& linesOffset, unsigned int first_burstInd, bool inputWithInvalidPixels = false);
bool Overlap(std::pair<unsigned long, unsigned long>& linesUp, std::pair<unsigned long, unsigned long>& linesLow,
std::pair<unsigned long, unsigned long>& samplesUp, std::pair<unsigned long, unsigned long>& samplesLow, unsigned int burstIndUp,bool inputWithInvalidPixels = false);
bool WorldToLineSampleYZ(const Point3DType& inGeoPoint, Point2DType& cr, Point2DType& yz) const;
bool WorldToLineSample(const Point3DType& inGEoPOint, Point2DType& cr) const;
bool WorldToSatPositionAndVelocity(const Point3DType& inGeoPoint, Point3DType& satellitePosition, Point3DType& satelliteVelocity) const;
bool LineToSatPositionAndVelocity(const double line, Point3DType& satellitePosition, Point3DType& satelliteVelocity) const;
static bool WorldToCartesian(const Point3DType& inGeoPoint, Point3DType& outCartesianPoint);
static bool ImageLineToDeburstLine(const std::vector<std::pair<unsigned long, unsigned long>>& lines, unsigned long imageLine, unsigned long& deburstLine);
static void DeburstLineToImageLine(const std::vector<std::pair<unsigned long, unsigned long>>& lines, unsigned long deburstLine, unsigned long& imageLine);
```
ossimSARSensorModel is implemented by 3 sensor:
* ossimSentinel1Model
* ossimCosmoSkyMedModel
* ossimTerraSarXModel
##### ossimGeometricSARSensorModel
ossimGeometricSARSensorModel is another implementation of SAR models in OssimPlugins, it is used by older sensors (?), including:
* ossimRadarSatModel
* ossimRadarSat2Model
* ossimAlosPalsarModel
* ossimErsSarModel
* ossimEnvisatASarModel
while ossimGeometricSARSensorModel is not used by SARSensorModelAdapters, it is available through the ossimPlugins projection factory, which means OTB is able to transform sensor points to/from geo points for these sensors. As metadata parsing is not implemented (it has been removed at some point ?) for these models (excepting RadarSat2), it is only possible to use these sensor by providing a geom file containing the required metadata to instantiate the ossimSensorModel. These sensor are not tested in OTB.
Questions:
* are AlosPalsar, ErsSar, Envisat Asar and RadarSat deprecated ? Should we support them in OTB 8.0 ?
* There is a RPC model defined in RadarSat 2 metadata, can we use it ? There is a ossimRadarSat2RPCModel, but it is not used by the ossimProjection factory.
### Refactoring plan
- [ ] #2158 >> Bug in TerraSar-X calibration
- [x] #2154 >> Implement SAR metadata parsing from geom files.
- [x] #2159 >> Use ImageMetadata in SarCalibration filters
- [x] #2150 >> Implement a factory for SensorTransformBase (only RPCForwardTransform and RPCInverseTransform are defined at this point), and use it in `GenericRSTransform`.
- [x] #2160 >> Implement `otb::SarSensorModel`, this class provides the same algorithms as `SARSensorModelAdapters`. This is the main issue of the refactoring, and it requires refinement.
- [x] #2151 >> Implement `otb::SarForwardTransform` and `otb::SarInverseTransform` deriving from `SensorTransformBase`, using `otb::SarSensorModel` to map points
- [x] #2152 >> Refactor SAR filter and application to use `otb::SarSensorModel`, `otb::SarForwardTransform` and `otb::SarInverseTransform`.
- [X] ~~Implement `otb::SarGeometricSensorModel` ?~~ (out of scope of version 8.0)
- [x] #2153 >> Remove Ossim based SAR classes (including OssimPlugins)
While most of the code is in QGISPlugins and can be reused easily (some Ossim data structure should be refactored, and ITK or the standard library should be used), some functionalities might require more work, in particular, SAR model uses a lot of point projections utilities from Ossim, maybe Proj can be used as a replacement for these functionalities.
The impact of the refactoring on DiapOTB should also be investigated.
### Tests
Test should be added for all methods implemented in `SarSensorModel` test should also be added for `otb::SarForwardTransform` and `otb::SarInverseTransform` with different sensors. The baseline for these tests will be generated with Ossim and OTB 7.2.0.
Some test have been deactivated while implementing the geom file reading (!759), because they rely heavily on the new SAR model. We need to reactivate them while implementing this model.
- saTvSarDeburstImageFilterTest
- saTvSarDeburstImageFilterTest2
- saTvSarDeburstImageFilterTest3
- saTvSarBurstExtractionImageFilterTest1
- saTvSarBurstExtractionImageFilterTest2
- prTvSensorModel_ers2-1
- prTvSensorModel_sentinel1-1
- prTvOrthoRectification_sentinel1_DEMGTIFF
- prTvOrthoRectification_sentinel1_DEMSRTM
- prTvOrthoRectification_sentinel1_noDEM
- otbSARAmplitudeEstimationImageFilterTest (DiapOTB)
- otbSARAmplitudeEstimationTest (DiapOTB)
- otbSARCartesianMeanEstimationTest (DiapOTB)8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1723Release 8.0.02021-11-05T10:49:25ZVictor PoughonRelease 8.0.0This is the planning issue for release 8.0.0
### Breaking changes
* [x] Remove application `LSMSSmallRegionsMerging` !233 (deprecated in 7.0.0)This is the planning issue for release 8.0.0
### Breaking changes
* [x] Remove application `LSMSSmallRegionsMerging` !233 (deprecated in 7.0.0)8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/1506Remove OSSIM2021-11-05T10:49:25ZGuillaume PaseroRemove OSSIM### What changes will be made and why they would make a better Orfeo ToolBox?
The main idea is to remove OSSIM from OTB, in order to reduce the dependency risk, and also to refactor some parts of the library.
#### High level descriptio...### What changes will be made and why they would make a better Orfeo ToolBox?
The main idea is to remove OSSIM from OTB, in order to reduce the dependency risk, and also to refactor some parts of the library.
#### High level description
OSSIM is used for geometric sensor modelling and metadata parsing. It has been used for that since the beginning of the OTB. Then adapter classes have been added, to hide OSSIM headers from OTB public API. Now, it is time to plan the removal of this dependency, whose development cycle is difficult to follow. In the current state, only a small portion of OSSIM is used anyway.
This work can be performed in several steps:
1. [x] Remove unused classes (classes that are not really used, and that won't be ported) (!221)
1. [x] Implement a new metadata parsing framework (rely on GDAL)
1. [x] Move FilterFunctionValues and MetadataKey to OTBMetadata (!651)
1. [x] Refactor OTB metadata (check #2024)
1. [x] Implement geometric models
1. [x] Refactor GenericMapProjection to use GDAL (!224)
1. [x] Re-implement Utils::GetZoneFromGeoPoint() (!224)
1. [x] re-implement DEMHandler (#1762)
1. [x] RPC model (based on GDAL ?) #2040
1. [x] Generic SAR model #2040
1. [x] SensorModel factory for external models (#2039)
1. [x] Minor migrations
1. [x] Replace the Sleep() function from OTBOpenThreadsAdapters
1. [x] Remove classes or modules that are not necessary anymore (#2153)
#### Risks and benefits
Risks:
- Large refactoring, there is a substantial amount of work needed, mostly for the porting of geometric models.
- Impacts on the validation tests, and baselines.
- Several components are complex to migrate (DEMHandler, SarSensorModel,...)
Benefits:
- 4 mandatory dependencies will be removed from OTB: Ossim, OssimPlugins, GeoTiff, OpenThreads
- 2 modules will be removed: OTBOssimAdapters and OTBOpenThreadsAdapters
- Sanitize the code base
- Easier support of new sensors models
- Better metadata architecture (which allows better metadata processing in our pipelines)
- Separation between metadata parsing, and geometric modelling.
#### Alternatives for implementations
Here are some details and propositions regarding the implementation.
##### Unused classes #####
This is a list of classes that could be removed at the beginning of the refactoring without much trouble:
- Module OTBOssimAdapters
- otbDEMConvertAdapter
- otbGeometricSarSensorModelAdapter
- otbPlatformPositionAdapter
- Module OTBProjection
- old map projections: Eckert, Lambert3CartoSud, LambertConformalConic, Mollweid, SinusoidalMap, SVY21Map, TransMercator. We should rather have a set of WKT strings for the "common" map projections.
- Module OTBAppImageUtils
- Application DEMConvert
##### Metadata parsing #####
Currently, metadata is parsed from each child class of ossimSensorModel. This metadata
is then exported into an ImageKeywordlist. On the other hand, the otbXXXImageMetadataInterface
is designed to expose this metadata in a generic way. It would make sense to move the
metadata-parsing code from ossimSensorModels into otbXXXImageMetadataInterface ("IMI"), and have
a single class that represents a sensor metadata.
I have been thinking about a more generic way to parse metadata. In the end, there are
always corner cases where the metadata is read from some file and then transformed before
being recorded into the ImageKeywordlist. I propose the following workflow to replace the
current function ``ReadGeometryFromImage()``:
1. We use a classic IMIFactory to find if a given IMI (associated to a given sensor) can read
a product. For a given input image filename, each IMI can return the list of associated
metadata files and their type (DIMAP, SAFE, txt file, ... ). Note that the image itself can
also be part of the list if there are metadata embedded in the image.
2. Reading the metadata files. The purpose of this step is to parse each metadata file associated
with the image file and supply it as a (in-memory) XML tree. This tree is given to a IMI that
will look for needed information. The parsing can be done by different classes, so far we expect
a plain XMLReader, a TextMetadataReader and a GDALMetadataReader. They can all derive from a
base class. Depending on the format:
* XMLReader can be simply implemented using TinyXML
* TextMetadataReader will try to parse 'key=value' pairs and format it in a XML tree.
* GDALMetadataReader will use GDALDataset::GetMetadata() to extract 'key=value' pairs
and format them into a XML tree.
3. Parsing in IMI. This step consists in finding the relevant metadata in the different XML trees
and mapping them into a KeywordList (map<string,string>) with the usual metadata keys (used
in geom files). At this step, we keep the metadata as strings, but we may also do specific
conversions to check the numerical range. This parsing can be nicely written with generic
functions like ``add()`` used in ossimSentinel1Model.cpp. If the parsing returns successfully,
the generated ImageKeywordlist is given to the input image metadata dictionary.
4. Instanciation of a Transform. I propose to separate the classes used to store and access
the product metadata, and the classes used to represent a transform associated with a sensor
model. The truth is that among all the sensor models currently handled by Ossim, we mostly
use RPC based models. There are few physical sensor models, but we could implement them with
a generic "PushBroomSensorModel". Maybe complex physical models will actually deserve a separate
implementation. Regarding SAR sensors, we now tend to rely on SarSensorModel which is generic.
This separation would also be consistent if we want to handle both physical and RPC models associated
with a given sensor (no need to clone the metadata class). An other good point is that pipelines
that don't do any geometry or projection stuff will not have to instanciate and configure a sensor
transform that they won't use anyway. The SensorTransform will be initialized
from the ImageKeywordList obtained in step 3.
The following diagram sums up the different steps.
![workflow_metadata](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb-devutils/raw/master/Scripts/ossim/workflow_metadata.png)
In the case of a single image file with a geom, the steps 1, 2 and 3 are not needed, the geom is
directly injected into the ImageKeywordlist.
In a nutshell, the IMI classes will be in charge of identifying the correct sensor and parsing all the
relevant metadata. The SensorTransform classes will operate various geometric transforms, configured
from an ImageKeywordList.
The role of IMI classes can also be extended, for instance give the list of metadata keys that carry
per-band values. A concatenate file will then be able to gather these metadata together.
Note from RKH: use RapidXML instead of TinyXML.
##### Geometric Models #####
We move the code related to geometric transforms into several generic SensorTransforms. For now we can expect:
* a RPCSensorTransform (maybe based on ``GDALRPCTransform()``)
* a SarSensorTransform (a port of ossimSarSensorModel)
* (optional) a GeometricSarSensorTransform ? (for deprecated SAR models)
* (optional) a PushBroomSensorTransform ? (for Spot5)
* (optional) a MatrixSensorTransform ? (for airborne sensors)
For geometric transform, all the structures like ossimDpt, ossimGpt should be replaced with ITK classes.
Ideally, the new SensorTransform could derive from otb::SensorModelBase, but the current classes
SensorModelBase, ForwardSensorModel and InverseSensorModel are not really usefull. The difference between
forward and inverse transform could very well be implemented with a template specialization on
SensorModelBase. I propose to remove ForwardSensorModel and InverseSensorModel, refactor SensorModelBase,
and derive new SensorTransforms from SensorModelBase.
Like with ossimSensorModel, we can implement tuning parameters in otb::SensorModelBase to reproduce
the ossimAdjustableParameterInterface. The adjustment parameters are often used to configure a
residual affine transform that is composed with the geometric model.
##### DEMHandler #####
Role:
* Open a DEM directory (with SRTM tiles, single GeoTIFF file, ...)
* Open a geoid file (\*.egm)
* Handle a default elevation setting
* Provide the elevation at any coordinates (lon/lat)
Actions: there is nothing really equivalent in GDAL. If we go for a custom development, we could
also plan a different use of this DEM:
* Point-based: this is the current implementation. We query the DEMHandler for an elevation at each
position independently. This implementation requires some caching to be efficient (done by Ossim,
can also be done by GDAL).
* Grid-based: this is a new strategy that could be implemented. For instance, during an
orthorectification process, a deformation grid is computed, using calls to the DEMHandler
at each point. It would be more efficient to extract the elevation values block by block.
In this case, we can also foresee some relevant features for DEM processing (like no-data
filling, reprojection, smoothing,...).
With a grid-based approach, the DEMHandler would rather return an image pointer (probably the output
of a pipeline) than a single float. After a quick review of the calls to
``itk::Transform::TransformPoint()``, it seems difficult to avoid the point-based service in
the DEMHandler.
The best example is the computation of an ortho extent. This requires (at least) 4 forward
transforms for each image corner. Assuming a RPC model, this sensor-to-ground transform uses
an iterative approach, because the RPC gives the transform from ground to sensor. The DEMHandler
will have to supply several elevations for each image corner and the (x,y) coordinates of theses
elevations can't be easily predicted. Also, since the corners may represent a large extent on
the ground, it is risky to assume that the whole region can be buffered.
Then, there is also the question of the input to the DEMHandler. At the moment, we give a directory
and an (optional) path to a geoid file. Giving a path to a DEM file could be easier for the
users, but keeping the existing behaviour would ensure nothing is broken on user side.
Since a DEM is often composed of several tiles, the VRT solution seems the best to gather
the tiles in a single entry point. We can even create it in-memory to deal with backward
compatibility.
The solution proposed is to work with 3 classes:
* ``DEMHandler``: part of OTBTransform, singleton, holds the defaults settings, and an internal DEM reader
* ``DEMReaderInterface``: part of OTBTransform, defines the interface to request point-based and
grid-based elevation
* ``GDALDEMReader``: part of OTBImageIO, implements the DEMReaderInterface, with point-based and
grid-based services as a standard image reader.
This is the proposed skeleton for the future ``DEMHandler`` class:
```cpp
class DEMHandler : public itk::Object
{
public:
/** Retrieve the singleton instance */
static Pointer Instance();
/** Default DEM settings */
string GetDEMPath();
void SetDEMPath(string);
string GetGeoidPath();
void SetGeoidPath(string);
double GetDefaultHeight();
void SetDefaultHeight();
void SetReader(reader);
DEMReaderInterface* GetReader();
private:
static Pointer m_Singleton;
string m_Geoid;
string m_DEM;
double m_DefaultHeight;
/** internal DEMReaderInterface */
DEMReaderInterface::Pointer m_Reader;
};
```
The skeleton of ``DEMReaderInterface`` will look like:
```cpp
class DEMReaderInterface
{
public:
/** Point-based elevation */
virtual double GetHeight(double x, double y);
/** Grid-based elevation */
virtual otb::Image<double,2>* GetHeightGrid();
/** setup function */
void SetSource(dem, geoid, defaultHeight);
};
```
The ``GDALDEMReader`` class will be a composite filter, containing the following boxes:
* [optional] ``DirectoryToVRT``: small tool used if the DEM path is a directory. It will:
* find the first VRT file in that directory
* or create an in-memory VRT using image present in the directory
* ``ImageFileReader``: actually reads an input VRT (we may also use GDALImageIO directly)
* ``DEMResampler``: can be used to support interpolation, resampling the DEM to a different grid and
a different SRS.
* [optional] ``GeoidReader``: reads the input geoid file, this will likely be an ``ImageFileReader``,
it may use the full buffer if the size allows it.
* [optional] ``AddImageFilter``: combine DEM and geoid heights
This reader will be used during the point-based calls: a small patch of DEM will be extracted around
the requested location. During grid-based calls, it will return an ``otb::Image`` connected to a pipeline.
The ``DEMResampler`` can be really simple at the start. The only mandatory feature to be implemented
is the interpolation on a different grid (using the same SRS). Indeed, this is how the point-based
method ``GetHeight()`` will interpolate the correct elevation. Later on, we can improve it to support:
* transformation to a different SRS
* gap-filling
* smoothing
* ...
It could be a nice feature in the long term.
We will probably need to modify the GDAL_CACHE_SIZE variable for better performances. Also, this
cache size should be subtracted from our RAM parameter.
### Who will be developing the proposed changes?
TBD8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2239Link otb 8.0 with boost2022-03-08T07:15:13ZCédric TraizetLink otb 8.0 with boostSome users are unable to compile eternal code and standalone remote modules against OTB 8.0 without calling find-package Boost first.
I failed to reproduce this bug using the binary package for now
The user was using a compiled versio...Some users are unable to compile eternal code and standalone remote modules against OTB 8.0 without calling find-package Boost first.
I failed to reproduce this bug using the binary package for now
The user was using a compiled version using system libraries in a docker8.0.0Thibaut ROMAINThibaut ROMAINhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2261Cannot get the output statistics of ComputeImagesStatistics2022-04-12T05:45:56ZNicolas NarçonCannot get the output statistics of ComputeImagesStatisticsCurrently, the application ComputeImagesStatistics only prints the mean and standard deviation and can optionally save the result inside an XML.
I would like to be able to get the result in Python, as it is possible in ReadImageInfo, so...Currently, the application ComputeImagesStatistics only prints the mean and standard deviation and can optionally save the result inside an XML.
I would like to be able to get the result in Python, as it is possible in ReadImageInfo, something such as:
```python
stats = otbApplication.Registry.CreateApplication('ComputeImagesStatistics')
stats.SetParameterStringList('il', ['image.tif'])
stats.Execute()
mean = stats.GetParameterFloat('mean')
```8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2260Problem for TSX metadata in V8 (parseGeom)2022-04-12T05:45:55ZGaëlle USSEGLIOProblem for TSX metadata in V8 (parseGeom)### Description
DEM projection can raise a seg fault for TSX inputs. A Seg Fault is raised if metadata comes from a geom file (OTB = 7.4)
Hera a [patch](/uploads/eda60d4fa2481c7ef6d6866529428cb1/tsx_metadata.patch) to solve this issue ...### Description
DEM projection can raise a seg fault for TSX inputs. A Seg Fault is raised if metadata comes from a geom file (OTB = 7.4)
Hera a [patch](/uploads/eda60d4fa2481c7ef6d6866529428cb1/tsx_metadata.patch) to solve this issue in 7.4. FYI, I tried to remain compliant with OTB < 7.4 (different keys in geom) but I did not check it.
### Configuration information
OS centos 7.9 (HAL), OTB release-8.08.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2258Problem for CSK metadata in V82022-03-15T07:08:47ZGaëlle USSEGLIOProblem for CSK metadata in V8### Description
I have noticed a few problems using CSK metadata:
* CSK metadata interface always set HDF5:*filename*://SBI//S01 as image. It does not work if an extraction was made from a CSK product.
* Missing metadata in `parseGDAL` ...### Description
I have noticed a few problems using CSK metadata:
* CSK metadata interface always set HDF5:*filename*://SBI//S01 as image. It does not work if an extraction was made from a CSK product.
* Missing metadata in `parseGDAL` function such as NumberOfColums or GCPTimes
* Missing metadata in `parseGeom`
Here a [patch](/uploads/9a5ca82f7b5266d7734f2cc4252c98e5/csk_metadata.patch).
Let me know if there is a problem.
### Configuration information
OS centos 7.9 (HAL), OTB release-8.08.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2257Problem nb lines/columns in SARBurstExtraction2022-03-15T07:08:59ZGaëlle USSEGLIOProblem nb lines/columns in SARBurstExtraction### Description
A little confusion between *NumberOfLines* and *NumberOfColumns* in `SARBurstExtraction` : [patch](/uploads/9090f64ea919491b5383625c66fe7d66/burst_extraction_correction.patch)
### Configuration information
OS centos 7....### Description
A little confusion between *NumberOfLines* and *NumberOfColumns* in `SARBurstExtraction` : [patch](/uploads/9090f64ea919491b5383625c66fe7d66/burst_extraction_correction.patch)
### Configuration information
OS centos 7.9 (HAL), OTB release-8.08.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2256Problem with right_looking_flag on S1 products2022-03-15T07:09:12ZGaëlle USSEGLIOProblem with right_looking_flag on S1 products### Description
Rigth looking flag is always true for S1 products. Old metadata (from v7.4/geom) do not contain a key to specify this flag (such as [support_data.look_side](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/releas...### Description
Rigth looking flag is always true for S1 products. Old metadata (from v7.4/geom) do not contain a key to specify this flag (such as [support_data.look_side](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/release-8.0/Modules/Core/Metadata/src/otbSarImageMetadataInterface.cxx#L300)). Without this key, the flag is set, by default to false.
IMO, we can set this flag to true in `Sentinel1ImageMetadataInterface` : [patch](/uploads/eeeef091657813fb0fc30610410f9434/rightlookingflag_s1_from_geom.patch)
### Configuration information
OS centos 7.9 (HAL), OTB release-8.08.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2255Wrong projection in range for S1 IW products2022-03-15T07:09:24ZGaëlle USSEGLIOWrong projection in range for S1 IW products### Description
I have noticed huge differences between v7.4 and v8 projection using `WorldToLineSampleYZ` function from `SarSensorModel`.
A bug is present on [nearRangeTime value](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/bl...### Description
I have noticed huge differences between v7.4 and v8 projection using `WorldToLineSampleYZ` function from `SarSensorModel`.
A bug is present on [nearRangeTime value](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/release-8.0/Modules/Core/Metadata/src/otbSARMetadata.cxx#L194). If the input image has *AzimuthBandwidth*, *RangeBandwidth* or *AzimuthSteeringRate* as metadata, *nearRangeTime* value is wrong.
Here a [patch](/uploads/06c3493ae0e294b8a64277ef418ccfdd/nearRangeTime_correction.patch) to correct this bug. Projection on range dimension seems to be OK with this patch.
### Configuration information
OS centos 7.9 (HAL), OTB release-8.08.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2241Segfault when using several images with no projection In Monteverdi2022-01-04T19:38:51ZCédric TraizetSegfault when using several images with no projection In Monteverdi### Description
Monteverdi crash when trying to open two images with no projection
### Steps to reproduce
```
monteverdi '/mnt/datas/OTB2/otb/Data/Input/StereoMoving.png' '/mnt/datas/OTB2/otb/Data/Input/StereoFixed.png'
```
### Conf...### Description
Monteverdi crash when trying to open two images with no projection
### Steps to reproduce
```
monteverdi '/mnt/datas/OTB2/otb/Data/Input/StereoMoving.png' '/mnt/datas/OTB2/otb/Data/Input/StereoFixed.png'
```
### Configuration information
develop and release-8.0 branches.8.0.0Cédric TraizetCédric Traizethttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2240DOC: Improve documentation on how to recompile python bindings2022-02-07T08:32:43ZThibaut ROMAINDOC: Improve documentation on how to recompile python bindingsIn the cookbook, the users don't have the information about the system requirements to rebuild the python bindingsIn the cookbook, the users don't have the information about the system requirements to rebuild the python bindings8.0.0Cédric TraizetCédric Traizethttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2238Need to document how to build a remote module against binary packages2022-02-07T08:32:42ZYannick TANGUYNeed to document how to build a remote module against binary packages### Target documentation resources
CookBook should add a section (or paragraph here : https://www.orfeo-toolbox.org/CookBook-8.0/RemoteModules.html#installation-and-usage)
### Change requested
I've compiled my remote module againt an ...### Target documentation resources
CookBook should add a section (or paragraph here : https://www.orfeo-toolbox.org/CookBook-8.0/RemoteModules.html#installation-and-usage)
### Change requested
I've compiled my remote module againt an installed OTB, I've a "symbol not found" error when I launched the application.
This is due to a very tricky bug, discussed here : https://gitlab.orfeo-toolbox.org/s1-tiling/s1tilingsupportapplications/-/issues/4#note_93605
When I add this flag, it works fine :
`cmake ../../fast-lsd/ -DOTB_BUILD_MODULE_AS_STANDALONE=True -DCMAKE_CXX_FLAGS=-D_GLIBCXX_USE_CXX11_ABI=0`
This workaround should be documented.8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2237Implement a GetSize() function for the ImageMetadata class2022-01-04T19:38:50ZJulien OsmanImplement a GetSize() function for the ImageMetadata class### Description
With OTB 7.4, I use `keywordlist.GetSize()`. It is not possible with OTB 8.0 and ImageMetadata since it doens't have such a function.### Description
With OTB 7.4, I use `keywordlist.GetSize()`. It is not possible with OTB 8.0 and ImageMetadata since it doens't have such a function.8.0.0Julien OsmanJulien Osmanhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2233OTB plugin integration in QGIS >= 3.222022-02-07T08:32:43ZYannick TANGUYOTB plugin integration in QGIS >= 3.22### Target documentation resources
CookBook shall be updated to document recent QGIS changes in OTB plugin integration : https://www.orfeo-toolbox.org/CookBook/QGISInterface.html
### Change requested
QGIS 3.22 does not activate any mor...### Target documentation resources
CookBook shall be updated to document recent QGIS changes in OTB plugin integration : https://www.orfeo-toolbox.org/CookBook/QGISInterface.html
### Change requested
QGIS 3.22 does not activate any more OTB plugins (and some others) by default. As OTB plugin is an official plugin, it only needs to be activated (in the Plugins>Manage plugins menu) and then configured (as it used to be, in the Settings>Options menu).
This is a small change but users need to know how to reactivate OTB plugin.
Tests have been done on Ubuntu 20.04 / QGIS 3.22 RC / OTB 8.0a1 and it works fine.8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/22328.0.0-alpha2 fails to build without Python2021-11-08T09:17:16ZBas Couwenberg8.0.0-alpha2 fails to build without PythonOTB 8.0.0-alpha2 fails to build without Python.
The issue is fixed by only looking for Python when `OTB_WRAP_PYTHON` is enabled:
[python.patch](/uploads/7639a2c054f6f182dd2ed6a957d572cd/python.patch)OTB 8.0.0-alpha2 fails to build without Python.
The issue is fixed by only looking for Python when `OTB_WRAP_PYTHON` is enabled:
[python.patch](/uploads/7639a2c054f6f182dd2ed6a957d572cd/python.patch)8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2228Write no data as output in applications2021-11-05T10:49:26ZCédric TraizetWrite no data as output in applicationsSome applications still uses the itk::MetadataDictionary to store nodata information. While this work during the application, the metadata is not written in the output image at then end of the pipeline. For example this is the case with ...Some applications still uses the itk::MetadataDictionary to store nodata information. While this work during the application, the metadata is not written in the output image at then end of the pipeline. For example this is the case with the `Orthorectification` application.
Instead, the metadata `MDNum::NoData` should be used consistently inside the library8.0.0https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2226Wrap image metadata in SWIG2022-01-04T19:38:50ZCédric TraizetWrap image metadata in SWIG`ImageMetadata` sgould be wrapped in SWIG so metadata dictionaries can be accessed from Python. This would allow metadata manipulations in Python pipelines.
The whole ImageMetadata API should be exported, along with the metadata key enu...`ImageMetadata` sgould be wrapped in SWIG so metadata dictionaries can be accessed from Python. This would allow metadata manipulations in Python pipelines.
The whole ImageMetadata API should be exported, along with the metadata key enums. In addition, some metadata classes should be wrapped so they can be manipulated from Python (e.g. RPCParam, TimePoint, Duration).
Maybe we could use SWIG typemaps to wrap some of the exported types directly to Python types.8.0.0Julien OsmanJulien Osmanhttps://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/issues/2221Clean of elevation management geoid parameter2022-02-07T08:32:42ZaurelieemilienClean of elevation management geoid parameter### Description
This issue is a follow up of #2198
In the context of the [cars software](https://github.com/CNES/CARS), we are having troubles to unset the geoid parameter used in the otb applications.
CARS implements two homemade app...### Description
This issue is a follow up of #2198
In the context of the [cars software](https://github.com/CNES/CARS), we are having troubles to unset the geoid parameter used in the otb applications.
CARS implements two homemade applications that use the elevation management:
* [otbDEMReader](https://github.com/CNES/cars/blob/master/otb_remote_module/app/otbDEMReader.cxx) used in a CARS API function called [read_lowres_dem](https://github.com/CNES/cars/blob/master/cars/externals/otb_pipelines.py#L338)
* [otbConvertSensorToGeoPointFast](https://github.com/CNES/cars/blob/master/otb_remote_module/app/otbConvertSensorToGeoPointFast.cxx) used for example in the CARS API function [get_time_ground_direction](https://github.com/CNES/cars/blob/master/cars/core/projection.py#L574)
If several calls of the CARS API function are done in the same run session, the geoid parameter setting is not cleared between each otb application creation and execution. It seems that once `m_GeoidFile` is defined it is not unset when using [`DEMHandler::ClearDEM`](https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/blob/release-7.3/Modules/Adapters/OSSIMAdapters/src/otbDEMHandler.cxx#L111).
It might concerned usual otb application too but I couldn't print the geoid file path used by them (CARS also uses the `StereoRectificationGridGenerator` application).
### Steps to reproduce
Two consecutive calls of the application using the elevation management (I only tried with the CARS API function described above), one with the geoid file set and one without. The `m_GeoidFile` should be both time set to the first geoid file.
### Configuration information
Ubuntu, Centos, OTB 7.x (7.2), installation from binary version.8.0.0Cédric TraizetCédric Traizet