OTB 8.0 alpha 1 (Optical image processing) & alpha 2 (SAR image processing) and validation plan
The main goal of release 8.0 is to remove OSSIM from OTB dependencies. This implies the refactoring of a lot of classes and functions, many of which are in the core modules of the library. See this issue for a detailed plan of the refactoring. Before the actual release and release candidates, it is planned to release two alpha versions:
- OTB 8.0.0 alpha-1 : Optical processing, OTB still depends on Ossim, but only use the library in SAR processing
- OTB 8.0.0 alpha-2 : SAR processing, OTB does not depend on Ossim.
As many users do not use the SAR functionnalities of OTB, the first alpha will give them the opportunity to test the refactored functionalities as soon as possible. This includes:
- The Metadata framework
- The DEM Handler
- The radiometry framework (optical calibration and filters)
- The RPC framework
Given the scale of the refactoring, particular attention should be given to the validation of these changes.
Validation plan
CI validation:
The major part of the validation is done using the OTB CI.
Metadata framework
See https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/wikis/Remove-OSSIM#the-new-imagemetadata-object
The metadata parsing framework is mainly validated by the tests of the OTBMetadata
module, in particular :
-
otbImageMetadataTestXXX
tests are used to validate the API of theImageMetadata
class. -
ioTvImageMetadataInterfaceTest
tests are used to validate theOpticalImageMetadataInterface
classes. All the implemented sensors are tested (one test per sensor).
Broadly speaking, a lot of tests are impacted by the changes in metadata parsing, and non regression on these tests is also used for the validation.
Radiometry Framework
See https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/wikis/Remove-OSSIM#metadata-parsing
The only API changes in classes related to optical calibration is the use of ImageMetadata
objects instead of KeywordLists
. The test have not been modified and the changes are validated by verifying that there are no regression.
DEM Handler
See https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/-/wikis/Remove-OSSIM#dem-handler
The DEMHandler
class API is validated using the uaTvDEMHandlerXXX
tests, testing DEM request on different configuration (with or without geoids, with or without DEM).
-
TODO: these tests are not at the right place at the moment, they should be moved from the OSSIMAdapters module to the GDALIO
module.
The RPC framework
The RPC transforms are now based around the GDAL functions GDALCreateRPCTransformer
and RPCTransformPoint
. The otb::GDALRPCTransformer
(OTBIOGDAL
module) is a wrapper around these classes. It is tested by the TuGDALRPCTransformerTest
tests.
The main entry points for these classes are otb::ForwardRPCTransform
et otb::InverseRPCTransform
. These classes are used by the GenericRSTransform
class. See prTvTestCreateInverseForwardSensorModel
and prTvGenericRSTransform
tests.
GenericRSTransform
is used in higher level classes dedicated to geometries class, including ImageWarp filters, stereo filters, orthorectification filters, and vector data projection filters. See the tests from the modules OTBTransform
, OTBProjection
, OTBStereo
, OTBDisparityMap
, OTBAppProjection
, AppVectorDataTranslation
and OTBDEM
.
As of 2020-12-02, while most tests produce the same results, differences are still observed in some cases. See !744 (merged). The regressions could be caused by the different implementations of RPC models in Ossim and GDAL. If it appears to be the case we will need to update the test baselines. In this case it would interesting to validate the refactored RPC models using an external geometry library (which one ?)
RPC validation manual tests:
-
Verify that the transform
geo->sensor->geo
(forward model followed by inverse model), is close to the identity transform, and that the error is negligible compared to the sensor precision -
Produce GCPs between an ortho-image and the corresponding sensor Image using the Sift algorithm (
HomologousPointsExtraction
application) and use the pair of points to validate the model. For each pair compute the distance in pixel between the Forward transform of the ortho point and the sensor point, and the distance in meters between the Inverse transform of the sensor point and the ortho point. The potential biases of this test are:
- the accuracy of SIFT points
- the accuracy of the model used to compute the orthoimage
- the DEM used to compute the orthoimage
Some tests have been disabled. These tests are specific to SAR functionalities that cannot work while the SAR part of OTB has not been refactored.
Code quality :
We should ensure some quality metrics before the release alpha:
- Fix compilation warnings on all platforms
- Fix Bugs and CodeSmells from SonarQube in new code
- Ensure a good code coverage on new code (which value should we aim at ?)
Manual testing :
In addition to CI tests. It could be interesting to make additionnal validation tests manually. While OTB tests cover a large parts of use cases, these tests would have several benefits:
- Human based testing might highlight problems that CI have missed (It was the case or this bug)
- CI tests are done on small images. Several test are done on full product (Large inputs tests). However these tests never produce full output images, because it would take too much testing time and a lot of disk space. This means that OTB scaling is not tested.
- We could measure performances differences, in particular in RPC based tasks.
The test could be done using the OpticalCalibration
and Orthorectifcation
applications. The products from the OTB-LargeInput
repository could be used. Alternatively, we could use other products (some sample imagery is often available on providers websites).
Example of test procedure:
- Run
OpticalCalibration
on the product - Visual inspection of the result in a GIS software
- check the metadata of the output product with
gdalinfo -mdd all
(projection, no data if relevant, metadata written in the OTB domain) - Run the application with OTB 7.2.0 (before the refactoring) and verify that there is no regression using the
CompareImages
application.
The following sensors should be tested:
- Ikonos
- Spot 5
- Spot 6
- Spot 7
- Pleiades
- Worldview 2
- QuickBird 2
- Formosat