What changes will be made and why they would make a better Orfeo ToolBox?
Refactor the tests in order to have a faster validation procedure. Ideally, this refactoring should also simplify the testing strategy, and result in a better balance between over-tested and under-tested classes.
High level description
3 steps are proposed for this refactoring:
- Refactor the longest tests. 41 tests have been identified as they take more than 10s to run.
- Design a test framework without IO, only in-memory images
Tvto use the new test framework
Prior to these steps, 2 other taks should simplify this work:
- #1765 : the modules to be moved out of OTB should reduce the number of tests
- !299 (merged) : refactor all functor based filters, their test will be much simpler
Risks and benefits
The main risk is on the refactoring time, that could be significant as there are about 2300 tests.
On the bright side, it will be possible to greatly reduce the total testing time, as well as the need for large test data repository.
Alternatives for implementations
The list of 41 pre-selected tests to refactor is here (with associated timings in seconds):
10.1 ioTvDEMToImageGeneratorFromImageTest_SensorModel 10.38 bfTuMeanShiftSmoothingImageFilterROIQBMul4 10.49 coTvCoordinateToNameMultithreadTest 10.93 obTuMeanShiftStreamingConnectedComponentSegmentationOBIAToVectorDataFilter 11.1 ioTvKmzProductWriterWithGCP 11.27 apTvSeLargeScaleMeanShiftTest 11.6 leTvSVMMachineLearningModelReg 13.18 bfTvMeanShiftSmoothingImageFilterThreadingNonOpt 13.28 apTvSeSegmentationWatershedVector 14.37 dmTvDisparityMapEstimationMethod 14.72 leTeSEMModelEstimatorExampleTest 14.92 apTvSeSegmentationCCVector_ULOVW 15.2 leTeSOMClassifierExampleTest 15.29 apTvSeSegmentationMeanshiftVector 15.84 obTuMeanShiftConnectedComponentSegmentationFilter 16.47 feTvLocalHoughDraw2 16.63 apTvRaOpticalCalibration_UnknownSensor 16.79 apTvSeSegmentationCCVector_ULU 18.25 apTvClKMeansImageClassification_composite 20.58 apTvSeSegmentationCCVector 22.13 raTvReflectanceToSurfaceReflectanceImageFilter2 23.11 bfTuMeanShiftSmoothingImageFilterQBRoad 23.75 apTvCdbDSFuzzyModelEstimation_LI 26.45 apTvCdbDSFuzzyModelEstimation_LI_autoInit 27.77 apTvClSampleAugmentationReplicate 28.12 apTvFeLineSegmentDetectionNoRescale 29.79 apTvClSampleAugmentationSmote 29.96 leTvGradientBoostedTreeMachineLearningModel 31.1 maTeMarkovClassification3ExampleTest 32.73 leTeSVMImageEstimatorClassificationMultiExampleTest 33.31 apTvClSampleAugmentationJitter 34.57 hyTvMDMDNMFImageFilterTest2 36.32 apTvFeLineSegmentDetection 38.8 leTeSOMExampleTest 51.22 ioTeTileMapImageIOExampleTest 52.66 reTeImageRegistration2ExampleTest 60.84 ioTvSHPVectorDataFileReader3 60.98 ioTvTileMapImageIOWeb 88.13 dmTvFineRegistrationImageFilterTestWithMeanSquare 112.69 owTvQtWidgetShow 135.51 apTvClMethodGBTImageClassifierQB1
In module OTBTestKernel:
a set of functions to produce hardcoded images:
- template functions working on both otb::Image and otb::VectorImage
- start with image allocation (buffered region equals to largest possible region)
- fills the image with a formula, an array, ...
a set of functions to set hardcoded image metadata
- fills any image metadata (projection ref, origin, spacing, OSSIM keywordlist, ...)
a set of functions to produce hardcoded vector data:
- two families of functions : for otb::VectorData or otb::OGRDataSource
- fills the dataset with hardcoded geometries and fields
- hardcoded metadata (projection ref, ...)
new generic functions to check the result of a boolean condition (similar to an
assertstatement). There are blocking and non-blocking flavours (a blocking check will stop the test execution if it fails).
Tv tests can be re-written to use the in-memory test data. They are composed
of the following steps:
- Instanciate the filter to be tested.
- Instanciate a given fake image
- Set the fake image as input of the filter
- Set the filter parameters
- Call Update() on the filter
- Check the differences with the baseline (if any)
- Check other outputs of the filter (if any)
- If all checks passed: returns EXIT_SUCCESS
- If one check failed:
- write PNG file for the difference image (if any)
- return EXIT_FAILURE
With this strategy, the comparison of test and baseline images in CDash is preserved. The baselines don't have to be images on disk, they can also be hardcoded in the test (suited for tiny images, like 10 x 10 pixels).
Who will be developing the proposed changes?
@salbert & OTB Dev Team