Fix issue on reduced OTB-data
Summary
This MR is related to #1834 (closed).
After having imported test-data into the OTB repository under git lfs
, this MR:
- makes OTB switch to the
$/otb/Data/
directory (instead of the$/otb-data
repository); and - fixes broken tests due, mainly to the renaming of
.hdr
and.hd
files into.raw.hdr
and.raw.hd
(see !387 (merged)).
Rationale
This MR is a second step in migrating the otb-data
repository into the otb
one under git lfs
. This is needed in order to fix broken tests.
Implementation Details
Updated CMake
build configuration: if the OTB_DATA_ROOT
environment variable is undefined, it is initialized, by default, to $/otb/Data
. So, the user can still benefit from the flexibility to link to another data repository (this is especially usefull in order to manage the dashboard nightly builds of develop
, release
, feature-branches
builds and testing.
Fixed renamed filenames in CMake
test configuration.
Copyright
The copyright owner is CNES and has signed the ORFEO ToolBox Contributor License Agreement.
Check before merging:
- All discussions are resolved
- At least 2
votes from core developers, no vote. - The feature branch is (reasonably) up-to-date with the base branch
- Dashboard is green
- Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
Merge request reports
Activity
added feature label
Hi @salbert, I still think it would be a better approach to proceed in incremental step as discussed in !387 (comment 76412). Is there some reason this approach is not posssible?
Hello @poughov,
Sure, it would have been a better approach but this implies that de work would be redone again. I have scripted some parts but there are some manual editing and fixing between each step. So, as discussed with @aregimbeau, we decided to move test-data from
$/otb/Data
to$/OTB-Data
in order to ease dashboard submission (see step #4 (closed) of !387 (comment 76413) ; Steps #1 (closed), #2 (closed) and #4 (closed) have been done). Also steps #1 (closed), #2 (closed) and #4 (closed) have been done and management of the nightly build of feature-branches as a compromise of !387 (comment 76412).Note : Compression is not so much risky regarding testing because it is managed by GDAL itself and compression has been done with gdal_translate (even if risk 0% is not reachable).
Edited by Stéphane Albertadded 1 commit
- 10b46cf4 - ENH: 1834: Fixed broken tests due to filerename .hdr into .raw.hdr.
Hello,
I merged develop into this branch following sync on !383 (merged) and !384 (merged) in !387 (merged). So the migration of test-data should be sync in OTB-Data with code in OTB.
This feature-branch is tested and sync with develop/master. So, last step is to fix remaining broken tests.
added 99 commits
- adbf159e - BUG: 1818: Fixed otb::TestHelper::RegressionTestOgrFile().
- 7b89f7ee - BUG: 1818: Commented-out all failing tests due to bugfix.
- 4c100b7c - COMP: Fixed test compile-time errors.
- 32489d34 - Revert "COMP: Fixed test compile-time errors."
- f04b17ab - Revert "BUG: 1818: Commented-out all failing tests due to bugfix."
- 4fc21158 - REFAC: 1778: Reduced input-data of hyTvMDMDNMFImageFilterTest2.
- d9e8e336 - REFAC: 1778: Reduced input-data of MeanShiftSmoothingImageFilterQBRoad.
- bae78ab0 - BUG: nonsense to check if data files are the same
- 19d66737 - BUG: fix empty output projRef
- 77d82487 - TEST: when possible avoid using LargeInputs, change some output extensions
- bae2e6d7 - 1778: !366 (merged): Moved image-file from incorrect Files directory into Images directory.
- c502b6c5 - 1778: !366 (merged): Moved baseline image into correct dir (hyTvISRAUnmixingImageFilterTest).
- fdfdc2ef - 1778: !366 (merged): Moved baseline image into correct dir (hyTvUnConstrainedLeastSquareImageFilterTest).
- 8da80f36 - ENH: use tolerance on layer extent check
- 4bc7f24b - MRG: Synchronize with branch develop
- 195418a2 - TEST: replace LARGEINPUT when not needed
- 1652d807 - TEST: avoid using baselines as inputs
- 054a517f - TEST: fix epsilon issues, de-coupling some tests
- e1be19e3 - TEST: allow some tolerance in compare-ogr
- 3ff86ddf - TEST: avoid relative path for test outputs
- a4821239 - TEST: fix TvOGRExtendedFilename
- d71222a2 - Add solar variability input in the Optical Calibration module
- 8bfc4c7c - Add tests on optical calibration with solar variability in input
- 69627268 - 1808: Add a test on the solar distance input of optical calibration application
- d3aa6594 - TEST: add the FastICA test to Ctest
- 24927a96 - ENH: remove variance normalization before whitening
- 47d69915 - BUG: use the PCA output instead of the input image at the first iteration of the FastICA algorithm
- d7aede13 - BUG: modified the transformation matrix update step in accordance with the formula of the paper
- 393797b6 - ENH: FastICAInternalOptimizerVectorImageFilter is now a persistent filter,...
- 50a80547 - ENH: move the processing done in before/afterThreadedGenerateData() in Reset()/Synthetize()
- a1408c25 - MRG: Synchronize with branch develop
- 0be94a37 - TEST: allow tolerance for compare-ogr tests
- 3f0e8f50 - BUG: mismatch between numbers of StartTransaction and CommitTransaction
- c5d8e692 - BUG: do not call GetLeoggerLevel in constructor
- bcd81b27 - ENH: add user logger level setting
- f22028c7 - ENH: allow setting on windows with ""
- 1e319e65 - ENH: the contrast function and its derivative are now lambdas
- 6aa70e14 - BUG: swap line/column during the update of W
- a5602f58 - DOC: added reference to the algorithm
- d684608d - ENH: removed commented code of the VD algorithm, which is now in EndmemberNumberEstimation
- 6e1ac948 - DOC: replace contrast function by nonlinearity (in ICA literature the...
- 18e1bc5a - ENH: added nonlinearity choice for ica in DimensionalityReduction
- d11bd963 - ENH: the dimensioality reduction is now done in the pca step
- c2fbb028 - TEST: use only 3 components in the fastICA test
- 876c3e5e - Merge branch 'develop' into fix_fastica
- ca86f5c2 - COMP: the functional header was not included on Windows
- 9124d43f - MRG: Synchronize with branch develop
- 9c49c64c - MRG: Synchronize with branch develop
- 77f66407 - TEST: larger false alarm rate because image is smaller
- be3e473a - REFAC: use amst2.png for otb-data cleanup
- 471ad135 - REFAC: Examples inputs now all come from otb-data/Input
- 85adeebc - REFAC: Examples/DEM_srtm moved to Input/
- 0a2b3125 - Merge branch 'fix_fastica' into 'develop'
- 61fe5b1d - REFAC: remove example from OTB_DATA_PATHS
- cc6dac85 - MRG: Synchronize with develop
- 79b0498d - Merge branch '1818-TestHelper-regression-test-ogr-file' into 'develop'
- ea56ec26 - ENH: strip env var for windows and fix bug of recursive call in constructor
- 759fcf3b - Merge branch '1808_solar_variability_as_input' into 'develop'
- 8a24a576 - REFAC: remove EXAMPLEDATA variable
- 01436afe - REFAC: remove JHMIChDet.cxx
- 62170473 - REFAC: remove LHMIChDet.cxx and KullbackLeiblerSupervizedDistanceChDet.cxx
- caf6f506 - REFAC: remove GeometriesFilter.cxx and GeometriesChangeSpatialReference.cxx
- 94b57cae - REFAC: remove HDFReaderExample.cxx and ImageToKmzAndMapFileProductExample.cxx
- 133e7d74 - REFAC remove CoordinateToNameExample.cxx
- 72dbfe20 - REFAC: remove TreeContainer.cxx
- e4d22c45 - REFAC: remove Image6.cxx
- cbc09f48 - REFAC: move Examples/DataRepresentation/Image up
- ae7d379f - COMP: WARNING, this commit introduce a compilation on purposegit add .!
- 8d918e22 - Revert "COMP: WARNING, this commit introduce a compilation on purposegit add .!"
- 2b203c56 - COMP: WARNING, this commit introduce a compilation on purpose
- baa0d2d7 - ENH: try sending an error
- 3755a517 - Fix typo
- 11b70f29 - ENH: change to test to detect error
- ffb1d5ee - ENH: allow build to fail and to submit
- 42ddb26e - Merge branch '1778-hyTvMDMDNMFImageFilterTest2' into 'develop'
- ff7dffc2 - Merge branch '1778-MeanShiftSmoothingImageFilterQBRoad' into 'develop'
- f7c21902 - REFAC: Update otb data paths to Input
- 43af2b90 - 1808: Fix error in solar distance test
- e062106e - Merge branch '1808_fix_solar_distance_test' into 'develop'
- 1161d6cb - CI: use ninja
- 72ab2865 - ENH: catch otbImageFileReaderException in otbTestMain
- 76dced8f - REFAC: catch by const ref in otbTestMain.h
- 7a8866cb - Merge branch 'logger_bug' into 'develop'
- eeaa1265 - Merge branch 'otbTestMain-catch-ImageFileReaderException' into 'develop'
- bca41753 - Merge branch 'data_move_input' into 'develop'
- 0c18bc65 - Merge branch 'develop' into remove_stale_examples
- 2ca03c1e - Merge branch 'ci_ninja' into 'develop'
- 38c1f53f - COMP: allow Qt5 to compile on old kernels
- 05da6b1e - Revert "COMP: WARNING, this commit introduce a compilation on purpose"
- 9fbf0ecb - Merge branch 'CI_failstatus' into 'develop'
- ffcfca59 - Merge branch 'remove_stale_examples' into 'develop'
- 83519b4f - PKG: avoid dependency to libjson-c on Unix
- 186d3f92 - MRG: Synchronize with branch develop
- c6d7205d - CI: allow per site configuration
- a441f67e - SuperBuild: use internal libjson-c from GDAL
- fba7da66 - Merge branch 'ci_multi_config' into 'develop'
- 336b3534 - SuperBuild: USE_LOW_KERNEL_VERSION=ON by default, also used on macOS
- 8a7a9329 - Merge branch 'sb_low_kernel_version' into 'develop'
- fed69aa2 - Merge branch 'develop' into 1834-test-data-switch
Toggle commit listmentioned in merge request !395 (merged)
added 3 commits
-
1586b73c...28d146bd - 2 commits from branch
1834-test-data-lfs
- a785f1f2 - Merge branch 'develop' into 1834-test-data-switch
-
1586b73c...28d146bd - 2 commits from branch
added 7 commits
-
342cbffd...491efeb3 - 5 commits from branch
1834-test-data-lfs
- 50653d0d - Merge branch '1834-test-data-lfs' into 1834-test-data-switch
- 7410cc46 - Missing .raw in dmTvAdhesionCorrectionFilter test
-
342cbffd...491efeb3 - 5 commits from branch
- Resolved by Antoine Regimbeau