Skip to content
Snippets Groups Projects

Fix issue on reduced OTB-data

Merged Stéphane Albert requested to merge 1834-test-data-switch into 1834-test-data-lfs

Summary

This MR is related to #1834 (closed).

After having imported test-data into the OTB repository under git lfs, this MR:

  1. makes OTB switch to the $/otb/Data/ directory (instead of the $/otb-data repository); and
  2. fixes broken tests due, mainly to the renaming of .hdr and .hd files into .raw.hdr and .raw.hd (see !387 (merged)).

Rationale

This MR is a second step in migrating the otb-data repository into the otb one under git lfs. This is needed in order to fix broken tests.

Implementation Details

Updated CMake build configuration: if the OTB_DATA_ROOT environment variable is undefined, it is initialized, by default, to $/otb/Data. So, the user can still benefit from the flexibility to link to another data repository (this is especially usefull in order to manage the dashboard nightly builds of develop, release, feature-branches builds and testing.

Fixed renamed filenames in CMake test configuration.

Copyright

The copyright owner is CNES and has signed the ORFEO ToolBox Contributor License Agreement.


Check before merging:

  • All discussions are resolved
  • At least 2 :thumbsup: votes from core developers, no :thumbsdown: vote.
  • The feature branch is (reasonably) up-to-date with the base branch
  • Dashboard is green
  • Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
Edited by Antoine Regimbeau

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • added feature label

  • Hi @salbert, I still think it would be a better approach to proceed in incremental step as discussed in !387 (comment 76412). Is there some reason this approach is not posssible?

  • Hello @poughov,

    Sure, it would have been a better approach but this implies that de work would be redone again. I have scripted some parts but there are some manual editing and fixing between each step. So, as discussed with @aregimbeau, we decided to move test-data from $/otb/Data to $/OTB-Data in order to ease dashboard submission (see step #4 (closed) of !387 (comment 76413) ; Steps #1 (closed), #2 (closed) and #4 (closed) have been done). Also steps #1 (closed), #2 (closed) and #4 (closed) have been done and management of the nightly build of feature-branches as a compromise of !387 (comment 76412).

    Note : Compression is not so much risky regarding testing because it is managed by GDAL itself and compression has been done with gdal_translate (even if risk 0% is not reachable).

    Edited by Stéphane Albert
  • added 1 commit

    • 10b46cf4 - ENH: 1834: Fixed broken tests due to filerename .hdr into .raw.hdr.

    Compare with previous version

  • Hello,

    I merged develop into this branch following sync on !383 (merged) and !384 (merged) in !387 (merged). So the migration of test-data should be sync in OTB-Data with code in OTB.

    This feature-branch is tested and sync with develop/master. So, last step is to fix remaining broken tests.

  • Antoine Regimbeau changed the description

    changed the description

  • Antoine Regimbeau added 99 commits

    added 99 commits

    • adbf159e - BUG: 1818: Fixed otb::TestHelper::RegressionTestOgrFile().
    • 7b89f7ee - BUG: 1818: Commented-out all failing tests due to bugfix.
    • 4c100b7c - COMP: Fixed test compile-time errors.
    • 32489d34 - Revert "COMP: Fixed test compile-time errors."
    • f04b17ab - Revert "BUG: 1818: Commented-out all failing tests due to bugfix."
    • 4fc21158 - REFAC: 1778: Reduced input-data of hyTvMDMDNMFImageFilterTest2.
    • d9e8e336 - REFAC: 1778: Reduced input-data of MeanShiftSmoothingImageFilterQBRoad.
    • bae78ab0 - BUG: nonsense to check if data files are the same
    • 19d66737 - BUG: fix empty output projRef
    • 77d82487 - TEST: when possible avoid using LargeInputs, change some output extensions
    • bae2e6d7 - 1778: !366 (merged): Moved image-file from incorrect Files directory into Images directory.
    • c502b6c5 - 1778: !366 (merged): Moved baseline image into correct dir (hyTvISRAUnmixingImageFilterTest).
    • fdfdc2ef - 1778: !366 (merged): Moved baseline image into correct dir (hyTvUnConstrainedLeastSquareImageFilterTest).
    • 8da80f36 - ENH: use tolerance on layer extent check
    • 4bc7f24b - MRG: Synchronize with branch develop
    • 195418a2 - TEST: replace LARGEINPUT when not needed
    • 1652d807 - TEST: avoid using baselines as inputs
    • 054a517f - TEST: fix epsilon issues, de-coupling some tests
    • e1be19e3 - TEST: allow some tolerance in compare-ogr
    • 3ff86ddf - TEST: avoid relative path for test outputs
    • a4821239 - TEST: fix TvOGRExtendedFilename
    • d71222a2 - Add solar variability input in the Optical Calibration module
    • 8bfc4c7c - Add tests on optical calibration with solar variability in input
    • 69627268 - 1808: Add a test on the solar distance input of optical calibration application
    • d3aa6594 - TEST: add the FastICA test to Ctest
    • 24927a96 - ENH: remove variance normalization before whitening
    • 47d69915 - BUG: use the PCA output instead of the input image at the first iteration of the FastICA algorithm
    • d7aede13 - BUG: modified the transformation matrix update step in accordance with the formula of the paper
    • 393797b6 - ENH: FastICAInternalOptimizerVectorImageFilter is now a persistent filter,...
    • 50a80547 - ENH: move the processing done in before/afterThreadedGenerateData() in Reset()/Synthetize()
    • a1408c25 - MRG: Synchronize with branch develop
    • 0be94a37 - TEST: allow tolerance for compare-ogr tests
    • 3f0e8f50 - BUG: mismatch between numbers of StartTransaction and CommitTransaction
    • c5d8e692 - BUG: do not call GetLeoggerLevel in constructor
    • bcd81b27 - ENH: add user logger level setting
    • f22028c7 - ENH: allow setting on windows with ""
    • 1e319e65 - ENH: the contrast function and its derivative are now lambdas
    • 6aa70e14 - BUG: swap line/column during the update of W
    • a5602f58 - DOC: added reference to the algorithm
    • d684608d - ENH: removed commented code of the VD algorithm, which is now in EndmemberNumberEstimation
    • 6e1ac948 - DOC: replace contrast function by nonlinearity (in ICA literature the...
    • 18e1bc5a - ENH: added nonlinearity choice for ica in DimensionalityReduction
    • d11bd963 - ENH: the dimensioality reduction is now done in the pca step
    • c2fbb028 - TEST: use only 3 components in the fastICA test
    • 876c3e5e - Merge branch 'develop' into fix_fastica
    • ca86f5c2 - COMP: the functional header was not included on Windows
    • 9124d43f - MRG: Synchronize with branch develop
    • 9c49c64c - MRG: Synchronize with branch develop
    • 77f66407 - TEST: larger false alarm rate because image is smaller
    • be3e473a - REFAC: use amst2.png for otb-data cleanup
    • 471ad135 - REFAC: Examples inputs now all come from otb-data/Input
    • 85adeebc - REFAC: Examples/DEM_srtm moved to Input/
    • 0a2b3125 - Merge branch 'fix_fastica' into 'develop'
    • 61fe5b1d - REFAC: remove example from OTB_DATA_PATHS
    • cc6dac85 - MRG: Synchronize with develop
    • 79b0498d - Merge branch '1818-TestHelper-regression-test-ogr-file' into 'develop'
    • ea56ec26 - ENH: strip env var for windows and fix bug of recursive call in constructor
    • 759fcf3b - Merge branch '1808_solar_variability_as_input' into 'develop'
    • 8a24a576 - REFAC: remove EXAMPLEDATA variable
    • 01436afe - REFAC: remove JHMIChDet.cxx
    • 62170473 - REFAC: remove LHMIChDet.cxx and KullbackLeiblerSupervizedDistanceChDet.cxx
    • caf6f506 - REFAC: remove GeometriesFilter.cxx and GeometriesChangeSpatialReference.cxx
    • 94b57cae - REFAC: remove HDFReaderExample.cxx and ImageToKmzAndMapFileProductExample.cxx
    • 133e7d74 - REFAC remove CoordinateToNameExample.cxx
    • 72dbfe20 - REFAC: remove TreeContainer.cxx
    • e4d22c45 - REFAC: remove Image6.cxx
    • cbc09f48 - REFAC: move Examples/DataRepresentation/Image up
    • ae7d379f - COMP: WARNING, this commit introduce a compilation on purposegit add .!
    • 8d918e22 - Revert "COMP: WARNING, this commit introduce a compilation on purposegit add .!"
    • 2b203c56 - COMP: WARNING, this commit introduce a compilation on purpose
    • baa0d2d7 - ENH: try sending an error
    • 3755a517 - Fix typo
    • 11b70f29 - ENH: change to test to detect error
    • ffb1d5ee - ENH: allow build to fail and to submit
    • 42ddb26e - Merge branch '1778-hyTvMDMDNMFImageFilterTest2' into 'develop'
    • ff7dffc2 - Merge branch '1778-MeanShiftSmoothingImageFilterQBRoad' into 'develop'
    • f7c21902 - REFAC: Update otb data paths to Input
    • 43af2b90 - 1808: Fix error in solar distance test
    • e062106e - Merge branch '1808_fix_solar_distance_test' into 'develop'
    • 1161d6cb - CI: use ninja
    • 72ab2865 - ENH: catch otbImageFileReaderException in otbTestMain
    • 76dced8f - REFAC: catch by const ref in otbTestMain.h
    • 7a8866cb - Merge branch 'logger_bug' into 'develop'
    • eeaa1265 - Merge branch 'otbTestMain-catch-ImageFileReaderException' into 'develop'
    • bca41753 - Merge branch 'data_move_input' into 'develop'
    • 0c18bc65 - Merge branch 'develop' into remove_stale_examples
    • 2ca03c1e - Merge branch 'ci_ninja' into 'develop'
    • 38c1f53f - COMP: allow Qt5 to compile on old kernels
    • 05da6b1e - Revert "COMP: WARNING, this commit introduce a compilation on purpose"
    • 9fbf0ecb - Merge branch 'CI_failstatus' into 'develop'
    • ffcfca59 - Merge branch 'remove_stale_examples' into 'develop'
    • 83519b4f - PKG: avoid dependency to libjson-c on Unix
    • 186d3f92 - MRG: Synchronize with branch develop
    • c6d7205d - CI: allow per site configuration
    • a441f67e - SuperBuild: use internal libjson-c from GDAL
    • fba7da66 - Merge branch 'ci_multi_config' into 'develop'
    • 336b3534 - SuperBuild: USE_LOW_KERNEL_VERSION=ON by default, also used on macOS
    • 8a7a9329 - Merge branch 'sb_low_kernel_version' into 'develop'
    • fed69aa2 - Merge branch 'develop' into 1834-test-data-switch

    Compare with previous version

  • Antoine Regimbeau changed title from WiP: Resolve "Extract & reduce test-data from the otb-data repository" to WiP: Fix issue on reduced OTB-data

    changed title from WiP: Resolve "Extract & reduce test-data from the otb-data repository" to WiP: Fix issue on reduced OTB-data

  • added 1 commit

    • 1586b73c - Rename input file for test (.raw.hdr)

    Compare with previous version

  • Victor Poughon mentioned in merge request !395 (merged)

    mentioned in merge request !395 (merged)

  • added 3 commits

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • added 7 commits

    • 342cbffd...491efeb3 - 5 commits from branch 1834-test-data-lfs
    • 50653d0d - Merge branch '1834-test-data-lfs' into 1834-test-data-switch
    • 7410cc46 - Missing .raw in dmTvAdhesionCorrectionFilter test

    Compare with previous version

  • Antoine Regimbeau unmarked as a Work In Progress

    unmarked as a Work In Progress

  • everything seems to be green on the dashboard, this MR is going to be merged in the branch 1834-test-data-lfs.
    If you have anything to review on the CMake part please do.

  • Antoine Regimbeau resolved all discussions

    resolved all discussions

  • added 1 commit

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading