S1Tiling issueshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues2020-09-22T15:56:13Zhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/13Add unitary test on python code2020-09-22T15:56:13ZMickael SavinaudAdd unitary test on python code* Add unitary test on the python code : free function, class, ...
* Define and use a framework : unittest, pytest, ...
* In order to avoid too long test, don't run OTB app just verify the input parameters.
Additional few end to end test...* Add unitary test on the python code : free function, class, ...
* Define and use a framework : unittest, pytest, ...
* In order to avoid too long test, don't run OTB app just verify the input parameters.
Additional few end to end tests can be used to test the full chain and perform acceptance tests during release of a new version.
This end to end test can be used also in user oriented documentation tutorial to illustrate the functionalities of the chains.
Add coverage and pylint configuration. These outputs can be integrated in a CI which use OTB sonarcube instance.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/14Add gamma terrain flattening correction2021-02-19T13:39:13ZMickael SavinaudAdd gamma terrain flattening correctionAdd terrain flattening correction according to [Small, 2011](https://doi.org/10.1109/TGRS.2011.2120616)
This correction will develop as an OTB app which will be push to OTB.
Validation will be perform against SNAP implementationAdd terrain flattening correction according to [Small, 2011](https://doi.org/10.1109/TGRS.2011.2120616)
This correction will develop as an OTB app which will be push to OTB.
Validation will be perform against SNAP implementationLuc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/15Add the possibility to reproject into a new tile scheme with a specific srs2021-02-19T13:38:58ZMickael SavinaudAdd the possibility to reproject into a new tile scheme with a specific srsCurrently the output tiling is based on the S2 one (MGRS one) but several other one exist as for example : EEA one and landsat ARD one.Currently the output tiling is based on the S2 one (MGRS one) but several other one exist as for example : EEA one and landsat ARD one.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/16Move the format of configuration file to json2021-02-19T13:38:38ZMickael SavinaudMove the format of configuration file to jsonhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/19Add Jupiter Notebook2021-02-19T13:38:04ZMickael SavinaudAdd Jupiter NotebookAdd Jupiter Notebook to demonstrate the usage of S1 TilingAdd Jupiter Notebook to demonstrate the usage of S1 Tilinghttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/32Register S1 image downloading as Dask tasks2023-08-24T15:07:54ZLuc HermitteRegister S1 image downloading as Dask tasksCurrently (v0.2), `S1Processor.py` loops on all S2 tiles requested. For each S2 tiles, after a dependency analysis (#22), all intersecting S1 images are downloaded, then the processing is started.
Instead, it could be interesting to reg...Currently (v0.2), `S1Processor.py` loops on all S2 tiles requested. For each S2 tiles, after a dependency analysis (#22), all intersecting S1 images are downloaded, then the processing is started.
Instead, it could be interesting to register the downloading of S1 images as dask tasks that calibration+cutting depends upon. This would give more flexibility for S2 tiles for which of subset of intersecting S1 images are already downloaded: this would reduce the overall latency during the execution: we could download **and** process at the same time, and so on.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/53Register temporary files (like orthoready) for automatic removal2021-01-19T15:41:59ZLuc HermitteRegister temporary files (like orthoready) for automatic removalIt's important to clean-up temporary files as soon as they are no longer required in order to avoid disk saturation.
Workarounds like #52 are possible, but at the cost of disabling the factorisation of some processing.
We could remove ...It's important to clean-up temporary files as soon as they are no longer required in order to avoid disk saturation.
Workarounds like #52 are possible, but at the cost of disabling the factorisation of some processing.
We could remove the oldest ortho-ready products, but without any guarantee they won't be useful later. This could be possible that with requests on wide time ranges, the number of products for a same S1 tile is superior to the arbitrary threshold used to clean-up old files.
The only solution that would scale is to change the way tasks dependency tree is built. It needs to encompass all S2 tiles (instead of a single one at the time). This could be done after #32 (_Register image downloading as Dask tasks_). This way, once each S2 images that depend on an ortho-ready S1 product have been produced, the temporary product could be safely removed.
Ideas:
- the requested products should be sorted by time stamp
- removal of temporary files could be a new task, automatically generated through graph analysis (non-requested products used as inputs)https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/54Encapsulate OTB process execution into subprocessed2024-03-19T15:31:06ZLuc HermitteEncapsulate OTB process execution into subprocessedIn order to reduce memory leak risks to dask leaks, and alleviate chances for #49 to occur, the calls to OTB applications should be done in subprocesses.
No data, like `np.array`, are exchanged with OTB application through their Python ...In order to reduce memory leak risks to dask leaks, and alleviate chances for #49 to occur, the calls to OTB applications should be done in subprocesses.
No data, like `np.array`, are exchanged with OTB application through their Python bindings, but only filenames. This means there should be no problem to hide calls to OTB python bindings in subprocesses.1.2https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/55Tests data retrieval from datalake through eodag2022-10-04T14:43:00ZLuc HermitteTests data retrieval from datalake through eodagAlso check whether data is retrieved as symbolic link or zipped archive.Also check whether data is retrieved as symbolic link or zipped archive.Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/59Clean automatically older SAFE data in raw directory2024-03-21T17:57:53ZThierry KoleckClean automatically older SAFE data in raw directoryTo save disk space, one can keep only a given number of file (SAFE) in the raw directory.
Before downloading, the code should remove the older SAFE file in order to keep the same number of file in this directory. This number can be fixe...To save disk space, one can keep only a given number of file (SAFE) in the raw directory.
Before downloading, the code should remove the older SAFE file in order to keep the same number of file in this directory. This number can be fixed by the user in the config file.1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/61Speckle filtering processing2022-10-25T15:42:40ZThierry KoleckSpeckle filtering processingThe former version of S1Tiling included a multitemporal speckle filtering processing.
Once the ortho time-serie completed on a S2 tile, the multitemp processing consists in applying a speckle filter to generate a new time serie with a re...The former version of S1Tiling included a multitemporal speckle filtering processing.
Once the ortho time-serie completed on a S2 tile, the multitemp processing consists in applying a speckle filter to generate a new time serie with a reduced speckle.
Two OTB applications were developped OTBMultitempFilterOutcore and OTBMultitempFilterFiltering.
This issue consists in adding a optional processing, on each tile with these 2 applications.
One can also propose to implement spatial speckle filtering with OTBDespeckle application.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/64Support srtm files from usgs/EarthExplorer2024-02-05T14:56:54ZMickael SavinaudSupport srtm files from usgs/EarthExplorerCurrently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name ...Currently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name is also quite similar: for example `n41_w096_1arc_v3.tif`. For dt2 format it is the same: `n41_w096_1arc_v3.dt2`.
The grid is the same only the naming convention is different.
It could be nice to support this file without a new shp file. If I know what I want to process, I will provide the right elevation data.Thierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/65Support zip file in input directory2021-03-12T10:17:29ZMickael SavinaudSupport zip file in input directoryWhen I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would b...When I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would be nice to define that in the documentation and to add the support of a directory which contain zip file.1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/66Write output with UInt16 encoding2024-03-25T10:51:43ZMickael SavinaudWrite output with UInt16 encodingCurrently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.Currently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/74Detect and report saturated storage (for tmp/data_out/data_in)2021-05-04T12:36:04ZLuc HermitteDetect and report saturated storage (for tmp/data_out/data_in)In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py...In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py`.
Several approaches are possible
- define an allocated space in MB/GB in the configuration file, and stop processing+report when the quota is reached
- pros: simple, portable
- cons:
- the quota is per execution on S1Tiling. If two instances are running on the same disks no load balancing is possible
- this approach cannot know is the configured quota is compatible with the available diskspace allocated implicitly (e.g. we don't know how much space other processes on the same machine need) or explicitly on the storage (e.g. scratch has a limited capacity on HAL, per user!).
- => We need to check whether the allocated quota makes sense with the situation detected when S1Tiling process is started
- define a percentile on available disk-space/quota allocated to user
- pros: load balancing becomes implicit
- cons:
- having a portable way to detect how much one can use may be complex
- on shared machines (qsubed nodes) where no quota is allocated to a job, we don't know how much can be used
- let S1Tiling saturate, but intercept the error, and report the issue with the proper error code.
- cons:
- we may not be able to have a complete log file
- we may saturate a disk space shared with other people and project (qsubed nodes)
- pros: fail-safe mechanism that can be implemented on top of any of the two previous approaches.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/76Provide output in dB values2021-05-25T13:53:32ZMickael SavinaudProvide output in dB valuesdB values option can be nice to avoid conversion after S1 tilingdB values option can be nice to avoid conversion after S1 tilinghttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/90Integration Gamma Naught - RTC2021-07-20T14:52:46Zfabien contivalIntegration Gamma Naught - RTCIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcfabien contivalfabien contivalhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/97Help page deploying job may fail due to "artifacts for pages are too large"2021-10-13T12:03:05ZLuc HermitteHelp page deploying job may fail due to "artifacts for pages are too large"Sometimes page deploying job will fail because of the weight of built help pages.
A workaround is to remove documentation of older versions with `mc rm -r --force minio-otb/s1-tiling/www/0.2.0rc42/` for instance. But this is only a work...Sometimes page deploying job will fail because of the weight of built help pages.
A workaround is to remove documentation of older versions with `mc rm -r --force minio-otb/s1-tiling/www/0.2.0rc42/` for instance. But this is only a workaround, and the problem needs to be fixed.
If we take a look closer at the `_static` directories, we see they occupy 12Mb each. It's not even from the few S1 and S2 images used in the documentation but mostly from `fonts/` and `css/` data.
We need to find a way to factorize there two information (or may be `_static/` directory in order to reduce the weight of _page_ artefacts.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/102Apply Precise Orbit files before processing2022-09-28T12:59:59ZThierry KoleckApply Precise Orbit files before processingAfter downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel...After downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files)
- [ ] @koleckt Evaluate the effect of applying orbit file on orthorectifyed products
- [ ] @lhermitte Implement the downloading and process to apply orbit fileThierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/106Improve handling of S1Tiling version2022-01-28T16:15:38ZLuc HermitteImprove handling of S1Tiling version- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is tagged- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is tagged