S1Tiling issueshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues2024-03-15T11:46:56Zhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/152Support OTB 92024-03-15T11:46:56ZLuc HermitteSupport OTB 9- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [ ] Check the baseline is compatible wi...- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [ ] Check the baseline is compatible with future (#150) OTB 8 compatible baseline1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/151Use precise orbit files for establishing grid of ECEF sensor position matchin...2024-03-19T15:23:22ZLuc HermitteUse precise orbit files for establishing grid of ECEF sensor position matching S2 grid (LIA)Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products...Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products
- We have no documentation that tells how wide is the Orbit State Vector (OSV) series found in S1 products
- Downloading one S1 product just for its embedded orbit information is a bit overkill.
Instead we should use precise orbit files.
Notes: (non-issues)
- As we typically expect the LIA maps to be computed beforehand, we don't need to wait for 3 days before obtaining orbit data for a given pair of _S2 tile_ + _relative-orbit-id_. We can just use any orbit data published long before.
- OTB 9 (/SARDEMProjection2's) `SarSensorModel` can compute sensor positions from the precise orbit data -- we don't need extra data like GCPs, slantRangeTime...
The resolution of this issue can be decomposed as such:
- [ ] Fetch Precise Orbit Files from an/the official provider
- https://github.com/scottstanie/sentineleof may help
- [ ] Change download routine to support this new source of data
- [ ] Create a new OTB application
- that reuses SARDEMProjection2 code related to `-withxyz`, `-withsatpos` and that supports DEM in any spatial reference
- that decodes OSV data from EOF files to construct `FastSarSensorModel` accordingly
- [ ] Adapt the LIA `ComputeGroundAndSatPositionsOnDEM` step to use the new OTB application1.2https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/131Implement SLC support2023-08-28T07:58:50ZLuc HermitteImplement SLC supportAt the moment, only GRD products are supported. SLC products should also be supported.At the moment, only GRD products are supported. SLC products should also be supported.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/123Copy Geoid file with DEM files2023-01-30T14:19:06ZThierry KoleckCopy Geoid file with DEM filesWhen the option `cache_srtm_by:` is set to `copy`, the DEM files are copied to the tmp directory. This issue proposes to copy also the geoid file in the tmp directoryWhen the option `cache_srtm_by:` is set to `copy`, the DEM files are copied to the tmp directory. This issue proposes to copy also the geoid file in the tmp directory1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/118Chain concatenation and spatial despeckling in-memory, when possible.2022-11-10T16:33:41ZLuc HermitteChain concatenation and spatial despeckling in-memory, when possible.Concatenation is a non trival step that:
- recognizes 2 compatibles inputs and changes the output filename in consequence
- rename orthorectification output when there is only one input.
To be chained in-memory SpatialDespeckle would ne...Concatenation is a non trival step that:
- recognizes 2 compatibles inputs and changes the output filename in consequence
- rename orthorectification output when there is only one input.
To be chained in-memory SpatialDespeckle would need to:
- recognize 2 compatibles inputs and change the output filename in consequence
- start from the renamed file (instead of expecting to be chained in-memory) when there is only one input.
Relates to #116 where in-memory chaining of concat+despeckle isn't supported yet.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/106Improve handling of S1Tiling version2022-01-28T16:15:38ZLuc HermitteImprove handling of S1Tiling version- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is tagged- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is taggedhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/102Apply Precise Orbit files before processing2022-09-28T12:59:59ZThierry KoleckApply Precise Orbit files before processingAfter downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel...After downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files)
- [ ] @koleckt Evaluate the effect of applying orbit file on orthorectifyed products
- [ ] @lhermitte Implement the downloading and process to apply orbit fileThierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/90Integration Gamma Naught - RTC2021-07-20T14:52:46Zfabien contivalIntegration Gamma Naught - RTCIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcfabien contivalfabien contivalhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/74Detect and report saturated storage (for tmp/data_out/data_in)2021-05-04T12:36:04ZLuc HermitteDetect and report saturated storage (for tmp/data_out/data_in)In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py...In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py`.
Several approaches are possible
- define an allocated space in MB/GB in the configuration file, and stop processing+report when the quota is reached
- pros: simple, portable
- cons:
- the quota is per execution on S1Tiling. If two instances are running on the same disks no load balancing is possible
- this approach cannot know is the configured quota is compatible with the available diskspace allocated implicitly (e.g. we don't know how much space other processes on the same machine need) or explicitly on the storage (e.g. scratch has a limited capacity on HAL, per user!).
- => We need to check whether the allocated quota makes sense with the situation detected when S1Tiling process is started
- define a percentile on available disk-space/quota allocated to user
- pros: load balancing becomes implicit
- cons:
- having a portable way to detect how much one can use may be complex
- on shared machines (qsubed nodes) where no quota is allocated to a job, we don't know how much can be used
- let S1Tiling saturate, but intercept the error, and report the issue with the proper error code.
- cons:
- we may not be able to have a complete log file
- we may saturate a disk space shared with other people and project (qsubed nodes)
- pros: fail-safe mechanism that can be implemented on top of any of the two previous approaches.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/66Write output with UInt16 encoding2024-03-25T10:51:43ZMickael SavinaudWrite output with UInt16 encodingCurrently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.Currently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/65Support zip file in input directory2021-03-12T10:17:29ZMickael SavinaudSupport zip file in input directoryWhen I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would b...When I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would be nice to define that in the documentation and to add the support of a directory which contain zip file.1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/64Support srtm files from usgs/EarthExplorer2024-02-05T14:56:54ZMickael SavinaudSupport srtm files from usgs/EarthExplorerCurrently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name ...Currently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name is also quite similar: for example `n41_w096_1arc_v3.tif`. For dt2 format it is the same: `n41_w096_1arc_v3.dt2`.
The grid is the same only the naming convention is different.
It could be nice to support this file without a new shp file. If I know what I want to process, I will provide the right elevation data.Thierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/61Speckle filtering processing2022-10-25T15:42:40ZThierry KoleckSpeckle filtering processingThe former version of S1Tiling included a multitemporal speckle filtering processing.
Once the ortho time-serie completed on a S2 tile, the multitemp processing consists in applying a speckle filter to generate a new time serie with a re...The former version of S1Tiling included a multitemporal speckle filtering processing.
Once the ortho time-serie completed on a S2 tile, the multitemp processing consists in applying a speckle filter to generate a new time serie with a reduced speckle.
Two OTB applications were developped OTBMultitempFilterOutcore and OTBMultitempFilterFiltering.
This issue consists in adding a optional processing, on each tile with these 2 applications.
One can also propose to implement spatial speckle filtering with OTBDespeckle application.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/59Clean automatically older SAFE data in raw directory2024-03-21T17:57:53ZThierry KoleckClean automatically older SAFE data in raw directoryTo save disk space, one can keep only a given number of file (SAFE) in the raw directory.
Before downloading, the code should remove the older SAFE file in order to keep the same number of file in this directory. This number can be fixe...To save disk space, one can keep only a given number of file (SAFE) in the raw directory.
Before downloading, the code should remove the older SAFE file in order to keep the same number of file in this directory. This number can be fixed by the user in the config file.1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/54Encapsulate OTB process execution into subprocessed2024-03-19T15:31:06ZLuc HermitteEncapsulate OTB process execution into subprocessedIn order to reduce memory leak risks to dask leaks, and alleviate chances for #49 to occur, the calls to OTB applications should be done in subprocesses.
No data, like `np.array`, are exchanged with OTB application through their Python ...In order to reduce memory leak risks to dask leaks, and alleviate chances for #49 to occur, the calls to OTB applications should be done in subprocesses.
No data, like `np.array`, are exchanged with OTB application through their Python bindings, but only filenames. This means there should be no problem to hide calls to OTB python bindings in subprocesses.1.2https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/53Register temporary files (like orthoready) for automatic removal2021-01-19T15:41:59ZLuc HermitteRegister temporary files (like orthoready) for automatic removalIt's important to clean-up temporary files as soon as they are no longer required in order to avoid disk saturation.
Workarounds like #52 are possible, but at the cost of disabling the factorisation of some processing.
We could remove ...It's important to clean-up temporary files as soon as they are no longer required in order to avoid disk saturation.
Workarounds like #52 are possible, but at the cost of disabling the factorisation of some processing.
We could remove the oldest ortho-ready products, but without any guarantee they won't be useful later. This could be possible that with requests on wide time ranges, the number of products for a same S1 tile is superior to the arbitrary threshold used to clean-up old files.
The only solution that would scale is to change the way tasks dependency tree is built. It needs to encompass all S2 tiles (instead of a single one at the time). This could be done after #32 (_Register image downloading as Dask tasks_). This way, once each S2 images that depend on an ortho-ready S1 product have been produced, the temporary product could be safely removed.
Ideas:
- the requested products should be sorted by time stamp
- removal of temporary files could be a new task, automatically generated through graph analysis (non-requested products used as inputs)https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/32Register S1 image downloading as Dask tasks2023-08-24T15:07:54ZLuc HermitteRegister S1 image downloading as Dask tasksCurrently (v0.2), `S1Processor.py` loops on all S2 tiles requested. For each S2 tiles, after a dependency analysis (#22), all intersecting S1 images are downloaded, then the processing is started.
Instead, it could be interesting to reg...Currently (v0.2), `S1Processor.py` loops on all S2 tiles requested. For each S2 tiles, after a dependency analysis (#22), all intersecting S1 images are downloaded, then the processing is started.
Instead, it could be interesting to register the downloading of S1 images as dask tasks that calibration+cutting depends upon. This would give more flexibility for S2 tiles for which of subset of intersecting S1 images are already downloaded: this would reduce the overall latency during the execution: we could download **and** process at the same time, and so on.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/19Add Jupiter Notebook2021-02-19T13:38:04ZMickael SavinaudAdd Jupiter NotebookAdd Jupiter Notebook to demonstrate the usage of S1 TilingAdd Jupiter Notebook to demonstrate the usage of S1 Tilinghttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/16Move the format of configuration file to json2021-02-19T13:38:38ZMickael SavinaudMove the format of configuration file to jsonhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/15Add the possibility to reproject into a new tile scheme with a specific srs2021-02-19T13:38:58ZMickael SavinaudAdd the possibility to reproject into a new tile scheme with a specific srsCurrently the output tiling is based on the S2 one (MGRS one) but several other one exist as for example : EEA one and landsat ARD one.Currently the output tiling is based on the S2 one (MGRS one) but several other one exist as for example : EEA one and landsat ARD one.