S1Tiling issueshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues2024-03-22T10:08:02Zhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/154Review documentation2024-03-22T10:08:02ZLuc HermitteReview documentation1.1Thierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/152Support OTB 92024-03-28T15:28:00ZLuc HermitteSupport OTB 9- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [x] Check the baseline is compatible wi...- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [x] Check the baseline is compatible with future (#150) OTB 8 compatible baseline1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/151Use precise orbit files for establishing grid of ECEF sensor position matchin...2024-03-19T15:23:22ZLuc HermitteUse precise orbit files for establishing grid of ECEF sensor position matching S2 grid (LIA)Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products...Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products
- We have no documentation that tells how wide is the Orbit State Vector (OSV) series found in S1 products
- Downloading one S1 product just for its embedded orbit information is a bit overkill.
Instead we should use precise orbit files.
Notes: (non-issues)
- As we typically expect the LIA maps to be computed beforehand, we don't need to wait for 3 days before obtaining orbit data for a given pair of _S2 tile_ + _relative-orbit-id_. We can just use any orbit data published long before.
- OTB 9 (/SARDEMProjection2's) `SarSensorModel` can compute sensor positions from the precise orbit data -- we don't need extra data like GCPs, slantRangeTime...
The resolution of this issue can be decomposed as such:
- [ ] Fetch Precise Orbit Files from an/the official provider
- https://github.com/scottstanie/sentineleof may help
- [ ] Change download routine to support this new source of data
- [ ] Create a new OTB application
- that reuses SARDEMProjection2 code related to `-withxyz`, `-withsatpos` and that supports DEM in any spatial reference
- that decodes OSV data from EOF files to construct `FastSarSensorModel` accordingly
- [ ] Adapt the LIA `ComputeGroundAndSatPositionsOnDEM` step to use the new OTB application1.2https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/131Implement SLC support2023-08-28T07:58:50ZLuc HermitteImplement SLC supportAt the moment, only GRD products are supported. SLC products should also be supported.At the moment, only GRD products are supported. SLC products should also be supported.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/130Mirror github repository2023-05-23T16:46:56ZRémi CressonMirror github repositoryGithub repo is not in sync with this repo, which could be confusing for the users.Github repo is not in sync with this repo, which could be confusing for the users.Thierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/128Incorrect analysis of S2 results regarding with S1 product need to be downloaded2023-04-05T10:04:55ZLuc HermitteIncorrect analysis of S2 results regarding with S1 product need to be downloadedThe current implementation `does_final_product_need_to_be_generated_for()` to tell whether a S1 product need to be downloaded is incorrect.
- if the S2 product is found and the S2 filtered product isn't found on disk, then the S1 produc...The current implementation `does_final_product_need_to_be_generated_for()` to tell whether a S1 product need to be downloaded is incorrect.
- if the S2 product is found and the S2 filtered product isn't found on disk, then the S1 product won't be marked for download.
=> This is not an issue as the filtered product can be generated from the other one
- if the S2 product is not found while the S2 filtered product is found on disk, then the S1 product won't be marked for download.
=> This is an issue if the S2 product is required (`keep_non_filtered_products : True`): we won't have the required S1 data to generate the S2 product
Note: this scenario is unlikely, but not impossiblehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/123Copy Geoid file with DEM files2023-01-30T14:19:06ZThierry KoleckCopy Geoid file with DEM filesWhen the option `cache_srtm_by:` is set to `copy`, the DEM files are copied to the tmp directory. This issue proposes to copy also the geoid file in the tmp directoryWhen the option `cache_srtm_by:` is set to `copy`, the DEM files are copied to the tmp directory. This issue proposes to copy also the geoid file in the tmp directory1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/122Error KilledWorker2023-12-14T07:09:06ZfloveilError KilledWorkerHey,
When I launch S1 Tiling, I have an error on `distributed.scheduler.KilledWorker` :
`distributed.scheduler.KilledWorker: Attempted to run task /home/florent/s1chain_data/data/tmp/S2/36LVQ/s1a_36LVQ_vh_DES_050_20220804t033102.tif o...Hey,
When I launch S1 Tiling, I have an error on `distributed.scheduler.KilledWorker` :
`distributed.scheduler.KilledWorker: Attempted to run task /home/florent/s1chain_data/data/tmp/S2/36LVQ/s1a_36LVQ_vh_DES_050_20220804t033102.tif on 3 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://127.0.0.1:34493. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.`
I also add [S1Processor-main.debug.log](/uploads/fcbda81572b1eaaa5c1b9c6658e843a0/S1Processor-main.debug.log)
If you have an idea of the source of the problem.
Thankshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/120Support option "keep_non_filtered_product" in cfg file2022-11-10T16:33:41ZThierry KoleckSupport option "keep_non_filtered_product" in cfg filehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/118Chain concatenation and spatial despeckling in-memory, when possible.2022-11-10T16:33:41ZLuc HermitteChain concatenation and spatial despeckling in-memory, when possible.Concatenation is a non trival step that:
- recognizes 2 compatibles inputs and changes the output filename in consequence
- rename orthorectification output when there is only one input.
To be chained in-memory SpatialDespeckle would ne...Concatenation is a non trival step that:
- recognizes 2 compatibles inputs and changes the output filename in consequence
- rename orthorectification output when there is only one input.
To be chained in-memory SpatialDespeckle would need to:
- recognize 2 compatibles inputs and change the output filename in consequence
- start from the renamed file (instead of expecting to be chained in-memory) when there is only one input.
Relates to #116 where in-memory chaining of concat+despeckle isn't supported yet.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/113Incorrect pre-download filter for LIAMap2022-10-25T14:26:25ZLuc HermitteIncorrect pre-download filter for LIAMapWhen filtering files to download from `LIAMap` application, files to downloads aren't filtered correctly.
We should:
- Select the pair with the best cover in the required time-range
- Don't look at `fname_fmt.concatenation`, but at `f...When filtering files to download from `LIAMap` application, files to downloads aren't filtered correctly.
We should:
- Select the pair with the best cover in the required time-range
- Don't look at `fname_fmt.concatenation`, but at `fname_fmt.lia_product`
- and then download the selected pair if need be.
The issue has been observed with the following scenario, while the last product (S1A_IW_GRDH_1SDV_20201221T044221_20201221T044246_035779_043013_4BE6) is not yet on disk
```
[DataSource]
download : True
first_date : 2020-12-01
last_date : 2020-12-30
[Processing]
calibration: sigma
tiles : 33NVB
fname_fmt.concatenation : {flying_unit_code}_{tile_name}_{polarisation}_{orbit_direction}_{orbit}_{acquisition_stamp}_{calibration_type}.tif
```
See the [log file](/uploads/58708065b12c9eb7c01b78b6ed05ffae/S1Processor-main.debug.log).https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/106Improve handling of S1Tiling version2022-01-28T16:15:38ZLuc HermitteImprove handling of S1Tiling version- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is tagged- [ ] Version number should be written at the top of log files
- We may have to use version number from `__meta__.py` and/or current commit hash
- [ ] It should be possible to deduce it from git tags when the current commit is taggedhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/102Apply Precise Orbit files before processing2022-09-28T12:59:59ZThierry KoleckApply Precise Orbit files before processingAfter downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel...After downloading the GRD data, we can download the Precise Orbit files to update the GDR products.
[https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files](https://sentinel.esa.int/web/sentinel/-/access-to-sentinel-1-processor-and-orbit-auxiliary-files)
- [ ] @koleckt Evaluate the effect of applying orbit file on orthorectifyed products
- [ ] @lhermitte Implement the downloading and process to apply orbit fileThierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/97Help page deploying job may fail due to "artifacts for pages are too large"2021-10-13T12:03:05ZLuc HermitteHelp page deploying job may fail due to "artifacts for pages are too large"Sometimes page deploying job will fail because of the weight of built help pages.
A workaround is to remove documentation of older versions with `mc rm -r --force minio-otb/s1-tiling/www/0.2.0rc42/` for instance. But this is only a work...Sometimes page deploying job will fail because of the weight of built help pages.
A workaround is to remove documentation of older versions with `mc rm -r --force minio-otb/s1-tiling/www/0.2.0rc42/` for instance. But this is only a workaround, and the problem needs to be fixed.
If we take a look closer at the `_static` directories, we see they occupy 12Mb each. It's not even from the few S1 and S2 images used in the documentation but mostly from `fonts/` and `css/` data.
We need to find a way to factorize there two information (or may be `_static/` directory in order to reduce the weight of _page_ artefacts.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/90Integration Gamma Naught - RTC2021-07-20T14:52:46Zfabien contivalIntegration Gamma Naught - RTCIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcIl conviendrait de pouvoir intégrer le module Gamma Naught RTC.
Il s'agit de l'insérer juste avant l'étape de rectification S2 puis de pouvoir l'activer/la désactiver en config.
https://gitlab.orfeo-toolbox.org/s1-tiling/gamma0-rtcfabien contivalfabien contivalhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/76Provide output in dB values2021-05-25T13:53:32ZMickael SavinaudProvide output in dB valuesdB values option can be nice to avoid conversion after S1 tilingdB values option can be nice to avoid conversion after S1 tilinghttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/74Detect and report saturated storage (for tmp/data_out/data_in)2021-05-04T12:36:04ZLuc HermitteDetect and report saturated storage (for tmp/data_out/data_in)In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py...In order to avoid saturating disk space, S1Tiling should detect disk saturation and report it appropriately. Program exit code have been reserved in !33. Use `exits.OUTPUT_DISK_FULL` and `exits.TMP_DISK_FULL` from `s1tiling.libs.exits.py`.
Several approaches are possible
- define an allocated space in MB/GB in the configuration file, and stop processing+report when the quota is reached
- pros: simple, portable
- cons:
- the quota is per execution on S1Tiling. If two instances are running on the same disks no load balancing is possible
- this approach cannot know is the configured quota is compatible with the available diskspace allocated implicitly (e.g. we don't know how much space other processes on the same machine need) or explicitly on the storage (e.g. scratch has a limited capacity on HAL, per user!).
- => We need to check whether the allocated quota makes sense with the situation detected when S1Tiling process is started
- define a percentile on available disk-space/quota allocated to user
- pros: load balancing becomes implicit
- cons:
- having a portable way to detect how much one can use may be complex
- on shared machines (qsubed nodes) where no quota is allocated to a job, we don't know how much can be used
- let S1Tiling saturate, but intercept the error, and report the issue with the proper error code.
- cons:
- we may not be able to have a complete log file
- we may saturate a disk space shared with other people and project (qsubed nodes)
- pros: fail-safe mechanism that can be implemented on top of any of the two previous approaches.https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/66Write output with UInt16 encoding2024-03-25T10:51:43ZMickael SavinaudWrite output with UInt16 encodingCurrently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.Currently the output is written as float. To save disk space, it could be interesting to save the output as Uint16 with a specific formula to retrieve the real value.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/65Support zip file in input directory2021-03-12T10:17:29ZMickael SavinaudSupport zip file in input directoryWhen I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would b...When I put a zip file in the input directory, the product is not detected. Even I unzip the file, the product is not detected. I need to create a directory with the product name without SAFE suffix to be able to run s1tiling.
I would be nice to define that in the documentation and to add the support of a directory which contain zip file.1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/64Support srtm files from usgs/EarthExplorer2024-02-05T14:56:54ZMickael SavinaudSupport srtm files from usgs/EarthExplorerCurrently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name ...Currently the srtm provided to S1Tiling must be provided according a shp file. The default one is only for srtm in hgt format.
Personally when I want to retrieve a srtm data, I go to EarthExplorer and I retrieve the tif files. The name is also quite similar: for example `n41_w096_1arc_v3.tif`. For dt2 format it is the same: `n41_w096_1arc_v3.dt2`.
The grid is the same only the naming convention is different.
It could be nice to support this file without a new shp file. If I know what I want to process, I will provide the right elevation data.Thierry KoleckThierry Koleck