S1Tiling issueshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues2024-03-22T10:08:02Zhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/154Review documentation2024-03-22T10:08:02ZLuc HermitteReview documentation1.1Thierry KoleckThierry Koleckhttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/153Bad error message when "tiles_list_in_file" parameter is wrong2024-03-21T18:01:57ZThierry KoleckBad error message when "tiles_list_in_file" parameter is wrongIn the config file, there is a key in [Processing] group that allow to give a file for tiles list.
This parameter "tile_list_in_file" must have a filename containing all the tiles to be processed.
If this file doesn't exist, S1Tiling r...In the config file, there is a key in [Processing] group that allow to give a file for tiles list.
This parameter "tile_list_in_file" must have a filename containing all the tiles to be processed.
If this file doesn't exist, S1Tiling returns the following error:
```
CRITICAL - No option 'tiles' in section: 'Processing'
Please fix the configuration file '/home/il/koleckt/tropisco/preprocess/GA/AS.cfg'.
```
It should be
```
CRITICAL - File not found for 'tiles_list_in_file' in section: 'Processing'
Please fix the configuration file '/home/il/koleckt/tropisco/preprocess/GA/AS.cfg'.
```1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/152Support OTB 92024-03-28T15:28:00ZLuc HermitteSupport OTB 9- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [x] Check the baseline is compatible wi...- [ ] Make sure S1Tiling is compatible with OTB 9
- [ ] Update the relevant documentation (e.g. document which modules are required)
- [ ] Update the installation scripts
- [ ] Update the dockers
- [x] Check the baseline is compatible with future (#150) OTB 8 compatible baseline1.1https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/151Use precise orbit files for establishing grid of ECEF sensor position matchin...2024-03-19T15:23:22ZLuc HermitteUse precise orbit files for establishing grid of ECEF sensor position matching S2 grid (LIA)Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products...Knowing which input S1 files to download in order to generate grid of XYZ sensor positions (aligned on S2 grid) can be a bit tricky:
- S1 product footprints seldom cover S2 tiles -- hence S1Tiling, that _concats_ data from 2 S1 products
- We have no documentation that tells how wide is the Orbit State Vector (OSV) series found in S1 products
- Downloading one S1 product just for its embedded orbit information is a bit overkill.
Instead we should use precise orbit files.
Notes: (non-issues)
- As we typically expect the LIA maps to be computed beforehand, we don't need to wait for 3 days before obtaining orbit data for a given pair of _S2 tile_ + _relative-orbit-id_. We can just use any orbit data published long before.
- OTB 9 (/SARDEMProjection2's) `SarSensorModel` can compute sensor positions from the precise orbit data -- we don't need extra data like GCPs, slantRangeTime...
The resolution of this issue can be decomposed as such:
- [ ] Fetch Precise Orbit Files from an/the official provider
- https://github.com/scottstanie/sentineleof may help
- [ ] Change download routine to support this new source of data
- [ ] Create a new OTB application
- that reuses SARDEMProjection2 code related to `-withxyz`, `-withsatpos` and that supports DEM in any spatial reference
- that decodes OSV data from EOF files to construct `FastSarSensorModel` accordingly
- [ ] Adapt the LIA `ComputeGroundAndSatPositionsOnDEM` step to use the new OTB application1.2https://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/150Incorporate otb 8 compatible images in test baseline2024-03-28T16:05:44ZLuc HermitteIncorporate otb 8 compatible images in test baselineThe current test baseline has images generated with SARCalibration and OrthoRectofication applications from OTB 7.x
It should be updated to:
- either incorporate OTB 8 based reference images as well -- this will take some space on the ...The current test baseline has images generated with SARCalibration and OrthoRectofication applications from OTB 7.x
It should be updated to:
- either incorporate OTB 8 based reference images as well -- this will take some space on the S3
- or supersede OTB 7 images with OTB 8 ones -- this won't permit to continue testing S1TIling with OTB 7.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/149Integrate new LIA workflow directly done in S2 geometry2024-03-21T14:51:43ZLuc HermitteIntegrate new LIA workflow directly done in S2 geometryGenerating XYZ coordinate positions in S1 geometry (through SARDEMProjection + SARCartesianMeanEstimation) doesn't result in normals good enough. Artefacts are observed in rugged areas. See https://gitlab.orfeo-toolbox.org/s1-tiling/norm...Generating XYZ coordinate positions in S1 geometry (through SARDEMProjection + SARCartesianMeanEstimation) doesn't result in normals good enough. Artefacts are observed in rugged areas. See https://gitlab.orfeo-toolbox.org/s1-tiling/normlim_sigma0/-/issues/3
Another approach seems to yield much better results:
1. gdalwrap dem + geiod on S2 geometry
2. BandMath sum them
3. Generate XYZ ground positions + satellite position on the S2 geometry with new version of SARDEMProjection (https://gitlab.orfeo-toolbox.org/s1-tiling/normlim_sigma0/-/issues/9)
4. Compute LIA + sin(LIA) from these informations directly in S2 geometry1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/148Allow to choose the folder name for filtered products2024-03-28T18:41:35ZThierry KoleckAllow to choose the folder name for filtered productsIn the config file, we can choose the filename for filtered products with the key `fname_fmt.filtered`
However, the folder is still static, `{dataout}/filtered/{tilename}` and the user cannot change it through the configuration file.
T...In the config file, we can choose the filename for filtered products with the key `fname_fmt.filtered`
However, the folder is still static, `{dataout}/filtered/{tilename}` and the user cannot change it through the configuration file.
This issue propose to improve the key `fname_fmt.filtered` to take into account the folder description.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/146Support for any DEM should support DEM in any SpatialReference2024-02-06T13:56:01ZLuc HermitteSupport for any DEM should support DEM in any SpatialReferenceCurrent implementation of !76 is restricted to WGS84 DEMs. Any CRS needs to be supported, like Lambert 93...Current implementation of !76 is restricted to WGS84 DEMs. Any CRS needs to be supported, like Lambert 93...1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/145Entrypoint not working with singularity2023-11-16T14:54:42ZCéline RailléEntrypoint not working with singularityIssue linked to the project [Theia s1tiling](https://gitlab.cnes.fr/Theia/s1tiling/-/tree/main).
On the cluster HPC, the workdir is overwritten using singularity, so the entrypoint returns an error because the virtual environment cannot...Issue linked to the project [Theia s1tiling](https://gitlab.cnes.fr/Theia/s1tiling/-/tree/main).
On the cluster HPC, the workdir is overwritten using singularity, so the entrypoint returns an error because the virtual environment cannot be activated. To avoid duplicating the entrypoint file between the two projects, would it be possible to use an absolute path for the virtual environment in the entrypoint instead of a relative path?1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/144Old worker logs are not cleaned2023-11-06T21:33:49ZLuc HermitteOld worker logs are not cleanedThis is regression from the changes on logs handling.This is regression from the changes on logs handling.1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/143Missing DEM are not reported when creating the overlapping DEM VRT2023-11-07T16:35:51ZLuc HermitteMissing DEM are not reported when creating the overlapping DEM VRTIf DEM are missing when creating the DEM VRT in AgglomerateDEM, an error shall be reported.If DEM are missing when creating the DEM VRT in AgglomerateDEM, an error shall be reported.1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/142Cannot load python library exceptions.py on branch 18-support-various-dem-as-...2023-11-08T11:54:48ZThierry KoleckCannot load python library exceptions.py on branch 18-support-various-dem-as-inputWhen executing the branch 18-support-various-dem-as-input, there is a error when `from s1tiling.libs import exceptions` in S1FileManager.py line 59, because exceptions.py doesn't exists in this branch.
```
(s1tiling-copdem) [koleckt@tre...When executing the branch 18-support-various-dem-as-input, there is a error when `from s1tiling.libs import exceptions` in S1FileManager.py line 59, because exceptions.py doesn't exists in this branch.
```
(s1tiling-copdem) [koleckt@trex017 Brazil]$ S1Processor /home/il/koleckt/tropisco/preprocess/Brazil/Brazil.cfg
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling-copdem/bin/S1Processor", line 33, in <module>
sys.exit(load_entry_point('S1Tiling', 'console_scripts', 'S1Processor')())
File "/work/scratch/env/koleckt/.conda/envs/s1tiling-copdem/bin/S1Processor", line 25, in importlib_load_entry_point
return next(matches).load()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling-copdem/lib/python3.8/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/work/scratch/env/koleckt/.conda/envs/s1tiling-copdem/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/il/koleckt/s1tiling-copdem/s1tiling/s1tiling/S1Processor.py", line 68, in <module>
from s1tiling.libs.S1FileManager import S1FileManager, WorkspaceKinds
File "/home/il/koleckt/s1tiling-copdem/s1tiling/s1tiling/libs/S1FileManager.py", line 59, in <module>
from s1tiling.libs import exceptions
ImportError: cannot import name 'exceptions' from 's1tiling.libs' (/home/il/koleckt/s1tiling-copdem/s1tiling/s1tiling/libs/__init__.py)
```1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/141Add (c) mentions in test scripts2023-11-07T17:55:54ZLuc HermitteAdd (c) mentions in test scripts1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/140EODAG doesn't generate exception when http request failed2023-12-13T15:51:10ZThierry KoleckEODAG doesn't generate exception when http request failedSometimes, EODAG crashes because of timeout to access to PEPS provider (see log below). And no error are reported to S1Tiling.
To solve this problem, two changes are required:
- add `raise_errors=True` parameter to the search functions...Sometimes, EODAG crashes because of timeout to access to PEPS provider (see log below). And no error are reported to S1Tiling.
To solve this problem, two changes are required:
- add `raise_errors=True` parameter to the search functions (see [EODAG documentation](https://eodag.readthedocs.io/en/stable/notebooks/api_user_guide/4_search.html)). This parameter allows generating exceptions by EODAG.
- manage the exception by retrying (infinitely ?) to access to PEPS
```
load conda/22.11.1 : OK
envs_dirs : /work/scratch/env/koleckt/.conda/envs
pkgs_dirs : /work/scratch/data/koleckt/.conda/pkgs
load python/3.8.4_raw : OK
load otb-depends/7.4.2-python3.8.4 : OK
load otb/7.4.2-python3.8.4 : OK
mkdir: cannot create directory ‘S1Tiling’: File exists
INFO - OTB version detected on the system is 7.4.2
WARNING - No time and orbit compatible products found on disk!
INFO - Requested tiles: ['47PNR']
INFO - The following tiles will be processed: ['47PNR']
INFO - SRTM ok
INFO - Check SRTM coverage for 47PNR
INFO - -> 47PNR coverage = 1.0 => OK
INFO - 0 images to process on ['47PNR'] tiles
INFO - Required SRTM tiles: ['N14E100', 'N14E099', 'N13E100', 'N13E099', 'N14E098', 'N13E098']
2023-10-30 15:13:59,254 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2023-10-30 15:13:59,808 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40667
2023-10-30 15:13:59,808 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:38435
2023-10-30 15:13:59,808 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40667
2023-10-30 15:13:59,808 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:38435
2023-10-30 15:13:59,808 - distributed.worker - INFO - Worker name: 0
2023-10-30 15:13:59,808 - distributed.worker - INFO - Worker name: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - dashboard at: 127.0.0.1:40543
2023-10-30 15:13:59,809 - distributed.worker - INFO - dashboard at: 127.0.0.1:39533
2023-10-30 15:13:59,809 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,809 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - Threads: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - Threads: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - Memory: 10.00 GiB
2023-10-30 15:13:59,809 - distributed.worker - INFO - Memory: 10.00 GiB
2023-10-30 15:13:59,809 - distributed.worker - INFO - Local Directory: /tmp/slurm-1880419/dask-scratch-space/worker-4aareetw
2023-10-30 15:13:59,809 - distributed.worker - INFO - Local Directory: /tmp/slurm-1880419/dask-scratch-space/worker-6q027am5
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,816 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,816 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,817 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,817 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,818 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40947
2023-10-30 15:13:59,818 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40947
2023-10-30 15:14:00,495 - distributed.worker - INFO - Starting Worker plugin _WorkerSetupPlugin-a75f7451-7733-4b31-93eb-3395836094c7
2023-10-30 15:14:00,495 - distributed.worker - INFO - Starting Worker plugin _WorkerSetupPlugin-a75f7451-7733-4b31-93eb-3395836094c7
INFO - Processing tile 47PNR (1/1)
INFO - Sending search request: https://peps.cnes.fr/resto/api/collections/S1/search.json?polarisation=VV VH&sensorMode=IW&startDate=2023-10-01&completionDate=2023-10-16&geometry=POLYGON ((98.9998 13.4780, 98.9998 14.4728, 100.0186 14.4728, 100.0186 13.4780, 98.9998 13.4780))&productType=GRD&maxRecords=20&page=1
ERROR - Skipping error while searching for peps QueryStringSearch instance:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 466, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 461, in _make_request
httplib_response = conn.getresponse()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='peps.cnes.fr', port=443): Read timed out. (read timeout=5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/eodag/plugins/search/qssearch.py", line 961, in _request
response = requests.get(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='peps.cnes.fr', port=443): Read timed out. (read timeout=5)
INFO - 0 remote S1 products returned in page 1: []
INFO - 0 remote S1 product(s) found and filtered (IW && VV VH): []
INFO - 0 remote S1 product(s) will be downloaded
INFO - Downloading images related to 47PNR took 7.200475322082639sec
INFO - Intersecting raster list w/ 47PNR took 1.5857519004493952sec
INFO - No intersection with tile 47PNR
INFO - Processing of tile 47PNR took 8.788740198127925sec
INFO - Execution report: no error detected
INFO - -> Nothing has been executed
INFO - Stopping worker at tcp://127.0.0.1:40667. Reason: nanny-close
INFO - Stopping worker at tcp://127.0.0.1:38435. Reason: nanny-close
2023-10-30 15:14:09,299 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:40667. Reason: nanny-close
2023-10-30 15:14:09,299 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:38435. Reason: nanny-close
[2023-10-30 15:14:09.737736626+00:00] /koleckt/1880419: Account (env) =
[2023-10-30 15:14:09.738837876+00:00] /koleckt/1880419: User (env) = koleckt
[2023-10-30 15:14:09.739657538+00:00] /koleckt/1880419: GroupId = biomass(5099)
[2023-10-30 15:14:09.740474140+00:00] /koleckt/1880419: UserId = koleckt(1687)
[2023-10-30 15:14:09.741516560+00:00] /koleckt/1880419: Account = sco
[2023-10-30 15:14:09.742531210+00:00] /koleckt/1880419: Account = sco
[2023-10-30 15:14:09.743865107+00:00] /koleckt/1880419: Partition = cpu2022
[2023-10-30 15:14:09.745402943+00:00] /koleckt/1880419: QOS = cpu_2022_1280
[2023-10-30 15:14:09.746615481+00:00] /koleckt/1880419: JobName = jobThailand.slurm
[2023-10-30 15:14:09.747606771+00:00] /koleckt/1880419: SubmitTime = 2023-10-30T15:13:38
[2023-10-30 15:14:09.748467723+00:00] /koleckt/1880419: RunTime = 00:00:30
[2023-10-30 15:14:09.749347165+00:00] /koleckt/1880419: TimeLimit = 01:00:00
[2023-10-30 15:14:09.750203856+00:00] /koleckt/1880419: EligibleTime = 2023-10-30T15:13:38
[2023-10-30 15:14:09.751024628+00:00] /koleckt/1880419: StartTime = 2023-10-30T15:13:39
[2023-10-30 15:14:09.751834341+00:00] /koleckt/1880419: EndTime = 2023-10-30T15:14:09
[2023-10-30 15:14:09.753293647+00:00] /koleckt/1880419: ExitCode = 0:0
[2023-10-30 15:14:09.754064039+00:00] /koleckt/1880419: NodeList = trex027
[2023-10-30 15:14:09.754861262+00:00] /koleckt/1880419: BatchHost = trex027
[2023-10-30 15:14:09.755633654+00:00] /koleckt/1880419: TRES =
[2023-10-30 15:14:09.756412867+00:00] /koleckt/1880419: Command = /home/il/koleckt/theia/jobThailand.slurm
[2023-10-30 15:14:09.757175429+00:00] /koleckt/1880419: WorkDir = /home/il/koleckt/theia
[2023-10-30 15:14:09.757922832+00:00] /koleckt/1880419: StdErr = /home/il/koleckt/theia/slurm-1880419.out
[2023-10-30 15:14:09.758671885+00:00] /koleckt/1880419: StdOut = /home/il/koleckt/theia/slurm-1880419.out
#########################################
# EVIDEN - CNES #
# Job Accounting #
#########################################
Cluster : trex
JobIDRaw : 1880419
JobID : 1880419
JobName : jobThailand.slurm
Account : sco
User : koleckt(1687), biomass(5099)
Partition : cpu2022
QOS : cpu_2022_1280
Nodelist : trex027 (1)
State : COMPLETED (Exitcode=0:0, DerivedExitCode=0:0, Restarts=0)
Submit date : 2023-10-30T15:13:38
Start time : 2023-10-30T15:13:39
End time : 2023-10-30T15:14:09
Elapsed time : 00:00:30 (Timelimit=01:00:00, Suspended=00:00:00)
TEI : 0:02:00
Command : /home/il/koleckt/theia/jobThailand.slurm
StdOut : /home/il/koleckt/theia/slurm-1880419.out
StdErr : /home/il/koleckt/theia/slurm-1880419.out
STEP(s) : 2
---------
StepID | JobName State Exit Start End Elapsed TotalCPU CPUTime ConsumedEnergy MaxDiskRead MaxDiskWrite MaxRSS MaxRSSTask MaxRSSNode NNodes UserCPU NodeList
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
batch | batch COMPLETED 0:0 2023-10-30T15:13:39 2023-10-30T15:14:09 00:00:30 00:14.532 00:02:00 0.08625K 277.99M 152.40M 263684K 0 trex027 1 00:11.960 trex027
extern | extern COMPLETED 0:0 2023-10-30T15:13:39 2023-10-30T15:14:09 00:00:30 00:00:00 00:02:00 0.08625K 0.00M 0 1672K 0 trex027 1 00:00:00 trex027
```1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/139Warn a "Failed to produce" even if the downloading is correct2023-12-13T15:51:09ZThierry KoleckWarn a "Failed to produce" even if the downloading is correctIn this log file, the product S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB is correctly downloaded, but the output is not generated (see WARNING message).
Notice that the downloading of this product has failed ear...In this log file, the product S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB is correctly downloaded, but the output is not generated (see WARNING message).
Notice that the downloading of this product has failed earlier in the processing (previous tiles) because of PEPS timeout (unknown reason).
However, for this tile, the download is OK but the product is not generated.
`
2023-10-24 09:39:47,589 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230203T112122_20230203T112147_047071_05A589_2EFE, provider=peps)...
2023-10-24 09:39:47,589 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230207T230109_20230207T230134_047136_05A7BF_07EE, provider=peps)...
2023-10-24 09:42:07,198 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230207T230109_20230207T230134_047136_05A7BF_07EE/S1A_IW_GRDH_1SDV_20230207T230109_20230207T230134_047136_05A7BF_07EE.SAFE correctly downloaded
2023-10-24 09:42:07,198 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 14.285714285714286%
2023-10-24 09:42:07,201 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230215T112121_20230215T112146_047246_05AB64_34F0, provider=peps)...
2023-10-24 09:42:20,288 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230203T112122_20230203T112147_047071_05A589_2EFE/S1A_IW_GRDH_1SDV_20230203T112122_20230203T112147_047071_05A589_2EFE.SAFE correctly downloaded
2023-10-24 09:42:20,288 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 28.571428571428573%
2023-10-24 09:42:20,291 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230219T230109_20230219T230134_047311_05ADA7_DF79, provider=peps)...
2023-10-24 09:43:43,001 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230215T112121_20230215T112146_047246_05AB64_34F0/S1A_IW_GRDH_1SDV_20230215T112121_20230215T112146_047246_05AB64_34F0.SAFE correctly downloaded
2023-10-24 09:43:43,002 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 42.857142857142854%
2023-10-24 09:43:43,005 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB, provider=peps)...
2023-10-24 09:43:53,076 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230219T230109_20230219T230134_047311_05ADA7_DF79/S1A_IW_GRDH_1SDV_20230219T230109_20230219T230134_047311_05ADA7_DF79.SAFE correctly downloaded
2023-10-24 09:43:53,076 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 57.142857142857146%
2023-10-24 09:43:53,079 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230227T112122_20230227T112147_047421_05B161_4294, provider=peps)...
2023-10-24 09:45:15,845 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB/S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB.SAFE correctly downloaded
2023-10-24 09:45:15,845 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 71.42857142857143%
2023-10-24 09:45:15,848 - root - INFO - Starting download of EOProduct(id=S1A_IW_GRDH_1SDV_20230303T230109_20230303T230134_047486_05B392_DE6E, provider=peps)...
2023-10-24 09:45:21,537 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230227T112122_20230227T112147_047421_05B161_4294/S1A_IW_GRDH_1SDV_20230227T112122_20230227T112147_047421_05B161_4294.SAFE correctly downloaded
2023-10-24 09:45:21,538 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 85.71428571428571%
2023-10-24 09:46:49,166 - s1tiling.filemanager - INFO - /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230303T230109_20230303T230134_047486_05B392_DE6E/S1A_IW_GRDH_1SDV_20230303T230109_20230303T230134_047486_05B392_DE6E.SAFE correctly downloaded
2023-10-24 09:46:49,167 - s1tiling.filemanager - INFO - --> Downloading products for 47PQS... 100.0%
2023-10-24 09:46:49,179 - s1tiling.filemanager - INFO - Remote S1 products saved into [<bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230207T230109_20230207T230134_047136_05A7BF_07EE/S1A_IW_GRDH_1SDV_20230207T230109_20230207T230134_047136_05A7BF_07EE.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230203T112122_20230203T112147_047071_05A589_2EFE/S1A_IW_GRDH_1SDV_20230203T112122_20230203T112147_047071_05A589_2EFE.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230215T112121_20230215T112146_047246_05AB64_34F0/S1A_IW_GRDH_1SDV_20230215T112121_20230215T112146_047246_05AB64_34F0.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230219T230109_20230219T230134_047311_05ADA7_DF79/S1A_IW_GRDH_1SDV_20230219T230109_20230219T230134_047311_05ADA7_DF79.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB/S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230227T112122_20230227T112147_047421_05B161_4294/S1A_IW_GRDH_1SDV_20230227T112122_20230227T112147_047421_05B161_4294.SAFE>, <bound method Outcome.value of Success: /tmp/slurm-1743092/s1tiling/raw/S1A_IW_GRDH_1SDV_20230303T230109_20230303T230134_047486_05B392_DE6E/S1A_IW_GRDH_1SDV_20230303T230109_20230303T230134_047486_05B392_DE6E.SAFE>]
2023-10-24 09:46:49,651 - root - INFO - Downloading images related to 47PQS took 423.96629132004455sec
2023-10-24 09:46:49,653 - s1tiling.filemanager - WARNING - Don't generate s1a_47PQS_*_ASC_099_20230227txxxxxx.tif, because [Failed to produce EOProduct(id=S1A_IW_GRDH_1SDV_20230227T112057_20230227T112122_047421_05B161_2AAB, provider=peps): None: None]
`1.0rc3Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/138Apply pylint2024-01-30T14:02:40ZLuc HermitteApply pylint1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/137DEM VRT should use local DEM copy or link2024-03-11T12:26:37ZThierry KoleckDEM VRT should use local DEM copy or linkWhen building the DEM VRT (for LIA/RTC processing), S1Tiling use the original DEM filenames.When building the DEM VRT (for LIA/RTC processing), S1Tiling use the original DEM filenames.1.1Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/136Warning/Error in Synthesize app with OTB82023-11-16T15:00:30ZThierry KoleckWarning/Error in Synthesize app with OTB8The Synthesize app generate Warning/Error in OTB8.1 because otbSARMetadata.cxx cannot find some metadata.
Anyway, the processing continue....
`DEBUG - Register app: Synthetize (from ['/work/scratch/data/koleckt/tmp/S2/47NMJ/s1a_47NMJ_v...The Synthesize app generate Warning/Error in OTB8.1 because otbSARMetadata.cxx cannot find some metadata.
Anyway, the processing continue....
`DEBUG - Register app: Synthetize (from ['/work/scratch/data/koleckt/tmp/S2/47NMJ/s1a_47NMJ_vh_ASC_070_20201107t113519_gamma.tif', '/work/scratch/data/koleckt/tmp/S2/47NMJ/s1a_47NMJ_vh_ASC_070_20201107t113544_gamma.tif']) -ram 4096 -il '/work/scratch/data/koleckt/tmp/S2/47NMJ/s1a_47NMJ_vh_ASC_070_20201107t113519_gamma.tif' '/work/scratch/data/koleckt/tmp/S2/47NMJ/s1a_47NMJ_vh_ASC_070_20201107t113544_gamma.tif' -out '/work/scratch/data/koleckt/outputS1/47NMJ/s1a_47NMJ_vh_ASC_070_20201107txxxxxx.tif'`
`INFO - 2023-09-01 09:37:55 (INFO) Synthetize: Default RAM limit for OTB is 256 MB`
`INFO - 2023-09-01 09:37:55 (INFO) Synthetize: GDAL maximum cache size is 51563 MB`
`INFO - 2023-09-01 09:37:55 (INFO) Synthetize: OTB will use at most 2 threads`
`WARNING - 2023-09-01 09:37:55 (WARNING): Input image has SAR sensor metadata, but OTB was not able to read it: /work/scratch/data/osmanji/otb-build-script-workdir/otb-8.1/otb/Modules/Core/Metadata/src/otbSARMetadata.cxx:43:`
`WARNING - otb::ERROR: Unable to find 'SAR.AzimuthTimeInterval' in the input keywordlist`
`WARNING - 2023-09-01 09:37:55 (WARNING): Input image has SAR calibration metadata, but OTB was not able to read it: /work/scratch/data/osmanji/otb-build-script-workdir/otb-8.1/otb/Modules/Core/Metadata/src/otbSARMetadata.cxx:43:`
`WARNING - otb::ERROR: Unable to find 'SARCalib.RadiometricCalibrationIncidenceAngle' in the input keywordlist`
`INFO - 2023-09-01 09:37:55 (INFO): Loading metadata from official product`
`INFO - 2023-09-01 09:37:55 (INFO) Synthetize: Image #1 has 1 components`
`WARNING - 2023-09-01 09:37:55 (WARNING): Input image has SAR sensor metadata, but OTB was not able to read it: /work/scratch/data/osmanji/otb-build-script-workdir/otb-8.1/otb/Modules/Core/Metadata/src/otbSARMetadata.cxx:43:`
`WARNING - otb::ERROR: Unable to find 'SAR.AzimuthTimeInterval' in the input keywordlist`
`WARNING - 2023-09-01 09:37:55 (WARNING): Input image has SAR calibration metadata, but OTB was not able to read it: /work/scratch/data/osmanji/otb-build-script-workdir/otb-8.1/otb/Modules/Core/Metadata/src/otbSARMetadata.cxx:43:`
`WARNING - otb::ERROR: Unable to find 'SARCalib.RadiometricCalibrationIncidenceAngle' in the input keywordlist`Luc HermitteLuc Hermittehttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/135Processing SLC data2023-08-28T07:07:34ZThierry KoleckProcessing SLC dataThis issue proposes a way to process Single Look Complex (SLC) products rather Ground Range Detected (GRD).
The objective of using SLC are to:
- improve the resolution of final product, despite a reduced Equivalent Number of Look (ENL)
...This issue proposes a way to process Single Look Complex (SLC) products rather Ground Range Detected (GRD).
The objective of using SLC are to:
- improve the resolution of final product, despite a reduced Equivalent Number of Look (ENL)
- provide VV*conj(VH) images as outputshttps://gitlab.orfeo-toolbox.org/s1-tiling/s1tiling/-/issues/134Incorrect pre-analysis when final products exist2023-11-16T15:01:03ZLuc HermitteIncorrect pre-analysis when final products existIn the case of a final product that depends on 2 S1 input products with one on disk, and the other one to be downloaded, the analysis of what needs to be done fails.
This very rare case may happen in testing scenarios. It's quite unlike...In the case of a final product that depends on 2 S1 input products with one on disk, and the other one to be downloaded, the analysis of what needs to be done fails.
This very rare case may happen in testing scenarios. It's quite unlikely to happen (only 1 of two input products on disk while the output product has already been generated).Luc HermitteLuc Hermitte