EODAG doesn't generate exception when http request failed
Sometimes, EODAG crashes because of timeout to access to PEPS provider (see log below). And no error are reported to S1Tiling.
To solve this problem, two changes are required:
- add
raise_errors=Trueparameter to the search functions (see EODAG documentation). This parameter allows generating exceptions by EODAG. - manage the exception by retrying (infinitely ?) to access to PEPS
load conda/22.11.1 : OK
envs_dirs : /work/scratch/env/koleckt/.conda/envs
pkgs_dirs : /work/scratch/data/koleckt/.conda/pkgs
load python/3.8.4_raw : OK
load otb-depends/7.4.2-python3.8.4 : OK
load otb/7.4.2-python3.8.4 : OK
mkdir: cannot create directory ‘S1Tiling’: File exists
INFO - OTB version detected on the system is 7.4.2
WARNING - No time and orbit compatible products found on disk!
INFO - Requested tiles: ['47PNR']
INFO - The following tiles will be processed: ['47PNR']
INFO - SRTM ok
INFO - Check SRTM coverage for 47PNR
INFO - -> 47PNR coverage = 1.0 => OK
INFO - 0 images to process on ['47PNR'] tiles
INFO - Required SRTM tiles: ['N14E100', 'N14E099', 'N13E100', 'N13E099', 'N14E098', 'N13E098']
2023-10-30 15:13:59,254 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy
2023-10-30 15:13:59,808 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:40667
2023-10-30 15:13:59,808 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:38435
2023-10-30 15:13:59,808 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:40667
2023-10-30 15:13:59,808 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:38435
2023-10-30 15:13:59,808 - distributed.worker - INFO - Worker name: 0
2023-10-30 15:13:59,808 - distributed.worker - INFO - Worker name: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - dashboard at: 127.0.0.1:40543
2023-10-30 15:13:59,809 - distributed.worker - INFO - dashboard at: 127.0.0.1:39533
2023-10-30 15:13:59,809 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,809 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - Threads: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - Threads: 1
2023-10-30 15:13:59,809 - distributed.worker - INFO - Memory: 10.00 GiB
2023-10-30 15:13:59,809 - distributed.worker - INFO - Memory: 10.00 GiB
2023-10-30 15:13:59,809 - distributed.worker - INFO - Local Directory: /tmp/slurm-1880419/dask-scratch-space/worker-4aareetw
2023-10-30 15:13:59,809 - distributed.worker - INFO - Local Directory: /tmp/slurm-1880419/dask-scratch-space/worker-6q027am5
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,809 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,816 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,816 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,817 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:40947
2023-10-30 15:13:59,817 - distributed.worker - INFO - -------------------------------------------------
2023-10-30 15:13:59,818 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40947
2023-10-30 15:13:59,818 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:40947
2023-10-30 15:14:00,495 - distributed.worker - INFO - Starting Worker plugin _WorkerSetupPlugin-a75f7451-7733-4b31-93eb-3395836094c7
2023-10-30 15:14:00,495 - distributed.worker - INFO - Starting Worker plugin _WorkerSetupPlugin-a75f7451-7733-4b31-93eb-3395836094c7
INFO - Processing tile 47PNR (1/1)
INFO - Sending search request: https://peps.cnes.fr/resto/api/collections/S1/search.json?polarisation=VV VH&sensorMode=IW&startDate=2023-10-01&completionDate=2023-10-16&geometry=POLYGON ((98.9998 13.4780, 98.9998 14.4728, 100.0186 14.4728, 100.0186 13.4780, 98.9998 13.4780))&productType=GRD&maxRecords=20&page=1
ERROR - Skipping error while searching for peps QueryStringSearch instance:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 466, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 461, in _make_request
httplib_response = conn.getresponse()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 798, in urlopen
retries = retries.increment(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 714, in urlopen
httplib_response = self._make_request(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 468, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/urllib3/connectionpool.py", line 357, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='peps.cnes.fr', port=443): Read timed out. (read timeout=5)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/eodag/plugins/search/qssearch.py", line 961, in _request
response = requests.get(
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/work/scratch/env/koleckt/.conda/envs/s1tiling/lib/python3.8/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='peps.cnes.fr', port=443): Read timed out. (read timeout=5)
INFO - 0 remote S1 products returned in page 1: []
INFO - 0 remote S1 product(s) found and filtered (IW && VV VH): []
INFO - 0 remote S1 product(s) will be downloaded
INFO - Downloading images related to 47PNR took 7.200475322082639sec
INFO - Intersecting raster list w/ 47PNR took 1.5857519004493952sec
INFO - No intersection with tile 47PNR
INFO - Processing of tile 47PNR took 8.788740198127925sec
INFO - Execution report: no error detected
INFO - -> Nothing has been executed
INFO - Stopping worker at tcp://127.0.0.1:40667. Reason: nanny-close
INFO - Stopping worker at tcp://127.0.0.1:38435. Reason: nanny-close
2023-10-30 15:14:09,299 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:40667. Reason: nanny-close
2023-10-30 15:14:09,299 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:38435. Reason: nanny-close
[2023-10-30 15:14:09.737736626+00:00] /koleckt/1880419: Account (env) =
[2023-10-30 15:14:09.738837876+00:00] /koleckt/1880419: User (env) = koleckt
[2023-10-30 15:14:09.739657538+00:00] /koleckt/1880419: GroupId = biomass(5099)
[2023-10-30 15:14:09.740474140+00:00] /koleckt/1880419: UserId = koleckt(1687)
[2023-10-30 15:14:09.741516560+00:00] /koleckt/1880419: Account = sco
[2023-10-30 15:14:09.742531210+00:00] /koleckt/1880419: Account = sco
[2023-10-30 15:14:09.743865107+00:00] /koleckt/1880419: Partition = cpu2022
[2023-10-30 15:14:09.745402943+00:00] /koleckt/1880419: QOS = cpu_2022_1280
[2023-10-30 15:14:09.746615481+00:00] /koleckt/1880419: JobName = jobThailand.slurm
[2023-10-30 15:14:09.747606771+00:00] /koleckt/1880419: SubmitTime = 2023-10-30T15:13:38
[2023-10-30 15:14:09.748467723+00:00] /koleckt/1880419: RunTime = 00:00:30
[2023-10-30 15:14:09.749347165+00:00] /koleckt/1880419: TimeLimit = 01:00:00
[2023-10-30 15:14:09.750203856+00:00] /koleckt/1880419: EligibleTime = 2023-10-30T15:13:38
[2023-10-30 15:14:09.751024628+00:00] /koleckt/1880419: StartTime = 2023-10-30T15:13:39
[2023-10-30 15:14:09.751834341+00:00] /koleckt/1880419: EndTime = 2023-10-30T15:14:09
[2023-10-30 15:14:09.753293647+00:00] /koleckt/1880419: ExitCode = 0:0
[2023-10-30 15:14:09.754064039+00:00] /koleckt/1880419: NodeList = trex027
[2023-10-30 15:14:09.754861262+00:00] /koleckt/1880419: BatchHost = trex027
[2023-10-30 15:14:09.755633654+00:00] /koleckt/1880419: TRES =
[2023-10-30 15:14:09.756412867+00:00] /koleckt/1880419: Command = /home/il/koleckt/theia/jobThailand.slurm
[2023-10-30 15:14:09.757175429+00:00] /koleckt/1880419: WorkDir = /home/il/koleckt/theia
[2023-10-30 15:14:09.757922832+00:00] /koleckt/1880419: StdErr = /home/il/koleckt/theia/slurm-1880419.out
[2023-10-30 15:14:09.758671885+00:00] /koleckt/1880419: StdOut = /home/il/koleckt/theia/slurm-1880419.out
#########################################
# EVIDEN - CNES #
# Job Accounting #
#########################################
Cluster : trex
JobIDRaw : 1880419
JobID : 1880419
JobName : jobThailand.slurm
Account : sco
User : koleckt(1687), biomass(5099)
Partition : cpu2022
QOS : cpu_2022_1280
Nodelist : trex027 (1)
State : COMPLETED (Exitcode=0:0, DerivedExitCode=0:0, Restarts=0)
Submit date : 2023-10-30T15:13:38
Start time : 2023-10-30T15:13:39
End time : 2023-10-30T15:14:09
Elapsed time : 00:00:30 (Timelimit=01:00:00, Suspended=00:00:00)
TEI : 0:02:00
Command : /home/il/koleckt/theia/jobThailand.slurm
StdOut : /home/il/koleckt/theia/slurm-1880419.out
StdErr : /home/il/koleckt/theia/slurm-1880419.out
STEP(s) : 2
---------
StepID | JobName State Exit Start End Elapsed TotalCPU CPUTime ConsumedEnergy MaxDiskRead MaxDiskWrite MaxRSS MaxRSSTask MaxRSSNode NNodes UserCPU NodeList
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
batch | batch COMPLETED 0:0 2023-10-30T15:13:39 2023-10-30T15:14:09 00:00:30 00:14.532 00:02:00 0.08625K 277.99M 152.40M 263684K 0 trex027 1 00:11.960 trex027
extern | extern COMPLETED 0:0 2023-10-30T15:13:39 2023-10-30T15:14:09 00:00:30 00:00:00 00:02:00 0.08625K 0.00M 0 1672K 0 trex027 1 00:00:00 trex027