Skip to content

Conflict in logging when running many jobs

When running more than one job (on HAL for instance), there is a conflict when the code writes log files:

INFO - Execute PrepareForOrtho -> /tmp/pbs.7287662.admin01/s1tiling/S1/s1a-iw-grd-vv-20200606t111928-20200606t111953-032896-03cf78-
001_OrthoReady.tiff from /tmp/pbs.7287662.admin01/s1tiling/raw/S1A_IW_GRDH_1SDV_20200606T111928_20200606T111953_032896_03CF78_58BB/
S1A_IW_GRDH_1SDV_20200606T111928_20200606T111953_032896_03CF78_58BB.SAFE/measurement/s1a-iw-grd-vv-20200606t111928-20200606t111953-
032896-03cf78-001.tiff
ERROR - --- Logging error ---
INFO - Execute PrepareForOrtho -> /tmp/pbs.7287662.admin01/s1tiling/S1/s1a-iw-grd-vv-20200618t111928-20200618t111953-033071-03d4c1-
001_OrthoReady.tiff from /tmp/pbs.7287662.admin01/s1tiling/raw/S1A_IW_GRDH_1SDV_20200618T111928_20200618T111953_033071_03D4C1_A4BA/
S1A_IW_GRDH_1SDV_20200618T111928_20200618T111953_033071_03D4C1_A4BA.SAFE/measurement/s1a-iw-grd-vv-20200618t111928-20200618t111953-
033071-03d4c1-001.tiff
ERROR - --- Logging error ---
INFO - Execute PrepareForOrtho -> /tmp/pbs.7287662.admin01/s1tiling/S1/s1a-iw-grd-vh-20200618t111903-20200618t111928-033071-03d4c1-
002_OrthoReady.tiff from /tmp/pbs.7287662.admin01/s1tiling/raw/S1A_IW_GRDH_1SDV_20200618T111903_20200618T111928_033071_03D4C1_4B37/
S1A_IW_GRDH_1SDV_20200618T111903_20200618T111928_033071_03D4C1_4B37.SAFE/measurement/s1a-iw-grd-vh-20200618t111903-20200618t111928-
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - --- Logging error ---
ERROR - Execution of PrepareForOrtho -> /tmp/pbs.7287662.admin01/s1tiling/S1/s1a-iw-grd-vh-20200712t111905-20200712t111930-033421-0
3df63-002_OrthoReady.tiff from /tmp/pbs.7287662.admin01/s1tiling/raw/S1A_IW_GRDH_1SDV_20200712T111905_20200712T111930_033421_03DF63
_2D9F/S1A_IW_GRDH_1SDV_20200712T111905_20200712T111930_033421_03DF63_2D9F.SAFE/measurement/s1a-iw-grd-vh-20200712t111905-20200712t1
11930-033421-03df63-002.tiff failed
Traceback (most recent call last):
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/logging/__init__.py", line 1029, in emit
    self.flush()
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/logging/__init__.py", line 1009, in flush
    self.stream.flush()
OSError: [Errno 116] Stale file handle

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/logging/__init__.py", line 1029, in emit
    self.flush()
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/logging/__init__.py", line 1009, in flush
    self.stream.flush()
OSError: [Errno 116] Stale file handle

In addition, here is a sample of stderr output. It seems there is a mix of 2 concurrent writing in the file

Downloading products:  96%|█████████▌| 25/26 [45:07<02:27, 147.15s/product]
22.9GB [46:34, 14.3MB/s]
Extracting files from S1A_IW_GRDH_1SDV_20200712T111905_20200712T111930_033421_03DF63_2D9F.zip:   0%|          | 0/29 [00:00<?, ?fil
Extracting files from S1A_IW_GRDH_1SDV_20200712T111905_20200712T111930_033421_03DF63_2D9F.zip: 100%|██████████| 29/29 [00:12<00:00,
  2.30file/s]les from S1A_IW_GRDH_1SDV_20200712T111905_20200712T111930_033421_03DF63_2D9F.zip:  45%|████▍     | 13/29 [00:12<00:17,
Downloading products: 100%|██████████| 26/26 [47:12<00:00, 108.95s/product]Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00002ba33e557700 (most recent call first):
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 204 in commit_frame
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 486 in save
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 884 in _batch_setitems
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 859 in save_dict
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 504 in save
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 662 in save_reduce
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 549 in save
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 885 in _batch_setitems
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 859 in save_dict
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 504 in save
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 662 in save_reduce
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 549 in save
  File "/home/il/koleck/.conda/envs/s1tilingV0.2/lib/python3.7/pickle.py", line 885 in _bFatal Python error: aCannot recover from s
tack overflow.t
c
hCurrent thread 0x_00002b3c411e6700s (most recent call first):
e  File t"i/theommse
/  File i"l//hkoomlee/cikl/./ckonodlae/cekn/v.sc/osn1dtai/leinnvgsV/0s.12t/illiibn/gpVy0t.h2on/3l.i7b//ppiyctkhloen.3p.y7"/, line p
204i in cckolmem.ipty_"f, line r859a in msea
v  File e"_/dhioFatal Python error: cmCannot recover from stack overflow.te

/
  File iCurrent thread 0x"l00002b7e63ba0700// (most recent call first):
  File hk"/oohmloeem/ce/ikil/l/./kckooollnecedkca/k/./eco.nndcva/osenn/vdssa1//tsei1nltviisnli/gnsVg10V0t..i22l//illniigbbV//0pp.yy
2tth/holon3in.b37/./p7py/ictpkhiloce.nkp3ly.e"7., line /p204py in i"coc, line mmk486itl in _fesra.ampveye
  File "
", line   File /504"h in /omshe/aoivmlee/k
/o  File il"lec//k/hk.oocomlneeda/c/eiknl/vs/./kcsoo1lntiedlicank/g/eV0.n.2cv/osln/idsb/a1p/tyeitnlhovinsn3./g7sV/p10ict.ki2lel/.il
pyni"gb, line V/4860p in s.ya2tv/he
lo  File in"b3//.hop7mey//itplhi/koconkle3lck.e/7../pcopyndi"a/c, line ek884nl in vse_/s.b1patiytli"cng, line hV662_0 in s.2se/atli
vibet/p_eytrmhesond
3u  File .c"7/e/p
hic  File ok"mle/e.h/pyoi"ml, line e/884/k in _ioblla/etkccokh_l/see.ticctkoe/nm.dsca
o/  File "ne/dnhavom/sFatal Python error: ee/Cannot recover from stack overflow./ns
iv1
lstCurrent thread 0x//i00002b0b3e431700ksl (most recent call first):
o1i  File ltn"eig/clVhki0o/n.m.g2ecV//o0lin.ild2b/a//k/lpoeiylnbtev/hcspok/yn/st3.1h.cto7oin/nl3pdi.ian7c/g/keVpln0iev.c.s2kp//lysl
e"1i., line tbp859i/y in lp"siy, line ant549vgh in eVos_0nad.3vi2.ec/7
tl/  File
ip"  File bi/"/ch/pkohylmoteemh./eopi/nyli3"/l., line k/7859ok/ in Fatal Python error: lopsCannot recover from stack overflow.eleia
cccv
kkkeCurrent thread 0x//l_00002ba806730700..ed (most recent call first):
cc.i  File oopc"nnyt/dd"
haa, line   File o//204"mee in /ennch/vvooissmml//me/ssi/k11tiott_lliif/ellrkciiaoknnml/ggee.VV0
cc0.  File ko.2"/n2//.d/lhcalioo/ibmneb/edn/p/avpyi/sytle/th/nshokv1onostn3l/i3.esl.7c1i7/ktn/p/igpi.lVicci0ckon.klng2ledV/e.a0l.p/
.ipye2by"n//", line vlp, line 504siy885 in /bt in ss/h_a1pobvtynaeit3t
lh.c  File io7h"nn/_/g3pshV.ieo07ctm./kie2plt//ieeilc.mlikps/bly
k/e"  File op., line "lyp204/etFatal Python error: y in hchCannot recover from stack overflow."cokon
, line om/3