let-it-snow issueshttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues2024-03-21T15:35:04Zhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/130Rastertools shaded snow algorithm is missing some shadow especially on mounta...2024-03-21T15:35:04ZGuillaume Eynard-BontempsRastertools shaded snow algorithm is missing some shadow especially on mountain ridgesWhile comparing results of LIS blue in #21, I observed some shadow detection problem with the new rastetools algorithm used in 1.11.0 version.
The overall observation is that:
- In 1.10.2 version, saga gis algorithm (or the threshold we...While comparing results of LIS blue in #21, I observed some shadow detection problem with the new rastetools algorithm used in 1.11.0 version.
The overall observation is that:
- In 1.10.2 version, saga gis algorithm (or the threshold we use) tends to overestimated shaded zones, and on casted shadow, there seems to be some little shift on the area detected in shadow.
- In 1.11.0 version using rastertools, the shaded areas are much more conservative: casted shadow are really precise, but on slopes and especially mountain ridges, some shaded areas are missed, leading to strange patterns when viewing the results.
The following images illustrate these results and problems near Goulier area in France on the S2 image taken on 19 Nov 2019.
The hillshade mask is green is issued from saga-gis, wherease the one in blue on the top is from rastertools:
![LIS_hillshade_mask_goulier_2019_comparison](/uploads/020555e6fc1ef4b7194c292246483cb7/LIS_hillshade_mask_goulier_2019_comparison.PNG)
On the left, rastertools mask is missing shadow near the mountain ridges. On the top, we can see that saga-gis mask detects also slopes with lower illumination. On the right, rastertools mask is better on casted shadow.
It is hard to argue on some areas which one of the mask is the best, a choice between over estimation and conservative one. But the problem of the ridges of rastertools mask is clearly visible on the final result (LIS 1.11 in blue, LIS 1.10 in green):
![LIS_result_goulier_1.11](/uploads/be12b73c14665a9265f778a74d9db8fd/LIS_result_goulier_1.11.PNG)![LIS_result_goulier_1.10](/uploads/9a26693fc938d9c4073fa52cb009415c/LIS_result_goulier_1.10.PNG)https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/129synthesis code is verbose2024-03-08T16:12:34ZSimon Gascoinsynthesis code is verbose`let_it_snow_synthesis.py` outputs a lot of text to STDOUT even when setting "log_level": "WARNING" in the [config file](`doc/synthesis_launch_example.json`). It seems that the log level of OTB applications is not maintained throughout t...`let_it_snow_synthesis.py` outputs a lot of text to STDOUT even when setting "log_level": "WARNING" in the [config file](`doc/synthesis_launch_example.json`). It seems that the log level of OTB applications is not maintained throughout the code.https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/128QCFLAGS is not masked2024-03-08T11:57:49ZSimon GascoinQCFLAGS is not maskedThe current QCFLAGS file is not masked where the FSC product is nodata. It may be a bit confusing for the user? Masking QCFLAGS should somewhat reduce the size of the QCFLAGS file too.
FSC
<img src="/uploads/6f054b2687fc6eba7a56db54b2...The current QCFLAGS file is not masked where the FSC product is nodata. It may be a bit confusing for the user? Masking QCFLAGS should somewhat reduce the size of the QCFLAGS file too.
FSC
<img src="/uploads/6f054b2687fc6eba7a56db54b2ad3c8d/image.png" alt="FSC" width="300" height="auto">
QCFLAGS bit 2
<img src="/uploads/2fe79e53b82a8ca88e10d535b40d3f39/image.png" alt="FSC" width="300" height="auto">https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/127NaN in metadata file2024-03-06T09:28:36ZSimon GascoinNaN in metadata fileNot really a "LIS" issue but in LIS*.xml files produced by CNES (/work/THEIA/hesperides/prod/HYMOTEP/) incidence angles copied from L2A product are always NaN
<Incidence_Angles>
<ZENITH_ANGLE unit="deg">NaN</ZENITH_ANGLE>
...Not really a "LIS" issue but in LIS*.xml files produced by CNES (/work/THEIA/hesperides/prod/HYMOTEP/) incidence angles copied from L2A product are always NaN
<Incidence_Angles>
<ZENITH_ANGLE unit="deg">NaN</ZENITH_ANGLE>
<AZIMUTH_ANGLE unit="deg">NaN</AZIMUTH_ANGLE>
</Incidence_Angles>https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/126rastertools default windows_size parameters throws error on some S2 tiles2024-03-21T14:04:19ZGuillaume Eynard-Bontempsrastertools default windows_size parameters throws error on some S2 tilesDepending on elevation and sun angle, Radius automatically computed might be too big.
See following error:
```
INFO:hillshade.py::process_file:Processing file /usr/local/test_lis/dtm/32TLR/dem_20m.tif
2024-02-26T13:43:42.708 ERROR:sn...Depending on elevation and sun angle, Radius automatically computed might be too big.
See following error:
```
INFO:hillshade.py::process_file:Processing file /usr/local/test_lis/dtm/32TLR/dem_20m.tif
2024-02-26T13:43:42.708 ERROR:snow_detector.py::detect_snow:Rastertools hillshade methods encountered an error : The radius (option --radius, value=629) must be strictly less than half the size of the window (option --window_size, value=1024)
2024-02-26T13:43:42.709 ERROR:let_it_snow_fsc.py::let_it_snow_fsc:The radius (option --radius, value=629) must be strictly less than half the size of the window (option --window_size, value=1024)
Traceback (most recent call last):
File "app/let_it_snow_fsc.py", line 166, in let_it_snow_fsc
detect_snow(config, output_dir, h2_chain_version, product_counter)
File "/usr/local/lib/python3.8/site-packages/s2snow/snow_detector.py", line 162, in detect_snow
compute_hillshade_mask_rastertools(relief_shadow_mask, dem, config.metadata, tmp_dir, tg_resolution,
File "/usr/local/lib/python3.8/site-packages/s2snow/hillshade.py", line 309, in compute_hillshade_mask_rastertools
hillshade_str = tool.process_file(dem_path)
File "/usr/local/lib/python3.8/dist-packages/eolab/rastertools/hillshade.py", line 128, in process_file
raise ValueError(f"The radius (option --radius, value={radius}) must be strictly "
ValueError: The radius (option --radius, value=629) must be strictly less than half the size of the window (option --window_size, value=1024)
```
We must:
- increase default windows_size to a sensible value (4096?)
- Check that it works on different conditions, with correct results
- Eventually make rastertools more robust to this case.1.12.0Michael ERBLANGMichael ERBLANGhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/124Docker build: problem in the Python environment2023-12-15T16:24:18ZGuillaume Eynard-BontempsDocker build: problem in the Python environmentWhen building, there is a warning:
> ERROR: scipy 1.10.1 has requirement numpy<1.27.0,>=1.19.5, but you'll have numpy 1.17.4 which is incompatible. ERROR: rasterio 1.3.8 has requirement numpy>=1.18, but you'll have numpy 1.17.4 which is...When building, there is a warning:
> ERROR: scipy 1.10.1 has requirement numpy<1.27.0,>=1.19.5, but you'll have numpy 1.17.4 which is incompatible. ERROR: rasterio 1.3.8 has requirement numpy>=1.18, but you'll have numpy 1.17.4 which is incompatible.
This warning is also present when running the image:
> /usr/local/lib/python3.8/dist-packages/scipy/__init__.py:143: UserWarning: A NumPy version >=1.19.5 and <1.27.0 is required for this version of SciPy (detected version 1.17.4)
> warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
Even if it does not seem to produce errors in the results, this should be corrected.1.11.0Céline RailléCéline Railléhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/123Compress output files2024-03-05T08:33:26ZGuillaume Eynard-BontempsCompress output filesEven if let-it-snow outputs are not really big compared to other products, it is alsways a good pratice to use optimized file formats as ouput.
Minimum would be to apply some compression on TIF outputs, this should be easily done, and e...Even if let-it-snow outputs are not really big compared to other products, it is alsways a good pratice to use optimized file formats as ouput.
Minimum would be to apply some compression on TIF outputs, this should be easily done, and every library for writing or reading TIF already support this.
Check if using Cloud optimized GeoTIFF makes sense here.
This could be an option first, but at some point the default.1.12.0Céline RailléCéline Railléhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/122Ability to configure a tmp working directory different from output directory2023-10-05T10:22:32ZGuillaume Eynard-BontempsAbility to configure a tmp working directory different from output directoryCurrent output for Snow FSC product example layout:
```
fsc_config.json
LIS_METADATA.XML
LIS_S2-SNOW-FSC_T31TCJ_20230204T105857_1.10.2_1.tif
LIS_S2-SNOW-FSC-QCFLAGS_T31TCJ_20230204T105857_1.10.2_1.tif
tmp/
```
At least tmp folder path s...Current output for Snow FSC product example layout:
```
fsc_config.json
LIS_METADATA.XML
LIS_S2-SNOW-FSC_T31TCJ_20230204T105857_1.10.2_1.tif
LIS_S2-SNOW-FSC-QCFLAGS_T31TCJ_20230204T105857_1.10.2_1.tif
tmp/
```
At least tmp folder path should be configurable if needed (typically if we don't want to check the files here). This is especially a good thing when working on an HPC cluster.
Maybe json and XML files should have the product id in their file names.
Should we create a parent folder with product id?
Same should be done for other products.https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/121RedHat8 system compatibility2024-01-22T11:29:22ZGuillaume Eynard-BontempsRedHat8 system compatibilityTest and validate compilation and execution of LIS on Rhel 8 operating systems.
This is needed to deploy let-it-snow on the new CNES cluster TREX.Test and validate compilation and execution of LIS on Rhel 8 operating systems.
This is needed to deploy let-it-snow on the new CNES cluster TREX.1.11.0Céline RailléCéline Railléhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/120OTB 8 compatibility2024-03-28T15:54:36ZGuillaume Eynard-BontempsOTB 8 compatibilityMake let-it-snow compatible with OTB 8.x versions.
Not a hard requirement for a deployment on Trex new CNES cluster, but nonetheless it's important to follow OTB developements.Make let-it-snow compatible with OTB 8.x versions.
Not a hard requirement for a deployment on Trex new CNES cluster, but nonetheless it's important to follow OTB developements.1.12.0Céline RailléCéline Railléhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/119Remove SAGA-GIS dependency2024-01-25T11:06:27ZGuillaume Eynard-BontempsRemove SAGA-GIS dependencySaga gis library was needed to quickly implement a correct computation of shaded snow. However this is not a simple dependency, it has some Python incompatibilities and it complexifies the installation of let-it-snow.
Thus, it should be...Saga gis library was needed to quickly implement a correct computation of shaded snow. However this is not a simple dependency, it has some Python incompatibilities and it complexifies the installation of let-it-snow.
Thus, it should be replaced and removed in the next LIS major version.1.11.0Michael ERBLANGMichael ERBLANGhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/118Tests set parameters (and other part of the files here) are tightly linked to...2024-01-22T15:50:32ZGuillaume Eynard-BontempsTests set parameters (and other part of the files here) are tightly linked to CNES HPC infrastructureA lot of test set configuration files point to CNES HPC infrastructure shared storage, in particular for auxiliary data and saga_gis software.
It can be verified using simple grep commands:
```
grep -R work .
grep -R soft .
```
```
......A lot of test set configuration files point to CNES HPC infrastructure shared storage, in particular for auxiliary data and saga_gis software.
It can be verified using simple grep commands:
```
grep -R work .
grep -R soft .
```
```
...
./Input-Data-Test/SNOW_PRODUCTS/SENTINEL2A_20180131-105416-437_L2A_T31TCH_D_V1-4/param_test.json: "dem": "/work/OT/siaa/Theia/Neige/DEM/S2__TEST_AUX_REFDE2_T31TCH_0001.DBL.DIR/S2__TEST_AUX_REFDE2_T31TCH_0001_ALT_R2.TIF"
...
./Input-Data-Test/S2/shaded_snow/global_parameters_shaded_snow_saga.json: "sagagis_tools_libs": "/softs/rh7/spack_install/linux-centos7-x86_64/gcc-10.2.0/python-3.8.4-4ujdnstfskok34hg3kerp662wwdaezso/lib/saga"
...
```
It would be better to extract all this to be able to run all tests outside of CNES infrastructure.1.12.0Michael ERBLANGMichael ERBLANGhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/117LIS_configuration_for_expert documentation n'est pas à jour2023-07-12T12:43:41ZAurore DupuisLIS_configuration_for_expert documentation n'est pas à jourIl manque `sagagis_tools_libs` et `sagagis_thresh`.Il manque `sagagis_tools_libs` et `sagagis_thresh`.1.10.2https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/116Ajout d'un message d'erreur + exception si pas de ta_lighting trouvé dans le ...2023-07-12T10:05:32ZAurore DupuisAjout d'un message d'erreur + exception si pas de ta_lighting trouvé dans le repertoire saga_toolsAjout d'un message d'erreur + exception si pas de ta_lighting trouvé dans le repertoire saga_toolsAjout d'un message d'erreur + exception si pas de ta_lighting trouvé dans le repertoire saga_tools1.10.1https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/115Erreur au lancement de lis 1.10 sur des produits L8 avec le nouveau fichier d...2023-08-11T11:43:34ZAurore DupuisErreur au lancement de lis 1.10 sur des produits L8 avec le nouveau fichier de configurationLa détection des ombres intégrées à la version 1.8 et les suivantes utilise les bandes bleues et NIR, ainsi que les métadonnées.
La configuration "mission" interne de LIS, permettant de récupérer les bandes nécessaires au traitement lors...La détection des ombres intégrées à la version 1.8 et les suivantes utilise les bandes bleues et NIR, ainsi que les métadonnées.
La configuration "mission" interne de LIS, permettant de récupérer les bandes nécessaires au traitement lors de la lecture du produit n'a pas été mise à jour pour Landsat 8.
De ce fait, les bandes sont à None et le traitement tombe en erreur.
- [ ] Mettre à jour la configuration mission pour landsat 8 (voir ci dessous)
- [ ] Ajouter un test ou modifier le test landsat 8 avec le nouveau fichier de configuration
```
L8_parameters_new_format = {"multi": 1,
"green_band": ".*FRE_B3.*\.tif$",
"green_band_number": 1,
"red_band": ".*FRE_B4.*\.tif$",
"red_band_number": 1,
"swir_band": ".*FRE_B6.*\.tif$",
"swir_band_number": 1,
"blue_band": ".*SRE_B2.*\.tif$",
"blue_band_number": 1,
"nir_band": ".*SRE_B5.*\.tif$",
"nir_band_number": 1,
"cloud_mask": ".*CLM_XS.*\.tif$",
"dem": ".*ALT_R2\.TIF$",
"div_mask": ".*MG2_XS.*\.tif$",
"div_slope_threshold": 64,
"shadow_in_mask": 32,
"shadow_out_mask": 64,
"all_cloud_mask": 1,
"high_cloud_mask": 128,
"resize_factor": 8,
"metadata": ".*MTD_ALL.*\.xml$" }
L8_parameters = {"multi": 1,
"green_band": ".*ORTHO_SURF_CORR_PENTE.*\.TIF$",
"green_band_number": 3,
"red_band": ".*ORTHO_SURF_CORR_PENTE.*\.TIF$",
"red_band_number": 4,
"swir_band": ".*ORTHO_SURF_CORR_PENTE.*\.TIF$",
"swir_band_number": 6,
"blue_band": ".*SRE_B2.*\.tif$",
"blue_band_number": 1,
"nir_band": ".*SRE_B5.*\.tif$",
"nir_band_number": 1,
"cloud_mask": ".*NUA.*\.TIF$",
"div_mask": ".*DIV.*\.TIF$",
"div_slope_threshold": 8,
"dem": ".*\.tif",
"shadow_in_mask": 64,
"shadow_out_mask": 128,
"all_cloud_mask": 1,
"high_cloud_mask": 32,
"resize_factor": 8,
"metadata": ".*MTD_ALL.*\.xml$"}
```1.10.2Guillaume Eynard-BontempsGuillaume Eynard-Bontempshttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/114Gitlab-CI migration of the CI/CD pipeline2023-07-12T10:04:33ZCéline RailléGitlab-CI migration of the CI/CD pipeline1.10.2Céline RailléCéline Railléhttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/113Update FSC formula into configuration file2023-04-11T15:41:04ZAurore DupuisUpdate FSC formula into configuration file1.10.0Aurore DupuisAurore Dupuishttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/112Integrate shaded computation with saga-gis2023-04-17T19:58:27ZAurore DupuisIntegrate shaded computation with saga-gisIntegrate saga-gis and shaded mask computation, using task 117 in #21
- integrate saga_gis into environment on hpc and docker file
- include saga-gis code to compute shaded masm
- add configuration item to add path to saga-gis library an...Integrate saga-gis and shaded mask computation, using task 117 in #21
- integrate saga_gis into environment on hpc and docker file
- include saga-gis code to compute shaded masm
- add configuration item to add path to saga-gis library and threshold
- add error management : if no path defined => compute only hillshade mask, if invalid path => error message1.10.0Pierre TysebaertPierre Tysebaerthttps://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/111plusieurs erreurs inoffensives dans hillshade.py2023-04-17T20:31:34ZSimon Gascoinplusieurs erreurs inoffensives dans hillshade.pyDans le code hillshade.py les angles sont lus et écrits avec un mauvais géoréférencement ce qui ne produit pas d'erreur car la grille est traitée comme un numpy array avec la bonne taille. Pour avoir un code cohérent voici les 4 modifs:
...Dans le code hillshade.py les angles sont lus et écrits avec un mauvais géoréférencement ce qui ne produit pas d'erreur car la grille est traitée comme un numpy array avec la bonne taille. Pour avoir un code cohérent voici les 4 modifs:
0) lire le DEM sans rééchantillonnage GRA_Lanczos ni buffer et retypage (int16 peut être gardé)
```
dem_array = dem.ReadAsArray()
```
au lieu de [ligne 157](https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/blob/develop/python/s2snow/hillshade.py#L157) :
```
dem_array = np.copy(dem.ReadAsArray(buf_xsize=dem.RasterXSize, buf_ysize=dem.RasterYSize, resample_alg=gdal.GRA_Lanczos).astype(dtype=np.float64))
```
1) importer osr
```
from osgeo import osr
```
2) remplacer les deux fonctions `get_grid_values_from_xml` et `resample_angles` par celles-ci
```
def resample_angles(angle_array, angle_geoTransform, angle_spatialRef, ref_dem, output_angle):
"""
Resample the small grid of angles to the size of the dem to take into consideration the variation of sun angles given in the metadata files.
:param angle_array: The angles from the metadata file given as an array
:param ref_dem: the gdal Dataset object that will serve for the target resolution and georeferencing during the resampling
:param output_angle: The path of the file where the resampled angles will be written.
:return Gdal object: the gdal Dataset with the resampled angles
"""
driver = gdal.GetDriverByName("GTiff")
angles = driver.Create(output_angle, angle_array.shape[0], angle_array.shape[1], 1, gdal.GDT_Float32)
angles.SetGeoTransform(angle_geoTransform)
angles.SetProjection(angle_spatialRef.ExportToWkt())
angles.GetRasterBand(1).WriteArray(angle_array)
angles.FlushCache()
gdal.Warp(output_angle, angles, resampleAlg=gdal.GRIORA_Cubic, width=ref_dem.RasterXSize, height=ref_dem.RasterYSize)
return gdal.Open(output_angle)
```
```
def get_grid_values_from_xml(xml_path, angle="Zenith"):
tree = etree.parse(xml_path)
# get col and row steps (5 km)
node = tree.xpath(f".//Sun_Angles_Grids/{angle}/COL_STEP")[0]
colstep = float(node.text)
node = tree.xpath(f".//Sun_Angles_Grids/{angle}/ROW_STEP")[0]
# assume it's a north up image
rowstep = -1*float(node.text)
# get EPSG code
epsg = tree.find(".//HORIZONTAL_CS_CODE").text
# get grid corners coordinates
# warning: in array geometry the lower left corner is the upper left in raster geometry
ulx = float(tree.find(".//*[@name='upperLeft']/X").text)
uly = float(tree.find(".//*[@name='upperLeft']/Y").text)
lrx = float(tree.find(".//*[@name='lowerRight']/X").text)
lry = float(tree.find(".//*[@name='lowerRight']/Y").text)
node = tree.xpath(f".//Sun_Angles_Grids/{angle}/Values_List")[0] # VALUES
arrays_lst = []
for values in node:
values_arr = [float(k) for k in values.text.split(" ")]
arrays_lst.append(values_arr)
# We assume that the above coordinates correspond to the *centers* of corner pixels
# otherwise the 23x23 grid would have an extra row and column somewhere
ulxMTD = ulx - colstep/2
ulyMTD = uly - rowstep/2
# define the affine transformation coefficients
geoTransform = (ulxMTD, colstep, 0, ulyMTD, 0, rowstep)
# create spatial reference
spatialRef = osr.SpatialReference()
spatialRef.ImportFromEPSG(int(epsg))
return np.array(arrays_lst), geoTransform, spatialRef
```
3) appeler ces fonctions de la façon suivante
```
angle_array, angle_geoTransform, angle_spatialRef = get_grid_values_from_xml(
xml_path, angle="Zenith")
zenith = resample_angles(angle_array, angle_geoTransform, angle_spatialRef, dem, os.path.join(output_folder, ZENITH_PATH)).ReadAsArray()
angle_array, angle_geoTransform, angle_spatialRef = get_grid_values_from_xml(
xml_path, angle="Azimuth")
azimuth = resample_angles(angle_array, angle_geoTransform, angle_spatialRef, dem, os.path.join(output_folder, AZIMUTH_PATH)).ReadAsArray()
```1.10.0https://gitlab.orfeo-toolbox.org/remote_modules/let-it-snow/-/issues/110doc toc2023-04-11T15:23:04ZSimon Gascoindoc toc"If not defined for Sentinel-2 snow detection, only FSC-OG (on ground) will be computed."
It should be *FSC-TOC* (top-of-canopy)"If not defined for Sentinel-2 snow detection, only FSC-OG (on ground) will be computed."
It should be *FSC-TOC* (top-of-canopy)1.10.0