Memory issue in Monteverdi with images with subdatasets (HDF4)
Mantis Issue 1303, reported by mgrizonnet, assigned to gpasero, created: 2016-10-19
I faced some memory usage issue in monteverdi with HDF4 images.
Let me describe what I've observed when I compare opening a subdataset from a product in hdf or in tiff:
I've a hdf4 image with 6 subdatasets in float 32 ([4236x4128]). The size of the hdf is around 420Mb.
If I open it in Monteverdi and select one subdataset, I can observe a memory consumption after navigating in resolution of about 1.4Gb. I don't have overview for this dataset (overview generation for images with subdataset is not yet supported in monteverdi).
If I extract this subdataset and convert it to tif float 32 (around 65Mb) and I navigate in the product I can see a memory consumption arount 200Mb (without ovr).
Why such a difference?
My first idea was that for some reason Monteverdi or Ice load in memory not only the subdataset visualized but the 5 other subdatasets (it could explain the memory comsuption ratio between tif and hdf).
It is probably not the problem here as I've also noticed that opening more than one subdataset in Monteverdi increase even more the ram consumption (more than 4Gb!!)
1476869507 - rashadkmhmm. what if you open subdataset in montverdi using extended file name?
1476947615 - grizonnetmGood point, I've forgot to mention that I've also tested what you suggest:
mymonteverdi"input.hdf?&sdataidx=2" and I see the same memory consumption arounf 1.4 Gb.
Note that I've observed this memory issue on Linux with a local build with gdal 2.1 and hdf 4.2 (ubuntu 16.04) and also on Windows with binary package 5.6.1.
1477409201 - gpaseroI reproduced the issue with a smaller HDF (122MB). The memory usage is a bit high compared to the size of the dataset. When I load all sub-datasets, monteverdi uses 1050MB.
Plus I get a lot of error messages from GDAL : ERROR 1: /home/gpasero/Projet_OTB/build/Grille_Explo_05_Step_02_seuil_taux_initial_0_final_0_interpolateur_SINUS_CARDINAL.hdf:Not a TIFF or MDI file, bad magic number 782 (0x30e)
1477928876 - gpaseroI have a first clue : the resolution level is set by appending "?$resol=x" to the