Tiled GeoTIFF support and streaming
Description
For our application we'd like to switch from stripped to tiled GeoTIFFs. OTB supports these automatically via GDAL, of course, but it doesn't seem to configure an adequate streaming mode for them. Notice:
$ gdalinfo ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF | rg Block
Band 1 Block=256x256 Type=Int16, ColorInterp=Gray
Band 2 Block=256x256 Type=Int16, ColorInterp=Undefined
Band 3 Block=256x256 Type=Int16, ColorInterp=Undefined
Band 4 Block=256x256 Type=Int16, ColorInterp=Undefined
$ otbcli_Convert -in ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF -out features.tif int16
$ gdalinfo features.tif | rg Block
Band 1 Block=4500x1 Type=Int16, ColorInterp=Gray
Band 2 Block=4500x1 Type=Int16, ColorInterp=Undefined
Band 3 Block=4500x1 Type=Int16, ColorInterp=Undefined
Band 4 Block=4500x1 Type=Int16, ColorInterp=Undefined
This might seem harmless, but it actually leads to a massive read amplification effect. For example:
$ time otbcli_ConcatenateImages -il ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000.TIF -out features.tif int16
Writing features.tif...: 100% [**************************************************] (14 seconds)
otbcli_ConcatenateImages -il -out features.tif int16 14.67s user 2.30s system 113% cpu 14.972 total
$ time otbcli_ConcatenateImages -il ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000_stripped.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000_stripped.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000_stripped.TIF ~/SPOT4_HRVIR1_XS_20130206_N2A_ORTHO_SURF_CORR_ENV_EUkraineD0000B0000_stripped.TIF -out features.tif int16
Writing features.tif...: 100% [**************************************************] (5 seconds)
otbcli_ConcatenateImages -il -out features.tif int16 7.06s user 0.79s system 137% cpu 5.718 total
This case shows a 3x slowdown, but this is on an SSD and with a newer OTB version. In our use case we've seen a difference of 50-100x. Here is another example of the same thing on a different machine:
# tiled
Writing features.tif...: 100% [**************************************************] (1e+02 seconds)
88.57user 24.77system 1:47.15elapsed 105%CPU (0avgtext+0avgdata 312232maxresident)k
0inputs+1642256outputs (0major+89734minor)pagefaults 0swaps
# stripped
Writing features.tif...: 100% [**************************************************] (6 seconds)
12.78user 0.80system 0:07.66elapsed 177%CPU (0avgtext+0avgdata 288848maxresident)k
0inputs+1125064outputs (0major+83077minor)pagefaults 0swaps
It's also interesting that adding ?&streaming:type=tiled
doesn't seem to have any effect on the output block size.
Note that I haven't looked into the code yet, so it might be by design. But unless I'm missing something, it might make sense to select a region size that's a multiple of the GeoTIFF block size.
Configuration information
Linux x64, OTB 6.4.0