Skip to content

GitLab

Open
Opened by Laurențiu Nicola@lnicolaDeveloper

GDAL VSI support

I've recently heard someone insist that OTB is a bad framework and isn't "cloud-ready" because it doesn't support the "technologies of the future", like object storage. The thing is, GDAL does support cURL, Amazon S3 and other storage drivers. Unfortunately, OTB doesn't seem to.

I tried a couple of tests:

$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"        
2018-10-24 13:26:34 (INFO): Default RAM limit for OTB is 128 MB
2018-10-24 13:26:34 (INFO): GDAL maximum cache size is 393 MB
2018-10-24 13:26:34 (INFO): OTB will use at most 4 threads
2018-10-24 13:26:35 (INFO): 
Image general information:
	Number of bands : 3
	No data flags : Not found
	Start index :  [0,0]
	Size :  [65536,65536]
	Origin :  [0.5,0.5]
	Spacing :  [1,1]
	Estimated ground spacing (in meters): [547.072,296.427]

Image acquisition information:
	Sensor : 
	Image identification number: 

Image default RGB composition:
	[R, G, B] = [0,1,2]

Ground control points information:
	Number of GCPs = 0
	GCPs projection = 

Output parameters value:
indexx: 0
indexy: 0
sizex: 65536
sizey: 65536
spacingx: 1
spacingy: 1
originx: 0.5
originy: 0.5
estimatedgroundspacingx: 547.0715332
estimatedgroundspacingy: 296.4269409
numberbands: 3
sensor: 
id: 
time: 
town: 
country: 
rgb.r: 0
rgb.g: 1
rgb.b: 2
projectionref: 
keyword: 
gcp.count: 0
gcp.proj: 
gcp.ids: 
gcp.info: 
gcp.imcoord: 
gcp.geocoord

$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt gdalinfo http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif                
Driver: GTiff/GeoTIFF
Files: none associated
Size is 514, 515
Coordinate System is:
PROJCS["unnamed",
    GEOGCS["NAD27",
        DATUM["North_American_Datum_1927",
            SPHEROID["Clarke 1866",6378206.4,294.9786982138982,
                AUTHORITY["EPSG","7008"]],
            AUTHORITY["EPSG","6267"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4267"]],
    PROJECTION["Cylindrical_Equal_Area"],
    PARAMETER["standard_parallel_1",33.75],
    PARAMETER["central_meridian",-117.333333333333],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]]]
Origin = (-28493.166784412522247,4255884.543802191503346)
Pixel Size = (60.022136983193739,-60.022136983193739)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  -28493.167, 4255884.544) (117d38'27.05"W, 33d56'37.74"N)
Lower Left  (  -28493.167, 4224973.143) (117d38'27.05"W, 33d39'53.81"N)
Upper Right (    2358.212, 4255884.544) (117d18'28.38"W, 33d56'37.74"N)
Lower Right (    2358.212, 4224973.143) (117d18'28.38"W, 33d39'53.81"N)
Center      (  -13067.478, 4240428.844) (117d28'27.71"W, 33d48'15.38"N)
Band 1 Block=514x15 Type=Byte, ColorInterp=Gray

Notice that OTB reports wrong origin, ground spacing and size values.

$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -e trace=file -f otbcli_ReadImageInfo -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif"

[snip]
[pid  6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid  6069] getcwd("/home/user", 4096) = 16
[pid  6069] lstat("/home/user/http:", 0x7ffed725d380) = -1 ENOENT (No such file or directory)
[pid  6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid  6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.til", F_OK) = -1 ENOENT (No such file or directory)
[pid  6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  6069] access("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", F_OK) = -1 ENOENT (No such file or directory)
[pid  6069] openat(AT_FDCWD, "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  6069] stat("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x7ffed725dc50) = -1 ENOENT (No such file or directory)
[pid  6069] readlink("http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif", 0x245a730, 2048) = -1 ENOENT (No such file or directory)
2018-10-24 13:26:57 (INFO): No kwl metadata found in file http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif

OTB tries to open HTTP URLs as files.

$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_ComputeImagesStatistics -il "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true"

[snip]
[pid  6110] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqrr HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96

OTB gets into an infinite loop trying to request the wrong URL (notice the qqqqqqrr suffix).

CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt strace -s2048 -f otbcli_Convert -in "http://download.osgeo.org/geotiff/samples/gdal_eg/cea.tif?skipgeom=true" -out foo.tif

[snip]
[pid  6171] sendto(4, "GET /geotiff/samples/gdal_eg/cea.tifqqqqqqqq HTTP/1.1\r\nHost: download.osgeo.org\r\nAccept: */*\r\n\r\n", 96, MSG_NOSIGNAL, NULL, 0) = 96

Same issue, different suffix. Uninitialized memory?

Of course, I expect performance to be pretty poor when doing network access, but sometimes it's useful, and the alternative is to use a FUSE emulation driver that might have its own problems.

Related: #1746

Related: #1506

Related: #1642

Edited by Laurențiu Nicola