OTB is opening too many files
Description
We're trying to use OTB on file systems where metadata operations are very slow (e.g. network drives or worse). One issue I've noticed is that the geom
stuff is opening a lot of files:
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOTDIR (Not a directory)
[pid 26658] stat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 3
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 3
[pid 26658] openat(AT_FDCWD, "/home/me", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif.aux.xml", O_RDONLY) = 4
[pid 26658] access("/home/me/foo.geom", R_OK) = -1 ENOENT (No such file or directory)
[pid 26658] lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 26658] lstat("/home/me", {st_mode=S_IFDIR|0700, st_size=12288, ...}) = 0
[pid 26658] lstat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] stat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26658] access("/home/me/product.xml", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/PHRDIMAP.XML", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/METADATA.DIM", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/metadata.dim", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.TIL", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.til", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] access("/home/me/foo.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_vertices.kwl", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.geom", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] access("/home/me/foo.NTF", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.ntf", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 6
[pid 26658] access("/home/me/foo.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_vertices.kwl", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.IMD", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.IMD", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.IMD", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.IMD", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.IMD", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPB", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPB", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPB", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPB", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPB", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPA", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPA", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPA", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPA", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.RPA", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 6
[pid 26658] access("/home/me/foo.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.omd", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_e0.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif.ovr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo_vertices.kwl", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.hdr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/vdf_dat.001", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/METADATA.DIM", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/metadata.dim", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.ppj", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.hdr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif.hdr", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("foo_e0.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 26658] lstat("/home/me", {st_mode=S_IFDIR|0700, st_size=12288, ...}) = 0
[pid 26658] lstat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] stat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26658] access("/home/me/product.xml", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/PHRDIMAP.XML", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 5
[pid 26658] access("/home/me/foo.geom", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/METADATA.DIM", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/metadata.dim", F_OK) = -1 ENOENT (No such file or directory)
[pid 26658] access("/home/me/foo.tif", F_OK) = 0
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 4
[pid 26658] openat(AT_FDCWD, "/home/me", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
[pid 26658] openat(AT_FDCWD, "/home/me/foo.tif.aux.xml", O_RDONLY) = 5
skipgeom=true
makes things better:
[pid 26739] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOTDIR (Not a directory)
[pid 26739] stat("/home/me/foo.tif", {st_mode=S_IFREG|0644, st_size=69117019, ...}) = 0
[pid 26739] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 3
[pid 26739] openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 3
[pid 26739] openat(AT_FDCWD, "/home/me", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
[pid 26739] openat(AT_FDCWD, "/home/me/foo.tif.aux.xml", O_RDONLY) = 4
GDAL is even better:
openat(AT_FDCWD, "/home/me/foo.tif", O_RDONLY) = 3
openat(AT_FDCWD, "/home/me", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
openat(AT_FDCWD, "/home/me/foo.tif.aux.xml", O_RDONLY) = 4
This can be very painful. We have a file system (OpenStack Swift) where simply opening a file takes a couple of seconds. This is a good run:
otbcli_ReadImageInfo -in 1.60s user 0.07s system 15% cpu 10.723 total
Steps to reproduce
strace -f -e trace=file otbcli_ReadImageInfo -in ~/foo.tif > /dev/null
Configuration information
OTB 6.6
Edited by Laurențiu Nicola