RFC: Enable `zstd` support in SuperBuild
What changes will be made and why they would make a better Orfeo ToolBox?
ZSTD is a modern compression format that offers compression ratios comparable with zlib, but faster compression and decompression speeds: https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/. GDAL supports ZSTD since 2.3.
This is a proposal to include libzstd in the SuperBuild package, and enable GDAL's support of it. OTB applications will be able to use ZSTD compression seamlessly, by virtue of GDAL.
Results for a noisy Float32 10980x10980 TIFF, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
|---|---|---|---|---|
| Uncompressed | 482 330 028 b | N/A | N/A | N/A |
| DEFLATE | 419 729 334 b | 4.030s | 0.687s | 1.647s |
| ZSTD-9 | 415 737 011 b | 8.514s | 3.787s | 0.903s |
Results for a 15-bit Int16 10980x10980 TIFF, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
|---|---|---|---|---|
| Uncompressed | 226 122 884 b | N/A | N/A | N/A |
| DEFLATE | 201 822 038 b | 9.292s | 7.615s | 8.064s |
| ZSTD-9 | 207 553 979 b | 13.921s | 7.801s | 7.696s |
Results for a the file above, converted to 16-bit, no predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
|---|---|---|---|---|
| Uncompressed | 241 187 444 b | N/A | N/A | N/A |
| DEFLATE | 177 040 501 b | 1.736s | 0.396s | 0.869s |
| ZSTD-9 | 184 242 872 b | 12.715s | 3.077s | 0.558s |
Results for a the file above, converted to 16-bit, horizontal differencing predictor:
| Format | Size | Compression time | Compression time (MT) | Decompression time |
|---|---|---|---|---|
| Uncompressed | 241 187 444 b | N/A | N/A | N/A |
| DEFLATE | 159 937 804 b | 2.053s | 0.392s | 1.722s |
| ZSTD-9 | 153 871 845 b | 13.487s | 3.058s | 0.735s |
| ZSTD-3 | 154 626 632 b | 1.643s | 0.378s | 0.565s |
Times are best of three runs, taken on my computer, with GDAL from osgeo/gdal:ubuntu-full-3.2.0.
It seems that users would need to be very careful selecting the compression level, because the default is less efficient than DEFLATE-6.
Risks and benefits
- it makes SuperBuild more complex (one more library to keep track of)
- the SuperBuild package will be slightly larger (
libzstd.sois about 678 KB) - files saved with ZSTD will probably be unreadable by most software, but it's not used by default
- it allows users to take advantage of the faster compression and decompression speeds
Alternatives for implementations
Do nothing.
Who will be developing the proposed changes?
Not sure.