Improve user experience (!9) · Merge requests · Nicolas Narçon / pyotb

The source project of this merge request has been removed.

Vincent Delbar requested to merge (removed):develop into master Feb 14, 2022

This PR regroup several new improvements regarding user experience, especially if using an interactive environment.
I actually tried to test this as much as I could, but you really need a test module for this ^^
Everything seems ok for App, Input, Operation, in-memory...
Not sure about the Execute*() stuff though. I've commented 2 lines and it still looks OK with my new execute() method,
but may be you should be careful about that.

Changelog:

Module

Encoding declaration
Add numpy requirement
Improve readability when possible (alias otbApplication to otb, return early to avoid deep loops or condition tree, etc...)
Fix bug with subprocess call to list apps, raise different Exceptions, do not try to subprocess if not in interactive mode since it will fail
Catch errors better, avoid bare Exception, warn user and exit if OTB can't be found
Logger should log any error message, while providing user with information regarding context
User can set its own logging.basicConfig, just need to configure it before importing pyotb then declare logger var
Add script to remove fstrings from python code

apps.py

Edit code apps.py in order to avoid loading every functions and variables into pyotb namespace (use functions and _variable)
Apps created in apps.py are classes, not just constructor functions

core.py

Add App argument to make OTB stdout go silent
Allow user to set_parameters after object instantiation
Allow user to set a custom name property that will be displayed in logs instead of appname bm = BandMath() ; bm.name = "MyBandMath"
Break large function set_parameters into several, use class private methods
App set_parameters does not execute every time it is called (only if output parameter keys where passed to __init__)
Add App attribute parameters to store kwargs
Add App finished attribute to control if app has ran with success
Add App class functions to control workflow : execute, clear(memory, parameters), find_output

tools.py

To store helpers / general functions like find_otb

Module init

Import

Before :

In [1]: import pyotb
ModuleNotFoundError: No module named 'otbApplication'

After :

In [1]: import pyotb
2022-02-14 20:59:21 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=None
2022-02-14 20:59:21 (INFO) [pyOTB] : Searching for it...
2022-02-14 20:59:21 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 20:59:21 (INFO) [pyOTB] : Using OTB in /opt/otb
2022-02-14 20:59:22 (INFO) [pyOTB] : Successfully loaded 129 OTB applications

This will actually set OTB_APPLICATION_PATH, GDAL_DATA, PROJ_LIB and PYTHONPATH variables.

Exit if OTB can't be found on system

In [1]: import pyotb
2022-02-14 22:10:28 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 22:10:28 (INFO) [pyOTB] : Searching for it...
An exception has occurred, use %tb to see the full traceback.

SystemExit: Can't run without OTB. Exiting.

Warning to recompile SWIG bindings on fresh install

Before :

In [1]: import pyotb                                                                                                                   
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
(...)
ImportError: libpython3.8.so.rh-python38-1.0: cannot open shared object file: No such file or directory

After :

In [1]: import pyotb
2022-02-14 21:56:18 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 21:56:18 (INFO) [pyOTB] : Searching for it...
2022-02-14 21:56:18 (INFO) [pyOTB] : Found OTB-7.0.0-Linux64
2022-02-14 21:56:18 (INFO) [pyOTB] : Found OTB-8.0.0-Linux64
2022-02-14 21:56:19 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : An error occured while importing Python API
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : It seems like you need to symlink or recompile OTB SWIG bindings
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : Use 'cd /home/vidlb/Applications/OTB-8.0.0-Linux64 ; source otbenv.profile ; ctest -S share/otb/swig/build_wrapping.cmake -VV'

SystemExit: Can't run without OTB. Exiting.

Switch OTB version using env var OTB_ROOT or OTB_DIR

OTB_ROOT=/opt/otb ipython3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pyotb
2022-02-14 21:59:38 (INFO) [pyOTB] : Found OTB Python API in /opt/otb
2022-02-14 21:59:38 (DEBUG) [pyOTB] : Using library from /opt/otb/lib
2022-02-14 21:59:38 (DEBUG) [pyOTB] : /usr/bin/python3 -c 'import otbApplication; print(otbApplication.Registry.GetAvailableApplications())'
2022-02-14 21:59:38 (INFO) [pyOTB] : Successfully loaded 129 OTB applications

Find other OTB versions on system

In [1]: import pyotb
2022-02-14 21:52:50 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 21:52:50 (INFO) [pyOTB] : Searching for it...
2022-02-14 21:52:51 (INFO) [pyOTB] : Found Applications/OTB-7.0.0-Linux64
2022-02-14 21:52:51 (INFO) [pyOTB] : Found Applications/OTB-8.0.0-Linux64
2022-02-14 21:52:52 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 21:52:52 (INFO) [pyOTB] : Using OTB in /home/vidlb/Applications/OTB-8.0.0-Linux64
2022-02-14 21:52:52 (DEBUG) [pyOTB] : Using library from Applications/OTB-8.0.0-Linux64/lib
2022-02-14 21:52:52 (DEBUG) [pyOTB] : /usr/bin/python3 -c 'import otbApplication; print(otbApplication.Registry.GetAvailableApplications())'
2022-02-14 21:52:53 (INFO) [pyOTB] : Successfully loaded 117 OTB applications

--> Try to load OTB bindings or scan system, help user in case of failure, return env variables
    Precedence :                                    OTB_ROOT > python bindings directory
        OR search for releases installations    :    current directory > home directory
        OR (for linux)                          :    /opt/otbtf > /opt/otb > /usr/local > /usr

Application init

Every apps are `App` class children

Before :

In [6]: type(pyotb.BandMath)
Out[6]: function

After :

In [3]: type(pyotb.BandMath)
Out[3]: abc.ABCMeta

Init object without parameters

Before :

In [7]: b = pyotb.BandMath()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
(...)
RuntimeError: Exception thrown in otbApplication Application_Execute: ../Modules/Applications/AppMathParser/app/otbBandMath.cxx:198:
itk::ERROR: BandMath(0x129d360): No input Image set...; please set at least one input image

After :

In [6]: b = pyotb.BandMath()
2022-02-14 22:01:12 (WARNING) [pyOTB] : BandMath: No parameters where provided. Use App.set_parameters() then App.execute()

In [7]: b.set_parameters(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif")

In [8]: b.execute()
2022-02-14 22:01:15 (DEBUG) [pyOTB] : BandMath: run execute() with parameters={'il': ['RGB.tif'], 'exp': 'im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', 'out': 'test.tif'}
Writing test.tif...: 100% [**************************************************] (10s)
2022-02-14 22:01:26 (DEBUG) [pyOTB] : BandMath: execution succeeded
Out[8]: True

Wrong parameter name

Before :

In [11]: b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")
2022-02-14 22:03:16 (INFO): Loading metadata from official product
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
(...)
RuntimeError: Exception thrown in otbApplication Application_GetParameterType: ../Modules/Wrappers/ApplicationEngine/src/otbWrapperParameterGroup.cxx:470:
itk::ERROR: ParameterList(0x1660ca0): Could not find parameter typo

After :

In [15]: b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")                                                                                                                 
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-15-b9613f4f9166> in <module>
----> 1 b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")

(...)

Exception: BandMath: parameter 'typo' was not recognized

Edited Feb 15, 2022 by Vincent Delbar

Improve user experience

Changelog:

Module

apps.py

core.py

tools.py

Module init

Import

Exit if OTB can't be found on system

Warning to recompile SWIG bindings on fresh install

Switch OTB version using env var OTB_ROOT or OTB_DIR

Find other OTB versions on system

Application init

Every apps are App class children

Init object without parameters

Wrong parameter name

Merge request reports

Every apps are `App` class children