pyotb merge requestshttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests2022-01-25T15:34:02Zhttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/1FIX: enable importing pyotb after tensorflow has been imported2022-01-25T15:34:02ZNicolas NarçonFIX: enable importing pyotb after tensorflow has been importedCloses #2
This now works:
```python
import tensorflow
import pyotb
```
Limitations: It is not possible to use any Tensorflow related applications if Tensorflow itself has been imported beforehand in the script:
```python
import tens...Closes #2
This now works:
```python
import tensorflow
import pyotb
```
Limitations: It is not possible to use any Tensorflow related applications if Tensorflow itself has been imported beforehand in the script:
```python
import tensorflow
import pyotb
pyotb.BandMath(...) # this is ok
pyotb.TensorflowModelServe(...) # this will crash
```https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/2FIX: enable importing pyotb on python 3.6 and 3.72022-01-25T09:18:51ZNicolas NarçonFIX: enable importing pyotb on python 3.6 and 3.7Closes #3Closes #3https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/3FIX: for python 3.6 compat, replace capture_output arg by stdout arg2022-01-25T15:41:03ZNicolas NarçonFIX: for python 3.6 compat, replace capture_output arg by stdout argCloses #4Closes #4https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/4Add numpy-inspired functions2022-01-25T16:31:02ZNicolas NarçonAdd numpy-inspired functions2 functions added:
- all() : For only one image, this function checks that all bands of the image are True (i.e. !=0) and outputs
a singleband boolean raster
For several images, this function checks that all images are True (i.e....2 functions added:
- all() : For only one image, this function checks that all bands of the image are True (i.e. !=0) and outputs
a singleband boolean raster
For several images, this function checks that all images are True (i.e. !=0) and outputs
a boolean raster, with as many bands as the inputs
- any() : For only one image, this function checks that at least one band of the image is True (i.e. !=0) and outputs
a singleband boolean raster
For several images, this function checks that at least one of the images is True (i.e. !=0) and outputs
a boolean raster, with as many bands as the inputshttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/5Update install instructions and add numpy-inspired functions2022-01-26T11:30:58ZNicolas NarçonUpdate install instructions and add numpy-inspired functionshttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/6Optimizing arithmetic and logical operations2022-02-16T16:35:51ZNicolas NarçonOptimizing arithmetic and logical operations- Introduce a Slicer class when the user calls something such as raster[:, :, 0]. This class has a dual way of working : it can be an ExtractROI or a BandMath such as im1b1, depending on how the result is used. If the result is written t...- Introduce a Slicer class when the user calls something such as raster[:, :, 0]. This class has a dual way of working : it can be an ExtractROI or a BandMath such as im1b1, depending on how the result is used. If the result is written to disk or integrated in a pipeline, the ExtractROI app will be used. Otherwise, if the result is used in another Operation, we use the BandMath expression for sake of performance.
- Introduce a logicalOperation class that is a specialization of Operation class for boolean logical operations i.e. >, <, >=, <=, ==, !=, `&` and `|`.
- handle native operations (i.e. '+' etc...) with numpy objects
- more efficient pyotb.where function, based on Operation
- more efficient pyotb.any and pyotb.allhttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/7Enable use of object in in-memory pipelines even after it has been written to...2022-02-15T15:39:35ZNicolas NarçonEnable use of object in in-memory pipelines even after it has been written to diskCloses #7Closes #7https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/8Bump version to 1.0.22022-02-11T13:52:24ZNicolas NarçonBump version to 1.0.2https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/9Improve user experience2022-02-16T16:31:05ZVincent DelbarImprove user experienceThis PR regroup several new improvements regarding user experience, especially if using an interactive environment.
I actually tried to test this as much as I could, but you really need a test module for this ^^
Everything seems ok f...This PR regroup several new improvements regarding user experience, especially if using an interactive environment.
I actually tried to test this as much as I could, but you really need a test module for this ^^
Everything seems ok for App, Input, Operation, in-memory...
Not sure about the `Execute*()` stuff though. I've commented 2 lines and it still looks OK with my new execute() method,
but may be you should be [careful about that](https://gitlab.orfeo-toolbox.org/vidlb/pyotb/-/blob/develop/pyotb/core.py#L538).
### Changelog:
#### Module
- Encoding declaration
- Add numpy requirement
- Improve readability when possible (alias otbApplication to otb, return early to avoid deep loops or condition tree, etc...)
- Fix bug with subprocess call to list apps, raise different Exceptions, do not try to subprocess if not in interactive mode since it will fail
- Catch errors better, avoid bare Exception, warn user and exit if OTB can't be found
- Logger should log any error message, while providing user with information regarding context
- User can set its own `logging.basicConfig`, just need to configure it before importing pyotb then declare `logger` var
- Add script to remove fstrings from python code
#### apps.py
- Edit code apps.py in order to avoid loading every functions and variables into pyotb namespace (use functions and `_variable`)
- Apps created in apps.py are classes, not just constructor functions
#### core.py
- Add App argument to make OTB stdout go silent
- Allow user to set_parameters after object instantiation
- Allow user to set a custom name property that will be displayed in logs instead of appname `bm = BandMath() ; bm.name = "MyBandMath"`
- Break large function set_parameters into several, use class private methods
- App set_parameters does not execute every time it is called (only if output parameter keys where passed to `__init__`)
- Add App attribute `parameters` to store kwargs
- Add App `finished` attribute to control if app has ran with success
- Add App class functions to control workflow : `execute`, `clear(memory, parameters)`, `find_output`
#### tools.py
- To store helpers / general functions like `find_otb`
## Module init
### Import
Before :
```
In [1]: import pyotb
ModuleNotFoundError: No module named 'otbApplication'
```
After :
```
In [1]: import pyotb
2022-02-14 20:59:21 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=None
2022-02-14 20:59:21 (INFO) [pyOTB] : Searching for it...
2022-02-14 20:59:21 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 20:59:21 (INFO) [pyOTB] : Using OTB in /opt/otb
2022-02-14 20:59:22 (INFO) [pyOTB] : Successfully loaded 129 OTB applications
```
This will actually set OTB_APPLICATION_PATH, GDAL_DATA, PROJ_LIB and PYTHONPATH variables.
### Exit if OTB can't be found on system
```
In [1]: import pyotb
2022-02-14 22:10:28 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 22:10:28 (INFO) [pyOTB] : Searching for it...
An exception has occurred, use %tb to see the full traceback.
SystemExit: Can't run without OTB. Exiting.
```
### Warning to recompile SWIG bindings on fresh install
Before :
```
In [1]: import pyotb
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
(...)
ImportError: libpython3.8.so.rh-python38-1.0: cannot open shared object file: No such file or directory
```
After :
```
In [1]: import pyotb
2022-02-14 21:56:18 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 21:56:18 (INFO) [pyOTB] : Searching for it...
2022-02-14 21:56:18 (INFO) [pyOTB] : Found OTB-7.0.0-Linux64
2022-02-14 21:56:18 (INFO) [pyOTB] : Found OTB-8.0.0-Linux64
2022-02-14 21:56:19 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : An error occured while importing Python API
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : It seems like you need to symlink or recompile OTB SWIG bindings
2022-02-14 21:56:19 (CRITICAL) [pyOTB] : Use 'cd /home/vidlb/Applications/OTB-8.0.0-Linux64 ; source otbenv.profile ; ctest -S share/otb/swig/build_wrapping.cmake -VV'
SystemExit: Can't run without OTB. Exiting.
```
### Switch OTB version using env var OTB_ROOT or OTB_DIR
```
OTB_ROOT=/opt/otb ipython3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.31.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import pyotb
2022-02-14 21:59:38 (INFO) [pyOTB] : Found OTB Python API in /opt/otb
2022-02-14 21:59:38 (DEBUG) [pyOTB] : Using library from /opt/otb/lib
2022-02-14 21:59:38 (DEBUG) [pyOTB] : /usr/bin/python3 -c 'import otbApplication; print(otbApplication.Registry.GetAvailableApplications())'
2022-02-14 21:59:38 (INFO) [pyOTB] : Successfully loaded 129 OTB applications
```
### Find other OTB versions on system
```
In [1]: import pyotb
2022-02-14 21:52:50 (INFO) [pyOTB] : Failed to import otbApplication with PYTHONPATH=/home/vidlb/Projets/git/pyotb
2022-02-14 21:52:50 (INFO) [pyOTB] : Searching for it...
2022-02-14 21:52:51 (INFO) [pyOTB] : Found Applications/OTB-7.0.0-Linux64
2022-02-14 21:52:51 (INFO) [pyOTB] : Found Applications/OTB-8.0.0-Linux64
2022-02-14 21:52:52 (INFO) [pyOTB] : Found /opt/otb/lib/otb/
2022-02-14 21:52:52 (INFO) [pyOTB] : Using OTB in /home/vidlb/Applications/OTB-8.0.0-Linux64
2022-02-14 21:52:52 (DEBUG) [pyOTB] : Using library from Applications/OTB-8.0.0-Linux64/lib
2022-02-14 21:52:52 (DEBUG) [pyOTB] : /usr/bin/python3 -c 'import otbApplication; print(otbApplication.Registry.GetAvailableApplications())'
2022-02-14 21:52:53 (INFO) [pyOTB] : Successfully loaded 117 OTB applications
```
```
--> Try to load OTB bindings or scan system, help user in case of failure, return env variables
Precedence : OTB_ROOT > python bindings directory
OR search for releases installations : current directory > home directory
OR (for linux) : /opt/otbtf > /opt/otb > /usr/local > /usr
```
## Application init
### Every apps are `App` class children
Before :
```
In [6]: type(pyotb.BandMath)
Out[6]: function
```
After :
```
In [3]: type(pyotb.BandMath)
Out[3]: abc.ABCMeta
```
### Init object without parameters
Before :
```
In [7]: b = pyotb.BandMath()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
(...)
RuntimeError: Exception thrown in otbApplication Application_Execute: ../Modules/Applications/AppMathParser/app/otbBandMath.cxx:198:
itk::ERROR: BandMath(0x129d360): No input Image set...; please set at least one input image
```
After :
```
In [6]: b = pyotb.BandMath()
2022-02-14 22:01:12 (WARNING) [pyOTB] : BandMath: No parameters where provided. Use App.set_parameters() then App.execute()
In [7]: b.set_parameters(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif")
In [8]: b.execute()
2022-02-14 22:01:15 (DEBUG) [pyOTB] : BandMath: run execute() with parameters={'il': ['RGB.tif'], 'exp': 'im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', 'out': 'test.tif'}
Writing test.tif...: 100% [**************************************************] (10s)
2022-02-14 22:01:26 (DEBUG) [pyOTB] : BandMath: execution succeeded
Out[8]: True
```
### Wrong parameter name
Before :
```
In [11]: b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")
2022-02-14 22:03:16 (INFO): Loading metadata from official product
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
(...)
RuntimeError: Exception thrown in otbApplication Application_GetParameterType: ../Modules/Wrappers/ApplicationEngine/src/otbWrapperParameterGroup.cxx:470:
itk::ERROR: ParameterList(0x1660ca0): Could not find parameter typo
```
After :
```
In [15]: b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-15-b9613f4f9166> in <module>
----> 1 b = pyotb.BandMath(il='RGB.tif', exp='im1b1*0.2989 + im1b2*0.587 + im1b3*0.114', out="test.tif", typo="true")
(...)
Exception: BandMath: parameter 'typo' was not recognized
```https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/10Draft: Resolve "Regressions in Operation behavior"2022-02-17T10:09:15ZNicolas NarçonDraft: Resolve "Regressions in Operation behavior"Closes #8Closes #8Nicolas NarçonNicolas Narçonhttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/11Resolve "Regressions in Operation behavior"2022-02-17T10:10:48ZNicolas NarçonResolve "Regressions in Operation behavior"Closes #8Closes #8https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/12Fix `set_parameters` function2022-02-17T15:19:24ZNicolas NarçonFix `set_parameters` functionCloses #10Closes #10https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/13ADD: .gitlab-ci.yml2022-03-15T10:20:23ZRémi CressonADD: .gitlab-ci.ymlCloses #9Closes #9https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/14ENH: add the ability to propagate pixel types for App. Add propagation as...2022-02-24T08:47:40ZNicolas NarçonENH: add the ability to propagate pixel types for App. Add propagation as...Closes #11
Changelog:
- add the ability to propagate pixel types for App, as optional argument `propagate_pixel_type`. Add propagation as default behavior for Slicer and Input
- add a `to_numpy(propagate_pixel_type=False)` methods for ...Closes #11
Changelog:
- add the ability to propagate pixel types for App, as optional argument `propagate_pixel_type`. Add propagation as default behavior for Slicer and Input
- add a `to_numpy(propagate_pixel_type=False)` methods for all pyotb objects. This method is also called by `np.asarray()`
- fix the order when using slicing for ROI selection. Before, pyotb was following numpy convention, i.e. `obj[rows, cols]`. Now, pyotb follow the `obj[x, y]` which is more adequate for geoghraphic selection
- fix the `pyotb.get_pixel_type` functionhttps://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/15Resolve "Warning with pyotb.Input and propagating pixel type for unint8"2022-03-02T17:27:44ZNicolas NarçonResolve "Warning with pyotb.Input and propagating pixel type for unint8"Closes #13Closes #13https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/16Resolve "Improve OTBTF compatibility"2022-03-15T16:14:44ZNicolas NarçonResolve "Improve OTBTF compatibility"Closes #12Closes #12https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/17FIX: only set OTB_TF_NSOURCES if there is a valid source in args2022-03-16T07:16:18ZNicolas NarçonFIX: only set OTB_TF_NSOURCES if there is a valid source in argsCloses #14Closes #14https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/18Enable to set number of sources for all OTBTF applications2022-04-01T09:49:54ZNicolas NarçonEnable to set number of sources for all OTBTF applicationsCloses #15Closes #15https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/19Enable to access all scalar attributes of an application + better handle vect...2022-04-01T09:47:04ZNicolas NarçonEnable to access all scalar attributes of an application + better handle vector outputsCloses #16
Also, improvement for applications with vector outputs: Beforehand using `pyotb.AppName(..., out='output.shp')` was not writing output.shp. Now ExecuteAndWriteOutput is called for any output, not only rasters.Closes #16
Also, improvement for applications with vector outputs: Beforehand using `pyotb.AppName(..., out='output.shp')` was not writing output.shp. Now ExecuteAndWriteOutput is called for any output, not only rasters.https://gitlab.orfeo-toolbox.org/nicolasnn/pyotb/-/merge_requests/20ENH: subprocess is working even if non-interactive2022-04-04T07:27:57ZVincent DelbarENH: subprocess is working even if non-interactive