Skip to content

Resolve "Release the Python Global Interpreter Lock (GIL) during execution"

Julien Osman requested to merge 2078-release-the-python-global-interpreter-lock-gil-during-execution into develop

Summary

closes #2078

Implementation Details

See https://docs.python.org/3/c-api/init.html#thread-state-and-the-global-interpreter-lock, http://www.swig.org/Doc4.0/Python.html#Python_multithreaded and https://github.com/InsightSoftwareConsortium/ITK/pull/1065

In SWIG file otbApplication.i, setting module's parameter thread to "1" allows application wrappers to release Python GIL before entering wrapped code.

PyCommand has been modified to reacquire the GIL before calling its callback function. Patch taken from ITK : https://github.com/InsightSoftwareConsortium/ITK/commit/3eb1f6a6e59344a756dece628833b0214145057b

PythonLogOutput has not been modified, as setting thread=1 seems sufficient to solve bug #2078 (comment 91988). The difference with PyCommand is that we are using director classes in this case.

However, in the scope of this merge request the patch should be applied.

Additional notes

  1. In ITK, they also add %feature("nothreadallow");. However if we do the same in otbApplication.i it seems to prevent the GIL to be correctly released. In the end, setting thread to "1" seems enough.

  2. PyCommand is not very useful in OTB. It is not used in the wrapper. However it is part of the API, so we should solve the GIL bug in this case.

This class can be used in combination with AddObserver (from itk objects) to call Python code when a specific event is invoked by the object, for example

import otbApplication as otb
app = otb.Registry.CreateApplication("EdgeExtraction")
app.IN = input.tif
app.OUT = output.tif
def Update():
    print("hello")
iterationCommand = otb.itkPyCommand_New()
iterationCommand.SetCommandCallable(Update)
app.AddObserver(MyEvent, iterationCommand.GetPointer())
app.ExecuteAndWriteOutput()

will print "hello" when MyEvent is invoked.

This is useful in ITK where filters are wrapped in python. For example it is possible to print information at each iteration of a filter

In OTB the only wrapped class where this could be used is the Application class. This class does not invoke a lot of event, only AddProcessToWatchEvent in otb::Wrapper::Application, subclasses might invoke event when overriding methods (e.g. in DoExecute), this is never the case for otb Applications. Note that AddProcessToWatchEvent is already observed to trace the progress of the application (see the logger classes).

If someday we want to simplify the SWIG wrapper maybe PyCommand should be removed.

Copyright

The copyright owner is CNES and has signed the ORFEO ToolBox Contributor License Agreement.

Check before merging:

  • All discussions are resolved
  • At least 2 👍 votes from core developers, no 👎 vote.
  • The feature branch is (reasonably) up-to-date with the base branch
  • Dashboard is green
  • Copyright owner has signed the ORFEO ToolBox Contributor License Agreement
  • Optionally, run git diff develop... -U0 --no-color | clang-format-diff.py -p1 -i on latest changes and commit
Edited by Cédric Traizet

Merge request reports