Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
otb
otb
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 273
    • Issues 273
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 8
    • Merge Requests 8
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • Main Repositories
  • otbotb
  • Issues
  • #1817

Closed
Open
Opened Jan 23, 2019 by Stéphane Albert@salbertContributor

Huge Memory-leaks when in-memory chaining OTB-applications from Python

Description

Huge memory consumption when in-memory chaining OTB-Applications within loop from Python code.

Steps to reproduce

There are two processing pipelines defined by chaining OTB-applications in memory from Python code:

  1. create_water_mask:
graph water_mask_maker;
  PAN-->ExtractROI_PAN;
  XS-->ExtractROI_XS;
  ExtractROI_PAN-->SuperImpose;
  ExtractROI_XS-->WaterMaskMaker;
  WaterMaskMaker-->SuperImpose;
  1. create_color_image:
graph color_image_maker;
  PAN-->ExtractROI_PAN;
  XS-->ExtractROI_XS;
  ExtractROI_PAN-->BundleToPerfectSensor;
  ExtractROI_XS-->BundleToPerfectSensor;
  BundleToPerfectSensor-->Convert;

A main processing function runs alternatively those two pipeline on a set of (PAN ; XS) 2-uplets in a (Python) loop (see test_wwm_cim.py)

Memory-leak instrumentation done with vg-mem.sh.

Instrumentation of the WaterMaskMaker C++ OTB-Application

Command: vg-mem.sh ./bin/otbApplicationLauncherCommandLine WaterMaskMaker -progress true -in /home/salbert/data/bench-2/QB_TLSE_MUL_512x512_1024x1024.TIF -out /home/salbert/tmp/QB_TLSE_WMM.tif

Valgrind log: [otbApplicationLauncherCommandLine-20181213-105906.log](/uploads/7beb36ede3bca565a38fd58dc6d19c18/otbApplicationLauncherCommandLine-20181213-105906.log

Small memory-leaks in OSSIM projection and RTTI systems. Same for TIF and JP2000

Instrumentation of the BundleToPerfectSensor OTB-Application

Command: vg-mem.sh ./bin/otbApplicationLauncherCommandLine BundleToPerfectSensor -progress true -inp /home/salbert/data/bench-2/PRIMARY_BUNDLE_P_1024x1024.tif -inxs /home/salbert/data/bench-2/PRIMARY_BUNDLE_MUL_1024x1024.tif -out /home/salbert/tmp/PRIMARY_BUNDLE_BTPS.tif uint16

Valgrind log: otbApplicationLauncherCommandLine-20181220-130819.log

L924, L910: Memory-leaks in pipeline ::UpdateOutputInformation() from https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkImportImageContainer.hxx#L188

L896: Also inquire about leak of process-object instance of https://github.com/InsightSoftwareConsortium/ITK/blob/v4.12.2/Modules/Core/Common/include/itkInPlaceImageFilter.hxx#L38.

Instrumentation of SWIG OTB-Application wrapper

SWIG wrapper of OTB-applications have been instrumented using the otb-app-swig-mem-log.patch trace patch and all instances of OTB-Applications (be they composite or not) have been manually counted).

Command: vg-mem.sh ./test_OTB_application.py (see test_OTB_application.py)

Valgrind log: test_OTB_application_memory.py-20181214-113339.log

Command: vg-mem.sh ./test_OTB_application_memory_JP2.py (test_OTB_application_memory_JP2.py)

Valgrind log: test_OTB_application_memory_JP2.py-20181217-104219.log

There's no memory leak due to incorrect reference-counting of the SWIG wrapper. However:

  1. the reference-counter wrapper implementation [1] is out of date and could be replaced by direct SWIG 3 handling [2]
  2. ImageBaseType (which is a reference-counted instance) is not wrapped, but OTB-Application wrapper provides a ::GetParameterImage() returning a C++ raw-pointer without reference-counting handling, which may cause some memory leaks (see next section about ) in the Python environment, e.g.:
  def main( self ):
      app = ...
      img = app.GetParameterImage( ... ) # img is not reference-counted
      # If not giving img to a wrapped C++ function putting it into an itk::SmartPointer<> will cause a memory-leak which of all image-data.

Instrumentation of full Python sample code

Same memory-leaks as described above are traces. Moreover, the metadata-dictionary is lost when running multiple-times this instrumentation test without deleting the temporary output file.

Command: vg-mem.sh ./test_wmm_cim.py (2 iterations without removing ${HOME}/tmp files)

Valgrind log: test_wwm_cim.py-20181219-161831.log (see also: test_wwm_cim.py-20181217-174610.log L3666 which is related to a suspicious call to ::GetParameterImage() from otb::ImageFileReader<> when initializing pipeline on existing output-data file.)

Configuration information

OTB-6.6 release and ITK-4.12 on Ubuntu 18.04 LTS.

Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: orfeotoolbox/otb#1817