Skip to content

Strange memory use with applications from Python using Numpy outputs

Description

I noticed two anomalies one anomaly when using applications with Python, and getting the application outputs as Numpy arrays.

  1. Core dumped : Python crashes, when the application instance is deleted but its output still used
  2. Memory leak : Trying to workaround the aforementioned point, I've noticed something that looks like a memory leak: when the application instance is deleted, some memory is still used.

Steps to reproduce

Core dumped

The python code below enables to reproduce this issue.

The read_image() function returns a subset of an image using the ExtractROI application, as a numpy array. When we want to access the numpy array inside the read_image() function, everything is fine. But when we want to do the same thing outside, from the returned numpy array of the function, it crashes.

It looks like when the application is "deleted" (Does that exists in python?) after the read_image() function returns, the numpy array isn't referenced any more.

# -*- coding: utf-8 -*-
import sys
import otbApplication
import numpy as np

tile_size = 100

def disp_stats(np_array):
  print("\tMin:"+str(np.amin(np_array))+", Max:"+str(np.amax(np_array)) )
  
def read_image(fname):
  imageReader = otbApplication.Registry.CreateApplication('ExtractROI')
  imageReader.SetParameterString('in', fname)
  imageReader.SetParameterInt('sizex', tile_size)
  imageReader.SetParameterInt('sizey', tile_size)
  imageReader.Execute()
  outimg = imageReader.GetVectorImageAsNumpyArray('out', 'float')

  print("Compute stats in \"read_image()\"...")
  disp_stats(outimg)
  print("Compute stats in \"read_image()\"... ok")

  return outimg

if __name__ == "__main__":

  if len(sys.argv) != 2:
    print("Usage : <input_image.tif>")
    sys.exit(1)
    
  # Everything works fine
  arr1 = read_image(sys.argv[1])
  
  # Crash
  print("Compute stats in \"__main()__\"...")
  disp_stats(arr1)
  print("Compute stats in \"__main()__\"...ok")

Output:

python bug1.py /path/to/my_image.tif
2019-01-28 09:56:34 (INFO): No kwl metadata found in file /path/to/my_image.tif
Compute stats in "read_image()"...
	Min:0.0, Max:8135.0
Compute stats in "read_image()"... ok
Compute stats in "__main()__"...
Segmentation fault

From here, I found two workaround:

  • Declare the imageReader as global
  • Change the read_image() function to return a np.copy() of the actual application output as numpy array.

In both cases, I noticed a huge memory leak, that's what I am describing in the next point.

Memory leak

I've used psutil to perform some memory monitoring.

The code below show that there is some memory leak inside the read_image() function after the previously described workaround.

An input image of 10000x10000x4 Float32 pixels is used. This image weights approximately 1525 MB in RAM, and that is consistent with the RAM used before/after the np.copy() that copies the output-as-np-array of the ExtractROI application. However, there is a huge extra-memory that isn't released after the read_image() function... The output shows that before the call to the function, there is 5110 MB RAM available, and after the call to the function, only 7456 MB, that makes 2346 MB RAM used to read the 1525 MB Float32 image... where is the extra 2346-1525=821 MB ?

I am not surprised that during the read_image() function, the ExtractROI application consumes more than the output image size (late release of input buffer, etc.). But the memory used at the end (when the application instance should not exist anymore) is strange.

# -*- coding: utf-8 -*-
import sys
import otbApplication
import numpy as np
import time
import psutil
import os

tile_size = 10000
sleep_duration = 10
process = psutil.Process(os.getpid())

# This is wrong!
#def get_ram():
#  mem = psutil.virtual_memory()._asdict()
#  return mem["used"] / (1024*1024)

# This is correct
def get_ram():
  return process.memory_info().rss / (1024*1024)
  
def read_image(fname):
  
  print("imageReader.Execute()...")
  ram_before = get_ram()
  imageReader = otbApplication.Registry.CreateApplication('ExtractROI')
  imageReader.SetParameterString('in', fname)
  imageReader.SetParameterInt('sizex', tile_size)
  imageReader.SetParameterInt('sizey', tile_size)
  imageReader.Execute()
  print("imageReader.Execute()... ok. RAM in use: " + str(get_ram()) + " MB")
  time.sleep(sleep_duration)
  
  print("imageReader.GetVectorImageAsNumpyArray...")
  outimg = imageReader.GetVectorImageAsNumpyArray('out', 'float')
  print("imageReader.GetVectorImageAsNumpyArray...ok. RAM in use: " + str(get_ram()) + " MB")

  ram_after = get_ram()
  print("imageReader.Execute() + imageReader.GetVectorImageAsNumpyArray() used " + str(ram_after-ram_before) + " MB of RAM)")
  time.sleep(sleep_duration)
  
  print("Make a copy...")
  ram_before = get_ram()
  out = np.copy(outimg)
  ram_after = get_ram()
  print("Make a copy...ok. RAM in use: " + str(get_ram()) + " MB (np.copy() used " + str(ram_after-ram_before) + " MB of RAM)")
  mem_mb = outimg.size * 4 / (1024*1024);
  print("It is consistent with the image size " + str(outimg.shape) + " that should use " + str(mem_mb) + " mb of RAM")

  time.sleep(sleep_duration)

  return out
  
def test(fname):
  print("Beginning test. RAM in use: " + str(get_ram()) + " MB")
  ram_before = get_ram()
  arr1 = read_image(fname)
  ram_after = get_ram()
  print("End of test. RAM in use: " + str(get_ram()) + " MB")

  print("read_image() function has used " + str(ram_after-ram_before) + " MB of RAM")
  time.sleep(sleep_duration)

if __name__ == "__main__":

  if len(sys.argv) != 2:
    print("Usage : <input_image.tif>")
    sys.exit(1)
    
  print("Used RAM: " + str(get_ram()) + " MB")
  test(sys.argv[1])

Configuration information

Reproduced this on various Ubuntu like environments (Ubuntu 16.04 LTS and Ubuntu 18.04.1 LTS) and various OTB versions (6.6.1 (da1daa52) and 6.7 (a39cb76d)).

linked to #1817 ?

Edited by Rémi Cresson