Segmentation fault (core dumped) from pyvips about pyvips HOT 7 CLOSED

Hello again,

You can do a range of things to limit libvips memory use:

1. Don't use resize, use thumbnail instead. It's much faster, needs less memory, and the quality is better.
2. Disable the libvips operation cache. libvips caches the result of the last 1,000 operations, or 100MB of memory, whichever is smaller. Use pyvips.cache_set_max(0) somewhere near the start of your program.
3. Shrink the libvips thread pool. libvips has a pool of worker threads, sized to fill your machine, and each worker has a set of memory buffers. Image shinking is not that CPU intensive, so threads don't really help much anyway. Do export VIPS_CONCURRENCY=1 before running your program.

from pyvips.

I tried this program:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.leak_set(True)

image = pyvips.Image.new_from_file(sys.argv[1], access='sequential')
image = image.resize(0.8 / 1.0)
image.write_to_file(sys.argv[2])

leak_set makes libvips run a little slower, but it will check for leaks and report high-water memory usage. With a 10k x 10k pixel RGB image I see:

john@kiwi:~/try$ ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 53.48 MB

If I change the program like this:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.cache_set_max(0)
pyvips.leak_set(True)

image = pyvips.Image.thumbnail(sys.argv[1], 8000)
image.write_to_file(sys.argv[2])

I see:

john@kiwi:~/try$ VIPS_CONCURRENCY=1 ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 33.82 MB

That's running with ulimit -m 256000, which might be equivalent to your settings. I couldn't get it to crash.

from pyvips.

Sorry, I never answered your first question.

Out of memory errors are generally impossible to handle gracefully. Suppose memory is full and a fault is raised. What is going to handle the error? If no more memory can be used, no fault handler can execute. Like almost all software, libvips treats memory as an infinite resource and relies on the environment to manage it.

libvips does check large single allocations (for example, loading a 10gb image into memory), but does not check small ones, the reasoning being that, if a small allocation fails, the process is as good as dead.

from pyvips.

Thanks a lot. I tried your example code:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.cache_set_max(0)
pyvips.leak_set(True)

image = pyvips.Image.thumbnail(sys.argv[1], 8000)
image.write_to_file(sys.argv[2])

but my output was like:

Traceback (most recent call last):
  File "resize.py", line 20, in <module>
    image = pyvips.Image.thumbnails(sys.argv[1], 800)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/vimage.py", line 128, in call_function
    return pyvips.Operation.call(name, *args, **kwargs)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/voperation.py", line 132, in call
    op = Operation.new_from_name(operation_name)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/voperation.py", line 60, in new_from_name
    raise Error('no such operation {0}'.format(operation_name))
pyvips.error.Error: no such operation thumbnails
  VipsOperation: class "thumbnails" not found

I checked that I had vipsthumbnail command available. So how to use it through python?
What would be the best documentation for these? I saw that thumbnail() is there for pyvips.Image class but not the resize() and vips_thumbnail() is there for libvips API documentation. But how to get all for pyvips?

And where should I use ulimit -m 256000 in python to limit memory?

from pyvips.

  File "resize.py", line 20, in <module>
    image = pyvips.Image.thumbnails(sys.argv[1], 800)

It should be thumbnail, of course, not thumbnails, but I guess you fixed that.

Otherwise it's probably because you are using an old version of the libvips shared library. thumbnail was added in 8.4 I think.

ulimit is a shell command, so run that before you start python. It limits every process started from that shell.

from pyvips.

we are having same issue when creating large image files around 2 gb +

from pyvips.

Hello @abhijitgujar86,

Please open a new issue and include as much information as you can about the crash. Ideally, a reproducible test case.

from pyvips.