pyvips dzsave Bus error: 10 on OSX High Sierra with brew python3 about pyvips HOT 10 CLOSED

Hi @jason-green-io, I'd guess a stack overflow, though 140 isn't a very deep pipeline, so I'm slightly puzzled.

You can trim memory use a bit by adding access='sequential' to your new_from_buffer() call, that might help.

You can limit pipeline lengths by rendering intermediate images to memory, or a temporary file, depending on image size. For 20k x 20k, memory would probably be the simplest. How large are your PIL images?

I'll try making you a sample program.

from pyvips.

from pyvips.

I made a test program. This reuses memory buffers between numpy and vips rather than going via PNG, so it's a bit quicker:

import sys
import random
from PIL import Image
import numpy as np
import pyvips

output_size = 20000
bg = pyvips.Image.black(output_size, output_size, bands=4)

# pyvips.Image.new_from_memory doesn't take a ref to memory that py GC can see, 
# so we need to keep refs to the numpy arrays around to prevent them being 
# GCd away
numpy_arrays = []

for filename in sys.argv[2:]:
    pil_image = Image.open(filename)
    numpy_image = np.asarray(pil_image)
    height, width, bands = numpy_image.shape
    linear = numpy_image.reshape(width * height * bands)
    numpy_arrays.append(linear.data)
    vips_image = pyvips.Image.new_from_memory(linear.data,
                                              width, height, bands, 'uchar')
    x = random.randint(0, bg.width - vips_image.width)
    y = random.randint(0, bg.height - vips_image.height)
    vips_image = vips_image.embed(x, y, output_size, output_size)
    bg = bg.composite(vips_image, 'over')
    
bg.write_to_file(sys.argv[1])

I can run it with 200 1k x 1k PNGs like this:

$ mkdir sample
$ vips crop ~/pics/Opera-icon-high-res.png sample/x.png 0 0 1000 1000
$ for i in {1..200}; do cp sample/x.png sample/$i.png; done
$ time python composite.py x.dz sample/*.png

On this laptop running Ubuntu I see 19m runtime, and peak memory 2.8gb. It's a bit high, I guess partly because of all the numpy arrays it's keeping in memory. I didn't try more than 200.

from pyvips.

composite can compose an array of images, which is a bit quicker, since the pipeline will be shorter and it can avoid computing composites that are obscured.

If you change the program to be:

import sys
import random
from PIL import Image
import numpy as np
import pyvips

output_size = 20000
numpy_arrays = []
layers = []

for filename in sys.argv[2:]:
    pil_image = Image.open(filename)
    numpy_image = np.asarray(pil_image)
    height, width, bands = numpy_image.shape
    linear = numpy_image.reshape(width * height * bands)
    numpy_arrays.append(linear.data)
    vips_image = pyvips.Image.new_from_memory(linear.data,
                                              width, height, bands, 'uchar')
    x = random.randint(0, output_size - vips_image.width)
    y = random.randint(0, output_size - vips_image.height)
    vips_image = vips_image.embed(x, y, output_size, output_size)
    layers.append(vips_image)                 
    
final = layers[0].composite(layers[1:], ['over'] * (len(layers) - 1))
final.write_to_file(sys.argv[1])

For 200 1k x 1k PNGs I see runtime 6m20s, peak memory 1.8gb.

Unfortunately, composite in 8.6 is limited to stacks of 64 images, which I thought would be enough for anyone :( I had to up it to 300 for this test. I'll remove the limit.

The ['over'] * (len(layers) - 1) is a bit daft too. It should allow you to pass just a single mode and have it used for all the joins.

from pyvips.

I'll also make new_from_memory keep a ref that the py GC can see, so hopefully the numpy_arrays array can go.

from pyvips.

I fixed the number of images limit, and it now allows a single mode.

pyvips already keeps a ref to the data argument (I thought I remembered something like this), so I'm puzzled by the crash if you remove numpy_arrays.

https://github.com/jcupitt/pyvips/blob/master/pyvips/vimage.py#L380

I'll keep digging.

from pyvips.

Of course, the python GC can't look inside vips pipelines, so it doen't know that the image in layers depends on the memory from numpy. Anyway, this version works fine for me with HEAD of the 8.6 branch:

import sys
import random
from PIL import Image
import numpy as np
import pyvips

output_size = 20000
layers = []
memory = []

for filename in sys.argv[2:]:
    pil_image = Image.open(filename)
    numpy_image = np.asarray(pil_image)
    height, width, bands = numpy_image.shape
    linear = numpy_image.reshape(width * height * bands)
    data = linear.data
    memory.append(data)
    vips_image = pyvips.Image.new_from_memory(data,  
                                              width, height, bands, 'uchar')
    x = random.randint(0, output_size - vips_image.width)
    y = random.randint(0, output_size - vips_image.height)
    vips_image = vips_image.embed(x, y, output_size, output_size)
    layers.append(vips_image)                 

final = layers[0].composite(layers[1:], 'over')
final.write_to_file(sys.argv[1])

Or of course you can do a series of pair-wise composite operations.

from pyvips.

Working on building 8.6 HEAD from source. I've implemented your samples. I'll let you know how I make out. Will the composite limit removal make it into 8.6.2?

from pyvips.

Yes, next week probably. I’ve stumbled across another issue I ought to fix as well.

from pyvips.

OK, 8.6.2 is out, and I've updated pypi to 2.0.5 as well. Thanks for suggesting these improvements!

from pyvips.