Silent crash when using insert to create a large image about pyvips HOT 3 CLOSED

Hello Emil,

The problem is that on Windows you can't have more than 2048 files open at once, unfortunately. I think you have two choices:

1. Build your image as a series of passes. You could make your 90k x 40k image and paste 2000 images in, then save the output. Next, close Python, restart it (to make certain all 2000 files have been closed), load the image back, and paste another 2000 in. Once all 7000 are there, save in your output format. I would use vips images (.v) for the intermediates: it's a very fast format which can be loaded on demand.

2. Install linux in a VM and build the image there. I've built 30,000 image mosaics without issue.

There can also be issues with C stack overflow. libvips uses a recursive algorithm to work out which pixels to calculate next, and for very long pipelines this can overflow and cause a crash. Again, processing in chunks of 2000 at once should fix this.

from pyvips.

I tried writing in chunks and it works. Thanks! However, I would like to add some notes:

• I did not need to restart python between writing chunks.
• I had to use a chunk size of 500. If I used a larger chunk size, the first chunk would successfully be written but the program would exit soon after.
• Apparently I had a max of 512 open files. Changing it to 2048 did not allow me to increase the chunk size.

from pyvips.

I'm puzzled by your 512 limit, libvips is supposed to raise it for you. Anyway, I'm glad you got it working.

from pyvips.