Hi,

I am trying to convolve a binary mask with a tiff WSI to blackout certain regions for an experiment. However, I am not able to solve it with pyvips.conv operator.

Masks are gray-scale in png format (1BPP) and 100 times smaller than the original WSI. I am trying to load them as vips object and resizing them to the same size of WSI.

Here is the code:
wsi = vips.Image.tiffload(os.path.join(path_to_dataset, name))
mask = vips.Image.pngload(mask_path).resize(100, vscale=100)
new_wsi = wsi.conv(mask)

Here is the error:
pyvips.error.Error: unable to call conv
conv: matrix image too large

Any suggestions on this?

File this under VIPS "evangelism"...

As well as a closed-source GPU pipeline, Flickr also use their own BitmapFactory-inspired Ymagine open source library.

When alerted to VIPS, specifically the Speed and Memory Use page, Flickr's Head of Engineering @tedd4u said he "would be curious where [Ymagine] fits into that benchmark list ;)".

On the benchmark results test page there is a "Run time (secs real)" column. The time utility shows not only the real time but also user and sys time. Sum of user and sys time is total CPU time spent to the task. You can add this value to the table. I believe this could be useful for many users.

Any chance to add ExactImage to the comparison?

Hi!

Pillow-SIMD 4.3.0 with significant performance improvements is out. Some of improvements are releted to this benchamark.

I modified the test script a bit just to make clear what is happening.

#!/usr/bin/env python
from __future__ import print_function

import sys
from PIL import Image, ImageFilter, PILLOW_VERSION
import time

class Timer:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        self.start = time.time()
        self.cpu = time.clock()
        return self

    def __exit__(self, *args):
        print('>>>', self.name, time.time() - self.start)


# just to confirm we are getting the right version
print("pillow.py: PILLOW_VERSION =", PILLOW_VERSION)

def run():
    with Timer('open'):
        im = Image.open(sys.argv[1])
        im.load()

    with Timer('crop'):
        # Crop 100 pixels off all edges.
        im = im.crop((100, 100, im.size[0] - 100, im.size[1] - 100))

    with Timer('resize'):
        # Shrink by 10%
        im = im.resize((int(im.width * 0.9), int(im.height * 0.9)), Image.NEAREST)

    with Timer('apply filter'):
        # sharpen
        filter = ImageFilter.Kernel((3, 3),
                  (-1, -1, -1,
                   -1, 16, -1,
                   -1, -1, -1))
        im = im.filter(filter)

    with Timer('save'):
        # write back again
        # im.save(sys.argv[2], format='tiff')
        # I don't have pc with SSD for tests, write to memory
        im.tobytes()


with Timer('sum'):
    run()

The difference on my pc (only for example, of course):

$ pip install pillow-simd==4.2.1.post0
Successfully installed pillow-simd-4.2.1.post0

$ time ./pillow.py ./vips.tiff
pillow.py: PILLOW_VERSION = 4.2.1.post0
>>> open 0.106125116348
>>> crop 0.0340759754181
>>> resize 0.027755022049
>>> apply filter 0.59959602356
>>> save 0.0295100212097
>>> sum 0.803848028183

real	0m0.819s
user	0m0.748s
sys	0m0.068s

$ pip install pillow-simd==4.3.0.post0
Successfully installed pillow-simd-4.3.0.post0

$ time ./pillow.py ./vips.tiff 
pillow.py: PILLOW_VERSION = 4.3.0.post0
>>> open 0.0663890838623
>>> crop 0.0178101062775
>>> resize 0.0208790302277
>>> apply filter 0.0534710884094
>>> save 0.0313007831573
>>> sum 0.190583229065

real	0m0.223s
user	0m0.176s
sys	0m0.044s

Aggressive optimizations

Just for example of maximum performance. There are three tweaks and each of them can be used separately:

1. Avoid cropping and resize with box
2. Compile with AVX2
3. Use Pillow's memory pool and custom allocator

Difference in pillow_box_crop.py is:

    with Timer('crop'):
        # Crop 100 pixels off all edges.
        box = (100, 100, im.width - 100, im.height - 100)
        width, height = (im.width - 200, im.height - 200)

    with Timer('resize'):
        # Shrink by 10%
        im = im.resize((int(width * 0.9), int(height * 0.9)), Image.NEAREST, box=box)

# install and recompile with avx2
$ CFLAGS="-mavx2" pip install pillow-simd==4.3.0.post0 --no-cache-dir

# use custom memory allocator and tune memory settings
$ time LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 PILLOW_BLOCK_SIZE=4m PILLOW_BLOCKS_MAX=64 ./pillow_box_crop.py ./vips.tiff
pillow.py: PILLOW_VERSION = 4.3.0.post0
>>> open 0.0655601024628
>>> crop 7.15255737305e-06
>>> resize 0.0184090137482
>>> apply filter 0.035365819931
>>> save 0.0275781154633
>>> sum 0.147360086441

real	0m0.182s
user	0m0.124s
sys	0m0.052s

http://pike.lysator.liu.se/generated/manual/modref/ex/predef_3A_3A/Image/Image/apply_matrix.html

There is a comment in pillow.py:

starting with 2.7, Pillow uses a high-quality convolution-based resize for
BILINEAR ... the other systems in this benchmark are using affine + bilinear,
so this is rather unfair. Use NEAREST instead, it gets closest to what
everyone else is doing

I'm pretty sure that there are other libraries which do convolution resize. For example, ImageMagick and its successor GraphicsMagick. Maybe others.

Also, sips and libgd do supersampling resampling which quality is much closer to Bicubic convolution and complexity close to Bilinear convolution.

Using nearest neighbor only in Pillow unfair to other libraries. While resizing with 4-point bilinear interpolation (like in VIPS) is not suitable for scale factors < 0.5, for 0.9 (used in the test), it much closer (both quality and complexity) to Bilinear convolution rather than to nearest neighbor.

Rather than using nearest neighbor for all of this libraries, I believe it's better to use Bilinear for Pillow and add some note for VIPS, OpenCV and possibly others.

How to check

It's simple. Try to resize with scale factor 0.1. If a zoomed result will look like this, it's convolution or supersampling.

If it looks like this, it's 4-point bilinear.