Pyvips | Best way to create large histology image of size 5000000X5000000 pixels. about pyvips HOT 43 CLOSED

Hello @shivamchaubey,

Wow, that's a big image. If you want to paste many small images together at arbitrary positions, use insert. You will find there is a limit of about 2,000 images, set by the C stack size. You will probably have to assemble your image in sections.

If you can place the images in a grid, use arrayjoin. It can join at least 30,000 images at once.

composite is useful if your images have transparency. If you use it for images without an alpha channel, each image will just overwrite the previous one.

Yes, libvips has stuff for NCC-style matching, both 0 and 1st order. You make a series of pairwise joins along edges. It may not work for you, you'd need to give more information.

from pyvips.

Hello again, could you make a new issue? It's best to have one question per issue or other people will have trouble finding answers.

Include a complete runnable program and notes on how to make input data.

from pyvips.

1. Why embed and composite is not working in my case? I have given different x,y coordinate. Both images should be visible as they are not overlaying on each other. Also, I want gradient blending at the overlap of images, how this can be achieved?
2. Can you provide some example for insert function, I have tried but stuck in the arguments.
3. I have used arrayjoin it works fine, but now I want to increase the accuracy of the mosaic image by global optimization. So I need to locate all the images in large size image.
4. Is this NCC-style matching available in pyvips( for python), images are overlapping. What kind of information do you need? Is there any example available?
   Is there any website where I can find some examples of pyvips, I have gone through pyvips official website but sometimes I find hard to understand arguments.

from pyvips.

You need to give more information, but as I said my guess would be that your images have no alpha channel (they are JPG images, is that right?) so when you put one on top of the other, the top image hides the one below.

If you want to blend the edges, then composite would best. Add an alpha channel to each image and use gaussblur to feather the edges.

The main documention is on the libvips website:

http://jcupitt.github.io/libvips/API/current/

http://jcupitt.github.io/libvips/API/current/libvips-mosaicing.html

Composite will become rather slow if you have many thousands of images -- it was not really designed for huge image stacks. How many are you planning to assemble? It might need some development work to improve the speed in this case.

The NCC stuff in libvips is designed for doing pairwise joins, it sounds like you want to do something fancier with perhaps global optimisation of image positions. You could use something like fastcor:

http://jcupitt.github.io/libvips/API/current/libvips-convolution.html#vips-fastcor

to do the correlation, but the optimisation step would be up to you. Do you have a global frame of reference or are you hoping to derive that from a the pairwise correlations? How many images are you hoping to join? What size are they? What sort of transform do you need to do to match them? What platform are you using? What are your memory, disc and time constraints?

from pyvips.

Yes, Images are jpg of resolution 1280X720. Yes when I overlay the images on one another, top one hide the one below. I have to add aplha channel.

Ok, I will try libvips documentation, Is this implemented on C? Is bindings of livips available for pyvips, I want to use python, so how i can integrate libvips with python.

I want to assemble 500X500 images in a grid of resolution 1280X720 pixels.

I will use global reference but deriving the relative coordinates from pairwise correlation and using weighted least square optimization. Currently, I am using only geometrical transformation (X,Y) shift. I am figuring out the effect of a change in the field of view as I am using autofocus to capture the images, it might change the field of view of images. If it is affecting then I have to rescale the images too. I am using ubuntu 16.04, python2.7, opencv3.3, pyvips, skimage, scipy and other libraries. Currently I am developing in 16GB RAM but I want to run this application in 8GB RAM, 200 GB Hard disk, as of now time constraint is not important. I have stitched 156 images in 3.2 sec. Accuracy and handling a large number of images is my main priority for now.

from pyvips.

Yes, that describes the C API, but pyvips is just a very thin skin over that, so you can use the same docs.

Your project sounds interesting. I guess this is with a motorized microscope stage? Yes, you will need to lock the focus, I think. You might find you need to scale and rotate the tiles too, it depends a bit on the quality of your stage.

You will need to correct lighting too -- perhaps you are doing this already. You will need to do lighting correction in linear light. You will probably also need to correct for scatter in the optics.

You can use merge to join the frames:

http://jcupitt.github.io/libvips/API/current/libvips-mosaicing.html#vips-merge

It joins a pair of images with an x/y displacement and a feathered join. I would proceed like this:

1. Lay out your images in a 500x500 grid with your expected overlap.

2. Extract all the edge overlaps and use NCC to estimate the displacement for each join. You'll need a confidence rating for each estimated displacement -- libvips has quite a fancy thing for this, but unfortunately it's not available from pyvips. I should fix this.

3. Use weighted LMS to find a set tile positions that minimise the errors across the whole image mosaic.

4. For each row, make a series of 499 left-right joins, translating the global x/y positions from 3. into a local displacement.

5. Join up all rows with 499 top-bottom joins.

6. Write as a DZI.

from pyvips.

Actually you can get the results from the libvips edge analyzer in pyvips. Try this:

#!/usr/bin/python
  
import sys
import pyvips

# you can set the size of the window as well, but 10x10 pixels (the default) is
# probably fine

left = pyvips.Image.new_from_file(sys.argv[1])
right = pyvips.Image.new_from_file(sys.argv[2])

# the expected overlap, or the number of pixels the two images have in common
overlap = int(sys.argv[3])

# harea is the maximum displacement we detect, in pixels ... this is half
# the size of the area searched

# bandno is the band we search in ... green (band 1) is best for JPG images,
# since it has double the resolution

# dx0/dy0 are optional output args ... we set them True and pyvips will return
# their rvalues in result

join, optional = left.mosaic(right, 'horizontal',
                             left.width - overlap, 0, 0, 0,
                             harea=15, bandno=0,
                             dx0=True, dy0=True)

print 'horizontal offset =', optional['dx0']
print 'vertical offset =', optional['dy0']

print 'writing join to x.png ...'
join.write_to_file('x.png')

This is doing a 0th order (ie. just translation) join of two images left-right. It has a relatively robust thing for finding the overlap:

1. Split the overlap area into three sections, top middle and bottom.

2. Search the reference (left) edge for tie-points (high contrast areas) and collect the top 20 in each zone.

3. For each tie-point, search the corresponding area on the right, assuming the start overlap.

4. Put all the best match displacements into a model, find the best translation, and discard the worst pair of tie-points.

5. Repeat 4. until either we have a translation which fits all tie-points well, or we don't have enough tie-points left.

6. Do a feathered join, return the found best displacement.

With this pair of test images:

I see:

john@yingna ~/try $ ./find_overlap.py cd1.1.jpg cd1.2.jpg 100
horizontal offset = -426
vertical offset = 0
writing join to x.png ...

You don't need the joined images, so remove the write_to_file and it won't be generated.

In your LMS optimisation, you'll need to have something to handle missing displacements where the edges just didn't have enough information.

from pyvips.

Those test images need radiometric correction, obviously.

from pyvips.

Hello @jcupitt,
Thanks for the help. I will try all the methods you mentioned.

libvips has quite a fancy thing for this, but unfortunately it's not available from pyvips. I should fix this.

when will it be fixed?

from pyvips.

That example Python program does this, so it is available after all.

from pyvips.

Hi @jcupitt ,
I have checked your code:
join, optional = left.mosaic(right, 'horizontal', left.width - overlap, 0, 0, 0, harea=15, bandno=0, dx0=True, dy0=True)
It is working fine but not very robust.

I want to ask about merge function: out = ref.merge(sec, direction, dx, dy, mblend = int)
I want to create a reference image of size 1000X1000. Now I want to locate another secondary image of size 400X400 on top of a reference image. Let's say dx=100,dy=100, which is a left-top corner of the secondary image with respect to a top-left corner of the reference image (origin). How this can be achieved I have tried but getting different result with different setting.

Code I have tried:
output_size=1000 ref=pyvips.Image.black(output_size_x, output_size_y, bands=3) sec = pyvips.Image.new_from_file('image.jpg', access="sequential") output = ref.merge(sec, 'horizontal',100, 100, mblend = 100) output.write_to_file('output.tif')

I have tried replacing ref with sec and vice versa and dx=+/-dx,dy=+/-dy but I am not getting desired result. May be I have to add alpha channel ? If it is how I can proceed ?
What is the use of horizontal and vertical argument? dx & dy will provide enough information for shifting of an image.
How mblend parameter can be used?

from pyvips.

Hello again,

The mosaic and merge operations are for joining pairs of images left-right or up-down. They assume that there is a small (100 pixel?) overlap along the seam, and they blend along the midpoint of that line. They were originally written for a system with a fixed camera in front of a motorized easel, or a moving camera plus lighting. They should work for things like motorized microscope stages, or camera scan backs.

It sounds like you need something very different. Can you explain your setup?

from pyvips.

mblend is working fine, I am giving overlap region and the result is good. Just I need to locate any small image in a big image by giving coordinate x,y in reference to the top-left corner of big image( reference image). I want to use this reference image as a background image and overlay all the images giving x,y coordinates. How this can be done I am having RGB jpg images ?
My setup is having stepper control stage in X,Y,Z direction . Precision is +/- 40 pixels in X,Y direction. I am able to find the exact location of all the images just I need to overlay on one another.

from pyvips.

I don't think a reference image will work very well. I would test every overlap and estimate the displacement at each one plus a confidence, then use LMS to find a set of tile positions which minimise error.

Once you have the final xy position of every tile, join up a set of rows with left-right merge, then join the rows with top-bottom merge.

from pyvips.

I have done this part

I don't think a reference image will work very well. I would test every overlap and estimate the displacement at each one plus a confidence, then use LMS to find a set of tile positions which minimize error.

Just I need to overlay the images using a global coordinate system. The reason I want to use global coordinate is due to the constraint. I can't

join up a set of rows with a left-right merge, then join the rows with a top-bottom merge.

if I do so then I will have some black pixels at merging location of a top-bottom row because the first row will have some black pixels wherever there is a shift in a y-direction, the second row will have the same. So, when I will merge this top-bottom rows there will be some black pixels at merging line, and I don't want to crop.
The easiest way to overlay images on any big background image giving its coordinate referenced to the top-left corner (0,0) of a big background image.

from pyvips.

The merge operations use 0 to mean transparent and put 0 pixels in the exposed areas. You should be fine if there are vertical shifts.

You might want to add one to all pixels before merging to make sure there are no accidental zeros in your input images.

from pyvips.

I have done the top-bottom stitching just for reference. https://imgur.com/a/PDKeMcj please visit to see the image. I have not matched top and bottom row, just I want to show how black pixels will affect the image.
Code I have used:

im1 = pyvips.Image.new_from_file("check_2_0.tif", access="sequential")
out = im1.merge(im1, 'vertical',0, 100, mblend =500)
out.write_to_file('out1.tiff')

from pyvips.

What is wrong in doing this?

column=5
row=5
y=500
x=500
output_size_x=column*480
output_size_y=row*640
ref=pyvips.Image.black(output_size_x, output_size_y, bands=3)
sec = pyvips.Image.new_from_file("small_image.jpg", access="sequential") 
output = ref.merge(sec, 'vertical',-x, y, mblend = 100)
output.write_to_file('output.tif')

Parameters:

sec (Image) – Secondary image
(str) – Horizontal or vertcial merge 
dx(int) – Horizontal displacement from sec to ref 
dy(int) – Vertical displacement from sec to ref
mblend (int) – Maximum blend size

Here I assume Horizontal displacement from sec to ref -x (right to left side) , vertical displacement from sec to ref +y (bottom to top).
I am getting error:

Error: unable to call merge im_tbmerge: too much overlap

If I change x,y to +ve/-ve sign I get the image but the location is not correct, I am not getting a secondary image at (500,500) location.

If this function can't be used in my case what else can be used. insert, embed, composite ?

from pyvips.

I have tried to use insertfunction. Change my RGB image to alpha channel image. I am getting black image. How can this be done and what is the use of expand argument?

output_size = 2000
tile1 = pyvips.Image.new_from_file('1805_corrected.jpg', access="sequential")
tile1 = tile1.bandjoin(255)
bg = pyvips.Image.black(output_size, output_size, bands=4)
bg.insert(tile1, 500, 500,expand =100,background=[0,0,0,0])
bg.write_to_file('bg.tif')

When I check tile1.band it is 1. It should be 4 band ?

from pyvips.

Could you upload some test images somewhere, perhaps a 5x5 set? I'll join them for you as an example.

from pyvips.

As I said, the merge functions join pairs of images along a seam, you can't use them to put one image inside another.

Insert will do this, but it won't blend. expand is a boolean flag that sets whether it outputs the union or the bounding box of the input pixels.

https://jcupitt.github.io/libvips/API/current/libvips-conversion.html#vips-insert

Composite will do it and can be made to blend, but it will be rather slow.

If you upload some sample images, I'll join them for you to show how it works.

from pyvips.

I have uploaded a folder of 5 columns X 4 rows. https://drive.google.com/file/d/1OxoXAHIDV3CpBv6l6zsCNcvizDTpECd_/view

Yes, composite will be slow and expensive operation. It will place all the images on background image which will be very large and then each large images will be used to composite.
What would be a better way to do it?

from pyvips.

I have tried insert function but getting a black image. I will insert blending image, so I don't require to blend using this function. Just I need to place at the right location. I think the problem is a band.
Can you please show how to use insert function using given images?

Same happening with blend and composite function.
Code:

output_size=2000
tile1 = pyvips.Image.new_from_file('1805_corrected.jpg', access="sequential")
# tile1 = tile1.bandjoin(255)
# tile2 = tile2.bandjoin(255)
vips_image2 = tile1.embed(500, 500, output_size, output_size)   
vips_image1 = tile1.embed(0, 0, output_size, output_size)   
vips_image=vips_image2.composite(vips_image1,["over"])
vips_image.write_to_file('check01.jpg')

I have checked other arugments replcaing over given in vips document but I am getting only image at location (0,0), second image ( vips_image2) at location (500,500) is not shown in the resultant output.

from pyvips.

i joined up your mosaic:

The lighting seems to change a bit. I used globalbalance to automatically fix it:

I did the mosaic in nip2, but it would be the same in pyvips if you know the tile positions.

Here's the nip2 workspace that made that mosaic:

http://www.rollthepotato.net/~john/mosaic.ws

from pyvips.

So: use left-right merge to make a set of rows, then join the rows top-bottom. It should work fine.

from pyvips.

You can get help with the vips command-line interface:

john@kiwi:~/pics$ vips insert 
insert image @sub into @main at @x, @y
usage:
   insert main sub out x y [--option-name option-value ...]
where:
   main         - Main input image, input VipsImage
   sub          - Sub-image to insert into main image, input VipsImage
   out          - Output image, output VipsImage
   x            - Left edge of sub in main, input gint
			default: 0
			min: -10000000, max: 10000000
   y            - Top edge of sub in main, input gint
			default: 0
			min: -10000000, max: 10000000
optional arguments:
   expand       - Expand output to hold all of both inputs, input gboolean
			default: false
   background   - Color for new pixels, input VipsArrayDouble
operation flags: sequential 

So with these test images:

Entering:

john@kiwi:~/pics$ vips insert Gugg_coloured.jpg cramps.png x.png 500 100

Makes:

You can see the Cramps PNG has been clipped to the size of the background image. If you use expand, you get all pixels. Use background to set the colour of any new pixels (default black).

john@kiwi:~/pics$ vips insert Gugg_coloured.jpg cramps.png x.png 500 100 --expand --background 128

from pyvips.

Thanks!
I understand what you want to explain. Can you please check this image https://imgur.com/a/npfndvJ . Just merge this image vertical with the same image. How the black seam between this two images can be removed?
I have tried the code below but getting a black seam.

I have done the top-bottom stitching just for reference. https://imgur.com/a/PDKeMcj please visit to see the image. I have not matched top and bottom row, just I want to show how black pixels will affect the image.
Code I have used:

im1 = pyvips.Image.new_from_file("check_2_0.tif", access="sequential")
out = im1.merge(im1, 'vertical',0, 100, mblend =500)
out.write_to_file('out1.tiff')

from pyvips.

The dx / dy are the vector from the origin of the bottom image to the origin of the top image. You need to swap the sign.

vips -- merge x.png x.png y.png vertical 0 -600

from pyvips.

Yes. this is working in the terminal. But in python, this code is not working. Can you please figure what's the problem ? I have changed mblend parameters with different integers but getting the same result.

im1 = pyvips.Image.new_from_file("check_2_0.tif")
out = im1.merge(im1, 'vertical',0, 600, mblend =-1)
out.write_to_file('out1.tiff')

Output:

from pyvips.

I have checked other image. I am not able to figure out the problem. When I swipe the y-coordinate to -ve then black seam is removed.
Black seam code:

im1 = pyvips.Image.new_from_file("check_2_0.tif")
im2 = pyvips.Image.new_from_file("out1_1.tiff") 
out = im1.merge(im2, 'vertical',0, 600, mblend =100)
out.write_to_file('out1_4.jpg')

Result:

After swapping the sign:

im1 = pyvips.Image.new_from_file("check_2_0.tif")
im2 = pyvips.Image.new_from_file("out1_1.tiff") 
out = im1.merge(im2, 'vertical',0, -600, mblend =100)
out.write_to_file('out1_3.jpg')

Result:

What is the problem or where I am doing wrong ?
Where is the origin of an image in pyvips bottom-left corner?
im1 is the reference image so '600' is the +y direction from the bottom-left corner of im1?

from pyvips.

Yes, dx / dy is the displacement from the secondary (bottom) image to the top one, so the signs are reversed. It's a historical accident.

from pyvips.

So, do I have to translate the image in -y direction only? +y is having a seam.

from pyvips.

Yes, don't use +ve y, use -ve y.

from pyvips.

Can you please look at my insert and composite code I have posted?
Test in python why this is not working for me.

from pyvips.

This will composite one image into another with a feathered edge. But I would use the merge functions, they will be much, much faster.

#!/usr/bin/env python

import sys
import pyvips

if len(sys.argv) != 6:
    print "usage: join.py base sub out x y"
    sys.exit(1)

base = pyvips.Image.new_from_file(sys.argv[1], access="sequential")
sub = pyvips.Image.new_from_file(sys.argv[2], access="sequential")
x = int(sys.argv[4])
y = int(sys.argv[5])

# make a feathered alpha channel for sub ... we make a value 255 image with a
# black border the width of the blur radius, then blur it with int precision
white = pyvips.Image.black(sub.width - 15, sub.height - 15) + 255
alpha = white.gravity('centre', sub.width, sub.height)
feather = alpha.cast(sub.format).gaussblur(5, precision="integer")

# add the alpha
rgba = sub.bandjoin(feather)

# expand the rgba image to the size of the main image
big = rgba.embed(100, 100, base.width, base.height)

# and composite them together
out = base.composite(big, "over")

out.write_to_file(sys.argv[3])

from pyvips.

Thanks @jcupitt for your support. I have tried all the function supported by pyvips for image mosaicing and found merge is the fastest one.

from pyvips.

OK, sounds like we're done, I'll close.

from pyvips.

Hi @jcupitt ,

I am using merge function for joining images (4 row X 156 column). When I am exceeding images (120) in a row it's taking lots of ram (>16GB) to save the resultant mosaiced image. Stitching is working fine, I am facing problem in saving dzi or tiff file. What can be done to solve this issue?

Rough Code:

from os import listdir,remove
from os.path import isfile, join
import re
import cv2
import numpy as np 
import matplotlib.pyplot as plt
import pyvips
import time
mypath='../CASE4A_1_focused'
t1=time.time()    
onlyfiles = [ f for f in listdir(mypath) if isfile(join(mypath,f)) ]
images = np.empty(len(onlyfiles), dtype=object)
onlyfiles.sort(key=lambda var:[int(x) if x.isdigit() else x for x in re.findall(r'[^0-9]|[0-9]+', var)])
print onlyfiles, "onlyfiles"
# onlyfiles=onlyfiles[10*156:]
row=4
column=156
x,y=column*10,row*10
x0,y0=column*10,row*10
x,y=100,100
x00,y00=0,0
fx=0.5
output_size_x=column*480+column*10
output_size_y=row*640+row*10
bg = pyvips.Image.black(x0+480,y0+640)
coor=[]
# tiles=[]

for i in range(row):
    row_1=[]
    coor.append([])
    print coor, "coordinates"
    files=onlyfiles[i*column:(i+1)*column]
    if i%2==0:
        files.reverse()
     x,y=100,100   
    for j in range(column-1):
                #stitch images in a row
                im1=cv2.imread( join(mypath,files[j]))
                im2=cv2.imread( join(mypath,files[j+1]))  
                if j==0:
                    tile1 = pyvips.Image.new_from_file(join(mypath,files[j]), access="sequential")
                    bg = pyvips.Image.black(480,640)
                    tile1 = bg.merge(tile1, 'horizontal', -x, -y, mblend =False)
                    coor[i].append([x,y])
#                     tile1.write_to_file('result_black.tiff')
                tile2 = pyvips.Image.new_from_file(join(mypath,files[j+1]), access="sequential")              
                im1_s=cv2.resize(im1,None,fx=fx, fy=fx, interpolation = cv2.INTER_AREA)
                im2_s=cv2.resize(im2,None,fx=fx, fy=fx, interpolation = cv2.INTER_AREA)
                H,val=Stitch(im1_s,im2_s,fx,0)  #returns homography
                warped,im1_coor,im2_coor=warpImages(im1, im2, H) #returns coordinate
                print im1_coor,im2_coor, "im1 coor"
                x1,y1=im1_coor[2],im1_coor[0]
                x2,y2=im2_coor[2],im2_coor[0]
                print x,"xxxxx",x2,"xxxxx22222"            
                x=x+x2-x1
#                 y=y+y2
                y=y+y2-y1
                if y<0:
                    y=y2-y1
                
                print x,y, "xxxxx,yyyy"
                tile1 = tile1.merge(tile2, 'horizontal', -x, -y, mblend =False)
                coor[i].append([x,y])
       
    if i>0:
        #stitch row
        row_tile2=tile1
        print x0,y0,"x000000000 y000000000"
        im1=cv2.imread( join(mypath,prior_files[0]))
        im2=cv2.imread( join(mypath,files[0]))
        im1=np.rot90(im1)
        im2=np.rot90(im2)
        im1_s=cv2.resize(im1,None,fx=fx, fy=fx, interpolation = cv2.INTER_AREA)
        im2_s=cv2.resize(im2,None,fx=fx, fy=fx, interpolation = cv2.INTER_AREA)
        H,val=rowStitch(im1_s,im2_s,fx,0)  #returns homography
        warped,im1_coor,im2_coor=warpImages(im1, im2, H) #returns coordinate
        y1,x1=im1_coor[2],im1_coor[0]
        y2,x2=im2_coor[2],im2_coor[0]
        x_a,y_a=coor[i-1][0][0],coor[i-1][0][0]
        x_b,y_b=coor[i][0][0],coor[i][0][0]
        print im1_coor,im2_coor, "im1 coor"
        print coor[i-1][0],"first coor"
        print coor[i][0],"second coor"
        y_r=y00+y2        
        x_r=x00-x2+x1
        y_r=y_a+y_r-y_b
        x_r=x_a+x_r-x_b
        print x_r,y_r,"vertical coordinates"
        coor[i].append([x,y])
#         print tiles," tiles images"
        if i==1:
            result= row_tile1.merge(row_tile2, 'vertical', -x_r, -y_r, mblend = False)
        else:
            result= result.merge(row_tile2, 'vertical', -x_r, -y_r, mblend = False)
        x00,y00=x_r,y_r
    prior_files=files
    print coor,"coordinates"
    if i==0:
        row_tile1=tile1
print "saving result"        
result.write_to_file('result_stitch.tiff')
# result.dzsave("result")
# dzi_save(file_name)    
print time.time()-t1, "time taken for stitching %i images"%(row*column)

I have checked memory , it's taking very low memory <3GB in stitching images. When it saves the mosaiced image using result.write_to_file('result_stitch.tiff') it takes >16GB for 4X156 images.

from pyvips.

Ok sure, Thanks!

from pyvips.

Can you suggest how globalbalance can be used to remove contrast differences?
I have tried this code but got an error:

import pyvips
img = pyvips.Image.new_from_file("CASE-1A-7.jpg")
img=img.globalbalance()
img.write_to_file("balanced.jpg")

Error:

Error: unable to call globalbalance
im_global_balance: mosaic root not found in desc file
is this really a mosaiced image?

I am using merge function to join the images. Can you give an example how to use this function using merge function ?

from pyvips.

Could you make a new issue, please? It's best to split questions up to make answers easier to find.

from pyvips.

1. method

Hi @shivamchaubey Are you able to do perfect stitching of all the microscope image. i have the same case like you. i have 5000 images taken from digital microscope so i need to stitch them together to create one single image.Please help if you able to achieve it.

from pyvips.

hi @shivamchaubey
can you tell me how you reduce the overlap error between the images or how i can the find the exact overlap between the images with LMS.

from pyvips.