OverflowError in remove_background about tomopy HOT 7 CLOSED

Have you tried with a smaller data array?

from tomopy.

Hello Doga,

The full image size is 2560 by 2160. I initially ran 100 slices.

I did two tests after reading your email:

1. using a set of images with size of 1143 by 1040, 100 slices;

2. using the full image size of 2560 by 2160, but reduce slice number from 100 to 10.

In both cases, the remove_background finished without error!

My situation is: I need to process the full size of the images because the large size of the sample. I also need to correct the image tilt (I am thinking of using "d.correct_tilt") which I assume I have to use a reasonable large number of slices for the angle correction, right? What do you suggest?

By the way, I remember sometime before I was told to use only 180 degree of data. Can we process data with larger rotation angle now?

Thanks a lot.

Rong

From: Doga Gursoy [email protected]
Sent: Thursday, April 9, 2015 9:54 AM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

Have you tried with a smaller data array?

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91238754.

from tomopy.

As I remember, the function requires a lot of memory to process, so unfortunately you can't process all images at once. But you can apply it iteratively.

It's been a while I haven't used or checked correct_tilt. So I can't comment on that much. But I'll try to look into that.

You can sure process data with any angle. Just define your theta correctly.

from tomopy.

Dear Doga,

If I understand you correctly, increase my PC RAM should be helpful for this issue, right?

I guess somewhere tomopy probably changed the way of how the processes were handled. While I use the same PC to process the same data, the older tomopy works fine, but the currently cloned tomopy does not.

Thanks a lot for the help!

Best regards,

Rong

From: Doga Gursoy [email protected]
Sent: Friday, April 10, 2015 5:58 PM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

As I remember, the function requires a lot of memory to process, so unfortunately you can't process all images at once. But you can apply it iteratively.

It's been a while I haven't used or checked correct_tilt. So I can't comment on that much. But I'll try to look into that.

You can sure process data with any angle. Just define your theta correctly.

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91703821.

from tomopy.

Of course, but you may still need iterations for the full dataset. I suggest you to monitor the memory consumption for a specified number of slices. From that you can estimate roughly the required memory for the whole data.

I am not sure exactly what has changed. Maybe multiprocessing could have caused that. You can try remove_background(num_cores=1). It'll take more time but may save you some memory.

from tomopy.

Dear Doga,

Thank you for the very helpful discussions!

Best regards,

Rong

From: Doga Gursoy [email protected]
Sent: Saturday, April 11, 2015 12:58 AM
To: tomopy/tomopy
Cc: Rong Huang
Subject: Re: [tomopy] OverflowError in remove_background (#73)

Of course, but you may still need iterations for the full dataset. I suggest you to monitor the memory consumption for a specified number of slices. From that you can estimate roughly the required memory for the whole data.

I am not sure exactly what has changed. Maybe multiprocessing could have caused that. You can try remove_background(num_cores=1). It'll take more time but may save you some memory.

Reply to this email directly or view it on GitHubhttps://github.com//issues/73#issuecomment-91761255.

from tomopy.

You're welcome Rong.

from tomopy.