Unsupported image format about bimg HOT 13 CLOSED

Seems like there's some cache or memory issue while calling libvips C function vips-foreign-find-load-buffer().

Some libvips debugging could help a lot to clarify which could be the cause, maybe printing some memory/cache stats every minute. Also, some benchmarks could help to dissipate possible unknowns as well (indeed I've plans to do add more soon). You can try running thousands of image conversions in a loop.

Regarding to memory issues, I noticed that some users experimented some inconsistency in long-term running processes due to RSS memory issues, however as far I've tested I haven't detected evident memory leaks in bimg.

I'm currently using bimg through imaginary in multiple Heroku instances and I didn't experiment troubles when massively processing images (300-1000 per day). The unique concern is about dealing with a high concurrency.

I'll try to dig into it in a more detail, but any kind of debug traces you could share would be helpful.

from bimg.

just let me know what exactly do you need - bimg.VipsDebugInfo() will be sufficient enough after it start crashing? or I can make it write to file every n-minutes.

actually i'm running 2 separate processes behind load balancer - both of them crashing in the same time (load is spread evenly) - i'm scaling lot of small images 800x600 > 120x120

running looped conversion is not so easy, because i'm using a lot of cacheing mechanics everywhere. but will try to simulate it on localhost.

from bimg.

Sorry for the delay.

I'll try to make some exploratory fuzz testing during the weekend, however you could try with a small workaround that might mitigate the problem (probably originated by some memory leak).
This is as simple as force to the Go GC to release memory every, let's say, 2 minutes.
Here is the implementation: https://github.com/h2non/imaginary/blob/master/imaginary.go#L128

I've some suspicions about if this could be actually an internal issue of Go GC, since it seems like FFI to C level is generating some leaks and references are not properly detached from the GC index, and therefore not properly those memory regions are not flushed.

As I said, I'll try to dig into this and I'll let you know with the results.

from bimg.

I run some load testing on the system, and was able to push over 5k images without crashing system ( I tested on local build - mac os x, and then vm box with centos 7 - same story. But - same code still crashes on my AWS EC2 instance. I was checking memory - and it's not running out, in a "peak" got around 50% memory free, and no swap usage reported.

What else - tried to to do

bimg.Shutdown()
bimg.Initialize()

after crash, and redo images but also it didn't help. right now will try to run d.FreeOSMemory() after crash, ad se how it goes.

I'm not running it in production yet, so we have time to find issue :)

from bimg.

Hi! I've created a gist with a simple test code trying to reproduce your scenario and detect some kind of inconsistency with bimg. Here are the details:

Environment:

• OSX Yosemite
• libvips 7.42.3
• go 1.4.1
• bimg 0.1.17

Test:

• Running 10k crop operations sequentially
• Using a unique 2.4 MB JPG image

Result:
No error seen. I can't see any performance downgrade during the job.
RSS memory is always under 50MB.

A couple of questions that might help me to dig in detail on this:

• Are you using the same environment as me (specially libvips version)?
• Your bimg process is being called from a goroutine?
• Which level of concurrency you have, if any?

from bimg.

@slav123 Some new about this?

from bimg.

Your testing is not causing problems, but system still crashes after 1-2 days.
I'm running everything on Linux machine with latest vips library.
I figure that problem could be with determining image type format. So patched
this function that is not using vips binding. You can check out my fork on
github. Will let you know how it went.

from bimg.

Thanks for the update.

I saw your fix. The unique limitation checking the first bytes of the image header is that there's no way to detect if libvips already supports TIFF or Magick formats, based on its compilation bindings.
For instance, you can compile libvips with or without libmagick, and using the current way you can detect if a given format is actually supported or not by libvips

IMO this is a sort of workaround. If we're dealing here with a real bug, probably it came from libvips, so we should report it accordingly.

from bimg.

@slav123 Any update about this? Your change mitigated the issue?

from bimg.

So far my patch works correctly. I'm closing the ticket, because maybe it's only "my" special case.

About supporting TIFF or Magick - i strongly suggest to check this on init time, doesn't have to be checked each time when you check file format.

Anyway - thanks for help

from bimg.

Thanks! I'll consider add your patch

from bimg.

Supported images types are internally and statically defined in libvips based on the compilation bindings, so there's no extra significant and inefficient task there. The unique cost in this scenario is about FFI to C side, which is considerably expensive. Probably that could be mitigated checking the first bytes of the buffer in pure Go, as you did.

from bimg.

FYI @slav123 your hack is now supported in the last version of bimg. Let me know if you experiment new inconsistencies. Reference: #49

from bimg.