Creating pipes for GWakeup: Too many open files Trace/breakpoint trap (core dumped) about pipeless HOT 4 CLOSED

Yeah, that fixed the RAM overflow issue
Thanks a lot

from pipeless.

Hi @udhay24 ,
Does it work if you increase the default limit?

You can do something like:

ulimit -n 2048

By default, it is in most cases 1024

from pipeless.

Hi @miguelaeh ,
i tried increasing the ulimit and it fixed the issue.

Currently I am using a 15Gb ram and 4 VCPU spec, as i approach the 30+ streams. the RAM reaches the limit and it crashes the application. Do you have a suggested configuration or options i can use to make it efficient

Thanks

from pipeless.

Hi @udhay24 ,
I am glad it fixed it!

What you experience with the RAM is probably because some internal queues may be growing for all those streams, filling the memory with raw frames data. To avoid that you can set --stream-buffer-size <number> when starting pipeless. The number represents the maximum number of frames that can be maintained in memory per stream (note it is the number of frames, if your input frames have a big size you will need to set lower values). For example, if you set it 10 that means that, at maximum, each stream will have a queue of 10 frames to be processed (no matter the size of the frames). The frames that do not fit into that window will be discarded, so your streams will maintain real-time processing without filling the memory.

Finding the best value for your case requires you to play around with it a bit and test a few numbers.

There is an example of how to use the flag here: https://www.pipeless.ai/docs/docs/v1/examples/onnx-yolo-world#start-pipeless

Hope this helps!

from pipeless.