ARV_BUFFER_STATUS_TIMEOUT about camera_aravis HOT 10 OPEN

Hey,
I'm facing the same problem. Did you solve it?
Thanks!

from camera_aravis.

Hi @MaxBial,

did you rule out that this might be a bandwidth issue on your network?
We usually increase the mtu size to 9000 on the network card port to avoid any dropped frames (jumbo packets).

How many cameras are you trying to use in parallel?
Here it might be necessary to upgrade to a 10Gbit ethernet switch and 10Gbit network card if all camera streams are coming in over the same network card port to your host PC.

In my experience, we didn't have much success in improving our throughput with the timeout-time and buffer-size parameters.
You could see if setting all cameras with a different packet-delay improves the use of multiple cameras.
This is relevant if all cameras are triggered at the same time.

Let me know if that helped.

from camera_aravis.

Hi @MaxBial,

when do you get the '[ ERROR ] Control to aravis device lost'?

We recently encountered the same error. But in our case, this happens before any subscriber attaches to the node and, thus, before any big data junks are sent over the network. We have tried increasing the Timeout-Time, with no success. We haven't yet resolved the error but we assume that this might have something to do with the routing as the cameras are currently attached to a subnet with a Mesh-Wifi.

from camera_aravis.

@MaxBial

[ WARN ] Frame error: ARV_BUFFER_STATUS_TIMEOUT

I am flooded by those also in the following scenario:

• transport layer (ethernet) MTU was changed (e.g. to 9000)
• camera_aravis mtu was not changed
  • it is possible to observe this effect with dynamic reconfigure of mtu

So if you change one of them also make sure you change the second.

FWIW, I am on different fork but the code that is potentially causing those problems is shared.

Maybe somebody already faced with issue and could tell where in the code or how to change those settings, if this is the issue.

(DFK 33 GX249e - 2,3MP and 48 fps).

Looking at implementation of camera_aravis

• if processing time of buffer takes too much time you will get timeouts

Processing time is mainly:

• pixel format conversion from GiGE-Vision/GenICam to ROS
• ROS publishing
  • if you are pulling data over some compressed image_transport compression time adds up here

To diagnose you may:

• pull data at image_raw if you are using some compressed transport
• comment out pixel format conversion and check if it improves fps

from camera_aravis.

The problem was diagnosed here:

• Extend-Robotics#10 (comment)

At this point this is cause identified, no solution implemented yet.

from camera_aravis.

Back in the day I tried to change in the code the parameters for the "timeout-time" and the 2-3 other parameters at the same position in the code. Unfortunately it had only a small impact. For this reason I had to change to a different Camera with different ROS driver implementation.

The things that I noticed by installing the requirements for the new camera were:
to set "Receive Thread Priority Override"
and
"Increase the ring buffer size using the ethtool command."
and
"Configure the interrupt moderation rate using the ethtool command."

These things were not explicitly required for the camera_aravis driver but I guess maybe it shoulde be?

Best Regards
Max

from camera_aravis.

Transport level optimizations and larger buffers help but here the problem was blocking network communication from running while processing received data.

Solved in a fork today:

• Extend-Robotics#12

This fork is not easy to build in 20.04 + Noetic

• it depends on latest aravis (a bit different API).
  • latest aravis doesn't build trivially from source in 20.04
    • as it depends on recent Meson build system
      • not packaged in 20.04

But we released packaged version of latest aravis and camera_aravis for Noetic:

• https://github.com/Extend-Robotics/camera_aravis/releases/tag/ER4.1.3

As a side note, above fork had some breaking changes compared to this upstream.
I am not sure everything will work the same.

And finally our fork will become stale soon as I have implemented almost everything we needed.

Some last optimizations of pixel format conversions we need are pending.

from camera_aravis.