Raspberry Pi4 Kernel Panic/Crash When Capturing Color about spinnaker_sdk_camera_driver HOT 13 CLOSED

@fultonms Can you share your launch file and camera config file

Is the param color set to false in launch file and config file?

We didn’t test the code on any raspberry variant before.

Based on your description,

1. If total system memory is 4GB, you shouldn’t be setting usbfs_memory_mb to 4096 for one application.

Read more on
https://www.flir.com/support-center/iis/machine-vision/application-note/understanding-usbfs-on-linux/

2. Ideally you don’t need to run spinview before running this driver.
   To see if camera is detected, check in $lsusb
   Optionally you can run Acquisition sample binary installed by spinnaker sdk to check if spinnaker sdk is installed properly and to test image acquisition from camera.

I recommend setting usbfs limit to 1024 and param color to false and run the driver.

What FPS did you set in soft FPS param ?

Hope it helps

from spinnaker_sdk_camera_driver.

@mithunvarma Hi, thanks for your response.

I am testing using the acquisition_node.launch and test_param.yaml launch file as they are in the repository at this point, with the only change being my camera's serial number.

When I am testing without color, the color param is set to false. When I have tested with color, I have done so using the standard ROS method of passing parameters to the launch file, color:=true. I have also tried setting the parameter default in the launch file, with the same results.

A correction on my previous comment, the exact error message is as follows:

Terminate called after throwing and instance of 'Spinnaker :: Exception'
    what() : Spinnaker: Could not start acquisition. Please try reconnecting the device [-1010]

1. I've also tried at 1024, 2048, etc. I'll modify my configuration to set the usbfs_memory_mb to 1024 in the future though, thanks.

2. That's what concerns me. When running this driver on other computers, it has worked fine. However, for the Raspberry Pi, the driver does not seem to work dependably without running Spinnaker first. While doing some testing to confirm this, it did actually work (with no color) without running Spinnaker first, so maybe that problem isn't a consistent thing. I think the more persistent issue is the crashes when related to running the camera with color.

My camera is detected in lsusb. When running with color param set false, the camera does work, however I need color for my applications. The FPS is set to 20.

Thanks for your help

from spinnaker_sdk_camera_driver.

@fultonms Does the color mode work fine on other computers? If so, I think that the problem is with running color mode on Raspberry Pi. May be the Raspeberry Pi's USB is unable to handle the data coming through. You could may be try setting a lower FPS and see if that works?

from spinnaker_sdk_camera_driver.

@fultonms Could you try a couple more things. With color set to True can you try reducing the resolution by setting binning to 2 and/or reducing the soft_framerate to 5 or 10. This should help us determine if it is a driver / spinnaker issue or a system limitation.

Also another thing that comes to mind is that in our experience when the power to the camera is unstable it can throw errors like this. Can you connect the camera through a powered USB hub or give it external power via GPIO?

from spinnaker_sdk_camera_driver.

@PushyamiKaveti Yeah, I think that's the issue as well. I'm not expecting that this will work as well as it could on another system, just trying to get some level of performance out of this.

@vik748 I will try those things and report back. Thanks for the help!

from spinnaker_sdk_camera_driver.

@vik748 You are a lifesaver. Reducing the resolution by setting the binning to 2 was sufficient to get the camera working, at 20 FPS, in full color. Thank you so much!

I had not been trying resolution reduction, but thinking about the problem now, it really should have been one of the first things I tried.

from spinnaker_sdk_camera_driver.

@fultonms I still suspect that the root cause might be power. When in full color and full resolution, the camera might have a spike in power draw causing issues. Lowering either greyscale, resolution or framerate might be helping it reduce that power spike.

from spinnaker_sdk_camera_driver.

@vik748 I think you might be right. I'm doing some testing, but progress is being made.

A somewhat unrelated question: have you considered running cameras in a master/slave setup across multiple computers? So, several networked computers, one with the master cam, the others with slave cameras, all cameras connected via GPIO. The current status of this package does not allow for it. Is that a limitation in the Spinnaker SDK, or no?

from spinnaker_sdk_camera_driver.

We haven't tried it but should be possible. What you'd need to do is setup each computer as a single camera system. Have the 'master' as a regular system and have all other be setup to 'external trigger' as in the external_trigger branch. We haven't done this, so it might need a few tweaks in the code.

from spinnaker_sdk_camera_driver.

@vik748 Thank you! That has been working well so far.

However, after approximately 30 seconds, the externally triggered camera stops capturing frames. It appears to get to the call for acquisition::Capture::get_mat_images and not return from the call. For clarity, it does make it through that call for some period of time (apparently random), but it does fail eventually. Any ideas on what this might be caused by?

from spinnaker_sdk_camera_driver.

How are the cameras powered? We have seen if there is any power issues the cameras misbehave. We have found that using a powered USB hub helps with stability.

from spinnaker_sdk_camera_driver.

I experience the same issue whether I am powering the camera off the USB3 from the Raspberry Pi, or off Line 3 (Pin 1), for which I have to switch my trigger source to Line 2 ( Pin 3).

I have not experienced this issue when using SpinView on a laptop with either trigger/power setup. I'll be testing it with SpinView on the Pi4 tomorrow and will let you know what happens.

However at the moment, I don't think that it's a power issue, but I'm still testing to see if that could be the case.

The failure looks similar to the way that the camera stops capturing images if it does not have a trigger signal. However, I have checked with an oscilloscope to check that the trigger signal is being sent, so I think my debugging is just not in-depth enough?

from spinnaker_sdk_camera_driver.

Having looked a little more into it, I see the same issue with Spinview on the RPI, both powered and unpowered.

By looking a little more into it, I'm sometimes seeing warnings that frames are being skipped, then a warning: "Image incomplete with image status 9!" followed by a bad_alloc exception. Sometimes (without changing configs) I also see that warning and bad_alloc exception without first seeing the frame skip warnings, and still other times I see nothing, but the frame_grabbing function does not return.

from spinnaker_sdk_camera_driver.