Startup camera sync via initializing/deinitializing cycle does not work with GigE camera about spinnaker_sdk_camera_driver HOT 34 CLOSED

In the meantime we found the following regarding the errors with:

Image incomplete with image status 9!

The two cameras are flooding the IP connection with packets. To have more than one camera coexist in peace one has to divide the GigE bandwith between the cameras. This is done via the
PacketDelay parameter which may also be called GevSCPD (Gigevision Stream Channel Packet Delay) or the corresponding parameter DeviceLinkThroughputLimit (which is easier to understand and set a meaningful value).
You can set the DeviceLinkThroughputLimit to 61644209 (equals PacketDelay of 73760 for FLIR camera tick frequency) and this equals half of GigE bandwidth. Or you may use 40841495 (PacketDelay 147880) which is a third of GigE bandwidth to have some space.
Please note that limiting device bandwidth of course reduces the frame rate such that the necessary data transfer does not exceed the limit.

from spinnaker_sdk_camera_driver.

Hi @shahvi looks like the Event Data Spinnaker error was solved by increasing UDP buffer size to 25MB by 'sudo sysctl -w net.core.rmem_max=26214400' and 'sudo sysctl -w net.core.rmem_default=26214400'. I found this on a different forum. Sorry about the back and forth. I figured since the camera worked fine with Spinview that it would work fine with the ROS node.

from spinnaker_sdk_camera_driver.

Having the same issue, I commented out the assertion line, however, that leads to the node crashing eventually.

from spinnaker_sdk_camera_driver.

What is ur setup exactly? GIG E or ethernet. How many cameras?

from spinnaker_sdk_camera_driver.

@karanchawla the above question is for you.

from spinnaker_sdk_camera_driver.

I’m away from the computer at the moment, will send a complete bug report in a bit.

from spinnaker_sdk_camera_driver.

I asked the manufacturer FLIR about this and they told me (translated quote):

There actually is a difference between the GigE and USB3 variant: The GigE camera resets FrameID, if AcquisitionStop, AcquisitionStart is executed. The USB3 does this only on EndAcquisition, BeginAcquisition. The counter cannot be reset manually.

HTH

from spinnaker_sdk_camera_driver.

@llvs and @karanchawla are you guys using the GPIO cable to trigger the cameras as described in the ReadMe. If you are setting up the code for 2 camera and the trigger doesn't happen correctly, the code would timeout during the image acquisition on the slave camera and terminate.

@llvs before worrying too much about the frame id's it would be worth checking if it is an issue. If you are able to acquire images from 2 cameras, I suggest pointing them at a stopwatch and checking if they are synced.

from spinnaker_sdk_camera_driver.

No, as the problems using the node happen already with a single camera. However, in the future I assume that we will need to employ hardware triggering via GPIO.

from spinnaker_sdk_camera_driver.

My setup is as follows:
Camera: Blackfly S
Sensor: RGB
Connection: GIG E

I'm trying to use this repository to acquire images however, the node would crash after a few minutes after giving a few warnings about image might be incomplete!

from spinnaker_sdk_camera_driver.

@karanchawla which particular model are you using. How many megapixels camera is it?

from spinnaker_sdk_camera_driver.

Specs for the camera I'm using: https://www.ptgrey.com/blackfly-s-color-32-mp-gige-vision-sony-imx265

from spinnaker_sdk_camera_driver.

So does it work fine for a few mins?

from spinnaker_sdk_camera_driver.

Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://ThinkPad-L380-Yoga:43951/

SUMMARY
========

PARAMETERS
 * /acquisition_node/binning: 2
 * /acquisition_node/cam_aliases: ['cam0']
 * /acquisition_node/cam_ids: [18434273]
 * /acquisition_node/color: True
 * /acquisition_node/delay: 0
 * /acquisition_node/exp: 0
 * /acquisition_node/frames: 0
 * /acquisition_node/live: False
 * /acquisition_node/live_grid: False
 * /acquisition_node/master_cam: 18434273
 * /acquisition_node/max_rate_save: False
 * /acquisition_node/save: False
 * /acquisition_node/save_path: ~
 * /acquisition_node/save_type: bmp
 * /acquisition_node/skip: 30
 * /acquisition_node/soft_framerate: 15
 * /acquisition_node/time: False
 * /acquisition_node/to_ros: True
 * /acquisition_node/utstamps: False
 * /rosdistro: kinetic
 * /rosversion: 1.12.14

NODES
  /
    acquisition_node (spinnaker_sdk_camera_driver/acquisition_node)

ROS_MASTER_URI=http://localhost:11311

process[acquisition_node-1]: started with pid [4337]
[ INFO] [1539896282.597559438]: [ OK ] USB memory: 1000 MB
20181018[ INFO] [1539896282.597629787]: *** PARAMETER SETTINGS ***
[ INFO] [1539896282.597642561]: ** Date = 20181018
[ INFO] [1539896282.598052658]:   Save path set via parameter to: /home/skyryse
[ INFO] [1539896282.598085649]:   Camera IDs:
[ INFO] [1539896282.598526005]:     18434273
[ INFO] [1539896282.599001187]:   Camera Aliases:
[ INFO] [1539896282.599019405]:     18434273 >> cam0
[ INFO] [1539896282.599881492]:   Unique time stamps for each camera: 0
[ INFO] [1539896282.600393175]:   color set to: 1
[ INFO] [1539896282.600880416]:   Exporting images to ROS: 1
[ INFO] [1539896282.601281588]:   Showing live images setting: 0
[ INFO] [1539896282.601648232]:   Showing grid-style live images setting: 0
[ INFO] [1539896282.602022591]:   Max Rate Save Mode: 0
[ INFO] [1539896282.602444702]:   Displaying timing details: 0
[ INFO] [1539896282.602865232]:   No. of images to skip set to: 30
[ INFO] [1539896282.603289117]:   Init sleep delays set to : 30 sec
[ WARN] [1539896282.603729684]:   'fps' Parameter not set, using default behavior: fps=20.00
[ INFO] [1539896282.604159206]:   'exp'=0, Setting autoexposure
[ INFO] [1539896282.604585260]:   Binning set to: 2
[ INFO] [1539896282.604973468]:   Using Software rate control, rate set to: 15
[ INFO] [1539896282.605421087]:   Saving images set to: 0
[ INFO] [1539896282.606245677]:   Camera coeffs not provided correctly, camera info messages will not be published.
[ INFO] [1539896282.606282651]: Creating system instance...
[ INFO] [1539896282.606560907]: Retrieving the list of cameras...
[ INFO] [1539896284.732364455]: Numer of cameras found: 1
[ INFO] [1539896284.732453493]:  Cameras connected: 1
[ INFO] [1539896284.732554350]:   -18434273
[ INFO] [1539896284.739698339]: *** FLUSH SEQUENCE ***
[ INFO] [1539896284.739773712]: Initializing cameras...
[ INFO] [1539896285.389000848]: Deinitializing cameras...
[ INFO] [1539896287.166753612]: All cameras deinitialized.
[ INFO] [1539896287.166924233]: Initializing cameras...
[ INFO] [1539896287.645791210]: *** ACQUISITION ***
[ WARN] [1539896289.066276953]: Image incomplete with image status 9!
[ WARN] [1539896290.266493358]: Image incomplete with image status 9!
.
.
.
[ WARN] [1539896340.474110609]: Image incomplete with image status 9!
[FATAL] [1539896342.525220597]: Some exception occured. 
                                                         Exiting gracefully, 
  possible reason could be Camera Disconnection...
[ INFO] [1539896342.599320622]: Deinitializing cameras...
[ INFO] [1539896343.624576840]: All cameras deinitialized.
[ INFO] [1539896343.624668475]: Clearing camList...
[ INFO] [1539896343.624731904]: Releasing camera pointers...
[ INFO] [1539896343.624787122]: Releasing system instance...
terminate called after throwing an instance of 'std::bad_cast'
  what():  std::bad_cast

from spinnaker_sdk_camera_driver.

It works for less than a minute after which it crashes. I have posted the log in the comment above.

from spinnaker_sdk_camera_driver.

@karanchawla That issue seems to be with the Spinnaker SDK as opposed to the ROS driver. See
https://www.ptgrey.com/KB/11142 at the end under Troubleshooting.

• Can you run SpinView with similar frame rates and settings without any issues?
• Try running at lower frame rate just to see if this is related to the bandwidth.
• Also try to use binning to reduce image size. You can set it to 2 for testing.

from spinnaker_sdk_camera_driver.

@karanchawla and @llvs were you guys able to get the cameras to work? Do you have an update about the issues?

from spinnaker_sdk_camera_driver.

Hi @shahvi I am getting the exact same issue as @karanchawla. I have a Blackfly BFLY-PGE-20E4C. I had a frameID error to start with as above. I then commented line 59 in camera.cpp. Once I did that the camera would run for under a minute and then would crash again. Same failure as above.

I can run spinview at any framerate available. I have set frame:=10 for the ROS node.

from spinnaker_sdk_camera_driver.

@mikedef Its difficult to debug without us having that type of camera. One thing you could try would be to launch the code with the debug turned on by changing the first line in

See if the debug messages help...

from spinnaker_sdk_camera_driver.

@shahvi changing to debug_console.conf did not give me any updated debug messages.

Here is my latest error message: Spinnaker: Failed waiting for EventData on NEW_BUFFER_DATA event [-1011]
I see see that there was a closed ticket for this error, but no solution to it. Do you have any updated information on this buffer data issue?
Thanks

from spinnaker_sdk_camera_driver.

@mikedef , I am really out of ideas. It seems to be happening in the Spinnaker_SDK and I am not sure what it might be. Hard to know without having one to play with.

from spinnaker_sdk_camera_driver.

@shahvi Right I am not sure either. Also the color flag does not work for this camera as well.
Spinview works fine. So I am not sure why it's not working with this ROS app. Thanks for the help either way.

from spinnaker_sdk_camera_driver.

@mikedef thanks for the update. So does the driver work properly once you make this change?

from spinnaker_sdk_camera_driver.

@shahvi The driver does not work in color mode. It does work when in black and white. I had to make a few changes, mostly commenting out of error checking code as highlighted above.

I get the following errors when setting the color flag to true
[FATAL] [1549565095.362445492]: Unable to set PixelFormat to BGR8 (entry retrieval). Aborting...
[FATAL] [1549565095.362619558]: Error: Spinnaker: LogicalErrorException NULL pointer dereferenced [-2005]

from spinnaker_sdk_camera_driver.

Setting the eumeration takes place in:

Maybe these GigE cameras have a different enumeration entry ? You can check in SpinView, what enumerations the PixelFormat option accepts.

from spinnaker_sdk_camera_driver.

@shahvi Looks like my only option is BayerGB8. Where is this set exactly? I changed line 96 to PixelFormat_BayerGB8 and did a catkin_make, but I still get the BGR8 error as before.

from spinnaker_sdk_camera_driver.

You should be able to change it at

from spinnaker_sdk_camera_driver.

@shahvi Sorry it's actually BayerBG8 not GB. Either way it works when I change the pixel format to BayerBG8.
Thanks for the help. The ROS node is now running with the default settings and the color flag set to true.

from spinnaker_sdk_camera_driver.

@llvs That solves the image incomplete and event buffer timeout issue thanks!

from spinnaker_sdk_camera_driver.

@llvs how would you set that up with ros in terminal exactly? I can find the parameters in spinview but changing in spinview doesn't seem to help in roslaunching the application as the parameters are reset the next time I open spinview.

Many thanks.

from spinnaker_sdk_camera_driver.

@alankwok97 You can save SpinView settings to your camera by typing user into the search box in SpinView. Change the parameter UserSetSelector to UserSet0 or UserSet1. Then click the execute button for UserSetSave. Now your settings are saved and all you have to do now is change UserSetDefault to match your desired UserSet. Everytime your camera is booted up, the settings saved will be used. If you want to go back to Default, you can select Default under UserSetSelector and click Execute for UserSetLoad.

from spinnaker_sdk_camera_driver.

@kaalessi Thanks for your reply!

from spinnaker_sdk_camera_driver.

@llvs what could be causing this error if I’m using only one camera? You said in your case with multiple cameras they were flooding the network, but this shouldn’t happen with a single camera, right?

from spinnaker_sdk_camera_driver.

I can only guess here, but it may have to do with too small network buffers. The linux defaults are maybe too little. Or you may want to use Ethernet Jumbo Frames if the camera allows it to reduce network load. But those are only guesses.
HTH

from spinnaker_sdk_camera_driver.