bonsai-rx / sleap Goto Github PK

The next release of the editor will allow integrated package documentation so we should take advantage of that to organize the documentation for the Sleap package following existing DocFX templates.

Currently, all visualizers are tailored to single instances. For full topdown models would be interesting to have a way to visualize all animals with their respective ID.

In the GetMaximumConfidencePoseIdentity operator, when no matching identity is found in the set of all labeled instances, a default pose is generated with NaN confidence and NaN centroid. Identity is set to the specified target string, but IdentityIndex is left with its default value (zero), which can be misleading as it can be interpreted in different contexts as specifying the zero class label:

Here we should either set IdentityIndex to -1 to indicate detection failure, or set the default value of the IdentityIndex property to -1.

Sorry, I cannot find this file folder, The Sleap software should generate it, or shoudl create it somehowBecausese you say: "To use the Predict operators, you will need to provide the ModelFileName to the exported .pb file folder containing your pretrained SLEAP model, along with the corresponding PoseConfigFileName to the training_config.json file."
I have the .json file but not the .pb

Thanks

Following Sleap structure, there are currently four model types supported in Bonsai:

- Multi-instance:

• centroid
• top-down-model (Centroid + CenteredInstance)
• top-down-id-model (Centroid + CenteredInstance + ClassID)

- Single Instance:

• single instance (CenteredInstance)

Since Bonsai affords the ability to create custom centroid and instance cropping image processing routines, it would be nice to have the ability to run identification and pose estimation without the first centroid tracking step. This would be implemented through a custom single labeled instance network model. Similarly to the current single instance, it would be great if this first model always guarantees a single output per input image (even if full of NaNs).

As a side note, while certainly not necessary, is it possible to have a model wherein, from a cropped image, outputs only ClassID?

Currently the PoseIdentity class returns only the highest likelihood classification and identity class. The full vector should also be included to allow more sophisticated downstream filtering and processing of SLEAP output.

Sleap networks may have different input sizes than the videos they were trained on. This is done to accommodate two requirements:

1. The image size is divisible by the max stride of the model (usually 16 or 32) so we don't get mismatched tensor sizes across skip connections due to shape rounding/truncation
2. To ensure that all images in the same batch are of the same size so that we can handle variable image sizes.

This pre-processing step is automatically handled in the SLEAP API but not currently in Bonsai. They do it by padding, with zeros, at the bottom, and right of the image (which means coordinates should not change).
This pre-process step can be easily done in Bonsai by using the ResizeCanvas operator.

For the sake of usability, we could add this optional functionality to the predict nodes. Unfortunately, the training_config.json we are currently loading in bonsai with all the metadata does not have the network input size information.

Few possible solutions:

1. Load the info.json that results from the export sleap functionality. This would require loading and parsing an additional JSON file.
2. An alternative to 1. would be to simply merge the two json files in a single output file during the export stage, which would save users the bother to keep track of what metafiles correspond to what networks.
3. It might be possible to infer the network input size form the deserialized frozen network. Must investigate if this is a valid approach.

When parsing the training_config.json file, string values are read and cast to a numeric representation. E.g.:

https://github.com/bonsai-rx/sleap/blob/main/src/Bonsai.Sleap/ConfigHelper.cs#L43

In the future, we should strive to make such methods robust enough to parse the value irrespective of the regional preferences of the operating system.
This could be achieved by using a different overload in the Parse method:

float val = float.Parse(stringVal, CultureInfo.InvariantCulture.NumberFormat);

Hi, i'm trying to run a SLEAP single instance tracking node within a bonsai workflow and have issues getting it to use the GPU properly. I'll give some background on what i did so far to resolve a few issue because maybe some of those steps of installing and reinstalling software caused my current problem, and then describe the current issue.

So far:
I've set up SLEAP in a python virtual environment, where it is working fine. I've trained and exported a model, and in principle the Bonsai node worked. But it performed the tracking with only the CPU, and thus far to slow (~11 fps) for online video tracking (doing inference on the videos directly in the SLEAP GUI did use the graphics card, a geforce 3060). The errors i got were all in this style:
2023-07-19 17:05:50.946203: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
I ended up removing my current versions of tensorflow, CUDA and cuDNN, and installed tensorflow 2.10.0, CUDA v11.3 and cuDNN 8.2 (in the virtual env through miniconda, and in my base env through installs/copying .dlls into the bonsai extensions folder) as those were the last validated versions according to the bonsai/sleap install guide.

Current state:
Now bonsai/sleap detects and initializes the GPU, but then crashes with the following errors in bonsai itself:
Bonsai.WorkflowRuntimeException: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/single_instance_inference_model_1/single_instance_inference_layer_1/PartitionedCall_1/mod}}]]
[[Identity_1/_49]]
(1) UNKNOWN: JIT compilation failed.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/single_instance_inference_model_1/single_instance_inference_layer_1/PartitionedCall_1/mod}}]]
0 successful operations.
0 derived errors ignored. ---> TensorFlow.TFException: 2 root error(s) found.
(0) UNKNOWN: JIT compilation failed.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/single_instance_inference_model_1/single_instance_inference_layer_1/PartitionedCall_1/mod}}]]
[[Identity_1/49]]
(1) UNKNOWN: JIT compilation failed.
[[{{node StatefulPartitionedCall/StatefulPartitionedCall/single_instance_inference_model_1/single_instance_inference_layer_1/PartitionedCall_1/mod}}]]
0 successful operations.
0 derived errors ignored.
at TensorFlow.TFStatus.CheckMaybeRaise(TFStatus incomingStatus, Boolean last)
at TensorFlow.TFSession.Run(TFOutput[] inputs, TFTensor[] inputValues, TFOutput[] outputs, TFOperation[] targetOpers, TFBuffer runMetadata, TFBuffer runOptions, TFStatus status)
at TensorFlow.TFSession.Runner.Run(TFStatus status)
at Bonsai.Sleap.PredictSinglePose.<>c__DisplayClass20_1.b__1(IplImage[] input)
at System.Reactive.Linq.ObservableImpl.Select`2..OnNext(TSource value)
--- End of inner exception stack trace ---

And in the bonsai terminal:
Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
error: Can't find libdevice directory ${CUDA_DIR}/nvvm/libdevice
2023-07-19 17:53:06.603435: W tensorflow/core/framework/op_kernel.cc:1768] UNKNOWN: JIT compilation failed.

I've since tried adding the CUDA/v8.2.0.51/nvvm/libdevice folder (in the base environment, not the virtual env) to either the PATH or to it's own CUDA_DIR environmental variable, but with no success.
I know this is a rather lengthy one and i'm not super familiar with what i've been doing, so if anything is unclear just ask away. Any help in getting this to work would be much appreciated.

When no instance is found on a given frame (which can easily be simulated using SolidColor) Predict nodes crash.
The runner object returns a valid output. However, when trying to allocate the data from the TFTensor to a numeric array the operation returns: "Array must not be jagged". E.g.:

I believe the best solution is to return an empty Collection of tracked objects. Users can then filter this Collection by simply checking Count > 0.

for now, for single instance models, only pb files from sleap 1.2.9 work, not the most recent versions. Would be great to have this in the documentation (I learned the hard way by finding an issue after my training failed)