keijiro / tinyyolov2barracuda Goto Github PK

Tiny YOLOv2 on Unity Barracuda

License: The Unlicense

HLSL 7.18% C# 75.00% ShaderLab 17.82%

tinyyolov2barracuda's Introduction

TinyYOLOv2Barracuda

TinyYOLOv2Barracuda is a Unity sample project that shows how to run the YOLO object detection system on the Unity Barracuda neural network inference library.

This project uses a Tiny YOLOv2 model from ONNX Model Zoo. See the model description page for details.

System requirements

Unity 2021.3
Barracuda 3.0.0

How to run

This repository doesn't contain the ONNX model file to avoid hitting the storage quota. Download the model file from the ONNX Model Zoo page and put it in the Assets/ONNX directory.

Sample scenes

All these samples use the ImageSource class to feed images to the model. You can change the image source type from the drop down list.

VisualizerGpu

VisualizerGpu runs all the object detection & visualization processes (preprocess, inference, post-process, overlap removal, and visualization) solely on GPU. It minimizes the CPU load and visualization latency, but you can't do anything more complicated than simple visualization like drawing rectangles or something on detected objects.

VisualizerCpu

VisualizerCpu runs the object detection on GPU and then reads the detection results back to the CPU side. After that, it visualizes them using the Unity UI system. Even though this method runs slower than the GPU-only method, you can do complex processes using C# scripting.

Pixelizer

Pixelizer detects people from the input video stream and applies a pixelation effect to the person regions. It shows how to implement an image effect with the YOLO detector.

tinyyolov2barracuda's People

Contributors

Stargazers

Watchers

tinyyolov2barracuda's Issues

Questions

Hello, I am trying out your project and I have some questions:

Why are the classes limited to the ones in Config.cs? I think the original model has more (?).
Why doesn't the VisualizerGPU show any labels during detection like the VisualizerCPU?
I am trying to understand what the pipeline is (reverse engineering the code), and the differences between the two Visualizers are a bit confusing, any help would be appreciated.

Using a headset instead of webcam

Hello Keijiro!

This is not an issue with the project but with using a different source instead of a webcam.
I am using a mixed reality headset whose cameras are not accessible as web cameras. Therefore I can't use WebCamTexture, although the headset shows up in the list when I print the devices log. Have you tried with a different video source? Do you know an alternative to WebCamTexture?

Thanks!

Onnx file

When I put the onnx file into unity it didn't recognize it as NNModul and I get this error:
InvalidProtocolBufferException: Protocol message contained a tag with an invalid wire type.
Google.Protobuf.UnknownFieldSet.MergeFieldFrom (Google.Protobuf.CodedInputStream input) (at T:/src/github/protobuf/csharp/src/Google.Protobuf/UnknownFieldSet.cs:232)
Google.Protobuf.UnknownFieldSet.MergeFieldFrom (Google.Protobuf.UnknownFieldSet unknownFields, Google.Protobuf.CodedInputStream input) (at T:/src/github/protobuf/csharp/src/Google.Protobuf/UnknownFieldSet.cs:257)
Onnx.ModelProto.MergeFrom (Google.Protobuf.CodedInputStream input) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/Generated/Onnx.cs:1645)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:157)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:98)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/[email protected]/Barracuda/Editor/ONNXModelImporter.cs:65)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at :0)

Asset import failed, "Assets/ONNX/tinyyolov2-7.onnx" > InvalidProtocolBufferException: Protocol message contained a tag with an invalid wire type.
Google.Protobuf.UnknownFieldSet.MergeFieldFrom (Google.Protobuf.CodedInputStream input) (at T:/src/github/protobuf/csharp/src/Google.Protobuf/UnknownFieldSet.cs:232)
Google.Protobuf.UnknownFieldSet.MergeFieldFrom (Google.Protobuf.UnknownFieldSet unknownFields, Google.Protobuf.CodedInputStream input) (at T:/src/github/protobuf/csharp/src/Google.Protobuf/UnknownFieldSet.cs:257)
Onnx.ModelProto.MergeFrom (Google.Protobuf.CodedInputStream input) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/Generated/Onnx.cs:1645)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (Google.Protobuf.CodedInputStream inputStream) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:157)
Unity.Barracuda.ONNX.ONNXModelConverter.Convert (System.String filePath) (at Library/PackageCache/[email protected]/Barracuda/Runtime/ONNX/ONNXModelConverter.cs:98)
Unity.Barracuda.ONNXModelImporter.OnImportAsset (UnityEditor.AssetImporters.AssetImportContext ctx) (at Library/PackageCache/[email protected]/Barracuda/Editor/ONNXModelImporter.cs:65)
UnityEditor.AssetImporters.ScriptedImporter.GenerateAssetData (UnityEditor.AssetImporters.AssetImportContext ctx) (at :0)

Can I train this on custom classes?

Hi,

I've been searching everywhere but I can't find a way to train TinyYolov2 to detect 3 custom object classes. Any pointers would be great.

Alternatively, is there any way to train a model to detect 3 objects in a frame and have that model work with Barracuda? I see that only certain architectures are supported and TinyYOLOv2 seems to be the most suitable of the lot.

Thanks in advance!

Using video file instead of WebCam?

Hi there,

This is my first time using Unity. I got the project to run but I was wondering how to input a video file to the Pixelizer scene instead of the WebCamTexture? I notice in the README, the example seems to use a video recorded on a camera and not a webcam.

NullReferenceException

Got the following error when trying to run the Pixelizer scene

NullReferenceException: Object reference not set to an instance of an object
Unity.Barracuda.ModelLoader.Load (Unity.Barracuda.NNModel model, System.Boolean verbose, System.Boolean skipWeights) (at Library/PackageCache/[email protected]/Barracuda/Runtime/Core/ModelLoader.cs:29)
TinyYoloV2.ObjectDetector..ctor (TinyYoloV2.ResourceSet resources) (at Assets/Script/ObjectDetector.cs:37)
TinyYoloV2.Pixelizer.Start () (at Assets/Test/Pixelizer.cs:39)

Yolov2-tiny (COCO)

Hi,

I have a problem. I was struggling with the conversion from darknet to ONNX for several days, but today I finally succeed.

I have two versions of yolov2-tiny: voc (20 classes) and coco (80 classes) converted by me.
VOC model works out of the box, but COCO doesn't.
I changed:

ClassCount and _labels in Config file
CLASS_COUNT and anchors in Common.hlsl
model assigned to TinyYOLOv2 asset.
Here is my COCO model:
https://drive.google.com/file/d/1epFgd3Sx0NikGB-Y9oHsVWlACxclmjnL/view?usp=sharing

Could you @keijiro try to check my model and tell me what else do I need to change?

Can I ask if this can be implemented on MRTK-WEBRTC?

Hi there,

I have noticed if you have done great works with Barracuda in Unity and I would like to ask you if you have done any implementation with mrtk-webrtc and barracuda?
Right now, I am looking to run Barracuda-YOLO inference engine on MRTK-Webrtc stream from HoloLens and would like to know if its possible?
I am new to Unity and its development and would really appreciate your help.
Thanks.

Shader error in 'Hidden/TinyYOLOv2/Visualizer': Couldn't open include file '../Common.hlsl'. at line 6

Hi,

If i try to edit Common.hlsl file to change anchors and classes count for my custom model, then Visualizer and Pixelizer shaders crashes with error:
Shader error in 'Hidden/TinyYOLOv2/Visualizer': Couldn't open include file '../Common.hlsl'. at line 6

I tried to edit Common.hlsl file with Visual Studio, Atom and Notepad. Always when I save Common.hlsl file, shaders crashes.
I also tried to reimport shaders, but without success.

Can somebody tell me how can I edit this file?

How should I convert darknet model to onnx format?

Hi, I'm kinda struggling with how to use my custom model.

I'm testing with tiny YOLO v2(voc) files(.cfg, .weights) before I convert my custom weight.

First, I converted Darknet(yolov2-tiny-voc.cfg and yolov2-tiny-voc.weights) to TF model(.pb).

What I found out is that I have to freeze the TF graph so I did. Then I converted it to ONNX model.

I used DW2TF for conversion.
https://github.com/jinyu121/DW2TF

Could you help me how to get it work?

Here's the model that I used.
https://kookmin-my.sharepoint.com/:u:/g/personal/apex_kookmin_kr/EeoCfr_a6KlLitYRePduwVoBn1krOowxFsF4JVIkldGwAQ

Portrait mode

Hi,

I already have full screen preview with working image recognition in landscape mode, but I need portrait mode (probably portrait mode only without autorotation feature) in iOS app. I think I must rotate texture before execute ProcessImage function in ObjectDetector or maybe in Preprocess ComputeShader. But I don't know how. Has anyone managed to do it?

yolov3 or v4 integration?

i tried to use a yolov3 as i describe in this post, but i was wondering if you could tell me if i am doing something wrong or do you have a version with better results than the one this repo has (either more classes or better performance?, or do you think it is a problem of my environment, not realistic enough?)

You may be interested in our repository!

Hey, not an issue, just saw you working with Tiny Yolo V2 and we already got that working as well as Tiny Yolo V3 in the repo here, it may be useful to you!
https://github.com/derenlei/Unity_Detection2AR