"Ultra-Light-Fast-Generic-Face-Detector-1MB" face detector for Unity Barracuda
License: MIT License
UltraFaceBarracuda is a Unity sample project that shows how to run the UltraFace face detection neural network model on the Unity Barracuda.
For the details of the UltraFace ("Ultra-Light-Fast-Generic-Face-Detector-1MB") model, please see the original repository.
First, thank you for doing this. It was immensely helpful to see how to use the texture rendering chain to parse the outputs of UltraFace. I have a version of your work running, but I have modified it to work during runtime on a HoloLens 2, using MIcrosoft's MediaCapture APIs to capture the RGB camera fames from the front of the HL2. From these frame, I create a SoftwareBitmap and convert to a Unity Texture for use as an input into your rendering chain. When I do this, the model returns 0 faces, for reasons I cannot understand. Can you think of a way around this?
The code Looks something like this:
public async Task EvaluateVideoFrameAsync(byte[] bitmapFrame)
{
DetectedFaces result = new DetectedFaces();
try
{
Texture2D imageMediaTexture = new Texture2D(320, 240, TextureFormat.RGB24, false);
imageMediaTexture.LoadRawTextureData(bitmapFrame);
imageMediaTexture.Apply();
var pre = _resources.preprocess;
pre.SetInts("ImageSize", _config.InputWidth, _config.InputHeight);
pre.SetTexture(0, "Input", (Texture)imageMediaTexture);
pre.SetBuffer(0, "Output", _buffers.preprocess);
pre.DispatchThreads(0, _config.InputWidth, _config.InputHeight, 1);
// NNworker invocation
using (var t = new Tensor(_config.InputShape, _buffers.preprocess))
_worker.Execute(t);
// NN output retrieval
_worker.CopyOutput("scores", _buffers.scores);
_worker.CopyOutput("boxes", _buffers.boxes);
// Counter buffer reset
_buffers.post2.SetCounterValue(0);
_buffers.counter.SetCounterValue(0);
// First stage postprocessing: detection data aggregation
var post1 = _resources.postprocess1;
post1.SetTexture(0, "Scores", _buffers.scores);
post1.SetTexture(0, "Boxes", _buffers.boxes);
post1.SetDimensions("InputSize", _buffers.boxes);
post1.SetFloat("Threshold", .5f);
post1.SetBuffer(0, "Output", _buffers.post1);
post1.SetBuffer(0, "OutputCount", _buffers.counter);
post1.DispatchThreadPerPixel(0, _buffers.boxes);
// Second stage postprocessing: overlap removal
var post2 = _resources.postprocess2;
post2.SetFloat("Threshold", 0.5f);
post2.SetBuffer(0, "Input", _buffers.post1);
post2.SetBuffer(0, "InputCount", _buffers.counter);
post2.SetBuffer(0, "Output", _buffers.post2);
post2.Dispatch(0, 1, 1, 1);
ComputeBuffer.CopyCount(_buffers.post2, _buffers.countRead, 0);
List<DetectedFace> tempFaces = new List<DetectedFace>();
foreach(var detection in _readCache.Cached){
Rect tempBbox = new Rect();
tempBbox.X = (uint)(detection.x2 - detection.x1) / 2;
tempBbox.Y = (uint)(detection.y2 - detection.y1) / 2;
tempBbox.Width = (uint)(detection.x2 - detection.x1);
tempBbox.Height = (uint)(detection.x2 - detection.x1);
tempFaces.Add(new DetectedFace(){bbox = tempBbox, confidence = detection.score});
}
_readCache.Invalidate();
return new DetectedFaces(){originalImageBitmap = returnFrame.bitmap, Faces = tempFaces.ToArray()};
}
Hi Keijiro,
Congratulations on your great work once more.
I have been trying to adapt the code in this repository to apply the UltraFace-RFB-640 model instead of the UltraFace-RFB-320 model.
The only problem I find is when applying the first post-processing step through RenderTextures.
While for the 320x240 input network the maxdetections size is 4420, for the 640x480 input network the maxdetections size is 17640 (curiously it is not 4420 x 4 = 17680).
The problem I am facing is that is not possible to allocate a RenderTexture of 2x17680 or 4x17680.
I think it could be possible to reshape the output tensors "scores" and "boxes" to a more "square" format:
barracudaWorker.PeekOutput("scores").Reshape(new TensorShape(1, maxDetection / 4, 2 * 4, 1))
barracudaWorker.PeekOutput("boxes").Reshape(new TensorShape(1, maxDetection / 4, 4 * 4, 1))
But then I am a bit lost about how the ComputeShader should use the SV_DispatchThreadID to parse the BoundingBoxes data.
Do you think allocating more "square" RenderTextures is the best option or do you think it is possible to extract the output tensor data some other way?
Best regards.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.