WebGL implementation of masking about opendatacam HOT 10 CLOSED

forgot to ping you: @b-g @mmmmmmmmmmmmmmmm

from opendatacam.

🎉🔥 update ! ping @b-g @mmmmmmmmmmmmmmmm

I successfully managed to do a webgl implem of the masking you can check it out here: https://traffic-cam-goapmhhqhp.now.sh/webgl

➡ You need to click play on the top right box to start the video playback

How does it work

I'm using a rather simple fragment shader that computes the sum of the red, green blue channel of the difference between the frame and the average image, and is that difference is superior to a certain threshold, it does render the pixel from the average image, else it render pixel from the frame.

void main () {
  float y = uv.y * 3.0;
  vec3 averageColor = texture2D(average, uv).rgb;
  vec3 frameColor = texture2D(children, uv).rgb;
  vec3 canvas2DColor = texture2D(canvas2d, uv).rgb;
  if(canvas2DColor.r == 1.0) {
    if(dot(abs(frameColor - averageColor), vec3(1)) >= 0.25) {
      gl_FragColor = vec4(averageColor, 1.0);
    } else {
      gl_FragColor = vec4(frameColor, 1.0);
    }
  } else {
    gl_FragColor = vec4(frameColor, 1.0);
  }
}

The results is not super good but we could use more stuff from the previous research to improve the masking and get the car totally invisible... ( I think get rid of the average image and do the pixels background subtraction magic without it, reasoning on previous frames )

Also I use this amazing react API for webgl, incredible work: https://gl-react-cookbook.surge.sh/ , you can check out the examples, it expose a super nice API to integrate a webgl canvas in a react app and compose shaders, inject variables from the react app to the shaders ... It is only for 2D stuff, we render two triangles for the vertex shader and only get to play with the fragment shader, that's why I did the trick explained in the schema to have the positions, that was itself inspired by this crazy example: https://gl-react-cookbook.surge.sh/behindasteroids

After having myself wrote a simple frame comparison with raw webgl, this library enabled me at least to save 300 lines of code and 2 days of work to get to the present result I think 🙏..

Darkmode

To see the power of having webgl, I've added a darkmode (just check the box in little settings panel), which is not masking the car, just doing the simple difference of average with each frame

gl_FragColor = vec4(frameColor - averageColor, 1.0);

This shows how can we transform the scene on dynamically, in this case there is a re-render flash but we could switch shader without re-render.

Perfs and cross-browser compat:

With the simple shader without condition if else (like the one in the demo in the gl-react library: https://gl-react-cookbook.surge.sh/video ), it works really well on Chrome and mobile chrome, almost a 60 FPS on my 2-3y old samsung device... But with the more complex one (like the one I wrote), it is slower on my android, 20 FPS, and feels slow.. But I have read that if-else are not good in shader code, I should avoid using then.. so they is surely room for improvement.... It's kind of my first shader code I write, I have not experience in that field.

Then we have CORS issue with safari , safari mobile and firefox ( https://forums.developer.apple.com/thread/36725 ) , seems they don't implement the crossorigin="anonymous" correctly on the video element, maybe firefox can be fixed but safari definitely no..... But this would work if we serve videos from our server and not from vimeo. (more devops complexity...)

Also I noticed that when serving the video from the same domain to make it work, with safari and firefox the framerate drops sometime, it doesn't feel really smooth (comparing to chrome)... If we go further with the webgl implem I'll need to investigate that, as it doesn't seems to happen with the official example of the gl-react library: https://gl-react-cookbook.surge.sh/video

Next steps:

I'll likely not work more on that this week because I have lots in the pipeline, but this Proof of concept is pretty promising 😉 , the only thing that we need to take in account is that with the days budgeted for TrafficCam game, it seems hard to do the full mobile / desktop compatibility with webgl + all the other stuff in the roadmap. I think It could be realistic to have a desktop version (maybe chrome only) working well with WebGL and falling back on the SVG masking implem on mobile. We can discuss this "real world" time constraint on our next call.

But anyway the research here can be super useful for future projects.

from opendatacam.

@tdurand Just to let you know, in safari the video playback for the webgl implementation doesn't work. In Chrome and Firefox it does perfectly.

from opendatacam.

thanks @mmmmmmmmmmmmmmmm , yeah it is expected for safari, the only way is to host the video in the same domain and not on an external vimeo (or do a reverse proxy).

Firefox wasn't working correctly in local, but does work deployed, I suspect they allow the cross origin if you are on HTTPS ... Good news ! I just noticed some little FPS drops on firefox

from opendatacam.

@tdurand Great II!

IFs
No "ifs" in shaders, very slow and considered bad practice. The pattern I often used back in the day was something like this:

colorValue = (bool expression) * 1;
// bool expression is casted in shader to number and applied directly to pixel without an if in between

(Did you try whether the short form of if is faster? No idea ... but maybe worth a try)

gl_FragColor = (dot(abs(frameColor - averageColor), vec3(1)) >= 0.25) ? vec4(averageColor, 1.0) : vec4(frameColor, 1.0);

Color distance + threshold
Also if you are working with a color distance and a threshold ... I often had better result if I had squared the difference and then did a threshold gate thing. As then small changes become les important as big ones ... which often is exactly what you a need in the "color diff masking" realm.

CORS and Safari
A pitty ... not very keen for more dev ops. But maybe we can just implement it as an optional feature for all Chrome and Firefox users. Only if not too much effort. Lets discuss on Monday.

from opendatacam.

Great ! And cool you have some exp on shader code/perf and image processing, nice ideas ! Will try them on monday as I'm working on other stuff, or if you have little time and want to play with them you can just edit the shader code here : https://github.com/moovel/lab-traffic-cam/blob/master/app/components/webgl/BackgroundSubtraction.js . To run the app localy just npm install and npm run dev 😉

Agree for no more dev ops, though a reverse proxy could be simple to implement using some node library but I have not much background with that so maybe it can blow off some much needed time budget: https://github.com/Rob--W/cors-anywhere . Let's keep that in mind and see in the end if we have some time left !

from opendatacam.

The CORS reverse proxy doesn't look super crazy. Lets discuss on Monday. But I guess then the entire video traffic of all videos on iphone would have to be pipe trough our server. Seems not like the best idea ever.

from opendatacam.

Yep , and there is maybe more maintained library like this one: https://github.com/nodejitsu/node-http-proxy .. But indeed everything would pipe through our server, bandwith cost + also maybe would eat ressources from the other stuff. (or would need to have a separated instance on a subdomain , but more devops 😆 )

from opendatacam.

Update! (more of a memo for myself)

Shader perfs

Coded the shader without branching (ifs), but no perf gain on mobile, 20 FPS on my medium end android, not good enough.

precision highp float;
varying vec2 uv;
uniform sampler2D children;
uniform sampler2D average;
uniform sampler2D canvas2d;

float when_eq(float x, float y) {
  return 1.0 - abs(sign(x - y));
}

float when_gt(float x, float y) {
  return max(sign(x - y), 0.0);
}

float when_neq(float x, float y) {
  return abs(sign(x - y));
}

void main () {
  float y = uv.y * 3.0;
  vec3 averageColor = texture2D(average, uv).rgb;
  vec3 frameColor = texture2D(children, uv).rgb;
  vec3 canvas2DColor = texture2D(canvas2d, 

- uv).rgb;

  float isMaskedPixel = 0.0;
  vec3 outputColor = vec3(0);

  isMaskedPixel = 1.0 * when_eq(canvas2DColor.r, 1.0) * when_gt(dot(sqrt(abs(frameColor - averageColor)), vec3(1)), 0.90);
  
  outputColor += frameColor * when_neq(isMaskedPixel, 1.0);
  outputColor += averageColor * when_eq(isMaskedPixel, 1.0);

  gl_FragColor = vec4(outputColor, 1.0);
}

Shader feel:

@b-g Tried the squared root trick, it seems to help to separate things, but I think to get better results I need to workout better the pixel substraction maths + maybe try to get rid of the average images and reason only on the previous frame.

Next steps:

1. Integrate it to the game and see challenges and complexity
2. If 1. is validated, improve shader results and see if we can have push webgl perf to mobile also.

from opendatacam.

Kamino closed and cloned this issue to moovel/lab-beat-the-traffic

from opendatacam.