Mirror Webcam Video about ml5-next-gen HOT 4 CLOSED

I think it should be possible to have a feature request for this in p5.js directly although implementation wise we might need to think about a way to do it consistently and be performant. Do open a feature request issue on p5.js repo is desired.

There is a way to do it currently that does not rely on new implementation with a bit of extra steps but probably quite performant:

let img;

function preload(){
  img = createCapture(VIDEO);
  img.size(320, 240);
  img.hide();
}

function setup(){
  createCanvas(400, 400);
}

function draw(){
  background(200);
  let graphics = createGraphics(img.width, img.height);
  graphics.translate(img.width, 0);
  graphics.scale(-1, 1);
  graphics.image(img, 0, 0);
  image(graphics, 0, 0);
}

And instead of performing the ml on the img capture object, you can perform it on the graphics object which is basically a wrapped HTML canvas. The scale function is how we get performant mirroring.

from ml5-next-gen.

Thank you @limzykenneth for the input! I have implemented similar example codes for students using Teachable Machine, since the default video input in the training interface is mirrored. I wanted to address specific issues in case Peter @ziyuan-linn chooses this approach.

• createCapture() isn't affected by the preload() function so it can still remain in the setup() function. This will be a more beginner-friendly suggestion, as students might want to adjust the webcam’s dimensions based on their canvas size.
• Although createGraphics() constructs only one instance in JavaScript, it continually generates an additional HTML Canvas element if called repeatedly within the draw() function. Storing the graphics in a global variable and creating it once in setup() is a more efficient approach.
• I would recommend storing the graphic in a global variable just once within the setup() function. This way, students won't need to use flippedVideo.remove() after sending the modified graphic to the classification model. This simplifies the code, reduces confusion, and minimizes the need for additional explanations.

Below are the codes included in Teachable Machine after training a model.

function setup() {
  createCanvas(320, 260);
  // Create the video
  video = createCapture(VIDEO);
  video.size(320, 240);
  video.hide();

  flippedVideo = ml5.flipImage(video);
  // Start classifying
  classifyVideo();
}

function classifyVideo() {
  flippedVideo = ml5.flipImage(video)
  classifier.classify(flippedVideo, gotResult);
  flippedVideo.remove();
}

Based on my experience, I've observed that many students attempted to increase the canvas and capture size, and often generated unexpected errors. Additionally, some students may not immediately understand the rationale behind removing the video element; they often omit the line.

Lastly, when students work with webcam images, having mirrored webcam images is essential for a more natural interactivity. However, explaining this mirroring technique with transformation functions can often cause unnecessary complexity, and translate() students' focus and understanding far away from the canvas. If the flipped video function is implemented in p5.js, it will greatly benefit a number of beginner-level students!

from ml5-next-gen.

Thanks @ziyuan-linn for opening the discussion here! I'm excited about this possibility. @joeyklee previously implemented a flipImage() function as part of the ml5.js "utilities"—flipImage() source. This function, however, only flips the pixels (horiztontally) of a single image.

Your proposal, as I understand it, would instead return a canvas that is continuously updated just as the p5.js video element is? That I think would make it much easier to use and manipulate for ml5.js coders when wanting to mirror their video! (I just did a quick search and don't see this is a common OS feature or a setting within browsers around the camera. Some third party camera drivers/software do include a "mirror video" option.)

I hope we're not bothering you too much @limzykenneth and @Qianqianye, but since this is not a machine learning specific feature, I'm wondering if (a) this was something that might already be on a p5.js roadmap or (b) if you would like us to contribute this as a feature to p5.js... or keep things simple and integrate into ml5.js for our specific use cases?

Thank you!!

from ml5-next-gen.

Just noting that the discussion can continue in #106

from ml5-next-gen.