Javascript/WebGL lightweight face tracking library designed for augmented reality webcam filters. Features : multiple faces detection, rotation, mouth opening. Various integration examples are provided (Three.js, Babylon.js, FaceSwap, Canvas2D, CSS3D...).
Home Page: https://jeeliz.com
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
https://github.com/mrdoob/three.js/tree/dev/utils/exporters/blender
Should we update our tutorials?
I tried search
First, thanks for this good library.
Describe the bug
demos/threejs/miel_pops/index.html
I made a new page. Then I created an action to go to the video page. However, I found that the video page does not play or play when you open a new window
To Reproduce
Steps to reproduce the behavior:
<html>
<head>
<script>
</script>
</head>
<body>
<button style='font-size:10vw'; onclick='window.open("demos/threejs/miel_pops/index.html")'>new window</button>
</body>
</html>attempt: 1~6
First attempt = playing video
Second attempt = can't playing video
Thrid attempt = playing video
Fourth attempt = can't playing video
I tried other cases.
I tried with 'location.href' instead of 'window.open'. it's always playing video. I think
it's a problem with safari and 'window.open'.
When the video is played back, the video plays even if the page is refreshed. However, when the video is not playing, the video does not play when the page is refreshed.
Expected behavior
Everyday can playing video.
Desktop (please complete the following information):
Does not happen with Desktop
Smartphone (please complete the following information):
Occurs only on Mobile Safari(Does not happen with developer mobile tools)
Additional context
I was trying to get a point of getting the video rights. but it's in jeelizFaceFilterES6. I'm sorry that the situation is complicated.
Hello, thanks for this good library.
I have question at demo.
Device: Mac OS
Browser: Chrome
I think It occurs in most environments.
gltf_fullScreen/index.html
I added spotLight(castShadow on) in this demo. At this time, renderer have border and not full screen.
THREERENDERER.shadowMap.enabled = true;
var spotLight = new THREE.SpotLight( 0xffffff );
spotLight.castShadow = true;
THREESCENE.add(spotLight)I tried several attempts. It occurs by shadow. And this only occurs in this demo(maybe it's fullscreen). I hesitated very much to put this issue on here or on three js. I am sorry if it is a matter of three js.
Best,
whilemouse
Hey Jeeliz,
I've been using the Football Fan effect as a foundation for a project (it's very very good) and would love to be able to add some elements to the chin. When the mouth is opening, though, the texture image doesn't scale. Can you recommend the most efficient way of making the y scale of the 'mask' respond to the mouth movement?
Thanks so much!
Describe the bug
NOTE: Using an established video element
Error thrown (twice) on iOS (as seen in issue #14), but tracking initiates and works if facingMode has not changed:
WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL
If the facingMode has been changed ("user" or "environment", as seen below), tracking fails to start.
this.mediaDevices.getUserMedia({
"audio": false,
"video": this.params.constraints || {
minWidth: this.params.dest_width,
minHeight: this.params.dest_height,
facingMode: "user"
}
})
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Face tracking should start, regardless of if the camera has been re-bound with new constraints prior to init.
Smartphone (please complete the following information):
Hey again,
I'm curious is there's a way to modify the easing on elements when you move using the Football fan demo. I didn't see any speed variables and thought it might be a three.js value. As it currently works, it seems the mesh takes a second to catch up with me.
Also, when I turn my head left or right, the position of the elements scales in a strange way. To the left, everything gets smaller, to the right, everything gets bigger.
Thanks again!
While using iOS 12 and enabled WebGL2.0 in advanced setting, it goes to GL_INCOMPATIBLE error.
When using facefilter through npm I realized that it is not convenient to import and use the library in my project which is made up of many modules and components. To access the api I need to reference window element like this:
window['JEEFACEFILTERAPI']
which is not really an elegant solution
In my opinion it would be much better if the lib would export function which can be utilized within other scripts so it can be accesed like that:
import facefilter from 'dist/jeelizFaceFilter';
function foo(){
facefilter.init(...);
}
Hello again,
I'm doing some tests with 2d masks for prototypes and am getting inconsistent eye placement across devices. I'm using the jeeFaceFilterCanvas to hold it. Here's my tweaks for the object. This image is closely cropped.
var SETTINGS={
rotationOffsetX: 0, //negative -> look upper. in radians
cameraFOV: 40, //in degrees, 3D camera FOV
pivotOffsetYZ: [0,-0], //position the rotation pivot along Y and Z axis
detectionThreshold: 0.9, //sensibility, between 0 and 1. Less -> more sensitive
detectionHysteresis: 0.02,
mouthOpeningThreshold: 0.5, //sensibility of mouth opening, between 0 and 1
mouthOpeningHysteresis: 0.05,
scale: [4.5,4.5], //scale of the DIV along horizontal and vertical axis
positionOffset: [-1.55, -.9,-0.2] //set a 3D position fofset to the div
};
Any suggestions?
Thanks!
Hey guys,
Again, loving what you've done! Looking forward to sharing what I've been doing with it with you soon. :)
Do you have any projects in the works (or know of any APIs) that allow you to put a users face in another environment? Like this example :P: https://imgur.com/a/oDdS2N5
Thanks!
Uncaught DOMException: Failed to execute 'texImage2D' on 'WebGL2RenderingContext': Tainted canvases may not be loaded
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
First of all, thank you for the hard work you did and for sharing it :)
Is your feature request related to a problem? Please describe.
I am trying to make a photobooth where users can choose multiple filters (like hats, glasses, jewelry), and activate or desactivate some.
I'm using CSS3, which target an id:
DIV=document.getElementById('jeelizFaceFilterFollow');
Describe the solution you'd like
It would be nice to create as many as wanted tags, and to target them with a class, or with multiple ids.
Describe the bug
iPhone 7 and 8 for mobile safari face tracking seems to be broken.
Specifically, if you look at the 2d face paint demo, the canvas loads, shows the first few images, then freezes. Console shows the following error: WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I'd except the video stream / canvas to not freeze while tracking face. It works properly on desktop and android mobile.
Screenshots
n/a
Smartphone (please complete the following information):
Firstly, this is a fantastic library.. really well implemented and documented. Although i did get stuck in the following places.
Using the faceLowPoly.json can i highlight the meshlines in the video, currently when i try to do it. I get the entire mesh as 1 solid object.
Further extension of the one above, can i color certain triangles(facial markers) like lips eyes with colors? Basically detect the lips and then color them..
If so how do i go about doing this
Describe the solution you'd like
If it were possible to make background segmentation available it would be terrific.
Hello people, I'am facing a strange problem with multiple faces detection. After detect first face, the API take a longer time to detect the secound face, and sometimes detect only the secound face and ignore the first face.
I don't know if I'm doing something wrong, there's no diference from my code and the demos, except the fact that I'm using plane geometry and loading .png images instead of 3d models.
` var threeStuffs = THREE.JeelizHelper.init(spec, SETTINGS.maxFaces, detect_callback);
//Carrego o primeiro sticker
geometry = new THREE.PlaneGeometry(1,1,1);
material = new THREE.MeshBasicMaterial({
map : THREE.ImageUtils.loadTexture('assets/stickers/sticker1.png'),
transparent: true,
//color: 0x0000ff,
side: THREE.DoubleSide
});
plane = new THREE.Mesh(geometry, material);
plane.scale.set(3,3,3);
plane.position.set(0.0, 1.1, 0.0);
plane.frustumCulled = false;
threeStuffs.faceObjects.forEach(function(faceObject) {
faceObject.add(plane.clone());
});
//CREATE THE CAMERA
var aspecRatio=spec.canvasElement.width / spec.canvasElement.height;
THREECAMERA=new THREE.PerspectiveCamera(SETTINGS.cameraFOV, aspecRatio, 0.1, 100);`
Thank you very much!!!
I am trying with canvas2d face track example. Face rectangle comes somewhere below the face for the first few frames. Is there any reason for it?
Is your feature request related to a problem? Please describe.
How's our set position thing works?
Can you please explain how we can put sunglasses and hat on their exact place?
On the other hand, can you please explain me how can I create this three values for sunglasses?
hatMesh.scale.multiplyScalar(1.2);
hatMesh.rotation.set(0, -40, 0);
hatMesh.position.set(0.0, 0.6, 0.0);
Describe the solution you'd like
Should we create more tutorials?
Additional context
I want to place the frame below my face. I tried CANVAS2D width and height, but no effect.
Describe the bug
I have tried fiddling with Optionnal init arguments by addingidealwidth and idealheight in videoSetting but it does not seem to work.
function init_faceFilter(videoSettings){
JEEFACEFILTERAPI.init({
canvasId: 'jeeFaceFilterCanvas',
NNCpath: '../jeelizFaceFilter/dist/', // root of NNC.json file
callbackReady: function(errCode, spec){
if (errCode){
console.log('AN ERROR HAPPENS. ERR =', errCode);
return;
}
console.log('INFO : JEEFACEFILTERAPI IS READY');
init_threeScene(spec);
}, //end callbackReady()
//called at each render iteration (drawing loop) :
callbackTrack: function(detectState){
onUpdate();
THREE.JeelizHelper.render(detectState, THREECAMERA);
}, //end callbackTrack()
videoSetting:{'idealWidth':720,'idealHeight':1280}
// videoSetting:{'idealWidth':1280,'idealHeight':720}
}); //end JEEFACEFILTERAPI.init call
} // end main()<body onload="main()" style='color: white'>
<!-- CANVAS BEHIND : VIDEO FACEFILTER ONLY -->
<canvas id='jeeFaceFilterCanvas' style="width :100vw; height: auto;"></canvas>
<div id="reso" style ="font-size:2em; color: black"></div>
<!-- CANVAS ABOVE : AR WITH THREE.JS -->
<!-- <canvas width="600" height="600" id='threejsCanvas'></canvas> -->
<button id="btn" style="position: relative; font-size:inherit;">Record</button>
</body>Result
Webcam texture seems to be in landscape mode still.
Webcam texture in landscape mode
I am using Canvas2D tracking example.
Hi all,
Is there any chance you'll be adding a rear-facing camera option in the not-too-distant future? I've seen other webAR executions of this option (for example - https://www.vertebrae.com/mobile-web-ar-advertising/#more-2644).
Again, love what you've been doing and look forward to what's next!
Thanks so much for this API, it's great!
I'm curious (and not very experienced with threeJS) - do you know a way to import materials via the JSON model as opposed to loading a texture image? I'm trying to determine a workflow that's as streamlined as possible.
Also, do you use JSON models purely as a preference?
I'm trying to get the face detection to run with an exiting video element that gets video from the local webcam. I don't see an error but for some reason it doesn't detect anything. I'm using Chrome 70 on Windows.
(If I remove the videoSettings then it starts working right away)
I did research jeelizFaceFilter and jeelizGlassesVTOWidget. Also I did develop web application with jeelizFaceFilter. Because jeelizGlassesVTOWidget library is minified.
But jeelizFaceFilter is slower than jeelizGlassesVTOWidget in face recognition. How can I do it faster like jeelizGlassesVTOWidget in face recognition?
I uploaded video.
thanks for jeeliz developer
Hey there,
Thanks for the continued great work with the demos! Unfortunately, the Canvas2D demo doesn't work on my iPhone. Here's what I'm getting in the console:
[Error] WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL texImage2D b (jeelizFaceFilter.js:86:280) (anonymous function) (jeelizFaceFilter.js:88) l (jeelizFaceFilter.js:88:100) l (jeelizFaceFilter.js:46:152) pc (jeelizFaceFilter.js:53:355) N (jeelizFaceFilter.js:125:260) (anonymous function) (jeelizFaceFilter.js:131) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: generateMipmap: level 0 not power of 2 or not all the same size generateMipmap b (jeelizFaceFilter.js:86:332) (anonymous function) (jeelizFaceFilter.js:88) l (jeelizFaceFilter.js:88:100) l (jeelizFaceFilter.js:46:152) pc (jeelizFaceFilter.js:53:355) N (jeelizFaceFilter.js:125:260) (anonymous function) (jeelizFaceFilter.js:131) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL texImage2D f (jeelizFaceFilter.js:67:474) a (jeelizFaceFilter.js:79) d (jeelizFaceFilter.js:62) a (jeelizFaceFilter.js:79) a (jeelizFaceFilter.js:96:505) a (jeelizFaceFilter.js:89:142) (anonymous function) (jeelizFaceFilter.js:103:145) map Ac (jeelizFaceFilter.js:103:115) (anonymous function) (jeelizFaceFilter.js:131) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: drawElements: attempt to access out of bounds arrays drawElements f (jeelizFaceFilter.js:56:469) ha (jeelizFaceFilter.js:120:220) fa (jeelizFaceFilter.js:119:302) k (jeelizFaceFilter.js:129:453) (anonymous function) (jeelizFaceFilter.js:135:445) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: drawElements: attempt to access out of bounds arrays drawElements f (jeelizFaceFilter.js:56:469) ha (jeelizFaceFilter.js:120:303) fa (jeelizFaceFilter.js:119:302) k (jeelizFaceFilter.js:129:453) (anonymous function) (jeelizFaceFilter.js:135:445) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: drawElements: attempt to access out of bounds arrays drawElements f (jeelizFaceFilter.js:56:469) G (jeelizFaceFilter.js:92:180) (anonymous function) (jeelizFaceFilter.js:103:300) forEach G (jeelizFaceFilter.js:103:282) ha (jeelizFaceFilter.js:120:322) fa (jeelizFaceFilter.js:119:302) k (jeelizFaceFilter.js:129:453) (anonymous function) (jeelizFaceFilter.js:135:445) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: drawElements: attempt to access out of bounds arrays drawElements f (jeelizFaceFilter.js:56:469) wa (jeelizFaceFilter.js:112) G (jeelizFaceFilter.js:92:202) (anonymous function) (jeelizFaceFilter.js:103:300) forEach G (jeelizFaceFilter.js:103:282) ha (jeelizFaceFilter.js:120:322) fa (jeelizFaceFilter.js:119:302) k (jeelizFaceFilter.js:129:453) (anonymous function) (jeelizFaceFilter.js:135:445) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241) [Error] WebGL: INVALID_OPERATION: drawElements: attempt to access out of bounds arrays drawElements f (jeelizFaceFilter.js:56:469) wa (jeelizFaceFilter.js:112) G (jeelizFaceFilter.js:92:202) (anonymous function) (jeelizFaceFilter.js:103:300) forEach G (jeelizFaceFilter.js:103:282) ha (jeelizFaceFilter.js:120:322) fa (jeelizFaceFilter.js:119:302) k (jeelizFaceFilter.js:129:453) (anonymous function) (jeelizFaceFilter.js:135:445) (anonymous function) (jeelizFaceFilter.js:125:118) onreadystatechange (jeelizFaceFilter.js:19:241)
Is your feature request related to a problem? Please describe.
We have surveillance IP camera to detect faces from 10-20m distance. we want to detect faces at low lighting conditions using javascript framework.
Describe the solution you'd like
We use dlib for face recognition (using python framework). Using detected faces from jeeliz framework and process it using python framework.
Describe alternatives you've considered
we tried using haarcascade.xml for face recognition it is detecting non-faces as faces and the image quality is very low.
Pls suggest how can we use your js framework along with python framework.
when i set <canvas width="1080" height="1920"></canvas>
i find the face position downward migration
i tried using JEEFACEFILTERAPI.resize() but it not work
:)
Amazing library!
Nice to finally see an alternative to the existing (slow) ones.
Is there any intent to add the option to track multiple faces in the future?
Thanks.
I have a button, javascript function is called onclick. I want to stop the preview in javascript function.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Hey guys,
I'm playing with the div project, replacing the rectangle with an image. That was my only change.
I notice when rotating the camera away from my face, the image flies off wildly. I've played with the sensitivity settings and it still happens. Do you have any suggestions?
First, thanks for this good library.
Describe the bug
I used three.js demo and modified. video stopped when fullscreen mode with viewport. Probably it happened where only old iPhone.
To Reproduce
Steps to reproduce the behavior:
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
</head>function main(){
JeelizResizer.size_canvas({
canvasId: 'jeeFaceFilterCanvas',
isFullScreen: true,
callback: function(isError, bestVideoSettings){
init_faceFilter(bestVideoSettings);
}
})
}Expected behavior
Playing video
Screenshots
I don't upload. because screenshot can't know playing or stopped.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
My test device
is there anyway we can integrate this with Unity 3d?
I'm rebuilding the HeadCursor demo using head tracking to emulate mouse movement of a fake cursor i have on the page. I have code running, but the STABI.xy values never update.
Describe the solution you'd like
See some method of this demo using something like Vue or React.
Describe alternatives you've considered
I could always create this project the "old school" way not using a framework like Vue, but I really wanted to attempt recreating this using VueJS
Additional context
To see what I mean please run git clone https://github.com/tetreault/face-filter-experiments and then cd face-filter-experiments and npm i and then go to localhost:3000/headmouse and check out the console log for what I mean.
The small isolated code snippet can be found inside face-filter-experiments/pages/headmouse.vue
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Describe the bug
The actual behavior of the filter is using the user camera, this time i already have a video to apply the filter to, but can't manage to change the library source to take that video instead of the camera, i tried with the dog face filter demo and the babylon cube, but no luck, where can i change that source on those demos?
To Reproduce
Steps to reproduce the behavior:
This is actually not a error/bug
Expected behavior
what i'm expected to happen is the dog face filter or the babylon cube working using an actual video tag as source, i can send that video to a canvas if is necessary
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
-Chrome, Safari, Firefox
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
Congratulations for this incredible library!
Can I get precise coordinates of eyes, mouth and nose?
Let me explain what I want to do ... I'm not a blander/threeJs professional and I want to use CSS3 capabilities to position objects on camera using coordinates delivery from jellizFaceFilter library. This is possible?
Thanks for giving this awesome face filters.
I created a mobile app with gif face replacement but how to save the output?.
It has 2 canvas. I can able to record video from only 1 canvas at a time.
Then gif file size is too high and quality is very less. Is it possible to do face replacement with any video formats such as webm, mp4 etc instead of gif?
I tried in Moto G4 Play and Lenovo phone. The Camera is not opening even after 2mins.
Describe the bug
I've upgraded to Android 9. Jeeliz projects now throw a "GL_Incompatible" error in Chrome 69. They work as expected in Firefox on Android, so it seems to be Chrome specific. I have tested and seen the same result on 2 Pixel 2s.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Jeeliz projects to load
Screenshots
Smartphone (please complete the following information):
As usual, I'm a big fan of your work and hope it's a simple resolution.
First, thanks for this good library.
My environment:
NNC Version:
I was using it well on iOS 11. but I updated it to iOS 12 and I found the 3d model shaking. Is it a problem with ios12?
Describe the solution you'd like
Hi!
I want to be able to reload jeelizFaceFilter or pass another video source to it. Now it generates "already initialized" error when I try to start jeelizFaceFilter more than once.
Is it possible?
I noticed that using JEEFACEFILTER.init() only allows for detection of the face on the video stream accessed through getUserMedia(). As far as I understand there is no way to control the image/video which jeeliz analyses. I found out however that your sunglasses example allows to upload a custom picture.
Is it possible to input custom image/video stream to jeelizFaceFilter ? If yes then how can I do it ? I've looked through the readme and tutorials but with no luck :(
Describe the bug
I'm unable to run jeeliz face filter on safari. I'm getting the 'webcam_unavailable' error.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
You should see the camera, but you're only seeing a blank page.
Screenshots
Desktop (please complete the following information):
Smartphone (please complete the following information):
Is your feature request related to a problem? Please describe.
In your examples the face detection is broken when the head is rotated horizontally.
Describe the solution you'd like
The in plane face rotation should be properly detected.
Describe alternatives you've considered
There is no good alternative. Some random samples are:
https://github.com/auduno/clmtrackr
https://github.com/YadiraF/PRNet
https://github.com/cmusatyalab/openface
https://github.com/MarekKowalski/DeepAlignmentNetwork
Additional context
The rotation detection should not hinter accuracy or performance much. It also could be a separate pre-processing step.
First, thanks for this good library.
Can I mapping tracked face size and three js model size? People's face sizes are different. but three js model size is one.
ex) https://tastenkunst.github.io/brfv4_javascript_examples
I tried your other project. I want make 3d model more nicely.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.