I've been having an issue with eye tracking and getting it to function.

I built animations for eye movement using the "Animator" drop down when making animations. Using specifically an example, "Animator.Right Eye Down-Up".

I've gotten my jaw, and blinking to function correctly, but I believe my issue is that I'm not using specifically a blend shape and the animator instead.

I do have the Paramaters set up correctly and do see them changing in my debug menu correctly.

Is this my issue? Or am I missing something else?
Animator

Eye Tracking FX

Eye blinking for reference, Which is using a blendshape in the animation.

Thanks!

doc:
vive-eye-tracking-120hz

callbacks obviously reduce costs in game thread (in a plugin for another game)
...and how can i build this project?

Build 1102 seems to have broken the mod again.

`[04:25:26.986] [VRCFaceTracking] [ERROR] System.TypeInitializationException: The type initializer for 'VRCFaceTracking.Params.EyeTrackingParams' threw an exception. ---> System.TypeInitializationException: The type initializer for 'ParamLib.ParamLib' threw an exception. ---> System.InvalidOperationException: Sequence contains no matching element

at System.Linq.Enumerable.First[TSource] (System.Collections.Generic.IEnumerable1[T] source, System.Func2[T,TResult] predicate) [0x00011] in :0

at ParamLib.ParamLib..cctor () [0x00038] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

--- End of inner exception stack trace ---

at ParamLib.BaseParam.ResetParam () [0x00000] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

at ParamLib.BaseParam..ctor (System.String paramName, VRC.SDK3.Avatars.ScriptableObjects.VRCExpressionParameters+ValueType paramType) [0x00014] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

at ParamLib.FloatBaseParam..ctor (System.String paramName, System.Boolean prioritised) [0x00000] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

at VRCFaceTracking.Params.XYEyeParameter..ctor (System.Func`2[T,TResult] getValueFunc, System.String xParamName, System.String yParamName) [0x0000d] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

at VRCFaceTracking.Params.EyeTrackingParams..cctor () [0x00005] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

--- End of inner exception stack trace ---

at VRCFaceTracking.MainMod.VRChat_OnUiManagerInit () [0x00000] in <25c1fa7823c444ffa59d48d1e5a003d3>:0

at MelonLoader.MelonHandler.VRChat_OnUiManagerInit () [0x00041] in <61bd2bb7fbe24a28bc977e05ef482f41>:0 `

Could you please add support for face tracking with new htc face tracker? It could be straight forward as it uses SRanipal runtime as well.

Help please :( I only use my avatars ive built around face tracking lol (Not an ideal system seeing VRC gets an update or two a month)

I found crashes in some avatars and worlds while using this mod.
It used to happen very infrequently before, but after this 2019 Unity update, it's happened very often.

Error message is "Fatal error in GC / Collecting from unknown thread"

Sample World : [https://vrchat.com/home/world/wrld_95898adc-bf3d-42fb-b82b-8a4cd5a8f25f]

Maybe if you using VRCFaceTracking, you can't be in this world for more than an hour. (If you have someone else's avatar, symptoms will appear more quickly and may not last even 3 minutes.)

I've been testing all day to isolate this issue and have confirmed that the crash doesn't occur with the mod removed.

First of all thanks a ton for making this! It's the one good use I've gotten out of my Pro Eye's tracking hardware.

Running Melonloader v0.2.7.2 with VRChat version 2020.4.1p5, build 1010.

Despite this being the VRC update that was supposed to fix parameter sync, the following parameters no longer update based on my eye's tracking positions, either locally or to other players. All stay at their default values of zero:
-LeftEyeX
-LeftEyeY
-RightEyeX
-RightEyeY
-RightEyeLid

I'm not currently using any other tracking parameters on my avatar so haven't been able to test them in game, but I suspect they won't update either. I've attached my most recent MelonLoader log to this report. If you need any more info or
help with testing, I'm Azuth #1016 on discord or just Azuth in VRC.

MelonLoader_20-11-06_18-47-26.730.log

First of all I really appreciate the great mods you made.
Using this mode, I was able to apply eye tracking.

However, the only inconvenient thing is that if you apply this mode, CPU Frametime is very high.

My CPU is 9900k (with overclock to 5.1Ghz) and GPU is RTX 3090.
Nevertheless, the frame is hard to exceed 45 when applying this mode.

I don't think this mod itself has any performance-degrading issues.
I am wondering if it is possible to improve performance by manually disabling the unused one of the various parameters.
If that's difficult or doesn't work, I hope at least you add the ability to easily turn this mod on and off in-game. (Using UIExpansionKit)

Once again, I sincerely thank you for making this mod, and we look forward to improving it further.

Not an issue per se, but it would be super useful if this mod had the ability to read a specifically formatted serial input from a COM port to be able to drive the eye and face parameters.
This would single handedly blow open the door for setting up all sorts of eye and facial input devices by way of setting up and programming a microcontroller / microcomputer to take in face data via all sorts of means and send them along in a relatively easy way to VRChat.
Examples of eye tracking:
http://onloop.net/hairyplotter/
https://create.arduino.cc/projecthub/H0meMadeGarbage/eye-motion-tracking-using-infrared-sensor-227467
https://www.youtube.com/watch?v=vV4CiNsfUYY / https://arxiv.org/pdf/2003.08499.pdf
Facial tracking would just require any of the relatively straightforward pattern recognition solutions (Literally anything that can run OpenCV or TensorFlow) to be bootstrapped to something that could be mounted to a HMD.

I'm just wondering if it would be possible to implement Tobii Pro VR eye tracking. I don't know how difficult it would be, considering it doesn't support Sranipal. It uses the Tobii Pro SDK.

Tobii Pro SDK and Setup:
https://www.tobiipro.com/product-listing/tobii-pro-sdk/

Documentation:
https://developer.tobiipro.com/

Unity guide:
https://vr.tobii.com/sdk/develop/unity/getting-started/tobii-htc-dev-kit/

Hi. I try to use this mod and according setting, sometimes confuse to detect SRanipal is initialized or not.

i'm now trying to detect some geek way (existing param check and not zero), but also think if add boolean param to detect initialize is more confortable.

VRCFaceTracking currently utilizes SRAnipal to drive avatar expressitivity. I would like to propose the creation of a separate branch that would enable Desktop-Mode users to have the same expressitivity via OpenSeeFace.

For reference, me and a friend have created an application that gives Desktop-Mode users in NeosVR this functionality. Given the overlap our systems have (outputting values to drive parameters), I am curious to see if we could have WCFaceTrack interface with VRC's Avatar Params this way. Please let me know your thoughts!

I had the eye tracking working last week. Now it is failing to launch in Melonloader. The TobiiEye tracking is working in the corner and the calibration.

Idea

Support the HP G2 Reverb Omnicept VR headset's eye and lip tracking.

Notes

Neos supports the eye tracking (Neos-Metaverse/NeosPublic#2275) but not the lip sync as the API only exposes the raw camera capture not any data (or maybe it does but it isn't well documented).

There is a suggestion in Neos to make a generic API for eye/lip input that sounds perfect for this mod: Neos-Metaverse/NeosPublic#1140

Been following the guide and have blink working left and right no problem however I have issues with movement. I setup Left eye animations and tested them out and they work on their own however once I setup the Right eye the way the guide tells me only the right eye will move the left eye just sits still once I delete the blend type for the right eye the left eye tracks again. I've tried combining animations and making both eyes move at the same time in the same blend type but again the left eye just sits still while the right eye moves. I have no idea how to get this to work with both eyes

Currently when closing your eyes or the eyes lose tracking they revert to the bottom right corner... this would be fine if it wasn't for VRchat's network interpolation smoothing the eye tracking out to the point of causing it to be very noticable... I have two ideas for fixing this
1: make the blend tree parameters revert to 0.5 0.5 instead of 0 0... (this would somewhat fix it but still be a bit jank)
2: make the eyes retain their last look direction when losing tracking (this would probably look the best)

Noticed that when my Vive wireless battery dies and I swap it out, my parameters are no longer updating. Eye/Face trackers are still working outside of VRC, but to get my parameters to update I'm forced to restart VRC. Tried the force reinitialization button in your menu, but it didn't seem to help. Also tried lesser things like resetting avatar, changing avatars and changing back, nothing seemed to work except restarting VRC completely.

Thank you!

This wouldn't be a problem if it wasn't for how VRC interpolates parameters over the network... the smoothing causes the pupils on avatars with pupil dilation to shrink really tiny when you blink in a very un-natural way.

crashes vrchat upon start with Pimax Droolon Pi-1

I believe the latest VRChat update that dropped a few hours ago broke Eye Tracking. Trying to get others to confirm.

Does anyone know of a working Avatar to test of the Lip Tacking? I been trying to get my first avatar edit to work but really want to see if my Facial Tracker even works at this point.

Sometimes when first joining a session... combined params default to 1 or -1 instead of 0... this can be fixed by quickly moving your face around a lot but is annoying to do.

MelonLoader_21-05-04_00-46-19.627.log
there's the log, ah shit here we go again...

I will add this in the next update, though it won't be documented until HTC actually activates it.

Originally posted by @benaclejames in #28 (comment)

Pre-release build does not function at all when I try to use it
latest release build functions flawlessly
Logs
MelonLoader_21-04-24_14-16-57.886.log

Video of what happens in-game

It would be neat to have LeftEyeBlink and RightEyeBlink boolean parameters that indicate whether the eye is fully closed to allow for simple blink animations that use 1 parameter bit each instead of 8 as with the LeftEyeLid and RightEyeLid parameters.

Go through all parameters and make the decision as to whether to deprecate them. As of now, we have far too many parameters for any sane person to comprehend, so undocument the rarely-used ones and deprecate ones that have newer and more efficient counterparts.

Was using it fine yesterday; however today I'm not getting any Eye Tracking or Blink Detection. Not sure yet if it's just me or others as well.

[02:00:16.766] VRCEyeTracking v1.3.0
[02:00:16.768] by benaclejames
[02:00:16.781] SHA256 Hash: 1b942e30558ab5118613cb230bff6ecb3ce1bfdb5c7f3f9dd32dae04cc672589
[02:00:16.782] ------------------------------
[02:00:16.784] [emmVRCLoader] [emmVRCLoader] Checking if emmVRC can Load...
[02:00:17.406] [emmVRCLoader] [emmVRCLoader] emmVRC can be loaded!
[02:00:17.407] [emmVRCLoader] [emmVRCLoader] Downloading emmVRC...
[02:00:18.338] [emmVRCLoader] [emmVRCLoader] OnApplicationStart called

Steam_SetMinidumpSteamID: Caching Steam ID: ManuallyRemoved [API loaded no]
[02:00:21.321] [emmVRCLoader] VRChat build is: 1078
[02:00:21.321] [emmVRCLoader] Initializing network...
[02:00:21.661] [emmVRCLoader] [emmVRC] Initializing Quickmenu HUD
[02:00:22.631] [emmVRCLoader] Initialization is successful in 01.3s. Welcome to emmVRC!
[02:00:22.632] [emmVRCLoader] You are running version 2.8.5
[02:00:22.638] [VRCEyeTracking] Initializing SRanipal...
[02:00:30.759] [VRCEyeTracking] SRanipal SDK Started. Eye Tracking Active

When running the mod with only lip tracking (Using Valve Index) I recieve the following output/error

[05:10:21.262] [VRCEyeTracking] Initializing SRanipal...
[05:10:26.309] [VRCEyeTracking] [WARNING] Eye Tracking will be unavailable for this session. (NOT_SUPPORT_EYE_TRACKING)
[05:10:26.313] [VRCEyeTracking] SRanipal Lip Initialized!

After this, the UI is never given the configuration button.

Keeping this short because I don't know what else to say about this. When you go to the SRanipal tab in the Quick Menu, and you preview the Lip Tracker camera, it freezes the parameter values (as seen visually on the avatar and from the Debug panel).

For example:

• The LeftEyeOpen and RightEyeOpen get stuck at 1
• The TongueX and TongueY got stuck at NaN
• CombinedEyeWiden was stuck at 0.05

It seems to happen to me only when I preview the Lip Tracker camera, but a simple restart fixes it. The log and console do not output any error about this at all, so I won't upload a log file unless requested. Any thing else needed and I'll be happy to get back to you!

I thank always for your work.

I've made the same request before. (#28)
And I found that Eye_Frown parameter works in v2 version from vive forum.

https://forum.vive.com/topic/8493-no-way-to-track-eyebrows/ (This sample is unreal v2 SDK)

I still think if this parameter is added, it will help to express more emotion.
Could you please add this parameter?

Since Varjo has now started shipping their first "consumer" headset, the Aero, I wanted to ask if there's plans for supporting the Varjo Eye Tracking SDK?
I'd gladly volunteer for testing (due to the high prices of said HMD, I think this should be a given).

While I am a developer, I personally have never properly worked with C++ code (which is what the Varjo Native SDK is using), nor have I worked with interfacing between C++ and C# before. So while I'd usually just do this myself, it's sadly a bit out of my scope ^^"

UPDATE:
After doing some research, it looks like someone already written an implementation for this, but hasn't merged his changes into this project yet (nor have they implemented the fix from Dec 13). The fork I'm talking about is this one: m3gagluk/VRCFaceTracking/varjo_companion
I'll keep this issue open for now (as it's yet to find it's way into the main branch), but if anyone on the project thinks differently, you may as well close it.

Idea: add combined eyelid parameters (left right and combined) that includes widening openness and squeezing...
EG: 0-0.2 = widen... 0.2-0.8=blink... 0.8-1.0 = squeeze...

Reason: To allow for more room for extra parameters with lip tracking.

So not sure whats going on and i have tried to debug my situation but nothing seems to work. I have a Eye pro and i have had avatars that in the past have worked with the mod. but for some reason it no longer works... but it does. Please read before asking if i have done something.

1: The software used for the Eye pro is RUNNING.
2: Yes i have restarted my PC.
3: Yes i have uninstalled and reinstalled the mod.
4: Yes i have gone in to make sure that parameters were updated (incase there was a change), and that they are still properly named and have capitalizations where needed.

I know the mod itself is working as because in the quick menu where you can check the status, it clearly shows it knows where my eyes are looking. Just the avatars eyes do not move, and my eye lids do not blink. ( AGAIN my avatars worked in the past but for some reason do not work now)

Hi I just booted up vrchat today and everything seemed fine but my avatar wasnt moving with my face movement so I checked the camera preview and it looked wierd, I was wondering if this has been seen before here or if my face tracker is just broken already. The green light on the side goes on and the red light next to the cameras goes on but it just doesnt track my face and the preview is the still image as shown in the image attached below. Sorry if this doesn't have anything to do with the mod but I can't find any other way to see if its on my end or not.

I always get the UAC promo from sr_runtime.exe when I start VRChat in desktop or without my facial tracker plugged in and SRanipal running. When I want to use facial tracking I always check if I have the Runtime running anyway because it's a bit flaky with actually detecting the tracker. I would like to have a toggle in UI Expansion Kit mod settings or even a flag in the MelonPreferences would be enough to disable this behavior, so I can start VRChat without a connected facial tracker or in desktop mode without needing to walk back to my PC and accepting the UAC prompt. Maybe it even makes sense to not start the mod in desktop mode by default in any circumstance because you can't use facial or lip tracking without using an additional dll for facial tracking with a webcam.

VRChat updated to build 1149.
And borkoen VRFaceTracking mod in now.

[11:58:27.998] [VRCFaceTracking] [ERROR] System.TypeLoadException: Could not resolve type with token 01000078 (from typeref, class/assembly MulticastDelegateNPublicSealedVoGaVRBoUnique, )
at VRCFaceTracking.MainMod.UiManagerInit () [0x00011] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.MainMod.CheckUiManager () [0x000ab] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.MainMod.OnUpdate () [0x00014] in <1f220596fa2c49ba961a12295f311152>:0
at MelonLoader.MelonHandler+<>c.b__48_0 (MelonLoader.MelonBase x) [0x00000] in <67e16b0234254e2183a98652371f8451>:0
at MelonLoader.MelonHandler+<>c__DisplayClass52_0.b__1 (MelonLoader.MelonMod x) [0x00000] in <67e16b0234254e2183a98652371f8451>:0
at MelonLoader.MelonHandler.InvokeMelonMethod[T] (System.Collections.Generic.List1[T]& melons, MelonLoader.MelonHandler+InvokeMelonMethodDelegate1[T] method, System.Boolean remove_failed) [0x00032] in <67e16b0234254e2183a98652371f8451>:0
[11:58:28.009] [emmVRCLoader] VRChat build is: 1149
[11:58:28.010] [emmVRCLoader] VRChat build is: 1149
[11:58:28.433] [emmVRCLoader] Initialization is successful in 00.4s. Welcome to emmVRC!
[11:58:28.433] [emmVRCLoader] You are running version 3.0.0
[11:58:28.618] [UI_Expansion_Kit] Decorating menus
[11:58:33.078] [VRCFaceTracking] Eye Tracking Initialized
[11:58:33.079] [VRCFaceTracking] Lip Tracking Initialized
TypeRef ResolutionScope not yet handled (139) for .EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique in image data-000001B1E45B5040
[11:58:41.978] [VRCFaceTracking] [ERROR] System.TypeLoadException: Could not resolve type with token 0100008c (from typeref, class/assembly EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique, )
at VRCFaceTracking.QuickMenu.QuickModeMenu.InitializeMenu () [0x0000f] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.QuickMenu.QuickModeMenu.CheckIfShouldInit () [0x00007] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.MainMod.OnSceneWasLoaded (System.Int32 level, System.String levelName) [0x00004] in <1f220596fa2c49ba961a12295f311152>:0
at MelonLoader.MelonHandler+<>c__DisplayClass44_0.b__0 (MelonLoader.MelonMod x) [0x0000c] in <67e16b0234254e2183a98652371f8451>:0
at MelonLoader.MelonHandler.InvokeMelonMethod[T] (System.Collections.Generic.List1[T]& melons, MelonLoader.MelonHandler+InvokeMelonMethodDelegate1[T] method, System.Boolean remove_failed) [0x00032] in <67e16b0234254e2183a98652371f8451>:0
TypeRef ResolutionScope not yet handled (139) for .EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique in image data-000001B1E45B5040
[12:02:31.205] [VRCFaceTracking] [ERROR] System.TypeLoadException: Could not resolve type with token 0100008c (from typeref, class/assembly EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique, )
at VRCFaceTracking.QuickMenu.QuickModeMenu.InitializeMenu () [0x0000f] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.QuickMenu.QuickModeMenu.CheckIfShouldInit () [0x00007] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.MainMod.OnSceneWasLoaded (System.Int32 level, System.String levelName) [0x00004] in <1f220596fa2c49ba961a12295f311152>:0
at MelonLoader.MelonHandler+<>c__DisplayClass44_0.b__0 (MelonLoader.MelonMod x) [0x0000c] in <67e16b0234254e2183a98652371f8451>:0
at MelonLoader.MelonHandler.InvokeMelonMethod[T] (System.Collections.Generic.List1[T]& melons, MelonLoader.MelonHandler+InvokeMelonMethodDelegate1[T] method, System.Boolean remove_failed) [0x00032] in <67e16b0234254e2183a98652371f8451>:0
TypeRef ResolutionScope not yet handled (139) for .EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique in image data-000001B1E45B5040
[12:03:31.892] [VRCFaceTracking] [ERROR] System.TypeLoadException: Could not resolve type with token 0100008c (from typeref, class/assembly EnumNPublicSealedvaUnCoHeToTaThShPeVoUnique, )
at VRCFaceTracking.QuickMenu.QuickModeMenu.InitializeMenu () [0x0000f] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.QuickMenu.QuickModeMenu.CheckIfShouldInit () [0x00007] in <1f220596fa2c49ba961a12295f311152>:0
at VRCFaceTracking.MainMod.OnSceneWasLoaded (System.Int32 level, System.String levelName) [0x00004] in <1f220596fa2c49ba961a12295f311152>:0
at MelonLoader.MelonHandler+<>c__DisplayClass44_0.b__0 (MelonLoader.MelonMod x) [0x0000c] in <67e16b0234254e2183a98652371f8451>:0
at MelonLoader.MelonHandler.InvokeMelonMethod[T] (System.Collections.Generic.List1[T]& melons, MelonLoader.MelonHandler+InvokeMelonMethodDelegate1[T] method, System.Boolean remove_failed) [0x00032] in <67e16b0234254e2183a98652371f8451>:0

seen it working, love the addon.
my issue is that the tutorial isnt very beginner friendly, and is missing alot of steps.
have you considered making a video tutorial or a discord? itd help people who are just getting started and get some more traction for the plugin..
if not, i highly reccommend you do avatar modding commissions, i garuntee people would pay you to get their avatars setup with this. i know of several people whod like to have their avi modded for eye/face tracking but dont understand how. im sure if you dont do it someone else probably will and better the creator do it right than someone else do it wrong and have people thinking your plugin is at fault/not functional.

can someone please send me the names of some working avatars so i can search them in emm. i searched for working ones for so long and dident found one and im not very good in unity so i dont know how to make these avatars

Currently with the "Threaded Fix" version of the mod my pupil dilation parameter is constantly stuck at 0

Release: 1.2.0
VRC Version: Build 1033
DLLs loaded in MelonLoader?: Yes

I've got an Avatar 3.0 package with existing Parameter fields. To test, I've added LeftEyeX, LeftEyeY, RightEyeX, RightEyeY, and LeftEyeLid as Float. EyeLook has been disabled in the descriptor, and Additive shows "None (Runtime Animator Controller)".

Based on the provided documentation, it's unclear if anything else needs to be added to the avatar package. There's mention of simple blendstates, but it's unclear if the parameter items are all that needs to be added or not based on the docs.

Hi, I finished setting up my avatar's lip parameters a few weeks ago and set up the mod but when I click on the tab I get the following screen (pic attached) and nothing happens when I move my lips. I am using:

• Valve Index
• VIVE Facial Tracker
• VRChat v. 2021.3.2 (most recent)
• VRChat SDK v 2021.08.11.15.16 (most recent)
• Unity 2019.4.29f1

There is eye frown parameter in SRanipal SDK.
It seems to be used for an angry expression or for lowering and raising eyebrows.
If this parameter is added, it will help to express more emotion.

I have ran into an issue with when Vive Wireless disconnects/resets with the Eye Tracking mod will fail to re-establish the eye tracking parameters. In order to reset the eye tracking have to restart the Melon loader/VR Chat to reestablish the port. This is fairly common with Vive Wireless when battery dies. I have tried restarting the Tobii VRU02 Runtime service while in VRChat and still will not load the parameters.

Add support for external tracking module dlls to allow developers to create support for their own tracking methods without having to modify the base VRCFaceTracking libraries.

API+Document can download when install Pimax + Eye tracking module
Not SRanipal. Language is C#. Can you add support it?.