It would be nice to get a linux release so we can avoid compiling from source

Currently, when the UI is toggled on, the UI will also show up in whatever capture program is being used. This is not desirable for most cases unless someone is making a tutorial.

Note: my only experience is with OBS, so this proposal is tailored to OBS
Note: every OS has different implementations/interfaces for texture sharing

Implementation

• Windows: Spout
  • OBS plugin is available
  • I've already written an engine module that integrates with Spout
• Linux: Pipewire DMA-buff
  • Might be available in OBS on Linux if the user is using Pipewire?
  • Rust library available for interacting with Pipewire
  • Can reuse some code from my Spout engine module in order to capture the raw texture handle
• Mac: Syphon
  • OBS plugin might be available?
  • Can reuse some code from my Spout engine module in order to capture the raw texture handle
  • Will need to reference KlakSyphon in order to write a Rust/GDScript/C# library for Godot
    • KlakSyphon is for Unity

Hello, what am I doing wrong? There is only "0 Default camera" in camera selection, I start tracker but nothing happens. Otherwise cam works well, recognized by OBS with no issues. It is the same with github version and aur version.

v4l2-ctl --list-devices
Droidcam (platform:v4l2loopback_dc-000):
	/dev/video0

openseeface-gd
Godot Engine v3.4.3.rc.custom_build.31f3e358d - https://godotengine.org
OpenGL ES 3.0 Renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2
OpenGL ES Batching: ON
 
[INFO] 2022-1-25_15:20:40 Begin loading metadata
[INFO] 2022-1-25_15:20:40 Finished loading metadata
[INFO] 2022-1-25_15:20:40 Loading metadata: metadata.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: model.xml
this is a test print in Model.gd. Extend this by modifying resources/gui/Model.gd
[INFO] 2022-1-25_15:20:40 Loading gui file: tracking.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: features.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: presets.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: app-settings.xml
[INFO] 2022-1-25_15:20:40 Begin loading config for user:///.json
[INFO] 2022-1-25_15:20:40 Finished loading config
[INFO] 2022-1-25_15:20:40 Loading Default Model.
[INFO] 2022-1-25_15:20:40 Loading TSCN file.
[INFO] 2022-1-25_15:20:40 TSCN file loaded successfully.
[INFO] 2022-1-25_15:20:40 Press TAB to hide the UI
ERROR: Condition "name == """ is true.
   at: set_name (scene/main/node.cpp:914)
[INFO] 2022-1-25_15:20:47 Starting face tracker.
[INFO] 2022-1-25_15:20:47 Face tracker started, PID is 750575.
[INFO] 2022-1-25_15:20:47 Press spacebar to recenter the model if it's not looking correct!
[INFO] 2022-1-25_15:20:47 Listening for data at 127.0.0.1 :11573
Traceback (most recent call last):
  File "/usr/share/openseeface-gd/OpenSeeFaceFolder/OpenSeeFace/facetracker.py", line 125, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
[INFO] 2022-1-25_15:20:52 Saving config
[INFO] 2022-1-25_15:20:52 Finished saving config
[INFO] 2022-1-25_15:21:4 Stopping face tracker.
[INFO] 2022-1-25_15:21:4 Face tracker stopped, PID was 750575.
[INFO] 2022-1-25_15:21:6 Saving config
[INFO] 2022-1-25_15:21:6 Finished saving config
[INFO] 2022-1-25_15:21:6 Exiting. おやすみ。

I see the eyebrow tracking from OSF is referenced in the code, but I guess it's not fully implemented? Apparently it's not a VRM standard, but VSeeFace implements this for VRM models with two custom blendshape clips, "Brows up" and "Brows down": https://www.vseeface.icu/#special-blendshapes

Combined with #105 I'm sure it could bring a lot of life to some models! 🙂

This occurs for both VRM and normal .glb models. In the Model view, left column, bones should be displayed as selected if they are indeed selected.

Currently the app is hard coded to run a Windows-only exe. Add an OS check and instead run a python command to run OSF instead.

Outline with VRM models using MToon shader doesn't render correctly or at all

Example of correct and incorrect rendering:

So my model is twisted and leaning down with the arms T-Posed. I don't know why, but I'm 99% sure that isn't supposed to happen. Model works in other VRM programs just fine and in Unity without issue.

Screenshot:

The program does track, I can see the head move when I move, and the eyes blink and mouth moves when I speak. Its just the default values seem to all be wrong.

Could there be a button to add a chromakay-able background (Green, Magenta, and Cyan)? Running on a *nix OS makes stuff like OBS's Game Capture not possible (with the exception of Linux running WayLand instead of X11 and using an AMD GPU with an OBS plugin) so having and option to toggle those 3 colors should allow chromakeying of almost anything.

The majority of the godot-vrm addon code is actually the code from ~January 2021 and there have been significant updates since then. The application is actually currently able to run off of Godot's 3.x branch but loading .vrm files is broken because of the outdated addon.

After chatting with the V-Sekai team, it looks like my modifications to the addon were unnecessary, so it should be possible to use the godot-vrm addon as a git submodule.

• update godot-vrm to be a git submodule
• update VRMMappings.gd to pull bone mappings from vrm_meta.gd instead of intercepting the mappings inside of import_vrm.gd. VRMMappings.gd is used for mapping facial blend shapes (e.g. blinking, mouth open, etc) to facial tracking

I talked to Emilianna a little bit and she said that VRM part names don't actually matter, and you should use the humanoid bone mapping in the avatar descriptor.

https://github.com/vrm-c/vrm-specification/tree/9a02cc2302c4c656e404fc1d505e9d9c4895d04d/specification/0.0#vrm-extension-models-bone-mapping-jsonextensionsvrmhumanoid

There are some tests that cannot be run in ci (e.g. runtime loading GDNative libraries). These tests are toggled to return early by checking for the existence of an environment variable called VSS_ENV.

Currently, this does not appear to be working. A workaround is in place by checking for the Server feature, but this is not ideal.

Add support for static images (or a sequence of static images) to be used as the model.

Could probably be fancy by applying Godot's Skeleton2D to the textures so that the illusion of model rotation is still there. Will need to look into doing this at runtime. This might eventually pave the way for live2d-esque characters.

An addition to this would be listening for some audio device, VoiceMeeter on Windows and PipeWire/JACK on Linux, to make the images bounce on audio input.

I see you're aiming to add Live2D support, would you be able to also look into supporting Inochi2D? Its a FOSS alternative to Live2D that's currently being developed, and the dev told me she has a rough ETA of a working version around summer of this year, 2021. Current progress is avalible on github already, and its all written in D.

My working camera does not start face tracker, I followed QuickStart instructions and tried running it with the terminal.

Godot Engine v3.4.beta.custom_build.5fdd3ff95 - https://godotengine.org
OpenGL ES 3.0 Renderer: AMD Radeon(TM) Vega 8 Graphics (RAVEN, DRM 3.41.0, 5.13.8-artix1-1, LLVM 12.0.1)
OpenGL ES Batching: ON
...
Loading VRM file.
...
Model has ultra nested meshes: .
Model has ultra nested meshes: .
Starting face tracker.
Listening for data at 127.0.0.1:11573
Face tracker started.
No face tracking data found. <--- Here
...
Stopping face tracker.
Face tracker stopped.
Saving config
Finished saving config
Exiting.

No face tracking data found, latest version

As seen in #92 and on Discord, some Linux distros use python 3.10 as the default system python. This causes the automatic package installation to fail.

A Linux-only option needs to be added to specify a python path. If it is left empty, the scripts will attempt to find a valid python interpreter. If a python path is provided, the scripts should use the provided path.

Not applicable for Windows as an executable for OpenSeeFace is provided from upstream.

1. Change the default model to something other than the default duck
2. Delete/move the new model
3. Open the application again
4. Observe the crash

Description

Platform: Linux (Arch)
Version: 0.8.0
Python (tracking): 3.9

When using different models, I often encounter flickering in many areas of the models. This is independent of whether or not the tracking is running, and also changes in intensity depending on the angle the model is in. This makes me believe, that it is z-fighting going on there.

Examples

Tracking completely disabled (not listening for data)

Tracking enabled (tracker not running)

Tracking enabled (tracker running

So I have been trying to compile this for Linux, but upon choosing to export after installing the Linux preset I get the loading bars for a few seconds before going away and leaving me with a file that does nothing when launched. The progress bar gets to about 40% before vanishing (crashing?) The resulting executable has executable permission. If there's an error code its there too quickly for me to see it. I've tried both Godot from the official page and compiling your fork with the same results.

First of all, thanks a bunch for this awesome software! I got it working on Void Linux by compiling Python 3.8 myself (Void comes with 3.10 in the package repository), per issue #93. I used VRM_Addon_for_Blender to configure and export a VRM directly from Blender - no intermediate software was required. Nice!

Perhaps releasing the app as a Flatpak on Flathub (App Submission docs) would be a good idea? The appropriate Python and libc etc. dependencies could be included or pulled in from a runtime (AFAIK), and maybe the Python packages in the venv could be pre-installed, too. Additional niceties provided by this approach includes easy updates for users, and automatic install/uninstall of a .desktop entry for launching the application.

I might have a go later at building a Flatpak and see what it takes, but in any case, this issue could serve as the place to discuss Flatpak.

As far as I know, OpenSeeFace does not support any tracking other than face. Is arm tracking going to be added?

This can currently be worked around be using the hotkey shortcut ctrl + scroll as mentioned in the README.

i noticed that theoretically app-config could be overwritten by a normal config, if that config is called app-config
maybe all 'normal' configs should get a prefix attached when saving/loading to prevent this?

That's actually a good point. Maybe appending model_config_ to the start of all 'normal' configs would make sense.

Originally posted by @you-win in https://github.com/you-win/openseeface-gd/issues/61#issuecomment-974113503

I'm trying to package openseeface-gd for nixOS.
When building, godot complains about a missing export_presets.cfg file.
Other godot based project I've packaged always had that file in the repo but here I couldn't find one.
I've seen that you've added the export_presets.cfg to the gitignore, on purpose, I assume.
Since I'm not too familiar with godot, can I just copy that file from a different project or could you please provide me one to build for Linux.

Pull in alongside your own (display 2+ models at once).

The Vignette Project looks like they have Live2D working here under an MIT license.

It might be nice to see how feasible it is to use it as a reference for a Godot-compatible implementation. It could be in either GDScript or as a C++ module. I would like to stay away from mixing the general codebase (aka scripts) between GDScript and C#.

One of the final features still missing is automatic facial expression detection for VRM models.

The plan is to have a UI where a snapshot of the tracking data for a specific facial expression is taken. e.g. The user makes an angry face and the application saves that data as the "angry face" expression. Then during actual tracking, when the tracking data comes close enough to a saved expression, the VRM model will switch expressions accordingly.

I built the lip sync library on PopOS 21.10 which has a very recent glibc version. If the target OS has an older glibc version, the library will not run citing version issues.

• addons
• entities
• screens
• utils

1. Go to the pose menu
2. Select a bone to modify that has a "." in the name (e.g. Left.UpperArm)
3. Click on "Apply properties"
4. Observe that nothing happens

Suggested resolution: Store bone names elsewhere (i.e. not as the node name)

ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: set_axis_angle: The axis Vector3 must be normalized.
   At: core/math/quat.cpp:238.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: set_axis_angle: The axis Vector3 must be normalized.
   At: core/math/quat.cpp:238.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.
ERROR: set_axis_angle: The axis Vector3 must be normalized.
   At: core/math/quat.cpp:238.
ERROR: get_quat: Basis must be normalized in order to be casted to a Quaternion. Use get_rotation_quat() or call orthonormalized() instead.
   At: core/math/basis.cpp:771.

I don't know how to solve this,is there anyway to silence or fix it?

I just purchased a webcam (Logitech Brio) , and in OBS I can see things are looking quite good.. but when I come into OpenSeeFace-GD .. it seems not to quite be consistently able to trigger the camera on.. (the little light doesn't turn on for instance)

The only way I can get it to work is..

open the program
immediately go to the tracker settings
select the first camera in the list (there are 4 for some reason)
hit start tracker

that will work.. then I turn it off.. and try to re-engage it and.. nothing

python3 --version
Python 3.8.10

./OpenSeeFaceGD.x86_64 
Godot Engine v3.4.3.rc.custom_build.6658f5c47 - https://godotengine.org
OpenGL ES 3.0 Renderer: NVIDIA GeForce GTX 1060 6GB/PCIe/SSE2
OpenGL ES Batching: ON
 
[INFO] 2022-2-21_17:56:29 Begin loading metadata
[INFO] 2022-2-21_17:56:29 Finished loading metadata
ERROR: Can't open dynamic library: addons/real-time-lip-sync-gd/libreal_time_lip_sync_gd.so. Error: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/emcp/Tools/openseeface-gd_0.8.1_linux/libreal_time_lip_sync_gd.so)
   at: open_dynamic_library (drivers/unix/os_unix.cpp:403)
ERROR: No valid library handle, can't get symbol from GDNative object
   at: get_symbol (modules/gdnative/gdnative.cpp:510)
ERROR: No nativescript_init in "res://addons/real-time-lip-sync-gd/libreal_time_lip_sync_gd.so" found
   at: init_library (modules/gdnative/nativescript/nativescript.cpp:1510)
ERROR: In Object of type 'NativeScript': Attempt to connect nonexistent signal 'lip_sync_updated' to method 'Node._on_lip_sync_updated'.
   at: connect (core/object.cpp:1462)
ERROR: In Object of type 'NativeScript': Attempt to connect nonexistent signal 'lip_sync_panicked' to method 'Node._on_lip_sync_panicked'.
   at: connect (core/object.cpp:1462)
[INFO] 2022-2-21_17:56:29 Loading metadata: metadata.xml
[INFO] 2022-2-21_17:56:29 Loading gui file: model.xml
this is a test print in Model.gd. Extend this by modifying resources/gui/Model.gd
[INFO] 2022-2-21_17:56:29 Loading gui file: tracking.xml
[INFO] 2022-2-21_17:56:29 Loading gui file: features.xml
[INFO] 2022-2-21_17:56:29 Loading gui file: presets.xml
[INFO] 2022-2-21_17:56:29 Loading gui file: app-settings.xml
[INFO] 2022-2-21_17:56:29 Begin loading config for user:///hackz0r_final_seeface.json
[INFO] 2022-2-21_17:56:29 Finished loading config
[INFO] 2022-2-21_17:56:29 Loading VRM file.
VRM: Using builtin gltf module
/home/emcp/Downloads/hackz0r_final_seeface.vrm
WARNING: Octahedral compression cannot be used to compress a zero-length vector, please use normalized normal values or disable octahedral compression
     at: norm_to_oct (servers/visual_server.cpp:341)
[INFO] 2022-2-21_17:56:29 VRM file loaded successfully.
[INFO] 2022-2-21_17:56:30 Press ESCAPE to hide the UI
ERROR: Condition "name == """ is true.
   at: set_name (scene/main/node.cpp:914)
[INFO] 2022-2-21_17:56:30 Model has ultra nested meshes: .
[INFO] 2022-2-21_17:56:30 Model has ultra nested meshes: .

EDIT: Stranger still , I reopen the app and now see no cameras at all.. and in output I can see

There was no valid input.
[INFO] 2022-2-21_17:58:19 Finished saving config
[INFO] 2022-2-21_17:58:19 Exiting. おやすみ。
ERROR: No valid library handle, can't terminate GDNative object
   at: terminate (modules/gdnative/gdnative.cpp:417)

1. Add a Github workflow
2. Take the artifact from the latest 3.x branch on Godot Engine
3. Export the project
4. Combine with openseeface
5. Save as artifact

Many people have multiple cameras and with the default to id 0, this might break on some systems. There should be a camera selector dropdown that lets you change which camera OpenSeeFace uses.

The error in the logs is something to the effect of "Unable to call function _start_tracker"

The issue is somewhere in OpenSeeGD.gd at line 269 when generating run options.

When loading in a .vrm model, the head bone field is not populated. The model still tracks as expected.

Aka focal length. Not sure how Godot does it. Would be nice to have an option for this to simulate a wider lens camera, making objects appear more "flat" and less "fish-eye" distorted.

This will allow users to type chat and the avatar response with LipSync corresponding to Text to Speech.

• Use one archive
• Provide builds on Windows, Linux and Mac
• Does not also require installing openseeface separately
• Use https://github.com/chriskiehl/Gooey.git to make a gui for the python program https://github.com/emilianavt/OpenSeeFace

Add lip sync support. This will likely need to be done as a GDNative library for performance reasons. Rust will probably be the implementation language.

uLipSync (MIT) can be used as a reference even though it's a Unity library.

godot-audio-processing Can be used as a reference for feeding data from Godot and maybe interpreting the lip synced data?

NOTE: There's no license for godot-audio-processing, but I did ask on Discord if I could use the repo as a reference

personal fork of uLipSync

personal fork of godot-audio-processing

on linux with xorg and picom, enabling transparent background just makes the background black

weirdly, the godot splash screen has correctly functioning transparency

I think that it should be possible to turn on and off accessories like hairclips,clothes,helmets, glowing objects, objects on fire and not on fire and so on..

I have searched on the internet's to figure out if the vrm file standard supports this functionality, but no luck on finding a definite answer. It seems that it support "accessories", but there is nothing to be found on what that means.

I think that this would be a very nice feature to have.

Hi , I can see maybe I am an edgecase but.. running LTS 20.04 Kubuntu and trying to start this build

openseeface-gd 0.8.1

but it just gives an error about GLIBC.. I believe I have it but.. unclear how to proceed other than attempt to compile myself

GLIBC_2.33 wasn't found

1. Preset data breaks face tracking when loaded in
2. Preset data for other models is being displayed, which is a planned feature but should not be happening now

I've got the issue on Linux, that only one Camera shows up.

I've got a virtual camera from obs and my webcam.

# v4l2-ctl --list-devices
Dummy video device (0x0000) (platform:v4l2loopback-000):
	/dev/video0

Video Capture 4 (usb-0000:0a:00.3-1):
	/dev/video1
	/dev/video2
	/dev/media0

In the tracker camera select, only one camera shows up which is the virtual one.
I can tell that cause if the virtual camera in obs is deactivated, I get the following error when I start the tracker:
[ WARN:[email protected]] global /io/opencv/modules/videoio/src/cap_v4l.cpp (1000) tryIoctl VIDEOIO(V4L2:/dev/video0): select() timeout.
While it throws now warnings or errors if it's running.

If you need more information, to track this, please let me know.

dragonbones editor is a free software.And the runtime is opensource.you can add this file format support .
dragonbones can also make animation like live2d ,and it is opensource.

This functionality used to exist pre 0.7.0. Then I broke it when I rewrote the UI, so it needs to be refactored.

The functionality should probably be split into

1. Bone tracking <- easy, less accurate
2. Vertex tracking <- might melt your system idk, hyper accurate

OS:debian buster
When u try to run it on this,it can't be launched and print

./OpenSeeFace-gd.x86_64: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by ./OpenSeeFace-gd.x86_64)

It come with a package but upgrade that is too dangerous

On Windows, always setting --dcap (-D) to -1 just uses the default camera settings and ignores the fps/width/height params.

would love to see the same feature we have on vseeface to use AR (using phone ip) for face tracking

personally i use MeowFace on android, this would be huge for linux vtubers, hopefully it's possible!

maybe this will help you figure it out? https://github.com/StarfishHidari/neosvr-arkit