patniemeyer / vgg-memoji Goto Github PK

I'm interested in visual novels - and did some digging into crafting 2d characters and came across Japanese company live2d. http://www.live2d.com/en/showcase

While at first this may not seem related and a bit of tangent - hear me out. So under the hood of Apple's technology it seems some smarts from live2d have been leveraged. Have a look at this program which integrates the face tracking with illustrated character
https://github.com/matsune/Cubism3-ARKit

There's a pipeline of work I'm going to float here - and apologies if this off topic.

Potentially - something to explore could be reverse engineering memoji technology into live2d - then you'd have textures maps and ability to craft expressions programatically as you mentioned in blog. Each character has sets of params that can be tweaked as well expressions / animations.
be happy / be sad etc.

This is a texture map

https://github.com/matsune/Cubism3-ARKit/blob/master/model/Haru/Haru.2048/texture_00.png

and while it may not look like an memoji - if the memoji parts could be deconstructed - you could get something en par.

Inside Cubism3-ARKit / there's some real time parameters that hook into model

  func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
        guard let faceAnchor = anchor as? ARFaceAnchor else {
            return
        }
        // face angle
        let yaw = atan2(-faceAnchor.transform.columns.2.x, faceAnchor.transform.columns.0.x)
        let roll = atan2(-faceAnchor.transform.columns.1.z, faceAnchor.transform.columns.1.y)
        let pitch = asin(faceAnchor.transform.columns.1.x)
        model?.setParameter(.AngleX, value: -yaw * 40)
        model?.setParameter(.AngleY, value: (roll + 0.4) * 2 * 30)
        model?.setParameter(.AngleZ, value: pitch * 40)
        
        model?.setParameter(.BodyAngleX, value: -yaw * 10)
        model?.setParameter(.BodyAngleY, value: (roll + 0.4) * 2 * 10)
        model?.setParameter(.BodyAngleZ, value: pitch * 10)
        
        model?.setParameter(.BustX, value: -yaw)
        model?.setParameter(.BustY, value: (roll + 0.4) * 2)
        
 
    }
}

if you could have an algorithm to iterate through the parameter space - then you could craft illustrations live2d based off a picture - the above code is using face tracking - but consider use case of just a regular photo as per your project.

You mentioned there's 93 hair types
eg.
set the hair to be type 1

set the PARAM_HAIR_FRONT to be 0
set the hair to be 0.1

eg.
https://github.com/johndpope/ARKit-Live2D/blob/master/ARKit-Live2D/ViewController.swift

     live2DModel.setParam("PARAM_BODY_ANGLE_Z", value: (CGFloat)(10.0 * sin(t)))
        live2DModel.setParam("PARAM_HAIR_FRONT", value: (CGFloat)(sin(t)))
        live2DModel.setParam("PARAM_HAIR_BACK", value: (CGFloat)(sin(t)))
        live2DModel.setParam("PARAM_BREATH", value: (CGFloat)((cos(t) + 1.0) / 2.0))
        live2DModel.setParam("PARAM_BUST_Y", value: (CGFloat)(cos(t)))
        live2DModel.setPartsOpacity("PARTS_01_ARM_L_A_001", opacity: 0) // hide default position armL
        live2DModel.setPartsOpacity("PARTS_01_ARM_R_A_001", opacity: 0) // hide default position armR

There's some more information I've deduced pulling the moc files apart - which can rip out the texture mesh which glues the texture to (https://github.com/DragonBones/Tools)
This could open up a suite of possibilities to build out complex rigging using AI.
Sorry again if this a bit of tangent - if there's interest to explore this offline - hit me up.
If swift isn't your cup of tea - there's other SDKs in different languages.

Hi @patniemeyer,

I have an error when I run the script of run-eval.sh.
How to fix it?

Thank so much

sh: 1: screencapture: not found
sh: 1: sips: not found
qlua: /home/ml/torch/install/share/lua/5.1/image/init.lua:367: screen.png: No such file or directory
stack traceback:
        [C]: at 0x7fe1c5b64f40
        [C]: in function 'error'
        /home/ml/torch/install/share/lua/5.1/image/init.lua:367: in function 'load'
        /home/ml/chakkrit/vgg-memoji/util.lua:12: in function 'grabImage'
        eval-memoji.lua:52: in main chunk