I'm a rookie in neural network, so the question may be stupid. I changed the training data from referenced dataset and found the loss of decompressor reaches 1.9 after 500k iterations. Generated .bvh files also shows obvious difference. So I wonder if I choose bad training data and what can I do to reduce loss.

Thanks again Daniel for sharing this excellent demo.
I want to use my own data, so how should I make database.bin and features.bin?

is feature X a part of database build by generate_database.py

Thank you for a great job. In Figure 16 of the paper, you show the terrain you fitted by regression, which is used to input the terrain height under the toes. I have read the code of PFNN and still don't understand this part. For the fitted surface, how should I output the diverse terrain, can it be predicted by using the neural network

Hi, I've seen that a gamepad is mandatory, and that the "keyboard support patch" proposed here has a broken link.

Here are my changes in file controller.cpp:

vec3 gamepad_get_stick(int stick, const float deadzone = 0.2f)
{
// I've replaced all the original content of this function with the following:
vec3 pad(0,0,0);float am = 1.f;
if (IsKeyDown(KEY_RIGHT_SHIFT) || IsKeyDown(KEY_LEFT_SHIFT)) am*=0.5f;  // use SHIFT to slow down/walk
if (IsKeyDown(KEY_RIGHT_CONTROL) || IsKeyDown(KEY_LEFT_CONTROL)) am*=2.f; // use CTRL to speed up
if (stick==GAMEPAD_STICK_LEFT)	{
        // use ARROW keys to move the character
	if (IsKeyDown(KEY_RIGHT)) pad.x+=am;
	if (IsKeyDown(KEY_LEFT)) pad.x-=am; 
	if (IsKeyDown(KEY_UP)) pad.z-=am;
	if (IsKeyDown(KEY_DOWN)) pad.z+=am;
}
else if (stick==GAMEPAD_STICK_RIGHT)	{
        // use the number pad to move the camera
	if (IsKeyDown(KEY_KP_6)) pad.x-=am;
	if (IsKeyDown(KEY_KP_4)) pad.x+=am; 
	if (IsKeyDown(KEY_KP_8)) pad.z+=am;
	if (IsKeyDown(KEY_KP_2)) pad.z-=am;
}
return pad;
}

This works for me (I'm on Ubuntu Linux).
I'm not making a pull request. I've just posted my code here in case somebody is interested.

@orangeduck: feel free to close this issue whenever you want, and thanks for sharing the project!

Thanks again Daniel for sharing this excellent demo - one thing that jumped out while tuning sim/adjustment is how distorted trajectory futures become. I expect this is not avoidable if you want correction but wondered if you had any thoughts/guidance on it?

With rotation correction turning the root faster or slower than the animation your future position features become deflected (of course they are, because they are stored relative to the queried animation frame) - for our content this means my 180 idle-to-jog get tossed out in favor of visually worse 'turning jogs' because of this deflection. Intuitively it doesn't seem like a benefit to matching quality, the future positions are not actually deflected in the animation (the turn 180 stays along the axis).

I can iterate with animators to more closely match the sim of course to reduce the harm but I wondered if you've already put thought into this?

Hey Daniel, I hope you're doing well!

I was hoping to get your thoughts on a potential method to drastically speed up the training time of the projector. Currently, a call is made to a fitted search structure such as a BallTree. Whilst this does indeed accept batches of data, it still appears to be quite a large bottleneck. I have briefly profiled the train_projector.py script (only for 500 iterations with no results generated) and found that the total run time is roughly 20 seconds, with 12 seconds of that time being spent in a call to the BallTree.

I don't think there is any need to have this call to the nearest neighbor search during the training process. Rather, let's generate this offline first using optimized tools such as FAISS.

To make a fair comparison, I generated 16,000,000 queries (32 * 500,000) using the same method you do, a form of scaled Gaussian noise added to a sample of the feature database. I then query against a FAISS index which uses Euclidean distance measure under the hood. The output of this is 16,000,000 queries and their associated closest feature indexes. Generating this data takes roughly 15 minutes on an AMD Ryzen 9 5950X. I don't think FAISS is supported on GPU for Windows (at least not officially), but I think I remember them claiming that on average the GPU implementation is roughly 5x faster than CPU, so maybe there are more gains to be made...

Once this data has been generated, we can bypass the BallTree call altogether and just generate our batches from this offline data instead. So we can now train the projector on an NVIDIA 3080 with a batch size of 1024 for 50,000 iterations in roughly 3 minutes, with a final loss of 0.520.

Here is the code to produce the offline data:

import os
import struct
import time

import faiss
import numpy as np


class ProjectorDatasetBuilder:

    def __init__(self):
        pass

    def load_features_from_file(self, filename: str) -> None:

        with open(filename, 'rb') as f:
            
            self.n_frames, self.n_features = struct.unpack('II', f.read(8))
            self.features: np.ndarray = np.frombuffer(f.read(self.n_frames*self.n_features*4), dtype=np.float32, count=self.n_frames*self.n_features).reshape([self.n_frames, self.n_features])
    
        self.features_noise_std: np.ndarray = self.features.std(axis=0) + 1.0

    def make_queries(self, n_queries: int) -> None:
        samples = np.random.randint(0, self.n_frames, size=[n_queries]) # n_queries
        n_sigma = np.random.uniform(size=[n_queries, 1]).astype(np.float32)  # n_queries x 1
        noise = np.random.normal(size=[n_queries, self.n_features]).astype(np.float32) # n_queries x n_features
        
        # here we scale our noise and add to our samples
        queries = self.features[samples] + self.features_noise_std * n_sigma * noise
        self.queries = queries
    
    def build_faiss_index(self) -> None:
        start_time = time.time()
        self.index = faiss.IndexFlatL2(self.n_features)
        self.index.add(self.features)
        index_build_time = time.time() - start_time
        print(f'Time taken to build index: {index_build_time} seconds')
        self._sanity_check()
    
    def _sanity_check(self) -> None:
        """
        Here we are passing in the first 5 feature rows into our search.
        We expect to see the returned distances as 0, and the indexes to be 0 through 4,
        as these data points are explicitly part of the search index.
        """
        distances, indexes = self.index.search(self.features[:5], 1)
        print('Sanity check...')
        for i, (d, ii) in enumerate(zip(distances, indexes)):
            print(f'Closest neighbor of point {i} is {ii} with distance {d}')
    

    def get_projector_data(self) -> None:
        start_time = time.time()
        _, closest_indexes = self.index.search(self.queries, 1) # main call
        search_time = time.time() - start_time
        print(f'Time taken for search: {search_time} seconds.')
        self.closest_indexes = closest_indexes
    
    def save(
            self,
            dst_folder: str,
            query_filename: str ='test_queries.npy',
            nearest_indexes_filename: str ='test_nearest_indexes.npy'
            ):
        np.save(os.path.join(dst_folder, query_filename), self.queries)
        np.save(os.path.join(dst_folder, nearest_indexes_filename), self.closest_indexes)


if __name__ == "__main__":

    np.random.seed(1234)
    features_filename = "./features.bin"
    output_folder = "./output"
    num_queries = 16_000_000
    builder = ProjectorDatasetBuilder()
    builder.load_features_from_file(features_filename)
    builder.make_queries(num_queries)
    builder.build_faiss_index()
    builder.get_projector_data()
    builder.save(output_folder)

Just thought you might be interested!

My environment is Windows and I have the latest raylib 5.0 installed, and the latest raygui 4.0 installed. After configuring the environment as described in the issue-introduction, when I tried to run controller.exe, I encountered the following problem: after a few seconds of loading on the white screen, the window automatically crashed.

I found the console message at the time of the crash: process exited with code 3221225477 (0xc0000005). Google says this code represents an array out of bounds. But I don't know exactly what went wrong.

Here is the problem, if I just use my left stick to make my character move along one X axis(like i just make him move to left or right), eventually the character will have offsets along another Y axis, which is really not the resul I want. I wonder if there are ways to prevent this from happing? Thanks in advance ::)

What's the list of animations store in database.bin originally ? It's just the same as Files in Files to Process's list ?
I know that this neural network compresses the animation data tremendously. And in general, we compresses animation data with Catmull-Rom.
I'm just about to test which one compresses more data.

how to build feature.bin?
Thanks! your project is very important to me!

Thanks for your wonderful work!

Features are designed differently in each scenario according to the 'Learned Motion Matching' paper. Could you please kindly share your ideas on how to design features for 'Chair' scenario?

Features for 'Chair' are supposed to be 19-dimension rather than the 27-dimension feature for locomotion as described in the paper.

Thanks!

Hi Daniel. First of all, I really appreciate the fact you open all of this research to the public. Its really refreshing to see. On to the technical stuff...
In the compute_bone_position_feature in database.h, to get the position of a joint relative to the sim bone, you first calculate the global positions using fk. After that, you then rotate the diff between these two points by the global (global and local are the same in this case I assume as we're concerned with the sim bone) inverse rotation of the sim bone. Why is this? I thought it would have been sufficient to just use that diff? Am I misunderstanding the inputs and outputs of this method?
For reference the code is here:

// Compute a feature for the position of a bone relative to the simulation/root bone
void compute_bone_position_feature(database& db, int& offset, int bone, float weight = 1.0f)
{
    for (int i = 0; i < db.nframes(); i++)
    {
        vec3 bone_position;
        quat bone_rotation;
        
        forward_kinematics(
            bone_position,
            bone_rotation,
            db.bone_positions(i),
            db.bone_rotations(i),
            db.bone_parents,
            bone);
        
        bone_position = quat_mul_vec3(quat_inv(db.bone_rotations(i, 0)), bone_position - db.bone_positions(i, 0));
        
        db.features(i, offset + 0) = bone_position.x;
        db.features(i, offset + 1) = bone_position.y;
        db.features(i, offset + 2) = bone_position.z;
    }
    
    normalize_feature(db.features, db.features_offset, db.features_scale, offset, 3, weight);
    
    offset += 3;
}

Again, thank you for all your research!

Thank you for sharing.

When I add terrain information, the part of generating simulation bone in generate_database.py seems to be unavailable. In practice, it cannot generate the added Root node on the terrain, but drag the whole thing back to the plane. In this case, how should I modify this part so that the generated simulation bone can be on the fitted terrain.

Could you please add a small tutorial about how to build and use the code?

Hi, could you please offer some details about how to retarget to other character with the model output?

got to admit that I'm poor in C++!

How to run this code? I've downloaded raygui and installed raylib in ubuntu 18.04. Should I put raygui in the program folder, and make or something like this?

When running"g++ controller.cpp", errors show:

/tmp/ccomEtPp.o：in function 'GuiGroupBox':
controller.cpp:(.text+0x559)：Undefined reference 'GetColor'
controller.cpp:(.text+0x568)：Undefined reference  'Fade' 

What should I do? Is there a instructions like the one in #2 available?

Hi, thanks for your great work, it really helps me a lot.
I have some questions about inertialize_pose_update in the controller.cpp. First, I am confused with the terms of world space and space of the currently playing animation in line 447-450. And I don't know how to get more information about inertialization.

Sorry, I don't quite understand how to regenerate features.bin. I can't find any part of the code that describes this part.

I have retrained nets by attaching one hot code to the feature vector. And how to modify your controller when I want to change the character's gait?

Thank you so much for sharing this - I did find a copy-paste bug I think, feature bounding boxes are updating maximums wrong...

            db.bound_sm_min(i_sm, j) = minf(db.bound_sm_min(i_sm, j), db.features(i, j));
            db.bound_sm_max(i_sm, j) = maxf(db.bound_sm_min(i_sm, j), db.features(i, j));   // !! want sm_max here ?
            db.bound_lr_min(i_lr, j) = minf(db.bound_lr_min(i_lr, j), db.features(i, j));
            db.bound_lr_max(i_lr, j) = maxf(db.bound_lr_min(i_lr, j), db.features(i, j));   // !! want lr_max here ?

When I don't want to use the positive direction of the hip node as the direction of the trajectory, how should I change the Trajectory Directions? When I try to use the predicted three position information, the resulting bin file will directly change to nan when training the decompressor.

Use make build_depends && make mmatching && make run

CC=g++
INCLUDE_DIR = -I./
INCLUDE_DIR+= -I./depends/raylib/raylib/include
INCLUDE_DIR+= -I./depends/raygui/src
LIBRARY_DIR = -L./depends/raylib/raylib
LIBRARY_DIR+= -L./depends/raylib/raylib/external/glfw/src
LIBRARY_LINK = -lraylib -lglfw3 -ldl -pthread
BUILD_DEP = $(INCLUDE_DIR) $(LIBRARY_DIR) $(LIBRARY_LINK)

all: mmatching

clean:
	-rm mmatching

mmatching:
	$(CC) controller.c -o $@ $(BUILD_DEP)

run:
	./mmatching;



build_depends:
	mkdir -p depends/raygui
	mkdir -p depends/raylib
	git clone --depth 1 https://github.com/raysan5/raylib.git depends/raylib
	git clone --depth 1 https://github.com/raysan5/raygui depends/raygui
	cd depends/raylib && cmake . && make

patch-back:
	mv controller.old.c controller.c


.SILENT: clean

P.S.:

Im not have gamepad :( If you don't, then you can apply a patch. Only move and zoom works. The rest I did not touch. Basic keyboard support -> https://gist.github.com/fedor-elizarov/2a14307a34112f8d605ee2198855b8af

wasd move key pad 1 zoom in key pad 2 zoom out You need to apply a patch for it. Maybe it will be useful to someone

Why not code like below ? I guess this would be more accurate😁

Thanks Daniel!🤞

Thanks again for the open source code, it helped me a lot to understand motion match.

When I change database.bin, how should I modify the features.bin corresponding to it, or tell me what the 27-dimensional features of features.bin represent.

Thanks.

@orangeduck Thanks for your cool work !

• I want to make the character looks more like a humanoid robot, how can I change the character.bin ?
• Are there any recommended editing tools ?

Recently there was a ranking of Open Source game engines and raylib was ranked low compared to e.g. Godot.

Many Unity developers are moving to Godot4.

C++ code can be embedded as GDExtension in Godot

See one possible example

Hi,
Looking through the code, as far as I understood, the end of the clip problem has not been handled. Once the animation reaches the end of the clip, best_index is set to -1, and then trns_bone_positions are accessed by this index, which raises the assertion error of assert(i >= 0 && i < rows).

Thanks for your great work, following your way I managed to switch between running/jumping, but when I want to switch between other actions, such as lifting/kicking, it has some problems, I put the position of the end joint of the hand Information is used as features, and three models are retrained, and the cpp file you gave is also changed to make the dimension of features.bin larger (added hand position). But the results were not as expected. Please tell me what should I do to make the switching of other actions work normally.

Looking forward to your reply.