Domain Auto-labeling through Reinforcement Learning for the Inference of Gestures

This is the code repository for Yvan Satyawan's Master Thesis.

We must first prepare the dataset by generating indexes of the data and calculating its mean and standard deviation for standardization later on.

1. Generate the small dataset, if desired.
   1. Generate an index for the small dataset using src/data_utils/generate_dataset_index.py.
      • To generate the single-user leave out, run with generate_dataset_index.py [PATH_TO_DATA_ROOT] -u.
      • Use -s instead of -u to make a single-domain index.
      • Add -n 3 to only use 3 repetitions instead of the full dataset. Useful for debugging as it means a smaller dataset.
   2. Generate the smaller datasets using src/data_utils/generate_smaller_splits.py.
      • To generate the single-user split, run with generate_smaller_splits.py [PATH_TO_DATA_ROOT] single_user.
      • It is possible to replace single_user with whatever index file suffix was generated by generate_dataset_index.py
2. Otherwise, generate the dataset index using src/data_utils/generate_dataset_index.py. This will generate the index of the full dataset.
3. Calculate the mean and standard deviation of amplitude and phase using src/data_utils/calculate_mean_std.py
   • If not using the full dataset, copy both the indexes and generated mean_std.csv into the smaller split directory
4. Pregenerate transformations, as they take too long to generate during training, using src/data_utils/pregenerate_transform.py
   • Run using the config file for a given experiment with pregenerate_transform.py [PATH_TO_CONFIG_FILE].

Config files can be generated to ensure that runs are consistent. Our used experimental run configuration files are stored in the run_configs directory as YAML files. utils/config_parser.py contain the full list of possible configuration parameters.

DARLInG is trained by experiments/train_runner.py. It is run by using train_runner.py [PATH_TO_CONFIG_FILE].

Hyperparameter sweeps with Weights and Biases can be done by pointing weights and biases to utils/sweep_runner.py as the main file. This runner accepts command line arguments instead of a YAML file to initialize training. Internally, it's transforming the command line arguments into a dictionary and running the dictionary through the standard config parser.

All the other files and scripts are carefully documented with Python Docstrings to indicate what they do and how they work, but are not strictly necessary to run training usually. train_runner.py is for general model training, train.py hosts the main training loop code, and final_experiment_runner.py contains code to repeatedly call train_runner.py on multiple configurations without causing any memory issues. All other scripts in experiments/ also contain the motivation of the script, the question the script attempts to answer, and the answer.

These are mostly notes to myself to understand what the data looks like.

The files are stored in a lot of files, split into multiple folders. Folder naming scheme doesn't have much meaning, other than to split the dataset into when the data was captured.

Not all gestures are performed by all users. As such, we will only use gestures 1-6 in this work.

The CSI files are .dat files, which are simply CSI dumps from the tool used by the team to gather CSI data. The file naming convention is as follows:

id-a-b-c-d-Rx.dat

Each recorded CSI sequence can be understood as an tensor with the shape (i, j, k, 1).

• i is the packet number
• j is the subcarrier number
• k is the receiver antenna number

In the case of Widar3.0, the value of k is always 3 (3 antennas per receiver). Widar3.0 uses 1 transmitter and 6 receivers placed around the sensing area.

We use the package csiread to read the file. The .dat files can be read using csiread.Intel.

The BVP files are .mat files (MATLAB) that have been preprocessed by the authors. The file naming convention is as follows:

id-a-b-c-d-suffix.dat

suffix has no explanation. As far as I understand, I think it's just the configuration used to produce the BVP. There is no receiver ID since all 6 receivers were combined to produce the BVP.

Each file is a 20x20xT tensor.

We use Scipy to read the .mat files with scipy.io.loadmat().

BVP lengths are not consistent. We pad them to the all have the same length of 28.

The directory 20181130-VS contains no 6-link subdirectory.

As we are testing in vs out of domain performance, we will use the following data split for train, validation, and test.

We split it this way to make sure that the test set is truly unseen while the validation set is an unseen room-user combination instead of truly unseen.

We only use gestures 1-6, since these are the gestures which have samples from all participants.

For single domain, we use only User 2 in Room 1 with torso location 1 and face orientation 1. This was chosen as it has the largest number of samples. Test, validation, and training splits are randomly generated.