Action segmentation is the task of predicting the action occuring in each frame of a given sequence (e.g video or signal data).

This repo contains experiments and source code for work in Action Segmentation from Human-Robot Demonstrations for Grocery Picking by Sonia Mathews. This work proposes a multimodal model that learns from human-robot demonstrations (RGB images and Haptic feedback) to apply action segmentation for a grocery pick and place system. All Experiments are done in PyTorch and Pytorch Lightning.

This project contains the following files and folders

config/ contains baseline configurations for each implemented model.
cross-validation/ contains model weights during kfold cross-validation, used to ensemble the models and take the average predictions on a separate test dataset.
models/ contains the source code of all implemented models in this project.
unit_tests/ tests to check preprocessing and data loader implemented in this project.
utils/ a collection of scripts to run accuracy metrics, data-loading, data analysis and preprocessing.

The following scripts implement kfold cross-validation:

• cnn_tcn_lstm_kfold.py
• cnn_lstm_kfold.py
• cnn_tcn_kfold.py
• cnn_tcn_lstm_kfold.py
• lstm_kfold.py
• tcn_kfold.py

The following script implements a single train/val/test split to fit the model:

• lstm_test.py
• tcn_test.py

All experiments run in Google Colab:

• model_kfold.ipynb - to fit model using kfold cross-validation.
• model_test.ipynb - to fit a model using a single train/val/test split on dataset (Used by the author for model debugging).

Note:

• The accuracy metrics implemented in this project use source code from the study Temporal Convolutional Networks for Action Segmentation and Detection.
• The Temporal Convolution Network (TCN) used in this project uses source code provided from the study An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling.
• The dataset used in this project can be downloaded from CRISP Teleoperated Fruit Picking Dataset.