By Leonard Burns, Ciwan Ceylan, Matteo Iovino and Xiaomeng (Mandy) Zhu

The aim of this project is to create a computer vision system able to detect abandoned bags in a video stream. The algorithm has two main components:

• A neural network able to detect persons and bags in a single frame
• A simple tracker which keeps track of person and bag identities and associates bags with persons.

The system marks a detected bag as abandoned if the bag is without an owner or the assoiciated owner is too far away from the bag.

The detection algrithm is a Faster R-CNN neural network with a ResNet101 backbone. The model is implemented in the Detectron2 framework and is pretrained on MS COCO. To improve the detection algorithm, we combine the MS COCO dataset with the ADE20K dataset and fine-tune the model by training it to detect only two classes: persons and bags.

We have implemented a simple tracker which only recieves bounding boxes and labels for each frame. Simply put, the algorithm tracks the identities of persons and bags by associating each detection in the previous frame with the closest detection in the new frame, where closeness is Euclidean distance in pixel space. The are some more tricks which can be seen in the full implementation of the tracker, found in abandoned_bag_heuristic.py

• Linux system
• gcc & g++ >= 5
• Python >= 3.6
• CUDA 10.1 and cudNN installed for GPU support

* The instructions should also work for macOS but we have not tested this. You can also get the code running on Windows with some modifications but we do not provide instructions at this time.

Important: If you are planing to run the code on a GPU you need to first install CUDA and cudNN, and ensure that you have the CUDA_HOME path set. You can find installation instructions for cuda 10.1 here and instructions for cudNN here. CUDA installation can become a major headache on some systems so it is recommended that you go with the CPU version if you are not planing to run the training.

Start by creating a new fodler to contain the project and the detectron2 installation.

mkdir my-project-folder
cd my-project-folder

(Optional) Create a new virtualenv to contain the required python packages

python3 -m virtualenv venv
source venv/bin/activate

Clone the project and go into the project directory

git clone [email protected]:roym899/abandoned_bag_detection.git
cd abandoned_bag_detection

Install necessary python packages using

pip install -r requirements.txt

Next we you need to install Pytorch. You can use:

pip install torch torchvision

for CUDA 10.1 compatible installation in Linux or

pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

for a Linux CPU only installation. For any other setup consult the Pytorch installation page.

If you went with the GPU supported installation you can verify your setup by running

python -c 'import torch; from torch.utils.cpp_extension import CUDA_HOME; print(torch.cuda.is_available(), CUDA_HOME)'

. This should print True and the path to your CUDA 10.1 installation.

Before installing Detectron2 you also need to install pycocotools using

pip install -U 'git+https://github.com/cocodataset/cocoapi.git#subdirectory=PythonAPI'

Next we navigate back to my-project-folder and install detectron2 as follows:

cd ..  # Now you should be in my-project-folder
git clone https://github.com/facebookresearch/detectron2.git
cd detectron2 && python -m pip install -e .

For combining the MS COCO dataset with ADE20K with a unified format, Dataset-Converters is used. It is linked with our repo as a submodule and can be installed by navigating back into the abandinged_bag_detection directory and running the following git commands:

cd ../abandoned_bag_detection
git submodule init
git submodule update

Now everything should be installed and ready to go!

For training one first has to download MS COCO and ADE20K. For MS COCO you will need the 2017 train and val images, and the 2017 annotations. This are found on the MS COCO downloads page. Note that the data is more than 20 Gb so you might want to use gsutil rsync as suggested in the downloads page. The ADE20K dataset is packaged all into one zip-file of around ~4 Gb.

All scripts assume the dataset to be in abandoned_bag_detection/datasets/. After downloading and extracting the datasets the folder structure should look like this:

<abandoned_bag_detection>/datasets/ade20k/
<abandoned_bag_detection>/datasets/ade20k/images
<abandoned_bag_detection>/datasets/ade20k/index_ade20k.mat
<abandoned_bag_detection>/datasets/coco/annotations
<abandoned_bag_detection>/datasets/coco/annotations/instances_train2017.json
<abandoned_bag_detection>/datasets/coco/annotations/instances_val2017.json
<abandoned_bag_detection>/datasets/coco/images
<abandoned_bag_detection>/datasets/coco/images/train2017/
<abandoned_bag_detection>/datasets/coco/images/val2017/

If you have your datasets in a centralized folder you can create a symbolic link to said folder with

ln -s dataset_folder <abandoned_bag_detection>/datasets

First we need to use the Dataset-Converters to convert the ADE20K format to MS COCO format. For this run

# Ensure you are in the directory abandoned_bag_detection
python Dataset-Converters/convert.py -i <path_to_folder_ADE20K> -o <output_path> -I ADE20K -O COCO --copy

where you replace <path_to_folder_ADE20K> and <output_path> with the suitable paths, from inside the abandoned_bag_detection directory. <output_path> should be <abandoned_bag_detection>/datasets/ade20k_coco, otherwise filter_datasets.py has to be adjusted. After ADE20K has been converted to COCO format, you can run filter_datasets.py to create a merged dataset with only person and bag classes:

python filter_datasets.py

With this done, you can now run the training script to train a new model on the joint data:

python train.py

After training you should find a .pth file containing the weights of the trained model inside the output directory (<abandoned_bag_detection>/output/model_final.pth).

The final model can be run together with the heuristic by running the run_finetuned_model.sh bash script, i.e.,

source run_finetuned_model.sh

If you don't plan to run the training, can use the weights of our self trained model to run the demo derictly by running the run_trained_finetuned_model.sh bash script, i.e.,

source run_trained_finetuned_model.sh

To run on cpu, add MODEL.DEVICE cpu after --opts in run_trained_finetuned_model.sh or run_finetuned_model.sh bash scripts.