A detailed walk-through is available in Real-time Object Tracking with TensorFlow, Raspberry Pi, and Pan-tilt HAT.
- Raspberry Pi 4 (4GB recommended)
- Raspberry Pi Camera V2
- Pimoroni Pan-tilt Kit
- Micro SD card 16+ GB
- Micro HDMI Cable
- 12" CSI/DSI ribbon for Raspberry Pi Camera (optional, but highly recommended)
- Coral Edge TPU USB Accelerator (optional)
- RGB NeoPixel Stick (optional, makes lighting conditions more consistent)
An example of deep object detection and tracking with a Raspberry Pi
- Free software: MIT license
- Documentation: https://rpi-deep-pantilt.readthedocs.io.
Before you get started, you should have an up-to-date installation of Raspbian 10 (Buster) running on your Raspberry Pi. You'll also need to configure SSH access into your Pi.
- Install system dependencies
$ sudo apt-get update && sudo apt-get install -y \
cmake python3-dev libjpeg-dev libatlas-base-dev raspi-gpio libhdf5-dev python3-smbus python3-venv libopenjp2-7 libtiff5- Create new virtual environment
$ python3 -m venv .venv- Activate virtual environment
$ source .venv/bin/activate- Upgrade setuptools
$ python3 -m pip install --upgrade pip
$ pip install --upgrade setuptools- Install TensorFlow 2.4 (community-built wheel)
$ pip install https://github.com/bitsy-ai/tensorflow-arm-bin/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl
- Install the
rpi-deep-pantiltpackage.
pip install rpi-deep-pantilt- Install Coral Edge TPU
tflite_runtime(optional)
NOTE: This step is only required if you are using Coral's Edge TPU USB Accelerator. If you would like to run TFLite inferences using CPU only, skip this step.
$ pip install https://github.com/google-coral/pycoral/releases/download/release-frogfish/tflite_runtime-2.5.0-cp37-cp37m-linux_armv7l.whl=======
WARNING: Do not skip this section! You will not be able to use rpi-deep-pantilt without properly configuring your Pi.
- Run
sudo raspi-configand selectInterfacing Optionsfrom the Raspberry Pi Software Configuration Tool’s main menu. Press ENTER.
- Select the Enable Camera menu option and press ENTER.
- In the next menu, use the right arrow key to highlight ENABLE and press ENTER.
- Run
sudo raspi-configand selectInterfacing Optionsfrom the Raspberry Pi Software Configuration Tool’s main menu. Press ENTER.
-
Select the SPI menu option and press ENTER.
-
In the next menu, use the right arrow key to highlight Yes and press ENTER.
- Run
sudo raspi-configand selectInterfacing Optionsfrom the Raspberry Pi Software Configuration Tool’s main menu. Press ENTER.
-
Select the I2C menu option and press ENTER.
-
In the next menu, use the right arrow key to highlight Yes and press ENTER.
Alternatively, open /boot/config.txt and ensure that the following dtparams lines are uncommented:
dtparam=i2c1=on
dtparam=i2c_arm=onThe detect command will start a PiCamera preview and render detected objects as an overlay. Verify you're able to detect an object before trying to track it.
Supports Edge TPU acceleration by passing the --edge-tpu option.
rpi-deep-pantilt detect [OPTIONS] [LABELS]...
rpi-deep-pantilt detect --help
Usage: rpi-deep-pantilt detect [OPTIONS] [LABELS]...
rpi-deep-pantilt detect [OPTIONS] [LABELS]
LABELS (optional) One or more labels to detect, for example:
$ rpi-deep-pantilt detect person book "wine glass"
If no labels are specified, model will detect all labels in this list:
$ rpi-deep-pantilt list-labels
Detect command will automatically load the appropriate model
For example, providing "face" as the only label will initalize
FaceSSD_MobileNet_V2 model $ rpi-deep-pantilt detect face
Other labels use SSDMobileNetV3 with COCO labels $ rpi-deep-pantilt detect
person "wine class" orange
Options:
--loglevel TEXT Run object detection without pan-tilt controls. Pass
--loglevel=DEBUG to inspect FPS.
--edge-tpu Accelerate inferences using Coral USB Edge TPU
--rotation INTEGER PiCamera rotation. If you followed this guide, a
rotation value of 0 is correct.
https://medium.com/@grepLeigh/real-time-object-tracking-
with-tensorflow-raspberry-pi-and-pan-tilt-
hat-2aeaef47e134
--help Show this message and exit.
The following will start a PiCamera preview, render detected objects as an overlay, and track an object's movement with Pimoroni pan-tilt HAT.
By default, this will track any person in the frame. You can track other objects by passing --label <label>. For a list of valid labels, run rpi-deep-pantilt list-labels.
rpi-deep-pantilt track
Supports Edge TPU acceleration by passing the --edge-tpu option.
Usage: rpi-deep-pantilt track [OPTIONS] [LABEL]
rpi-deep-pantilt track [OPTIONS] [LABEL]
LABEL (required, default: person) Exactly one label to detect, for example:
$ rpi-deep-pantilt track person
Track command will automatically load the appropriate model
For example, providing "face" will initalize FaceSSD_MobileNet_V2 model
$ rpi-deep-pantilt track face
Other labels use SSDMobileNetV3 model with COCO labels
$ rpi-deep-pantilt detect orange
Options:
--loglevel TEXT Pass --loglevel=DEBUG to inspect FPS and tracking centroid
X/Y coordinates
--edge-tpu Accelerate inferences using Coral USB Edge TPU
--rotation INTEGER PiCamera rotation. If you followed this guide, a
rotation value of 0 is correct.
https://medium.com/@grepLeigh/real-time-object-tracking-
with-tensorflow-raspberry-pi-and-pan-tilt-
hat-2aeaef47e134
--help Show this message and exit.
rpi-deep-pantilt list-labels
The following labels are valid tracking targets.
['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']
The following command will detect human faces.
NOTE: Face detection uses a specialized model (FaceSSD_MobileNet_V2), while other labels are detecting using SSDMobileNetV3_COCO. You cannot detect both face and COCO labels at this time.
Watch this repo for updates that allow you to re-train these models to support a custom mix of object labels!
rpi-deep-pantilt detect face
Usage: cli.py face-detect [OPTIONS]
Options:
--loglevel TEXT Run object detection without pan-tilt controls. Pass
--loglevel=DEBUG to inspect FPS.
--edge-tpu Accelerate inferences using Coral USB Edge TPU
--help Show this message and exit.
The following command will track a human face.
rpi-deep-pantilt track face
Usage: cli.py face-detect [OPTIONS]
Options:
--loglevel TEXT Run object detection without pan-tilt controls. Pass
--loglevel=DEBUG to inspect FPS.
--edge-tpu Accelerate inferences using Coral USB Edge TPU
--help Show this message and exit.
The following section describes the models used in this project.
rpi-deep-pantilt detect and rpi-deep-pantilt track perform inferences using this model. Bounding box and class predictions render at roughly 6 FPS on a Raspberry Pi 4.
The model is derived from ssd_mobilenet_v3_small_coco_2019_08_14 in tensorflow/models. I extended the model with an NMS post-processing layer, then converted to a format compatible with TensorFlow 2.x (FlatBuffer).
I scripted the conversion steps in tools/tflite-postprocess-ops-float.sh.
If you specify --edge-tpu option, rpi-deep-pantilt detect and rpi-deep-pantilt track perform inferences using this model. Rounding box and class predictions render at roughly 24+ FPS (real-time) on Raspberry Pi 4.
This model REQUIRES a Coral Edge TPU USB Accelerator to run.
This model is derived from ssdlite_mobilenet_edgetpu_coco_quant in tensorflow/models. I reversed the frozen .tflite model into a protobuf graph to add an NMS post-processing layer, quantized the model in a .tflite FlatBuffer format, then converted using Coral's edgetpu_compiler tool.
I scripted the conversion steps in tools/tflite-postprocess-ops-128-uint8-quant.sh and tools/tflite-edgetpu.sh.
I was able to use the same model architechture for FLOAT32 and UINT8 input, facessd_mobilenet_v2_quantized_320x320_open_image_v4_tflite2.
This model is derived from facessd_mobilenet_v2_quantized_320x320_open_image_v4 in tensorflow/models.
If you run $ rpi-deep-pantilt test pantilt and see a similar error, check your Device Tree configuration.
File "/home/pi/projects/rpi-deep-pantilt/.venv/lib/python3.7/site-packages/pantilthat/pantilt.py", line 72, in setup
self._i2c = SMBus(1)
FileNotFoundError: [Errno 2] No such file or directoryOpen /boot/config.txt and ensure the following lines are uncommented:
dtparam=i2c1=on
dtparam=i2c_arm=onThe MobileNetV3-SSD model in this package was derived from TensorFlow's model zoo, with post-processing ops added.
The PID control scheme in this package was inspired by Adrian Rosebrock tutorial Pan/tilt face tracking with a Raspberry Pi and OpenCV
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent