Thermal Face

Thermal Face is a machine learning model for fast face detection in thermal images. It was built for Fever, the contactless fever thermometer with auto-aim.

Inference

The face detection model is using TensorFlow Lite for optimal performance on mobile/edge devices. The recommended inference setup is a Raspberry Pi 4 Model B with a Coral USB Accelerator.

The following is an example for inference from Python on an image file using the compiled model thermal_face_automl_edge_fast_edgetpu.tflite, downloaded from the latest release, and the Edge TPU API:

pip3 install Pillow
sudo apt-get install python3-edgetpu

from edgetpu.detection.engine import DetectionEngine
from PIL import Image

# One-time initialization:
face_detector = DetectionEngine('thermal_face_automl_edge_fast_edgetpu.tflite')

# Per-image detection:
image = Image.open('image.png').convert('RGB')
faces = face_detector.detect_with_image(image,
    threshold=0.5
    top_k=10,
    keep_aspect_ratio=True,
    relative_coord=False,
    resample=Image.BILINEAR)
for face in faces:
  # np.array([[left, top], [right, bottom]], dtype=float64)
  print(face.bounding_box)

Alternatively, you can use the TF Lite API directly on the compiled model or, in the absence of an Edge TPU, on the uncompiled model thermal_face_automl_edge_fast.tflite, which is also in the latest release.

Training

The model is trained with Cloud AutoML using a face dataset that combines a large set of images in visible light from the WIDER FACE database and a smaller set of thermal images from the Tufts Face Database and the FLIR ADAS Dataset.

1. Create the dataset

There are a total of 18,418 images and 164,915 face bounding box annotations in the combined dataset. The WIDER FACE set is large and diverse, but only contains visible-light images. The thermal images from the Tufts Face Database and FLIR ADAS Dataset are fewer and less diverse, so we mix the three sets before splitting them into training, validation, and test sets. The relative size of the test and validation sets are unusually small to achieve a better balance among the source datasets while still using a significant fraction of all available training data. The exact breakdown of images (and annotations) is as follows:

	Training set	Validation set	Test set
Tufts Face Database (IR)	1,247 (1,247)	155 (155)	155 (155)
Fraction of source images	80%	10%	10%
Fraction of combined	7% (1%)	39% (8%)	39% (8%)
FLIR ADAS (Faces)	617 (854)	77 (108)	77 (99)
Fraction of source images	80%	10%	10%
Fraction of combined	3% (1%)	20% (6%)	20% (5%)
WIDER FACE (Validation)	2,898 (28,859)	161 (1,611)	161 (1,741)
Fraction of source images	90%	5%	5%
Fraction of combined	16% (18%)	41% (86%)	41% (87%)
WIDER FACE (Training)	12,870 (130,086)	0 (0)	0 (0)
Fraction of source images	100%	0%	0%
Fraction of combined	73% (81%)	0% (0%)	0% (0%)
*Combined*	17,632 (161,046)	393 (1,874)	393 (1,995)
Fraction of combined sources	96% (98%)	2% (1%)	2% (1%)

1.1 Get the Tufts Face Database

Download the thermal images from the Tufts Face Database and upload them to Cloud Storage:

LOCATION="us-central1"
TDFACE_DIR="tufts-face-database"
TDFACE_BUCKET="gs://$TDFACE_DIR"

cd training

for i in $(seq 1 4)
do
  curl -O http://tdface.ece.tufts.edu/downloads/TD_IR_A/TD_IR_A_Set$i.zip
  curl -O http://tdface.ece.tufts.edu/downloads/TD_IR_E/TD_IR_E_Set$i.zip
done

mkdir $TDFACE_DIR
for f in TD_IR_*.zip
do
  unzip $f -d $TDFACE_DIR/$(basename $f .zip)
done

gsutil mb -l $LOCATION $TDFACE_BUCKET
gsutil -m rsync -r $TDFACE_DIR $TDFACE_BUCKET

Create a dataset spec of the images in the AutoML format using the separately created bounding box annotations and upload it:

TDFACE_ANNOTATIONS="$TDFACE_DIR/bounding-boxes.csv"
TDFACE_AUTOML="tdface-automl.csv"
TDFACE_TRAINING_FRACTION=0.8
TDFACE_VALIDATION_FRACTION=0.1
TDFACE_TEST_FRACTION=0.1

curl https://github.com/maxbbraun/tdface-annotations/releases/latest/download/bounding-boxes.csv -o $TDFACE_ANNOTATIONS

python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt

python automl_convert.py \
  --mode=TDFACE \
  --tdface_dir=$TDFACE_DIR \
  --tdface_bucket=$TDFACE_BUCKET \
  --tdface_annotations=$TDFACE_ANNOTATIONS \
  --training_fraction=$TDFACE_TRAINING_FRACTION \
  --validation_fraction=$TDFACE_VALIDATION_FRACTION \
  --test_fraction=$TDFACE_TEST_FRACTION \
  --automl_out=$TDFACE_AUTOML

gsutil cp $TDFACE_AUTOML $TDFACE_BUCKET

1.2 Get FLIR ADAS

Download the FLIR_ADAS_1_3.tar.* files and separately created bounding box annotations and process them:

FLIR_ADAS_REPO="flir-adas-faces"
FLIR_ADAS_DIR="flir-adas-database"
TMP_FLIR_ADAS_DIR="/tmp/$FLIR_ADAS_DIR"
FLIR_ADAS_BUCKET="gs://$FLIR_ADAS_DIR"
FLIR_ADAS_ANNOTATIONS="$FLIR_ADAS_REPO/bounding-boxes.csv"
FLIR_ADAS_AUTOML="flir-adas-automl.csv"
FLIR_ADAS_TRAINING_FRACTION=0.8
FLIR_ADAS_VALIDATION_FRACTION=0.1
FLIR_ADAS_TEST_FRACTION=0.1

git clone https://github.com/maxbbraun/flir-adas-faces.git $FLIR_ADAS_REPO
cd $FLIR_ADAS_REPO

curl https://github.com/maxbbraun/flir-adas-faces/releases/latest/download/bounding-boxes.csv -o $FLIR_ADAS_ANNOTATIONS

mkdir $FLIR_ADAS_DIR
tar -C $FLIR_ADAS_DIR -xvf FLIR_ADAS_1_3.tar.001 --strip-components=1

python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt

python flir_convert.py \
  --input_dir=$FLIR_ADAS_DIR/train \
  --output_dir=$TMP_FLIR_ADAS_DIR/train \
  --output_csv=/dev/null
python flir_convert.py \
  --input_dir=$FLIR_ADAS_DIR/val \
  --output_dir=$TMP_FLIR_ADAS_DIR/val \
  --output_csv=/dev/null
python flir_convert.py \
  --input_dir=$FLIR_ADAS_DIR/video \
  --output_dir=$TMP_FLIR_ADAS_DIR/video \
  --output_csv=/dev/null

cd ..
. venv/bin/activate

python automl_convert.py \
  --mode=FLIR \
  --tdface_dir=$TMP_FLIR_ADAS_DIR \
  --tdface_bucket=$FLIR_ADAS_BUCKET \
  --tdface_annotations=$FLIR_ADAS_ANNOTATIONS \
  --training_fraction=$FLIR_ADAS_TRAINING_FRACTION \
  --validation_fraction=$FLIR_ADAS_VALIDATION_FRACTION \
  --test_fraction=$FLIR_ADAS_TEST_FRACTION \
  --automl_out=$FLIR_ADAS_AUTOML

gsutil cp $FLIR_ADAS_AUTOML $FLIR_ADAS_BUCKET

1.3 Get WIDER FACE

Download and upload the WIDER FACE dataset:

WIDERFACE_DIR="wider-face-database"
WIDERFACE_BUCKET="gs://$WIDERFACE_DIR"

mkdir $WIDERFACE_DIR
for f in WIDER_*.zip wider_*.zip
do
  unzip $f -d $WIDERFACE_DIR/
done

gsutil mb -l $LOCATION $WIDERFACE_BUCKET
gsutil -m rsync -r $WIDERFACE_DIR $WIDERFACE_BUCKET

Create and upload the AutoML spec using the included bounding boxes:

WIDERFACE_TRAINING_AUTOML="widerface-training-automl.csv"
WIDERFACE_VALIDATION_AUTOML="widerface-validation-automl.csv"
WIDERFACE_TRAINING_TRAINING_FRACTION=1.0
WIDERFACE_TRAINING_VALIDATION_FRACTION=0.0
WIDERFACE_TRAINING_TEST_FRACTION=0.0
WIDERFACE_VALIDATION_TRAINING_FRACTION=0.9
WIDERFACE_VALIDATION_VALIDATION_FRACTION=0.05
WIDERFACE_VALIDATION_TEST_FRACTION=0.05

python automl_convert.py \
  --mode=WIDERFACE \
  --widerface_dir=$WIDERFACE_DIR/WIDER_train \
  --widerface_bucket=$WIDERFACE_BUCKET/WIDER_train \
  --widerface_annotations=$WIDERFACE_DIR/wider_face_split/wider_face_train_bbx_gt.txt \
  --training_fraction=$WIDERFACE_TRAINING_TRAINING_FRACTION \
  --validation_fraction=$WIDERFACE_TRAINING_VALIDATION_FRACTION \
  --test_fraction=$WIDERFACE_TRAINING_TEST_FRACTION \
  --automl_out=$WIDERFACE_TRAINING_AUTOML
python automl_convert.py \
  --mode=WIDERFACE \
  --widerface_dir=$WIDERFACE_DIR/WIDER_val \
  --widerface_bucket=$WIDERFACE_BUCKET/WIDER_val \
  --widerface_annotations=$WIDERFACE_DIR/wider_face_split/wider_face_val_bbx_gt.txt \
  --training_fraction=$WIDERFACE_VALIDATION_TRAINING_FRACTION \
  --validation_fraction=$WIDERFACE_VALIDATION_VALIDATION_FRACTION \
  --test_fraction=$WIDERFACE_VALIDATION_TEST_FRACTION \
  --automl_out=$WIDERFACE_VALIDATION_AUTOML

gsutil cp $WIDERFACE_TRAINING_AUTOML $WIDERFACE_BUCKET
gsutil cp $WIDERFACE_VALIDATION_AUTOML $WIDERFACE_BUCKET

1.4 Combine the datasets

Combine all AutoML dataset specs into one and upload it:

THERMAL_FACE_AUTOML="automl.csv"
MODEL_BUCKET="gs://thermal-face"
MODEL_NAME="thermal_face_automl_edge_fast"

gsutil mb -l $LOCATION $MODEL_BUCKET

rm -f $THERMAL_FACE_AUTOML
cat $TDFACE_AUTOML >> $THERMAL_FACE_AUTOML
cat $WIDERFACE_TRAINING_AUTOML >> $THERMAL_FACE_AUTOML
cat $WIDERFACE_VALIDATION_AUTOML >> $THERMAL_FACE_AUTOML
cat $FLIR_ADAS_AUTOML >> $THERMAL_FACE_AUTOML

gsutil cp $THERMAL_FACE_AUTOML $MODEL_BUCKET

2. Train the model

Use Cloud AutoML Vision with the following options:

New dataset name: tdface_x_widerface_x_fliradas
Model objective: Object detection
CSV file on Cloud Storage: $MODEL_BUCKET/$THERMAL_FACE_AUTOML
Model name: $MODEL_NAME
Model type: Edge
Optimize for: Faster predictions
Node budget: 24 node hours
Use model: TF Lite
Export to Cloud Storage: $MODEL_BUCKET/

3. Compile the model

Use Docker to compile the model for Edge TPU:

MODEL_FILE="$MODEL_NAME.tflite"
TPU_MODEL_FILE="${MODEL_FILE%.*}_edgetpu.${MODEL_FILE##*.}"
MODEL_DIR="$(pwd)/../models"
COMPILER_NAME="compiler"
OUT_DIR="/out"

gsutil cp $MODEL_BUCKET/**/*$MODEL_NAME*/model.tflite $MODEL_FILE

docker build . \
  --no-cache \
  --file=$COMPILER_NAME.Dockerfile \
  --tag=$COMPILER_NAME \
  --build-arg=MODEL_FILE=$MODEL_FILE \
  --build-arg=OUT_DIR=$OUT_DIR
docker run \
  --mount=type=bind,source=$MODEL_DIR,target=$OUT_DIR \
  --rm \
  $COMPILER_NAME
mv $MODEL_FILE $MODEL_DIR/

yingjun-zhang / thermal-face Goto Github PK

thermal-face's Introduction

Thermal Face

Inference

Training

1. Create the dataset

1.1 Get the Tufts Face Database

1.2 Get FLIR ADAS

1.3 Get WIDER FACE

1.4 Combine the datasets

2. Train the model

3. Compile the model

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent