StealthyIMU is a privacy threat that uses motion sensors to steal permission-protected private information from the Voice User Interfaces (VUIs) on smartphone. StealthyIMU can steal private information from 23 types of frequently-used voice commands to acquire contacts, search history, calendar, home address, and even GPS trace with high accuracy. Please refer to our StealthyIMU paper in NDSS 2023 for more details.

In this repo, we release our collected VUI response dataset, which contains the ground truth permission-protected private entities, audio recording, and corresponding accelerometer and gyroscope signals for each VUI response. Besides, we open-source the basic Spoken Language Model (SLU) DNN model that steals the permission-protected private entities from the motion sensor signals on smartphone.

Our open-sourced dataset contains 7 types of VUI responses as shown in the following table.

For each VUI response, we provide the following information:

• Metadata: ground truth of permission-protected private entities, transcription of audio recording
• Motion Sensor Signals (MSS): 6 DoF accelerometer (.acc) and gyroscope (.gyro) signals
• Audio Recordings: the audio (.wav) recorded by microphone on an Macbook Pro laptop
• Transcription File: the transcription file (.txt) of audio recordings via Amazon Transcribe or Google Speech-to-text
• The MSS signals after normalization (.accnpy, .gyronpy)
• The GPS trace/ location for the navigation command (.gps)

The dataset can be downloaded from the Google Drive.

Our implentation is based on the SpeechBrain Once you have created your Python environment (Python 3.7+), you can install the SpeechBrain via pip.

pip install speechbrain

Then make sure that you can access SpeechBrain with:

import speechbrain as sb

We provide a pretrained baseline Spoken Language Understanding (SLU) model and results.

You can also train a baseline Spoken Language Understanding (SLU) model for StealthyIMU with:

python train.py hparams/baseline.yaml

Note that

• You may need to change the "data_folder" in "hparams/baseline.yaml" to your download folder. If you want to modify more parameters in the model, please refer to "hparams/baseline.yaml".

• The pretrained tokenizer is provided in the file folder "pretrain". If you want to train a tokenizer by your self, please refer to SpeechBrain SLURP.

Once you train and test the model. You will receive a testing result "wer_test_real.txt".

You can evaluate StealthyIMU via our evaluation tool with

python eval/eval.py ./results/BPE51_all_opensource/1235/wer_test_real.txt

Here is an example results of our baseline model. You can improve this baseline results by designing a better SLU model or balance different types of VUI response data in the training dataset as discussed in our StealthyIMU paper.

Ke Sun, Chunyu Xia, Songlin Xu, Xinyu Zhang. StealthyIMU: Extracting Permission-protected Private Information from Smartphone Voice Assistant using Zero-Permission Sensors. In Proceedings of NDSS, 2023

@inproceedings{sun2023stealthyimu,
  title={{StealthyIMU}: Extracting Permission-protected Private Information from Smartphone Voice Assistant using Zero-Permission Sensors,
  author={Sun, Ke and Xia, Chunyu and Xu, Songlin and Zhang, Xinyu},
  year={2023},
  booktitle={NDSS},
}