Giter VIP home page Giter VIP logo

street-fighter-arduino-tflite's Introduction

Air Street Fighter with Arduino and TensorFlow Lite

This project is a prototype of Street Fighter game you play by moving your arms in the air, as if you were doing a "punch", "hadoken", or "shoryuken".

Live demo

To be able to use it, you need an Arduino 33 Nano BLE Sense, and follow the steps to pre-install the TFMicro Motion Kit on it.

Install and run

Flashing: Using the Arduino Nano Sense 33 BLE

  1. Install the Arduino IDE

  2. Setup Arduino board:

  • Plug in the board

  • Install the board by navigating to Tools > Board > Boards Manager and search for Arduino Mbed OS Nano Boards. Full instructions (including drivers required for Windows) here.

  • After the board is installed, select it under to Tools > Board > Arduino Mbed OS Nano Boards > Arduino Nano 33 BLE

  • Select the port by navigating to Tools -> Port -> dev/cu... (Arduino Nano 33 BLE)

  1. Install Arduino libraries
  • Navigate to Tools > Manage Libraries
  • Search for and install:
    • Arduino_LSM9DS1
    • ArduinoBLE
    • Arduino_TensorFlowLite

  1. Open the sketch and flash
  • Download the latest release here
  • Open the arduino/tf4micro-motion-kit and double click on <tf4micro-motion-kit.ino> file
  • Click the Right arrow in the top left corner to build and upload the sketch.
  • Warning: This process may take a few minutes. Also, warnings may populate but the upload should still succeed in spite of them.
  • If the sketch is installed, the LED on the board should flash red and green.
  1. Visit https://air-street-fighter.netlify.app

Running the app locally

  • Clone the repo
  • Install the dependencies with npm install
  • Start the server with node serve.js
  • The browser should open on port 8080

street-fighter-arduino-tflite's People

Contributors

charliegerard avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

gits00

street-fighter-arduino-tflite's Issues

Arduino BLE stuck on running IMU Tiny ML

Hi, I was following this tutorial of Arduino BLE 33 Sense to use it like a controler of Street Fighter on Raspberry Pi, I tried to use the BLE library to make it wireless with a baterry but I have problems running it like for two weeks, basically it get stuck on running the model I´m not sure if is because of the Bluetooth. Or something else but it don't show me the output from the gestures. It conects but after that it just show me on the serial monitor this:

Connecting ...
Connected

on my Python code that receives all the information I tried with a simple int and it works but when I tried to use the model don't give me any result from the gestureid.

I would be awesome if someone can help me with this.

#include <ArduinoBLE.h>
#include <Arduino_LSM9DS1.h>

#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_error_reporter.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <tensorflow/lite/version.h>

#include "model2.h"

BLEService sensorService("00001101-0000-1000-8000-00805f9b34fb");
//TX Characteristics
BLEStringCharacteristic txChar("00001143-0000-1000-8000-00805f9b34fb", BLERead | BLENotify,15);     

// last sensor data    
float oldXLevel = 0; 
float oldYLevel = 0; 
float oldZLevel = 0; 
long previousMillis = 0; 

const float accelerationThreshold = 2.5; // threshold of significant in G's
const int numSamples = 119;

int samplesRead = numSamples;
int gestureid=0;

// global variables used for TensorFlow Lite (Micro)
tflite::MicroErrorReporter tflErrorReporter;

// pull in all the TFLM ops, you can remove this line and
// only pull in the TFLM ops you need, if would like to reduce
// the compiled size of the sketch.
tflite::AllOpsResolver tflOpsResolver;

const tflite::Model* tflModel = nullptr;
tflite::MicroInterpreter* tflInterpreter = nullptr;
TfLiteTensor* tflInputTensor = nullptr;
TfLiteTensor* tflOutputTensor = nullptr;

// Create a static memory buffer for TFLM, the size may need to
// be adjusted based on the model you are using
constexpr int tensorArenaSize = 8 * 1024;
byte tensorArena[tensorArenaSize] __attribute__((aligned(16)));

// array to map gesture index to a name
const char* GESTURES[] = {
  "punch",
  "defend",
  "summon"
};

#define NUM_GESTURES (sizeof(GESTURES) / sizeof(GESTURES[0]))

void setup() {
  Serial.begin(115200);  
  while (!Serial);

  if (!IMU.begin()) {
    Serial.println("Failed to initialize IMU!");
    while (1);
  }

  pinMode(LED_BUILTIN, OUTPUT); 
  
  if (!BLE.begin()) {
    Serial.println("starting BLE failed!");
    while (1);
  }

  // get the TFL representation of the model byte array
  tflModel = tflite::GetModel(model);
  if (tflModel->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while (1);
  }

  // Create an interpreter to run the model
  tflInterpreter = new tflite::MicroInterpreter(tflModel, tflOpsResolver, tensorArena, tensorArenaSize, &tflErrorReporter);

  // Allocate memory for the model's input and output tensors
  tflInterpreter->AllocateTensors();

  // Get pointers for the model's input and output tensors
  tflInputTensor = tflInterpreter->input(0);
  tflOutputTensor = tflInterpreter->output(0);

  BLE.setLocalName("NanoBLE33");
  BLE.setAdvertisedService(sensorService);
  
  sensorService.addCharacteristic(txChar);
  BLE.addService(sensorService); 

  // initialize default data
  txChar.writeValue(String(0)); 

  // start advertising
  BLE.advertise();
  Serial.println("Bluetooth device active, waiting for connections...");
}

void loop() {
  // wait for a BLE central
  BLEDevice central = BLE.central();
  if (central) {
    Serial.print("Connected to central: ");
    Serial.println(central.address());
    digitalWrite(LED_BUILTIN, HIGH);

    while (central.connected()) {
      //long currentMillis = millis();
      updateGyroscopeLevel();
      delay(300);
    }

    digitalWrite(LED_BUILTIN, LOW);
    Serial.print("Disconnected from central: ");
    Serial.println(central.address());
  }
}

void updateGyroscopeLevel() {
  
  float aX, aY, aZ, gX, gY, gZ;

  // check if new acceleration AND gyroscope data is available
  if (IMU.accelerationAvailable() && IMU.gyroscopeAvailable()) {
    // read the acceleration and gyroscope data
    IMU.readAcceleration(aX, aY, aZ);
    IMU.readGyroscope(gX, gY, gZ);

    // normalize the IMU data between 0 to 1 and store in the model's
    // input tensor
    tflInputTensor->data.f[samplesRead * 6 + 0] = (aX + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 1] = (aY + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 2] = (aZ + 4.0) / 8.0;
    tflInputTensor->data.f[samplesRead * 6 + 3] = (gX + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 4] = (gY + 2000.0) / 4000.0;
    tflInputTensor->data.f[samplesRead * 6 + 5] = (gZ + 2000.0) / 4000.0;

    samplesRead++;

    if (samplesRead == numSamples) {
      // Run inferencing
      TfLiteStatus invokeStatus = tflInterpreter->Invoke();
      if (invokeStatus != kTfLiteOk) {
        Serial.println("Invoke failed!");
        while (1);
        return;
      }

      // Loop through the output tensor values from the model
      for (int i = 0; i < NUM_GESTURES; i++) {
        if(tflOutputTensor->data.f[i]>0.7)
        {
          gestureid=i+1;
          txChar.writeValue(String(gestureid));
          Serial.print(txChar.writeValue(String(gestureid)));
        }
      }

      Serial.print(gestureid);
      gestureid=0;
      //Serial.println();
    }
  }
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.