Feature Request: ml5 with Arduino Microcontrollers using webSerial about ml5-next-gen HOT 4 OPEN

Hi @hpssjellis yes, this is a wonderful idea! At the moment, the project is undergoing a rebuild in order to refine the models and update all the dependencies. There is a team of students at NYU in NY and also NYU Shanghai that are collaborating this summer on this work. This work is happening in this repo at the moment:

https://github.com/ml5js/ml5-next-gen

I think this would be a wonderful addition to the project, do you see it requiring new features in m5.js itself or is it more about providing an example with webSerial? @yining1023 may also be interested as she has taught a class around ML + physical computing.

from ml5-next-gen.

Thanks @shiffman for replying. I will be presenting my findings at an education conference in Italy https://indico.ictp.it/event/10185 July 3-7, 2023. I am a part of tinyML4D - tinyml.seas.harvard.edu/team bringing ML to university students in developing countries (Look for me Jeremy Ellis, the only high school teacher in the group ). I have already met one of the p5js contributors @gohai in this thread who used my suggestions about polyfill (an android Pixel phone fallback) for webSerial to bring it to p5js.

Thanks for the link, my research about webSerial with machine learning such as edgeimpulse.com or huggingface is here. I will be working on vanilla Javascript files, but thought you could bring webSerial for microcontrollers with machine learning to a wider audience. Hopefully we can kind of work together.

from ml5-next-gen.

I'm transferring the issue to our new repo: https://github.com/ml5js/ml5-next-gen for more updated discussion and possible issue assignment.

from ml5-next-gen.

@shiffman and @sproutleaf now that it is the fall and students are back, is anyone interested in merging TensorflowJS with Arduinos? I have basically done all the work, I just need someone to make it compatible with the p5/ml5 community.

The steps are:

1. TinyMLjs. This webpage uses WebSerial to connect an Arduino (Nano33BleSense) to a webpage and then train a tensorflowJS model and export the model.json and .bin shard files.

2. Gitpod tfjs-converter. This gitpod (docker in the browser) Uses tensorflowjs-converter to convert the model.json file to keras then tflite and finally to a c.header text readable file to load onto an Arduino

3. Arduino examples. This repository shows multiple arduino sketches that work with the c.header file (Note: This step I am working on as my working code has been deprecated by the latest arduino tensorflowlite)

None of this is easy or has been well documented but I am willing to help someone wanting to use the ideas for ml5. I will continue to improve my methods. Bes way to reach me is probably to continue this issues thread and add me @hpssjellis .

from ml5-next-gen.