javmarina / nintendo-switch-remote-control Goto Github PK

View Code? Open in Web Editor NEW

90.0 8.0 8.0 15.26 MB

Remote play system for the Nintendo Switch

License: MIT License

Makefile 1.44% C 23.63% Java 63.26% Python 6.91% Kotlin 4.75%

web-rtc arduino nintendo-switch java-fx streaming

nintendo-switch-remote-control's Introduction

Nintendo-Switch-Remote-Control

Remote play for the Nintendo Switch via WebRTC (with video streaming). Pro Controller emulation in AVR firmware.

This project uses the LUFA library and reverse-engineering of the Pro Controller for Nintendo Switch for remote control of the console. It consists on three main parts:

GUI: Java project with two programs, client and server.

The client takes input from a real controller, keyboard or a Discord bot and sends it to the server over WebRTC.

The server receives the input and sends it to an AVR microcontroller which acts as a controller. The MCU is plugged into the Switch dock and the console recognizes it as a Pro Controller. If you have a USB-C adapter, you should also be able to use this in handheld mode/Switch Lite.

Finally, the server sends a real-time video and audio stream acquired from an HDMI capture card, over WebRTC (the technology used for Stadia and Google Meet).
Firmware: firmware that runs on the microcontroller. The server PC sends the received commands to the microcontroller via UART. When the Switch requests HID reports, the microcontroller sends them. It uses the LUFA library.
Small Python code for fast prototyping (python folder).

The goal of this project is to provide a way to play Switch games remotely (two computers), or control the console locally or automate tasks (one computer).

A complete diagram is shown below:

NOTE: if you're using a FTDI-based serial adapter, yo need to reduce the latency timer value.

Additional information can be found on the specific README files.

The README inside the gui folder contains information about the Java project, required setup and steps, known issues and instructions for configuring a Discord bot.
Inside firmware folder there are instructions for compiling and flashing the firmware for different boards, as well as a list of changes made to the original code.

Is it fast/usable?

Depends on your setup. My goal has been to optimize the software, which is the part I can control, but performance can also be affected by the hardware.

Critical software parts, such as controller input, real-time video streaming and serial communication, use native libraries. Java is used for the user interface (in particular, JavaFX, which is fairly efficient).
Serial communication runs at 1 Mbps by default, which is the maximum baud rate the Arduino Uno R3 can safely work with. This means one controller packet is sent in under 0.1 ms.
For audio and video streaming, WebRTC is used. WebRTC is the state-of-the-art videoconference technology, used by Google Meet and, more importantly, Stadia. WebRTC has sub-500-millisecond latency, is open-source and has become an industry standard.
There is an option to select the video codec for the remote play. Users can take advantage of recent codecs, such as VP9 or AV1, with improved quality and reduced bandwidth.

Therefore, if you experience high latency and can't play games, your hardware setup is probably to blame. The most critical components are the HDMI capture card and the internet connection. Low quality HDMI capture devices introduce high latency that can affect the user experience. A high bandwith and low latency internet connection is also desirable.

Prerequisites

A LUFA-compatible microcontroller such as the Teensy 2.0++, Arduino UNO R3, or the Arduino Micro
A USB-to-UART (TTL) adapter. Popular ones are based on FTDI and CH340 chip. Can be easily found online.
An HDMI capture card compatible with libuvc (in general, any with USB output). Device quality can dramatically affect streaming performance.
A PC with Java and JDK installed (Java 11 or newer required).

TODO list

List of things that I might do in the future, no guarantee. Pull requests are welcome.

Automatically decrease read timeout for FTDI-based serial adapters (see https://github.com/projectgus/hairless-midiserial/blob/master/src/PortLatency_win32.cpp).
Power the board from the serial adapter and not USB (Switch), so that console can suspend without turning off the MCU. In that case, the Switch could be remotely turned on by long pressing the HOME button.
Power reduction techniques (section 9.9 from ATmega16U2 datasheet).
Refactoring of SerialAdapter.java. Blocking/non-blocking operations, use default procedures for syncing.
Migrate from Jamepad to sdl2gdx (?).

Acknowledgments

devopvoid for his work on the webrtc-java project.
mzyy94 for his work on Pro Controller emulation using a Raspberry Pi.
wchill for the SwitchInputEmulator project. My firmware was initially based on his work.
progmem for the Switch-Fightstick repository, which itself is the base of wchill work and created the opportunity to control the Switch with a LUFA-compatible MCU.
ItsDeidara author of the CommunityController repository and host of the namesake Twitch channel. Gave some tips for hardware configuration and some of the serial port code is based on his work.
abcminiuser, who created the LUFA library (Lightweight USB Framework for AVRs).

nintendo-switch-remote-control's People

Contributors

Stargazers

Watchers

Forkers

hisptoot tails32 goyeliseo1 straiforos doitandbedone rgdev llajas

nintendo-switch-remote-control's Issues

Issue after flashing in DFU mode for Arduino Uno r3

I successfully flashed the ATmega16u thanks to your directions and the Arduino is recognized correctly as a controller. However I’ve found that in both this build and that found in SwitchInputEmulator, I cannot get the Switch to receive commands.

I’ve done the wiring you described in your message to the SwitchInputEmulator dev and still no luck, were there any other modifications you made to get the setup to work?

I will add that I have an FT232 adaptor and Arduino Uno r3, and can confirm the setup works as I can send commands to the Switch successfully via the build in the Switch-Fightstick repository.

This is all amazing work and thanks in advance!

Switch does not recognize Arduino Uno

Hello! I recently came across this brilliant project, and have been working on implementing it with my Arduino Uno. I had no trouble generating the Joystick.hex file or flashing it onto the Arduino, and my computer now sees it as a Pro Controller. However, when I attempt to connect it using the setup diagram from the README or only with the USB B-> USB A cable, the Switch does not recognize the Arduino as a controller.

As a result, the Python client.py runs without error, but whenever it tries to send a button press, the read_bytes() function returns 0 and the button press is not received. Additionally, when the Arduino attempts to sync with the Switch, the program turns into an infinite loop when ser.in_waiting never changes from 0. (I have tested 19200 and 1Mbps baud values, changing both in avr.h and the client.py argument, but neither were successful).

As far as setup, I'm testing with client.py on my laptop, which is connected to the Uno with an FTDI USB->TTL Serial converter cable as follows: 5V/VCC->5V; GND->GND; TX->TX; RX->RX; CTS and RTS not connected. And the Uno is connected to the Switch dock with the standard USB B -> USB A cable. I tested changing the latency in the COM port settings to 1, 2, and 3 ms, and none of these values seemed to do anything. And from other testing, switching or entirely removing the TX and RX jumper wires did not change the output whatsoever, which makes sense considering that the Arduino's RX/TX lights do not turn on at any point in the process. Any help is greatly appreciated!

Python code gets stuck in loops.

I have tried to get the Python code to run, but the program keeps freezing.

The first time was at line 517. After commenting it out and finding the correct port index, it stopped at line 474 wait_for_data(). The method was called by sync(), which calls force_sync().

But when I start the Python program, the Arduino Uno R3 is recognised as a controller.

It worked via client.jar and server.jar, but unfortunately I had a CPU load of 100% and so I wanted to test python.
I would be very happy if the switch could be controlled via a python program.

Arduino Pro Micro rapidly disconnects

Hello! I've been following along with the guides for the project, but I've hit an annoying roadblock.

I compiled the firmware for the Arduino Pro Micro, with only one major change to make it compile for atmega32u4 - commenting out
UCSR1D = 0; in avr.h as UCSR1D appears to be undefined for certain chips and I couldn't find a good replacement.
I also tried to use #define UCSR1D _SFR_MEM8(0xCB) as this has been used in other code to patch it, but it did not help.
I suspect this is a vital line, as the controller never registers on the Switch, and when connected to PC it rapidly connects and disconnects as shown in the gif above. This also stops the Java program from recognising the serial communication.
Normally I would believe this to be the code crashing and rebooting, but all 3 LEDs (RX,TX and POWER) remain lit at all times.

Apologies if this is something obvious as I am not particularly knowledgable about C, but is there a fix or substitute that might solve this problem?

Controller at index 0 is not connected!

The Java Client component sadly does not work for me. It seems to be a problem with the Jamepad library, but since I can't really pin the issue down, I share this here for now. I tested under Windows 10 20H2, Feature Experience Pack 120.2212.1070.0 with a Xbox Series controller, a DualShock 4 controller and the virtual Xbox 360 controller from Parsec, all of them causing the same issue. To make sure I also used another machine entirely, also Windows 10. I also tried letting some else build the JAR for me, since I am not really sure if I did it correctly, but that didn't help either. Using the client with a keyboard works fine, I can control the Switch with that, but this doesn't cover my use case for this project.

NATIVE METHOD: loading from  "C:\Users\tails\AppData\Local\Temp\6964659601243710076.tmp"
             result 64  Failed loading libusb-1.0.dll: The specified module could not be found.

2 service(s) available
 0. Keyboard
com.studiohartman.jamepad.ControllerUnpluggedException: Controller at index 0 is not connected!
        at com.studiohartman.jamepad.ControllerIndex.ensureConnected(ControllerIndex.java:271)
        at com.studiohartman.jamepad.ControllerIndex.getName(ControllerIndex.java:250)
        at com.javmarina.client.services.DefaultJamepadService.toString(DefaultJamepadService.java:90)
        at com.javmarina.client.Client.showInitialFrame(Client.java:80)
        at com.javmarina.client.Client.main(Client.java:54)
 1. Disconnected

xhr poll error

I have everything mostly setup, however when I run the server it'll throw.
io.socket.engineio.client.EngineIOException: xhr poll error

I'm running on a Windows 11 PC using OpenJDK 20.

What's strange is it doesn't appear to be reaching out to the network at all to establish the websocket.

...

How to build Java parts?

Hey there, I've stumbled upon your great work while trying to build my own solution with the original reverse engineered code. As I am basically trying to do the same thing you did, I tried building yours.

I tried "gradle build" from the gui folder. I got some jar files but no executable jars. I've only ever worked with gradle in the context of android studio and generally don't have much java experience. Can you help me?

Thank in advance.

Curious about cross-compatibility with my project

Howdy!

I see we were both inspired by wchill's controller project! I have a similar fork of that meant for easy-to-setup UART->Nintendo Switch connections. Namely, my project UARTSwitchCon. I was wondering if you ever needed Joy-Con support (without IMU/NFC/IR support, but it could be added by someone) you could try out my ESP32-based project! Some games (Like Clubhouse Games) need Joy-Cons for the games to work, and I built that for simple games to be played with a Twitch Chat via ClubchatGames.

It should be a drop-in for the original wchill project, but I haven't been able to glean the differences between that and yours yet.

It can also act as a Pro Controller, but that's less important when we have the existing projects.

Just wanted to share this and see if it could help you at all! Take care and stay safe.

Couldn't sync with the AVR MCU

Hi, I seem to be having a similar issue to #5, but I've tested with two different Arduinos and two different UART adapters and got the same result, so I find it hard to believe it's a hardware issue (though I suppose it is possible).

My hardware:
Ardunio UNO R3 (a pretty old one, I think purchased from Radio Shack)
BIT-C Pro Micro
DSD TECH SH-U09C5
HiLetgo USB to TTL

I made sure to follow the steps here #5 (comment) exactly. I also tried many different baud rates, making sure they were compatible with both Ardunio and adapter as mentioned here #7 (comment)
I know I'm flashing the firmware correctly because once flashed, my Arduinos show up as Pro Controller in Windows.

However, no matter what I try, I'm always met with "Couldn't sync with the AVR MCU" with the console output simply saying:

Bytes written
Couldn't sync

I'm really running out of ideas here, so any help would be greatly appreciated!