UE5_Remote

POC - Video Streaming with UE5

Components

WebSocket Server

The WebSocket Server relays streaming binary images to clients.

Dependency: websockets/ws
WebSocket Server Server/app.js

cd Server
npm install
node app.js

HTML5 Client

The HTML5 client uses a WebSocket client to receive binary images and display in an img element.

(Future Feature) WebSocket clients can send player input to the Unreal Remote Host by using WebSocket messages.

UE5 Remote Host

The Unreal Remote Host is an UE5 project that uses a render texture and a WebSocket client to send binary image data to clients. Clients display the binary images. The UE5 project extends the third person perspective game template.

Unit Tests

Server/sendImage.js sends an image to the WebSocket Server that the client displays.

Change Log

Added UE5_Remote/Content/UI/BPWidget_UI.uasset - BP Widget
Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.h

Add RenderTarget include:

#include "Engine/CanvasRenderTarget2D.h"

Add BP Function:

 UFUNCTION(BlueprintCallable, Category = "Capture")
 UTextureRenderTarget2D* CreateRenderTarget(const int32 width, const int32 height);

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.cpp

Implement function:

UTextureRenderTarget2D* AUE5_RemoteCharacter::CreateRenderTarget(const int32 width, const int32 height)
{
 UTextureRenderTarget2D* RenderTarget = NewObject<UTextureRenderTarget2D>();

 RenderTarget->RenderTargetFormat = ETextureRenderTargetFormat::RTF_RGBA8;

 RenderTarget->InitAutoFormat(width, height);
 RenderTarget->UpdateResourceImmediate(true);

 return RenderTarget;
}

Updated UE5_Remote/Content/ThirdPerson/Blueprints/BP_ThirdPersonCharacter.uasset - Event Graph
BeginPlay: calls InitUI and SetupRenderTexture custom events.
InitUI: Loads UI BP Widget and saves to Widget variable.
SetupRenderTexture: Creates UTextureRenderTarget2D and saves to RenderTexture variable.
Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.h

Add class prototype:

class USceneCaptureComponent2D;

Add protected property for camera capture component:

UPROPERTY(VisibleAnywhere, BlueprintReadOnly, Category = Camera)
USceneCaptureComponent2D* CaptureComp;

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.cpp

Include SceneCaptureComponent2D:

#include "Components/SceneCaptureComponent2D.h"

Attach Capture Component to the Camera Boom:

CaptureComp = CreateDefaultSubobject<USceneCaptureComponent2D>(TEXT("CaptureComp"));
if (CaptureComp)
{
  CaptureComp->SetupAttachment(CameraBoom);
}

The BP Viewport will show the new capture component - UE5_Remote/Content/ThirdPerson/Blueprints/BP_ThirdPersonCharacter.uasset

The texture target of the CaptureComponent has not been set yet.

Create UE5_Remote/Content/Materials/RenderTarget2D_Capture.uasset render target at 480x270. This is 1/4 of 1080p by design.
Set the Texture Target to use the RenderTarget2D_Capture RenderTarget.

Set the Capture Source to Final Color (LDR) in RGB.

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.h

Add BP Functions:

UFUNCTION(BlueprintCallable, Category = "Capture")
bool GetRawData(UTextureRenderTarget2D* TexRT, TArray<uint8>& RawData);

UFUNCTION(BlueprintCallable, Category = "Capture")
void SendRenderTexture(UTextureRenderTarget2D* TextureRenderTarget);

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.cpp

Add implementation for getting raw data bytes from render texture.

// From: C:\Program Files\Epic Games\UE_4.21\Engine\Source\Runtime\Engine\Private\ImageUtils.cpp
bool AUE5_RemoteCharacter::GetRawData(UTextureRenderTarget2D* TexRT, TArray<uint8>& RawData)
{
 FRenderTarget* RenderTarget = TexRT->GameThread_GetRenderTargetResource();
 EPixelFormat Format = TexRT->GetFormat();

 int32 ImageBytes = CalculateImageBytes(TexRT->SizeX, TexRT->SizeY, 0, Format);
 RawData.AddUninitialized(ImageBytes);
 bool bReadSuccess = false;
 switch (Format)
 {
 case PF_FloatRGBA:
 {
  TArray<FFloat16Color> FloatColors;
  bReadSuccess = RenderTarget->ReadFloat16Pixels(FloatColors);
  FMemory::Memcpy(RawData.GetData(), FloatColors.GetData(), ImageBytes);
 }
 break;
 case PF_B8G8R8A8:
  bReadSuccess = RenderTarget->ReadPixelsPtr((FColor*)RawData.GetData());
  break;
 }
 if (bReadSuccess == false)
 {
  RawData.Empty();
 }
 return bReadSuccess;
}

Add headers:

#include "ImageUtils.h"
#include "IImageWrapper.h"
#include "IImageWrapperModule.h"
#include "RenderUtils.h"

Add ImageWrapper, RenderCore, and RHI modules to UE5_Remote/Source/UE5_Remote/UE5_Remote.Build.cs

PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "HeadMountedDisplay", "ImageWrapper", "RenderCore", "RHI" });

Regenerate the Visual Studio project files after a module change.

Include Websockets headers:

#include "WebSocketsModule.h"
#include "IWebSocket.h"

Add WebSockets module to UE5_Remote/Source/UE5_Remote/UE5_Remote.Build.cs

PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "HeadMountedDisplay", "ImageWrapper", "RenderCore", "RHI", "WebSockets" });

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.h
Add include for IWebSocket

#include "IWebSocket.h"

Add WebSocket protected data member.

TSharedPtr<IWebSocket> WebSocket;

Override BeginPlay and EndPlay:

virtual void BeginPlay() override;

virtual void EndPlay(const EEndPlayReason::Type EndPlayReason) override;

Updated Source: UE5_Remote/Source/UE5_Remote/UE5_RemoteCharacter.cpp
Implement BeginPlay:

void AUE5_RemoteCharacter::BeginPlay()
{
 Super::BeginPlay();

 if (!FModuleManager::Get().IsModuleLoaded("WebSockets"))
 {
  FModuleManager::Get().LoadModule("WebSockets");
 }

 WebSocket = FWebSocketsModule::Get().CreateWebSocket("ws://localhost:8080");

 WebSocket->OnConnected().AddLambda([]()
  {
   GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, "Successfully connected");
  });

 WebSocket->OnConnectionError().AddLambda([](const FString& Error)
  {
   GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Red, Error);
  });

 WebSocket->OnClosed().AddLambda([](int32 StatusCode, const FString& Reason, bool bWasClean)
  {
   GEngine->AddOnScreenDebugMessage(-1, 5.f, bWasClean ? FColor::Green : FColor::Red, "Connection closed " + Reason);
  });

 WebSocket->OnMessage().AddLambda([](const FString& MessageString)
  {
   GEngine->AddOnScreenDebugMessage(-1, 5.f, FColor::Green, MessageString);
  });

 WebSocket->Connect();
}

Implement EndPlay:

void AUE5_RemoteCharacter::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
 if (WebSocket->IsConnected())
 {
  WebSocket->Close();
 }

 Super::EndPlay(EndPlayReason);
}

Implement SendRenderTexture:

void AUE5_RemoteCharacter::SendRenderTexture(UTextureRenderTarget2D* TextureRenderTarget)
{
 //UE_LOG(LogTemp, Log, TEXT("Client sending over WebSocket"));

 if (WebSocket->IsConnected() && TextureRenderTarget)
 {
  if (TextureRenderTarget->GetFormat() != PF_B8G8R8A8)
  {
   EPixelFormat TextureFormat = TextureRenderTarget->GetFormat();
   UE_LOG(LogTemp, Log, TEXT("Render Target is not in the expected format should be 'PF_B8G8R8A8' instead found '%s'!"),
    GetPixelFormatString(TextureFormat));
  }
  else
  {
   check(TextureRenderTarget != nullptr);
   FRenderTarget* RenderTarget = TextureRenderTarget->GameThread_GetRenderTargetResource();
   FIntPoint Size = RenderTarget->GetSizeXY();

   TArray<uint8> RawData;
   bool bSuccess = GetRawData(TextureRenderTarget, RawData);

   IImageWrapperModule& ImageWrapperModule = FModuleManager::Get().LoadModuleChecked<IImageWrapperModule>(TEXT("ImageWrapper"));

   //TSharedPtr<IImageWrapper> ImageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::BMP); //crash probably too large
   //TSharedPtr<IImageWrapper> ImageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::PNG); // 10 FPS Max
   TSharedPtr<IImageWrapper> ImageWrapper = ImageWrapperModule.CreateImageWrapper(EImageFormat::JPEG); // 30 FPS

   ImageWrapper->SetRaw(RawData.GetData(), RawData.GetAllocatedSize(), Size.X, Size.Y, ERGBFormat::BGRA, 8);

   //const TArray64<uint8>& ImageData = ImageWrapper->GetCompressed(100); //largest size
   const TArray64<uint8>& ImageData = ImageWrapper->GetCompressed(0); //smallest size

   WebSocket->Send((void*)ImageData.GetData(), ImageData.GetAllocatedSize(), true);
  }
 }
 else
 {
  UE_LOG(LogTemp, Log, TEXT("HTTP module not available!"));
 }
}

Blueprints

Updated UE5_Remote/Content/ThirdPerson/Blueprints/BP_ThirdPersonCharacter.uasset - Event Graph
The SetupRenderTexture custom event sets up the RenderTexture called from the EventPlay event.

The SendRenderTextureToWebSocket custom event sends image data limited by the RefreshRate variable.

The EventTick calls the custom event.

linecode / ue5_remote Goto Github PK

ue5_remote's Introduction

UE5_Remote

Components

WebSocket Server

HTML5 Client

UE5 Remote Host

Unit Tests

Change Log

Blueprints

ue5_remote's People

Contributors

Watchers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent