Giter VIP home page Giter VIP logo

xv3dgs-ueplugin's Introduction

XVERSE 3D-GS UE Plugin

License

English | 中文

Introduction

XVERSE 3D Gaussian Splatting UE plugin(XVERSE 3D-GS UE Plugin)is an Unreal Engine 5 (UE5) based plugin developed by XVERSE Technology Inc. (Zhenshen, China), aiming to provide real-time visualization, management, editing, and scalable hybrid rendering of Gaussian Splatting model, which is a recently arised technique for reconstructing 3D scenes from multiple photos, more details can refer to here.

Currently, our plugin is implemented based on UE5 Niagara, fully exploiting the advantage of the features provided by UE5 to realize efficient rendering and managing for Guassian Splatting models.

Since we are at early access, current accessible features are summarized below:

  • Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
  • Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
  • Hybrid rendering with other UE assets
  • RTS and volume based crop
  • Crop a pure niagara for VFX
  • Support dynamic illumination of Gaussian Splatting scene
  • Automatic lod generation, supporting over 200,000 point clouds in niagara
  • Training tools in Windows system (XV3DTools): train a given mp4 video to GaussianSplatting ply file

We will actively release new features in this repo, please stay tuned. Some future updates will contain:

  • Dynamic Lod Rendering
  • Automatic collision generation
  • Support preview window
  • Interactively Edit 3D-GS assets
  • 3D-GS assets compression

Getting Started

Video Tutorial

Youtube(English):

B站(中文):

System Requirements

  • Windows 10 or 11
  • Unreal Engine 5.1.x / 5.2.x / 5.3.x
  • cuda11 or higher (if using the XV3DTools)

Plugin download

  1. Open github Release page

  1. Download the zip corresponding to the UE version you are using

  1. Unzip and move folder XV3dGS to folder Plugins

Demo Data Download Links:

We have provided two sets of test data available for download at:

Included in these downloads are demo_fox_1080p.mp4 and demo_office_1080p.mp4, serving as two sample MP4 files for testing XV3DTools' local training capabilities.

demo_fox_gs.ply and demo_office_gs.ply are 3DGS models trained by XV3DTools, which can be imported into Unreal Engine (UE) using this plugin for real-time rendering.

Open demo project

This repo contains a demo project with an example scene and level

  1. download using git clone
git clone https://github.com/xverse-engine/XV3DGS-UEPlugin.git
  1. Open XV3DGS.uproject to start UE
  2. Open demo level

Import your guassian splatting model

After v1.1.3, this plugin supports dragging multiple ply files directly into the Content Browser and importing them.

Click on the 'Load Assets' button on the navigation bar or navigate to Window/XV3dgs.

This action will bring up an integrated tool page by default set to load the PLY page. Click on the 'Load PLY' button.

This will trigger a resource selection window to appear.

Choose your Gaussian model file (in .ply format), then it will automaticly generate LOD and loaded into your Content Browser as a blueprint.

Drag the blueprint into Level Viewport and you can see your Gaussian scene.

Local Training on Windows Platform

We develop the XV3DTools(V1.1.1) to train MP4 videos into Gaussian Splatting ply.

Please Note: Installation of CUDA 11 or higher is required. CUDA Download Link

XV3DTools Usage Steps:

  1. Download XV3DTools(V1.1.1) tool from the cloud drive to your local machine:

  1. Click on the icon to open the interface and switch to the 'training' page.

  1. Click button to set the parameter "XV3DTools Folder" to the local path where XV3DTools tool is located. The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  2. Click button to select the MP4 file for processing and set it as the parameter "Input MP4 Path" (requires video frame rate > 20). The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  3. Click button to choose the output folder and set it as the parameter "Output Folder". The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  4. click Start Training button.

  1. Before training, you can open "Advanced Options" and click "Refresh Estimated Time" button to estimate the training time. The parameter "Training iteration per image" can be adjusted to control the number of iterations per image.

  1. Training takes time. Once completed, locate the trained ply file in the output folder.

Training Speed Reference:

  • Test Machine Configuration: Windows 10, CUDA 11.8, GeForce RTX 2080 (8GB VRAM), i9-9900 processor.
  • demo_fox.mp4: Video duration of 10 seconds, training time of 10 minutes.
  • demo_office.mp4: Video duration of 430 seconds, training time of 360 minutes.

Packaging UE level for Windows Using Our Plugins

You can refer to the following tutorial to package your own UE levels into exe files that can be run on windows System.

packing tutorial

We provide a packaged Windows executable demo, you can download pack_win_exe.zip from release, simply double-click the exe file to start it.

Feature Introduction

RTS (Rotation, Translation, Scaling) Transform

Select the 3D-GS asset and drag it into the scene. then you can Transform it.

You can also select a 3D-GS actor in the level viewport and transform it using GIZMO.

Model Clipping by Gaussian model

  1. Click to select your Gaussian model blueprint, then click to select the "Clip Volume Component", and open the "Visible" box in the "Clip Volume Component".

  1. With the "Clip Volume Component" selected, you can see a crop volume in the window. You can modify the rotation, translation, and scale of the volume by adjusting the Transform attribute. Alternatively, use the keyboard shortcuts WER to visually adjust the GIZMO for rotation, translation, and scale.

  1. Keep the "Clip Volume Component" selected, and choose different cropping options in the Crop Type. The default state is no cropping, but you can change it to crop the inside or outside of the volume. The example below shows cropping the outside of the volume.

  1. Click the "Crop GSPoints" button to clip the specified portion and obtain a clean Niagara.

  1. This clean Niagara can be used for VFX. For more details, please refer to our VFX tutorial (Coming soon).

Dynamic illumination

Dynamic lighting is not default opened.

you can enable it by turning on the withRelighting button.

Supports all the UE light types

The light source can be moved at will

Change color

Modify the AlbedoTint parameter to change the overall color.

LOD

For imported point clouds, LOD is automatically calculated, with the default being to explicitly display the maximum LOD within the limit of 2 million points for Niagara. The parameter "VisWhichLOD" can be used to force the use of a specific LOD.

Dynamic switching of LOD based on the view camera will be supported in future updates.

Release Note

v1.1.3

  • Support dragging multiple ply files into the content directory
  • Fix the crash issue when the buffer asset position is incorrect
  • Fix bug in XV3DTools while using wrong path

v1.1.2

  • Supports more types of ply header
  • Fixed flickers when moving objects
  • Update UI

v1.1.1

  • New Model Clipping allowing the creation of a clean Niagara for VFX.
  • XV3DTools v1.1.1: Users can now adjust the training iteration.

v1.1.0

  • Training tools in Windows system (XV3DTools v1.0) to train a given mp4 video to GaussianSplatting ply file
  • Automatic lod generation, supporting over 200,000 point clouds in niagara
  • fix bug in transform

v1.0.1

  • Fix bug in v1.0.0:Error while packaging for Windows in UE5.2 and UE5.3

v1.0.0

  • Niagara-Based High-quality real-time visualizing and rendering for 3D Gaussian Splatting
  • Easily importing and converting from the original Gaussian Splatting scene (.ply file) to ours
  • Hybrid rendering with other UE assets
  • RTS and Geometry Editing
  • Apply VFX effect to Gaussian Splatting scene
  • Support dynamic illumination of Gaussian Splatting scene
  • Automatic enhancement of Gaussian Splatting scene

Contributors

xv3dgs-ueplugin's People

Contributors

hrspythonix avatar kaca0083 avatar lxk121lalala avatar zhengzhongsun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xv3dgs-ueplugin's Issues

PLY file import error

Hello,
I tried to import ply Gaussian Splat file, but I get an error
"ply properties header format invalid, pls check it first"

The Gaussian Splat file was generated with Nerfstudio, then converted into PLY through SuperPlat.

I also tried to import PLY to Blender (worked) and re export to another PLY, but it still gets an error in XV3DGS plugin.

Unreal version 5.3.2
XV3DGS plugin 1.0
Windows 11

Thanks a lot for your help.
breezanil

Source Code

Is there a way to get the source code?
Unreal displays the .cpp files but they can not be opened.

Crash when importing .ply from PlayCanvas-SuperSplat

Describe the bug
When attempting to load the .ply file into the scene (using the "load GS ply model" button), the UE session immediately crashes. The ply file came from the "https://playcanvas.com/super-splat" web app. The file is relatively light, at just 44.7mb (uncompressed).

To Reproduce
Steps to reproduce the behavior:

  1. Go to "https://playcanvas.com/super-splat"
  2. Drag and drop a .ply file into the browser window
  3. When the scene loads, use the "brush" or "rect" selection tool to select a few dozen particles in the scene and press delete
  4. Scroll down the tool panel to "export" and click the "ply file" button, and save to desktop.
  5. Load the exported .ply file into UE using the XV3dGS plugin
  6. Crash occurs
  7. Reload scene, observe the content browser. new empty blueprint folder exists for the asset (suggesting the crash occurred during blueprint creation)

Expected behavior
Unreal should load the ply and create the new GS asset without crashing

Desktop (please complete the following information):

  • OS: win10
  • Browser: chrome
  • UE Version: 5.3.2

Feature Request: Add Option to Customize Training Iteration Steps

Hello Xverse,

I've been using XV3DGS-UEPlugin for a while now and I'm really impressed with its capabilities.

However, I've encountered a limitation regarding the inability to adjust the training iteration steps. Currently, the plugin runs for a fixed number of iterations, but for my use case, I need the flexibility to alter this based on different datasets and project requirements.

  • (The default number of steps for training is 12300)

Having the option to customize the number of training iterations would greatly enhance the plugin's utility, allowing users like myself to tailor the training process more closely to our specific needs.

Thank you for considering this request. I'm looking forward to any updates on this and am happy to provide more information if needed.

Best regards!

Crop problem in 5.4

Greetings!
Im on unreal engine 5.4 and when i crop a BPGaussianActor created with importer in the editor preview everything is fine and also cropping the points creates a niagara effect cropped correctly, but when i launch it in game the crop position is moved and i cannot find a way to crop the selected area...

To Reproduce
Steps to reproduce the behavior:

  1. Latest Plugin release for ue 5.4
  2. Imported/trained correctly aBPGaussianActor from an mp4
  3. Cropped and everything visualized fine in the editor preview and in the new cropped niagara asset created pushing on "Crop New Points In this Box" button from Clip Volume Component Box
  4. Launching the game in editor the crop box is putted way away from selected area and if i launch the game in standalone nothing is cropped, even if i select the cropped one as asset...

Expected behavior
Right cropped position also in game.

Btw great plugin!

for UE 5.4?

Release the newer version for UE 5.4 pls.

是否能增加批次匯入的功能

HI 感謝有這個插件,使用起來比Luma AI還好用
但我這邊有大量匯入PLY的需求
能不能加入類似Luma AI批次匯入方式(直接將PLY拉進資料夾或者匯入時可以選取多個檔案)

UE 5.4 and UE5.3 import PY showing plane. material problem ?

I have install the last Xverse for UE5.3.2 and UE5.4.
I using the demo_fox_ga.ply

FOR BOTH ue ENGINE,
when I import the ply,the first time I could see the fox but when I've drop the blueprint in level, the render is showing only plane.
Capture

even if I'm only using lumen,
I've try to disable raytracing r.RayTracing.ForceAllRayTracingEffects 0
BUT CHANGING NOTHING


in material M_GS_unlit
there is errors
[SM6] /Engine/Generated/Material.ush:3461:30: warning: implicit truncation of vector type [-Wconversion]
PixelMaterialInputs.Opacity = Local13;
^
[SM6] error: validation errors
[SM6] zzz.dxil:4463:5: error: Opcode Discard not valid in shader model lib_6_6(closesthit).
[SM6] note: at 'call void @dx.op.discard(i32 82, i1 true)' in block '#26' of function '�?MaterialCHS@@YAXUFPackedMaterialClosestHitPayload@@UFRayTracingIntersectionAttributes@@@z'.
[SM6] Validation failed.
[SM6] D3DCompileToDxil failed. Error code: Unspecified error (0x80004005).


when importing PLY the outpulog

LogTemp: Warning: create new guassian: demo_fox_gs
LogXverseGS: Warning: generate spatial result size 4076
LogXverseGS: Warning: GENERATE LOD [1] point num is 104122
LogXverseGS: Warning: generate spatial result size 2029
LogXverseGS: Warning: GENERATE LOD [2] point num is 52549
LogXverseGS: Warning: generate spatial result size 987
LogXverseGS: Warning: GENERATE LOD [3] point num is 26532
LogXverseGS: Warning: generate spatial result size 483
LogXverseGS: Warning: GENERATE LOD [4] point num is 13342
LogXverseGS: Warning: BuildNiagaraBlocks NUM 1
LogTemp: Warning: workspace_name demo_fox_gs
LogXverseGS: Warning: CreateVolumeComponent
LogXverseGS: Warning: CreateVolumeComponent
LogFileHelpers: InternalPromptForCheckoutAndSave started...
Cmd: OBJ SAVEPACKAGE PACKAGE="/Game/XV3dGS/demo_fox_gs/demo_fox_gs1_buffers" FILE="D:/UnrealWS_server/Goliath/Goliath/Content/XV3dGS/demo_fox_gs/demo_fox_gs1_buffers.uasset" SILENT=true
LogXverseGS: Warning: UGSRuntimeBuffer::Serialize START
LogXverseGS: Warning: UGSRuntimeBuffer::Serialize END
LogXverseGS: Warning: UGSRuntimeBuffer::Serialize START
LogXverseGS: Warning: UGSRuntimeBuffer::Serialize END
LogSavePackage: Moving output files for package: /Game/XV3dGS/demo_fox_gs/demo_fox_gs1_buffers
LogSavePackage: Moving 'D:/UnrealWS_server/Goliath/Goliath/Saved/demo_fox_gs1_buffersEF51F38947A794CD278AE8995672CDD2.tmp' to 'D:/UnrealWS_server/Goliath/Goliath/Content/XV3dGS/demo_fox_gs/demo_fox_gs1_buffers.uasset'
LogFileHelpers: InternalPromptForCheckoutAndSave took 273 ms (total: 853 ms)
LogXverseGS: Warning: CreateVolumeComponent
LogXverseGS: Warning: CreateVolumeComponent
LogUObjectHash: Compacting FUObjectHashTables data took 1.67ms
LogFileHelpers: InternalPromptForCheckoutAndSave started...
Cmd: OBJ SAVEPACKAGE PACKAGE="/Game/XV3dGS/demo_fox_gs/demo_fox_gs1" FILE="D:/UnrealWS_server/Goliath/Goliath/Content/XV3dGS/demo_fox_gs/demo_fox_gs1.uasset" SILENT=true****

Double/Incorrect Transformations

Describe the bug
When attempting to translate/move the BPGaussianActor in the scene, the Niagara particles will double-transform and if the object is rotated, the particles will transform on the incorrect axis when moved.

To Reproduce
Steps to reproduce the behavior:

  1. Load a .ply into the workspace and drag blueprint into scene
  2. Switch view to orthographic front/side view
  3. Translate the blueprint up and down with the manipulator tool
  4. notice the particles are moving 2x as fast as the manipulator tool
  5. reset position to 0, and rotate the blueprint 90 degrees on the Y axis
  6. Translate the blue print up and down in world space
  7. notice the actor is moving on multiple axis'

Expected behavior
The niagara particles should be inheriting the precise location of the root component, and not drift away from it when transforming the actor.

Desktop (please complete the following information):

  • OS: Win10
  • UE Version: 5.3.2

Additional context
I could be wrong, but perhaps this has to do with a problem converting local transforms to world space?

'ply properties header format invalid, pls check it first' From a polycam .ply file

          > Same issue here. Error code is the same. "ply properties header format invalid, pls check it first" UE Version: 5.2.1 XV3DGS Plugin 1.1.1 The Gaussian Splat file was generated with Nerfstudio, then exported as PLY from Nerfstudio. I can provide all necessary files if needed.

We added support for the generic ply header file in the just released v1.1.2 plugin, please try again and contact me if there are any bugs

Originally posted by @lxk121lalala in #16 (comment)

A fix for files from polycam would be very helpful. Thank you in advance!

UE5.2.1 encountered an error while packaging for Windows

I attempted to package for the Windows platform using UE5.2.1, but encountered an error when running the program after packaging. The error message is: Plugin 'XV3dGS' failed to load because module 'GSRuntime' could not be found. Please ensure the plugin is properly installed, otherwise consider disabling the plugin for this project.

性能优化是不是可以提升下,导入ply格式的小场景直接4090的GPU干满了

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Where is the demo project?

In the main branch there isn't any demo project as instructs says, and some links drives you to a 404 page...please, I need a demo project file to start if possible

about source code

Hi, thanks for your fantastic project!
I'd like to know if I can modify the source code and add other plugin functions.
Thanks!

XV3DTools does not work

Describe the bug
I can not train GS model via XV3DTools.

To Reproduce
When I click start training:

  1. finished extract the image from mp4.
  2. Exhaustive feature matching cannot finished,
    image

Expected behavior
when at Matching Block [1/12] [5/12] the terminal quited.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: Win11
  • UE 5.1
  • RTX4090
  • CUDA 12.4

關於輸出APK 到Oculus

之後有有機會可以輸出APK嗎?
在輸出APK時遇到這個錯誤
F:\UnrealProject_F\OculusAPK\Content\XV3dGS\Model_F\Model_F_buffers.uasset: Failed to cook /Game/XV3dGS/Model_F/Model_F_buffers for platform Android_ASTC. It imports class /Script/GSRuntime.GSRuntimeBuffer, which is in a module that is not available on the platform.
PackagingResults: Error: [AssetLog] F:\UnrealProject_F\OculusAPK\Content\XV3dGS\Model_F\Model_F.uasset: Failed to cook /Game/XV3dGS/Model_F/Model_F for platform Android_ASTC. It imports class /Script/GSRuntime.GaussianComponent, which is in a module that is not available on the platform.

4d Gaussian Support?

Hey guys, great project.

I really like being able to train from within unreal's IDE.

I've been looking into 4d gaussians recently and was wondering if you guys have any plans to try and implement it.
Having 3D gaussian splats be animated in unreal would be a game changer, i believe its worth looking into.
I'm going to try to get them working myself, but i doubt i'll have much success.

If you have looked into it, could you let me know how it went?

Thanks for the great plugin!

Colors not showing correctly in Unreal 5.3.2

Thanks for sharing this amazing plugin.

I just wanted to report, that I have tried it with Unreal 5.3.2 and i am experiencing some issues with the color shader:

The splats seam to be present but the colors are not rendered.

Steps to reproduce the behavior:

  1. Install the Plugin and activate it
  2. User plugin to import a PLY file.
  3. Drag imported asset into the open level or open the asset in the BluePrint Editor
  4. I can see the splat but it is grey.

I would expect to see the splat rendering in the viewport.

Screenshots
image

Desktop

  • OS: Windows 10
  • Unreal 5.3.2 DirectX 12
  • RTX2080Ti 11G VRAM / Driver Version 536.23

Additional context
When opening a splat for the first time in the Blueprint Editor after import, I was able to see the colors for a short moment. (less than a second) before they dissapear.

There is an error in the colose:

LogShaderCompilers: Warning: Failed to compile Material /XV3dGS/M_GS_unlit.M_GS_unlit for platform PCD3D_SM6, Default Material will be used in game.
/Engine/Generated/Material.ush:3466:30: warning: implicit truncation of vector type [-Wconversion]
PixelMaterialInputs.Opacity = Local12;
^
error: validation errors
zzz.dxil:4524:5: error: Opcode Discard not valid in shader model lib_6_6(closesthit).
note: at 'call void @dx.op.discard(i32 82, i1 true)' in block '#28' of function '�?MaterialCHS@@YAXUFPackedMaterialClosestHitPayload@@UFRayTracingIntersectionAttributes@@@z'.
zzz.dxil:4510:5: error: Opcode Discard not valid in shader model lib_6_6(closesthit).
note: at 'call void @dx.op.discard(i32 82, i1 true)' in block '#26' of function '�?MaterialCHS@@YAXUFPackedMaterialClosestHitPayload@@UFRayTracingIntersectionAttributes@@@z'.
Validation failed.
D3DCompileToDxil failed. Error code: Unspecified err0r (0x80004005).
zzz.dxil:4607:5: error: Opcode Discard not valid in shader model lib_6_6(closesthit).
note: at 'call void @dx.op.discard(i32 82, i1 true)' in block '#16' of function '�?PathTracingMaterialCHS@@YAXUFPackedPathTracingPayload@@UFRayTracingIntersectionAttributes@@@z'.
zzz.dxil:4593:5: error: Opcode Discard not valid in shader model lib_6_6(closesthit).
note: at 'call void @dx.op.discard(i32 82, i1 true)' in block '#14' of function '�?PathTracingMaterialCHS@@YAXUFPackedPathTracingPayload@@UFRayTracingIntersectionAttributes@@@z'.
LogSlate: Took 0.000193 seconds to synchronously load lazily loaded font '../../../Engine/Content/Slate/Fonts/Roboto-Light.ttf' (167K)

VR in UE 5.3.2.

Trying to make the plugin work in VR in UE 5.3.2.

GS is displaying fine in viewport and preview window, but rendering / displaying only on the left eye in the Quest headset.
The rest of the environment looks fine.

Any help would be appreciated.

Unreal Engine 5.4.2

Hello.

Yesterday I downloaded the latest Unreal Engine which is 5.4.2

Are you planing to release a plugin for 5.4 in the next time?

Greetings.

XV3DTools Detected as Trojan as of v1.1.1

Describe the bug
Windows Defender has detected XV3DTools' xv1_train.exe as Wacatac.B!ml trojan.

Screenshots
ApplicationFrameHost_RForFtlTxP

Desktop:

  • OS: Windows 11
  • XV3DTools Version: 1.1.1

Additional context
Version 1.1.0 still exists on my system and is not affected by Windows defender. Something in the last update has flagged it on Microsoft's malware ML database, and I'm hesitant to assume it's a false-positive considering it is an exe I downloaded from an otherwise unverified Google Drive zip. If there's any way that guarantees can be made about the safety of the executable, or if any of the internals can be open sourced, I would feel a lot better about using this tool going forward.

Depth of Field 'pops' into focus

Thanks for this awesome plugin.

I have a CineCameraActor correctly rendering depth of field for the rest of the scene (UE 5.3), but not for the splats. As I change the focus distance, the splat pops into focus all at once. Is this the expected behaviour?

使用open3d生成的点云数据不能被读取,能支持open3d生成的点云数据导入吗

当我使用open3d生成的以ply为后缀点云数据,通过xverse的ue插件导入到ue引擎中时被告知“ply properties header format invalid, pls checked it first”,请问哪些头属性是必须的?
下面是我通过open3d生成的ply文件的头信息

ply
format binary_little_endian 1.0
comment Created by Open3D
element vertex 921600
property double x
property double y
property double z
property uchar red
property uchar green
property uchar blue
end_header

以下是示例的ply文件的头信息

ply
format binary_little_endian 1.0
element vertex 206346
property float x
property float y
property float z
property float nx
property float ny
property float nz
property float f_dc_0
property float f_dc_1
property float f_dc_2
property float f_rest_0
property float f_rest_1
property float f_rest_2
property float f_rest_3
property float f_rest_4
property float f_rest_5
property float f_rest_6
property float f_rest_7
property float f_rest_8
property float f_rest_9
property float f_rest_10
property float f_rest_11
property float f_rest_12
property float f_rest_13
property float f_rest_14
property float f_rest_15
property float f_rest_16
property float f_rest_17
property float f_rest_18
property float f_rest_19
property float f_rest_20
property float f_rest_21
property float f_rest_22
property float f_rest_23
property float f_rest_24
property float f_rest_25
property float f_rest_26
property float f_rest_27
property float f_rest_28
property float f_rest_29
property float f_rest_30
property float f_rest_31
property float f_rest_32
property float f_rest_33
property float f_rest_34
property float f_rest_35
property float f_rest_36
property float f_rest_37
property float f_rest_38
property float f_rest_39
property float f_rest_40
property float f_rest_41
property float f_rest_42
property float f_rest_43
property float f_rest_44
property float opacity
property float scale_0
property float scale_1
property float scale_2
property float rot_0
property float rot_1
property float rot_2
property float rot_3
end_header

Crash on importing .ply file

Describe the bug
Unreal Engine 5.3.2 crashes on loading a .ply file

To Reproduce
Steps to reproduce the behavior:

  1. Go to gsgen - https://github.com/gsgen3d/gsgen
  2. Setup project and create a .ply file
  3. Create new UE5.3.2 project, enable XVERSE plugin and restart
  4. Import ply

Expected behavior
New blueprint containing the generated text-to-3d GSplat.

Log:
LoginId:9b69881d4eb82810a8475587b35d1a6b
EpicAccountId:98746074235e498cb3de68c55a3616ae

Assertion failed: NewNumMips <= GetFullMipCount(SizeX,SizeY,GetVolumeSizeZ()) [File:D:\build++UE5\Sync\Engine\Source\Runtime\Engine\Private\Texture.cpp] [Line: 4018]

UnrealEditor_Engine
UnrealEditor_Engine
UnrealEditor_GSImporter!UGaussianAsset::create_whole_world_point_cloud_texture() [C:\zyl_object\project\nerf\file\output\pluginPack\5.3\v03\XV3dGS\HostProject\Plugins\XV3dGS\Source\GSImporter\Private\GaussianAsset.cpp:249]
UnrealEditor_GSImporter!UGaussianAsset::serialize_gaussion_to_Texture2D() [C:\zyl_object\project\nerf\file\output\pluginPack\5.3\v03\XV3dGS\HostProject\Plugins\XV3dGS\Source\GSImporter\Private\GaussianAsset.cpp:276]
UnrealEditor_GSImporter!UGaussianAsset::load_gs_model_frome_ply_file() [C:\zyl_object\project\nerf\file\output\pluginPack\5.3\v03\XV3dGS\HostProject\Plugins\XV3dGS\Source\GSImporter\Private\GaussianAsset.cpp:417]
UnrealEditor_GSImporter!FGSImporterModule::PluginButtonClicked() [C:\zyl_object\project\nerf\file\output\pluginPack\5.3\v03\XV3dGS\HostProject\Plugins\XV3dGS\Source\GSImporter\Private\GSImporter.cpp:50]
UnrealEditor_GSImporter!TBaseRawMethodDelegateInstance<0,FGSImporterModule,void __cdecl(void),FDefaultDelegateUserPolicy>::ExecuteIfSafe() [C:\Program Files\Epic Games\UE_5.3\Engine\Source\Runtime\Core\Public\Delegates\DelegateInstancesImpl.h:534]
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_Slate
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
UnrealEditor_ApplicationCore
user32
user32
UnrealEditor_ApplicationCore
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
UnrealEditor
kernel32
ntdll

Desktop:

  • OS: Windows 11
  • GPU NVIDIA GeForce RTX 3050 Laptop

Moving gaussian splat to another folder location crashes editor

Describe the bug
After importing the gaussian splat through the import ply feature, if you move the folder where the blueprint is to another location in the project the editor will crash.

To Reproduce
Steps to reproduce the behavior:

  1. Import ply
  2. Move or copy generated folder with BP storing the gaussian splat
  3. Crash

Desktop (please complete the following information):

  • OS: Windows 11
  • Unreal Engine 5.3.2

3DGS-File trained with nerfstudio is not working with ue5.3

FYI

I trained a 3DGS-Modell with Nerfstudio which i loaded through your plugin to ue5.3 but the loaded Modell is super laggy and has fales colors. Somehow the turnaraound explained below fixed the problem:

SOLUTION: import it to blender with the gaussian-splatting addon from here:
https://github.com/ReshotAI/gaussian-splatting-blender-addon/blob/master/blender-addon/plyfile.py
and exporting it again. It creates a file with the same size than the original one from nerfstudio, but can now be loaded corretly to XV3DGS and ue5.3

thx for your work

M_GS_unlit and M_GS_relight materials fail to compile on 5.3.2

I just downloaded the "XV3DGS-v1.1.3-ue5.3" release package and am using it in a source build of 5.3.2.

I notice the splats show up and work, but the opacity custom code node is throwing errors on my end.

If I make opacity 1 or either the con_o input, I see the splat incorrectly, so i know everything else is working, except the material.

Any help would be appreicated!
Thank you!

-Brandon

image

高斯溅射生成的模型,打包后无法显示

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Smartphone (please complete the following information):

  • Device: [e.g. iPhone6]
  • OS: [e.g. iOS8.1]
  • Browser [e.g. stock browser, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Octree-GS pointcloud.ply suport

Hello, thank you for your work. Recently, I found Octree-GS which is another 3DGS. but they change the format of .ply.
Do you have interest to suport Octree-GS or give me some hint to fix it ?

[Bug] In Camera VFX project mode, blinking issue is shown.

Describe the bug
If the *.ply blueprint is loaded, loaded object is continueously blinking.
Then, play mode or rendering result is now shown that problem.

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'InCameraVFX project'
  2. Load *.ply as blueprint
  3. Click & Drag 'loaded ply blueprint' to display.
  4. See error

Screenshots ( video file )
https://github.com/xverse-engine/XV3DGS-UEPlugin/assets/26716759/9cc500f5-e427-4ce9-bc41-35d5b7198e3a

Desktop (please complete the following information):

  • OS: Windows 10

  • Version: 21H2

  • UE version: 5.3.2

  • NVIDIA driver version: 546.29 ( RTX GPU 2080ti )

Linux Support

Would love it if the plugin can be built with linux in mind so we can build a project for it.

Thanks.

Lower resolution than expected.

The resolution of the Gaussian Splatting capture is lower than expected.

Steps to reproduce the behavior:

  1. I have downloaded the plugin for UE5.2 and directly placed the 'XV3dGS' folder into the 'Plugins' folder.
  2. After enabling the plugin, the button to import appeared, and I used it to import the .ply file.
  3. I imported a .ply with 30,000 points.
    imagen
  4. The resolution of the image is much lower than expected, seems that it's about 1400 instead the 30000 points.
    imagen
    imagen

Do you have any suggestions to try to solve this?

Thank you!!

Windows 11
UE 5.2"

XV3DTools not working

UE 5.2 with CUDA 12.3 installed (Win 10, GTX 1080)

Trying to train 3DGS with XV3DTools. After clicking Start Training I get message in a few seconds Training complete but no ply was produced.

Here is log:
LogTemp: Error: -------------------------C:/Users/imons/Desktop/XV3DTools
LogTemp: Error: -------------------------K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/newtower720.mp4
LogTemp: Error: -------------------------K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024
LogTemp: Error: -------------------------startTraining
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/aa.mp4
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/aa.mp4
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/aa.mp4
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/input
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/input
LogTemp: Error: ---------------------------colmap_executable=C:/Users/imons/Desktop/XV3DTools/COLMAP/COLMAP.bat --ffmpeg_executable=C:/Users/imons/Desktop/XV3DTools/ffmpeg/ffmpeg.bat --source_path=K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/input
LogTemp: Error: --------------------------s K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video -m K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/out --resolution 1 --data_device cpu --lazy_loading --iterations 3000 --position_lr_max_steps 3000 --densify_until_iter 1500
LogTemp: Error: --------------------------s K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video -m K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/out --resolution 1 --data_device cpu --lazy_loading --iterations 3000 --position_lr_max_steps 3000 --densify_until_iter 1500
LogTemp: Error: -------------------------K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/out/point_cloud/iteration_3000/point_cloud.ply
LogTemp: Error: -------------------------K:/fotozhora/Bratislava/Bratislava Eurovea Tower/09.01.2024/temp/1_12_16_13_31/video/out/point_cloud/iteration_3000/point_cloud.ply

UE4 support

Thank you for the amazing work. I' m currently working on UE4.27 and I want to use this plugin in UE4. Will this plugin support UE4 in the future. Or Is it possible that I transplant it into UE4 ?

The culling cannot be disabled

Describe the bug

Even if I disabled all culling options in the settings panel, the culling cannot be disabled.

image

Screenshot

3D-GS-Culling

Environement:

  • OS: Windows 11
  • Version UE 5.3 + Plugin 1.1.3

"module that is not available on the platform"

I'm trying to export to Android build for VR (Quest 2), and get the prompt that the module is not available on the platform. Is there a way to build for Android (VR) to get around this error?

I'm using UE 5.3

PackagingResults: Warning: [AssetLog] C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\5.3\XV3dGS\Content\M_GS_unlit.uasset: Asset has been saved with empty engine version. The asset will be loaded but may be incompatible.
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Error: [AssetLog] D:\universal\xv\MyProject3\Content\XV3dGS\gs_cathedral\gs_cathedral_buffers.uasset: Failed to cook /Game/XV3dGS/gs_cathedral/gs_cathedral_buffers for platform Android_ASTC. It imports class /Script/GSRuntime.GSRuntimeBuffer, which is in a module that is not available on the platform.
PackagingResults: Warning: [AssetLog] C:\Program Files\Epic Games\UE_5.3\Engine\Plugins\5.3\XV3dGS\Content\BPGaussianActor.uasset: Asset has been saved with empty engine version. The asset will be loaded but may be incompatible.
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Error: [AssetLog] D:\universal\xv\MyProject3\Content\XV3dGS\gs_cathedral\gs_cathedral.uasset: Failed to cook /Game/XV3dGS/gs_cathedral/gs_cathedral for platform Android_ASTC. It imports class /Script/GSRuntime.GaussianComponent, which is in a module that is not available on the platform.
PackagingResults: Warning: CreateVolumeComponent
PackagingResults: Warning: AGaussianActor::InitNiagaraSystem()
PackagingResults: Error: Unknown Cook Failure

Problem packaging for Android - Failed to cook - UE 5.4

When trying to make the packaging of the project in UE 5.4, I get an error when cooking the material that prevents me from continuing the process.

UATHelper: Packaging (Android (ASTC)): LogInit: Display: LogCook: Error: [AssetLog] C:\Users+++\Documents\Unreal Projects\MyProjectTry3\Content\XV3dGS\champiCiudadEncantada\champiCiudadEncantada_buffers.uasset: Failed to cook /Game/XV3dGS/champiCiudadEncantada/champiCiudadEncantada_buffers for platform Android_ASTC. It imports class /Script/GSRuntime.GSRuntimeBuffer, which is in a module that is not available on the platform.
UATHelper: Packaging (Android (ASTC)): LogInit: Display: LogCook: Error: [AssetLog] C:\Users+++\Documents\Unreal Projects\MyProjectTry3\Content\XV3dGS\champiCiudadEncantada\champiCiudadEncantada.uasset: Failed to cook /Game/XV3dGS/champiCiudadEncantada/champiCiudadEncantada for platform Android_ASTC. It imports class /Script/GSRuntime.GaussianComponent, which is in a module that is not available on the platform.

UATHelper: Packaging (Android (ASTC)): LogInit: Display: LogCook: Error: Content is missing from cook. Source package referenced an object in target package but the target package was marked NeverCook or is not cookable for the target platform. UATHelper: Packaging (Android (ASTC)): Source package: /Game/XV3dGS/champiCiudadEncantada/Crop/croped_xv_ns/croped_xv_ns UATHelper: Packaging (Android (ASTC)): Target package: /XV3dGS/M_GS_unlit
UATHelper: Packaging (Android (ASTC)): Referenced object: /XV3dGS/M_GS_unlit.M_GS_unlit

image

To Reproduce

  1. Load a gaussian splatting using the plugin.
  2. Package > Android > ASTC

Expected behavior

Unreal generates an APK of the project

Desktop:

  • OS: Window11
  • Software: Unreal Engine
  • Version: 5.4

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.