Surprisingly, doing it in the main thread doesn't seem to be a huge disaster, but it would still be much better to decode images in the background.

Many properties on Cesium Actors and ActorComponents don't take effect immediately when edited. Instead, you have to hit Play, restart the editor, or remove/re-add the objects, which is really annoying. The most notable example of this is when modifying properties of raster overlays, but there are almost certainly others.

When recording videos, we'd like to wait for 3D Tiles to completely finish loading before snapping each frame.

From @AlbanBERGERET-Epic:

Synchronisation mechanism for rendering a movie waiting for streaming completed:
You can use the sequencer as you did, but you'll have to produce the video using the new Movie Render Pipeline.
https://docs.unrealengine.com/en-US/Engine/Sequencer/Workflow/RenderAndExport/HighQualityMediaExport/index.html

When using this pipeline, you can define your own setting class by implementing UMoviePipelineSettings. If you do that in your own code it should automatically show up in the dropdown list in the UI.
There's a project setting which lets you specify which classes are added to new jobs by default too, so you can ensure your setting always gets added.
When that setting class is initialized, you can register to the engine begin tick event, and you can flush the streaming every frame before the engine ticks and tries to use the data.

Similar to CesiumGS/cesium-native#112 but for Cesium for Unreal instead of cesium-native.

This might be old news / irrelevant, but I'm adding this info in case it's helpful for any reason. Please feel free to close this issue.

On one of my home machines (lower specs relative to my Cesium laptop), the crash rate for Project Everywhere 0.1 is very high: it crashes within a few seconds a majority of the time, sometimes instantly. Perhaps relatedly, I never see the Texture streaming pool over budget warning message on that machine.

It's possible that the program can't recover from whatever causes that warning, or it might just be another memory problem that happens first.

Specs for the home machine:

GeForce GTX 1070
Intel Core i5-4670K CPU @ 3.40GHz
16GB RAM

Rather than porting the terrain engine from CesiumJS, we should aim to use a single system for both 3D Tiles and globe terrain/imagery rendering.

We'll probably need a separate base material for the blend mode, at least.

In CesiumJS, we pretty much always render everything in its proper place on the globe, and many 3D Tiles datasets and even glTF models have world coordinates baked into their root transformation. The model itself ends up being in a tiny bounding sphere 6-7 million meters away from the origin.

In Unreal Engine, it may be valuable to render even these georeferenced datasets in a local coordinate system. Not always, of course, as proper globes are very valuable for some applications. But many users will only be concerned about a local area, and keeping that local area near the origin will increase the likelihood that other Unreal Engine plugins and app code work correctly. It will also be valuable to keep Z-up so that users can navigate the scene with standard Unreal cameras that aren't aware they're navigating a curved surface.

A side benefit is that, if we implement this earlier rather than later, it lets us implement and maybe even ship a valuable capability (local 3D Tiles rendering in Unreal Engine) prior to fully resolving #6.

Currently only Replacement refinement is supported.

Probably based on a subset of Cesium World Terrain plus Sentinel-2 imagery, transformed to 3D Tiles using https://github.com/CesiumGS/asset-pipeline/tree/a4b1fdd38acca71edec42e86f4dc80f6243b695f/Tools/QuantizedMeshTo3DTilesConverter

Example command:

./build/bin/quantizedmeshto3dtilesconverter -i cwt-subset.terraindb --imagery sentinel2.sqlite -o worldwide-test.3dtiles

CesiumJS parses tileset.json and creates tiles in the main thread, because it doesn't have much choice. In C++ / Unreal Engine, we can relatively easily do this in a separate thread, but currently we're not. This will be necessary if JSON parsing proves to be slower in C++ because it uses the heap heavily and allocations are slower in this non-garbage-collected environment.

In-editor UI to:

• Create a Cesium ion account
• Log in
• Query ion tilesets and add them to your game with a click.
• Upload data to Cesium ion and add it to your game.

Needs to be fleshed out.

An Unreal Engine Plugin for rendering 3D Tiles Batched 3D Models (b3dm) will likely consist of two main parts:

• GltfMeshComponent: A UPrimitiveComponent-derived class representing a single tile / glTF model. It may be helpful to derive from UMeshComponent or UCustomMeshComponent, but even if not, looking at their source code will be helpful. We'll also need a corresponding FPrimitiveSceneProxy-derived class, GltfMeshSceneProxy. Graphics Programming Overview has a good explanation of UPrimitiveComponent and FPrimitiveSceneProxy. In short, the former represents the model on Unreal's game thread, while the latter represents it on Unreal's rendering thread.
• Cesium3DTilesetActor: An Actor class that represents the entire tileset. Its Tick method can be used to select appropriate tiles each frame and create/destroy the primitives above as needed.

With these pieces in place, it should be simple for users to add a 3D Tiles tileset to the world using the Unreal Editor.

We may need to implement most of the code that loads a glTF file and creates Unreal Engine resources from it. While Unreal Engine has a built-in glTF importer plugin, this is meant to be used at design time rather than runtime. It converts the glTF to another form rather than creating a renderable resource directly. For 3D Tiles rendering, we need to be able to load glTFs dynamically. Depending on the license, we may be able to borrow parts of the glTF Importer or the third-party glTF for UE4 plugin, though. The Unreal Engine source code is installed with Unreal Engine and is very easy to explore with Visual Studio.

Rumor has it there is a "glTF runtime translator" coming in Unreal Engine 4.25 that will obviate the need for us to develop our own glTF support.

Useful Resources

• Unreal Engine programming in C++
• Graphics Programming Overview
• Creating Plugins

We need to use the availability information built into layer.json, individual quantized-mesh tiles, and the quantized-mesh metadata extension to learn ahead of time which tiles are available and avoid requesting ones that aren't. Currently, we just try to request everything.

A cleaned up version of this should go into a guide / README / whitepaper / marketing materials - as we unlock game engines for modeling & sim / real-world use cases, the accuracy of the rendering will come into question over and over again - game engines are known not to be accurate for global-scale WGS84, how did Cesium "fix" that?

Slack excerpt from @kring @pjcozzi:

1. Cesium for Unreal does math in double precision
2. Cesium for Unreal uses Unreal's origin rebasing feature to minimize the magnitude of the components in the the translate column in the view matrix so that the single-precision math in the Unreal Engine and on the GPU isn't so bad. Do we have any numbers around this, e.g., 1 cm for 131,071 meter tiles as Deron mentions for RTC, https://help.agi.com/AGIComponentsJava/html/BlogPrecisionsPrecisions.htm

That's right, and the precision should be close to the same as RTC. Either approach brings the translation component of the model-view matrix nearly to zero. The maximum distance of the camera from the view origin is user-configurable and defaults to 100 meters (which is probably overly conservative), so that doesn't change the expected precision meaningfully.

The biggest weakness of this approach is that Unreal's global origin is specified as a signed, 32-bit integer in centimeters, so the overall range is +/- 21 million meters. That's sufficient to get high precision for the whole Earth, but as the camera moves to geostationary orbit or to other parts of the solar system, we'll lose precision fast. I doubt anyone much cares at the moment though.

Also:

UE is single precision everywhere, but we're using the "origin rebasing" feature to keep the world origin close to the camera and the Cesium primitives know how to rebase in double precision

djusting the origin takes several milliseconds. We update it automatically based on the camera's distance from the previous origin.

CC @shehzan10 for your situational awareness.

• TODO: how will this change with Unreal 5 precision updates?

The glTF spec has clear instructions on how to generate model normals, tangents, and bitangents when they are not explicitly specified. We need to actually do what it says:
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#meshes

Tiles should be culled against the view frustum and tiles of an appropriate level of detail for the view should be selected from the 3D Tiles bounding-volume hierarchy.

Currently the imagery detail is limited by the terrain detail. CesiumJS handles this by "upsampling" leaf terrain tiles when further imagery refinement is desired.

If a Tileset Actor's Location is not 0,0,0, tiles will be culled incorrectly. The problem is that they're rendered as if the location is 0,0,0, but they're culled as if they're at the actual specified location.

Thanks to @AMX-MattWalker for reporting this.

Create ready-to-use UI elements that can be added to a game to allow the user to geocode some text, select a result from the list, and fly to that location.

Builds on CesiumGS/cesium-native#70.

UE's gravity points in the -Z direction. It might be possible to change this, but even so it will point in just one direction. But on Earth, the direction of gravity is different depending on where on the globe you're located. Can we simulate this in UE?

Some starting points:

• A custom gravity example: https://www.ue4resources.com/plugins
• A thread about adjusting gravity direction: https://forums.unrealengine.com/development-discussion/c-gameplay-programming/1702500-is-there-still-no-simple-solution-for-adjusting-gravity-direction
• PhysX documentation talks about radial gravity: https://docs.nvidia.com/gameworks/content/gameworkslibrary/physx/guide/Manual/RigidBodyDynamics.html#gravity

Tiles are currently requested as soon as they are needed, and no attempt is made to prioritize more recent / still visible requests. This means that with fast motion on a slower internet connection, useful tile requests can end up waiting behind requests for old tiles that are not even visible anymore.

Raster overlays don't detach properly in specific situations. The steps to minimally reproduce the issue:

• On a new terrain tileset with no current overlays, add two different raster overlays (make note of which overlay was added first).
• There should be a randomly interleaved mess of visible raster tiles, some from the first overlay and some from the second. So far, this is still to be expected since we don't yet claim to enforce a particular render ordering for overlays. We should also be getting "Too many raster overlays" warnings in the console at this point, which is also to be expected.
• Now delete the first overlay which was added. We would expect only the second overlay to show now, but in fact the first one shows within the currently visible region. Interestingly, loading new tiles (flying somewhere else) causes only the remaining (second) overlay to be used for new tiles as wanted. But the tiles that were already loaded before deletion will consistently show the wrong overlay (in fact there won't even be interleaving amongst the already loaded tiles now, only the wrong overlay will show!).
• To go a step further, you could delete the remaining overlay as well. Now we would expect nothing but the default grey material for the untextured terrain mesh, but in fact the initial overlay still sticks around on the tiles that were already loaded before its deletion. The second overlay however, deletes with no problem at this point.

We not only need to cache loaded 3D Tiles in memory, as CesiumJS does, but we likely need a tile cache on disk as well because we're not getting this for free as we do in the web browser.

https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#materialalphamode
Based on a quick look, Unreal Engine does not support toggling between back-face and front-face culling. It does have a "double-sided" property on materials, though, which is currently enabled for all materials. We can switch to front-face culling, when necessary, by flipping the order of the indices.

Specifically pnts, i3dm, and cmpt. We should track the development of the newer glTF-centric 3D Tiles spec, but we also should support these older tile payload types.

Currently the base material used for glTF is pretty simple:

(If you don't see GltfMaterial in the Content Browser in the Unreal Editor, click View Options -> Show Plugin Content.)

This is just a subset of the what a glTF material can do. It also requires the material to have a texture, which is not ok.

So this issue is to flesh out the base material - or create multiple base materials if necessary - in order to support the full set of glTF material options.

We should have a guide for using the Cesium for Unreal plugin.

These instructions may help in the short term but should be updated to follow best practices before the release:

Here are some steps for loading a tileset locally. You'll need to host the tileset with a local server. A sample tileset can be downloaded from here: https://drive.google.com/file/d/12xQr-q9hRZxcGAYJ0ZUnoTvwiYpeSfwX/view?usp=sharing

1. Load the TakeThree project and delete the CesiumOSMBuildings actor.
2. Add a Cesium 3DTileset actor to the scene
3. Set url to a local server path like http://localhost:8002/static/Desktop/agi/tileset.json (your path will be different). Alternatively, if you have a tileset hosted on Ceisum ion, pass in the asset id / access token, e.g. 142781 and eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI5YTRiMjI4OS04NTgzLTQ2YjgtOTNmNi02NTVmNjFkZjk4NzgiLCJpZCI6OSwic2NvcGVzIjpbImFzciIsImdjIl0sImlhdCI6MTU5Nzk2ODQ2N30.clxjNE-5Bc2Y9e7TDt9gN-WpYr6F3BmI-gediu1MTCw.
4. Set the transform location to 0, 0, 0
5. Optionally, set the origin parameters on the Georeference object. For the AGI tileset use Longitude: -75.5967069626 Latitude: 40.0387958754 Height: 70.257829797
6. Press play

Other tips:

• Disable lighting by pressing Shift + F2. Might look better for photogrammetry until #52 is addressed.

Project Anywhere used a completely unlit mode for night time. We should make an effort to add in night time lighting similar to CesiumJS

It should be possible to do CesiumJS-style picking of 3D Tiles. Applications should be able to determine which tileset, which tile, and which feature in the tile were picked. They should also be able to access properties of each feature.

Implement a camera similar to the one in CesiumJS, making it easy to navigate a global scene.

Set the material's shading model to unlit if the KHR_materials_unlit extension is present. The base color texture will need to hook into the emissive texture slot.

It crashes trying to use a nullptr CesiumGeoreference. This is true whether you drag in the tileset actor from the Place Actors panel, or duplicate an existing one by right clicking and choosing Edit -> Duplicate.

We're currently using the simplest possible 3D Tiles refinement algorithm, where we won't refine a tile until all of its children (visible or not) are loaded and able to be rendered. This prevents any LOD-skipping, and requires loading a lot more tiles than are actually rendered, which is inefficient. The main advantage of this algorithm (other than simplicity) is that we'll never find ourselves with holes in the mesh.

We should implement the selection algorithm from CesiumJS's terrain engine. It allows skipping any number of levels (and therefore much less loading) and will never allow scene geometry to disappear with camera movement (which looks terrible). But if the camera moves such that not-yet-loaded sections of model are now visible, those parts of the model will be missing entirely. The CesiumJS terrain engine renders "fill tiles" in this scenario, which will be very challenging for the more general case of 3D Tiles. For many 3D Tiles tilesets, though, it will be acceptable to simply render nothing until the new tiles are loaded.

Configuration options should allow the user to select between a guaranteed-complete mesh (current algorithm) or faster loading (this algorithm).

So that we can have a "base globe" tileset and then overlay smaller, more detailed areas from another tileset. For this issue, we only need something good enough for the primary use-case of "show a globe, let the user zoom in to a more detailed area" and we should aim to keep it simple. But if it's not much more time consuming to implement the latest approach in CesiumJS, we should do that instead.

cesium-native already works fine, but Cesium for Unreal (the plugin and the demo game) will take some more work.

We would like to get as close as possible to an apples-to-apples comparison on the amount of streaming data used for a similar application/camera flight in CesiumJS and Cesium for Unreal.

This is to validate whether one engine is streaming more data over the other. For example, does CesiumJS more mature algorithm (skip LOD etc) provide a benefit in the amount of data streamed? Does the camera/viewport in Cesium for Unreal request more data?

We want to assess all data streaming - 3D Tiles, Cesium World Terrain, and Bing Maps Imagery to start.

This will allow us to add information in white papers/communication materials that mentions how/why data streaming may or may not be different and how the roadmap may influence that.

@kring Let me know if you have any questions about the comparison.

Many data providers require that attribution be shown (or at least made available on a popup or something) when their data is shown. We need a system for surfacing this required attribution for tilesets and raster overlays, and we need to make it easy to display in UE.

In CesiumJS, terrain is rendered with a default screen-space error of 2.0, while 3D Tiles are rendered with a default SSE of 16.0. Using the 3D Tiles SSE for terrain would yield a hopelessly blurry mess, while using the terrain SSE for 3D Tiles would risk running out of memory and crashing. Currently there's a hack to divide the user-specified SSE by 8.0 when the "tileset" is a layer.json/quantized-mesh, but this is hacky. This is mostly an API/UX question.

In addition to the vertex precision challenges we're used to in CesiumJS, we have the additional one in Unreal that matrix and vector computations are done in single-precision even on the CPU.

Here are some problems we'll have as a result, off the top of my head (there may be more):

Imprecision of node transforms

If glTF nodes and 3D Tiles nodes are represented as attached UActorComponents, and the transforms are just local transforms relative to their parent (which is the obvious way to model it), then we'll lose significant precision when Unreal Engine multiplies these transforms together in single-precision.

Possible solution: give each actor component a flattened world-relative transform which we compute ourselves in double precision using glm or similar.

Relative-to-Center (RTC) jitters

The RTC rendering in CesiumJS works because we compute model-view matrices in double precision on the CPU. If the camera and a model are both far from the origin, both will have terms in their matrices with a large magnitude. But when the camera is near that model, the large magnitude terms effectively cancel out and we get a nice model-view matrix with no large-magnitude terms.

This canceling-out is reliable (avoids jittering) because we compute the model-view matrix in double precision. If Unreal Engine computes model-view in single-precision, the "canceling out" will be noisy and the model will appear to jitter with respect to the camera.

Position solution: The best workaround I'm aware of for this is Unreal's UWorld::SetNewWorldOrigin() method. It can be used to set the world origin (0,0,0) to a new location, expressed as signed 32-bit integer coordinates in centimeters. If we put the camera near the origin and then adjust (using double-precision) the models to be relative to that new origin, we should be able to avoid jittering.

Because the world origin is specified using 32-bit signed integers in centimeters, this approach gives us a maximum world radius of about 21 million meters. That's plenty for the Earth itself, but not enough for the geostationary belt in orbit, nevermind the Moon, sun, and planets. It's not clear to me what we can do about this, but we can probably just live with it in the short term given the expected use-cases for Cesium Unreal.

When a new world origin is set, UActorComponent::AddWorldOffset() is called on every component with the offset from the old origin expressed as floats. Ideally we'd be able to get our hands on the original integer coordinates and then compute the new single-precision local transform from the double-precision truth.

Some useful info on SetNewWorldOrigin is here: https://forums.unrealengine.com/development-discussion/content-creation/1668643-origin-rebasing-questions

We need to figure out what this means. Possibly:

• Use high-DPI versions of Bing Maps tiles with labels, so the text isn't so tiny.
• Double the screen-space error so that we don't render so many tiles. This is roughly consistent with what CesiumJS does by default. Maybe it should be optional?

Currently, once a tile is loaded, it is kept in memory forever.

We need a way to place non-Cesium Actors on a globe, which is weird because it has huge coordinate values and no consistent "up" direction.

Hopefully this can be done via an ActorComponent. The component would:

• Allow the Actor's position to be specified using longitude/latitude/height, double-precision Cartesian coordinates, or just using normal UE world coordinates if desired.
• Orient the Actor so that its local up is aligned with the WGS84 ellipsoid's local up at that position. And then the normal UE orientation would apply from there.
• Optionally adjust the object's height relative to the terrain at that location (see CesiumGS/cesium-native#72).

Currently, loaded raster images stick around pretty much forever. 😬

3D Tiles should be usable for collisions. For example, it should be possible to prevent pawns from passing through walls.

Related to #9, it should also be possible to determine which particular feature was collided with, and what properties it has.

Exiting or stopping play while tiles are in the progress of loading currently causes a crash. We need to defer destruction of the tiles until the load completes or is successfully canceled.

When I open cesium-unreal-demo in the editor and close it afterward, the editor displays crash reports after that.

The unreal version is 4.26.
Only OSM buildings and CWT terrain which are included in the demo are loaded

Crash report from unreal:

Fatal error: [File:D:/Build/++UE4/Sync/Engine/Source/Runtime/Engine/Private/Materials/MaterialShared.cpp] [Line: 2586] Cannot queue the Expression Cache when it is about to be deleted

UE4Editor_Engine
UE4Editor_Engine
UE4Editor_Engine
UE4Editor_Core
UE4Editor_Core
UE4Editor_RenderCore
UE4Editor_RenderCore
UE4Editor_Core
UE4Editor_Core
kernel32
ntdll

Something changed recently such that the Melbourne tileset doesn't immediately appear in the Melbourne level. Changing the CesiumGeoreference's OriginPlacement to "True Origin" and back to "Bounding volume center" fixes it, so it probably has something to do with an initially-incorrect transformation and I probably introduced it in #107.