- Web-based framework for environment-aware rendering and interaction in augmented reality based on WebXR using three.js
- The framework implements realistic rendering of 3D objects, handles with geometry occlusions, matches the lighting of the environment, casts shadows, and provides physics interaction with real-world objects.
- Target the web environment and have designed our solution to work on a vast range of devices as, monocular camera setups or devices with dedicated higher quality depth sensors.
- Single application while being capable of obtaining over 20 frames per second even on middle-range devices.
- Demonstration video of the features in the framework.
- Based on the experimental WebXR AR draft API available in Chrome 88+
- Download the repository from git
gh repo clone tentone/ar-occlusion - Install Node and NPM.
- Prepare a browser compatible with WebXR in a depth with support for
depth-estimation,hit-testandlighting. - Install dependencies from NPM by running
npm installand start the code runningnpm run start
-
To ensure consistency in the rendering of the virtual scene we implement a physically-based rendering pipeline.
-
Physically correct attributes are associated with each 3D object that, combined with lighting information captured by the device, enables the rendering of AR content that matches the real-world illumination.
-
Depth data provided by WebXR to dynamically place 3D objects around the environment, calculate geometry occlusion and build a model of the environment to enable physics interaction.
-
Calculate geometry occlusion on a pixel basis directly from the projected vertices coordinates to prevent loss in data precision
- To reuse the PBR pipeline of three.js we inject our code into the existing shaders generated by the library using the
onBeforeCompilemethod of the material. - We use the cannon.js physics simulation engine for all physics interactions.
- The environment is mapped using a probabilistic voxel based model that is updated every frame.
- The code from the project is MIT licensed. The license is available on the project repository,
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent