rerun-io / rerun Goto Github PK

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

License: Apache License 2.0

Shell 0.14% Rust 79.29% Python 11.14% HTML 0.37% JavaScript 0.47% Dockerfile 0.06% CSS 1.01% Just 0.11% WGSL 1.37% C 0.31% CMake 0.39% C++ 5.35%

computer-vision cpp multimodal python robotics rust visualization

rerun's Introduction

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

import rerun as rr  # pip install rerun-sdk

rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42))

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors))
…

Getting started

C++: Guide
Python: pip install rerun-sdk or on conda
Rust: cargo add rerun

Rerun Viewer binary

Both the Python and Rust library can start the Rerun Viewer, but to stream log data over the network or load our .rrd data files you also need the rerun binary.

It can be installed with pip install rerun-sdk or with cargo install rerun-cli.

You should now be able to run rerun --help in any terminal.

Documentation

Status

We are in active development. There are many features we want to add, and the API is still evolving. Expect breaking changes!

Some shortcomings:

Multi-million point clouds are slow.
The data you want to visualize must fit in RAM.
- See https://www.rerun.io/docs/howto/limit-ram for how to bound memory use.
- We plan on having a disk-based data store some time in the future.

Business model

Rerun uses an open-core model. Everything in this repository will stay open source and free (both as in beer and as in freedom). In the future, Rerun will offer a commercial product that builds on top of the core free project.

The Rerun open source project targets the needs of individual developers. The commercial product targets the needs specific to teams that build and run computer vision and robotics products.

How to Cite Rerun

When using Rerun in your research, please cite it to acknowledge its contribution to your work. This can be done by including a reference to Rerun in the software or methods section of your paper.

Suggested citation format:

@software{RerunSDK,
  title = {Rerun: A Visualization SDK for Multimodal Data},
  author = {{Rerun Development Team}},
  url = {https://www.rerun.io},
  version = {insert version number},
  date = {insert date of usage},
  year = {2024},
  publisher = {{Rerun Technologies AB}},
  address = {Online},
  note = {Available from https://www.rerun.io/ and https://github.com/rerun-io/rerun}
}

Please replace "insert version number" with the version of Rerun you used and "insert date of usage" with the date(s) you used the tool in your research. This citation format helps ensure that Rerun's development team receives appropriate credit for their work and facilitates the tool's discovery by other researchers.

Development

ARCHITECTURE.md
BUILD.md
rerun_py/README.md - build instructions for Python SDK
CODE_OF_CONDUCT.md
CODE_STYLE.md
CONTRIBUTING.md
RELEASES.md

Installing a pre-release Python SDK

Download the correct .whl from GitHub Releases
Run pip install rerun_sdk<…>.whl (replace <…> with the actual filename)
Test it: rerun --version

rerun's People

Contributors

Stargazers

Watchers

Forkers

tenx-s therustmonk zmilan yiannistheocharakis eastwooder xieren58 prajapatisarvesh muharremokutan jithinraj devdoshi krshrimali quanpinjie gmh5225 stjordanis spread0x meehawk dbuch alio24 mxrowneropun patoessy whitez7y specialbs jaedukseo linecode eltociear placrosse appdirectory taniamolina3737 techventurebuilder psy-repos-rust marcelapaola esquith rstkit valenrukra abhishek-peri dut3062796s denistome art-e-fact salvationz philglobal jempabroni mpizenberg barral-robotica-e-automacao mariocorsanici vikhegde vilaksh01 jleibs jleibs-test weblfe joe2hpimn clfnd pablovela5620 imstevenpmwork zrezke laujay sjouks pupubear007 vsuryamurthy ktfth winterchaufr dan255 hurricankai benjamindev codingonion rasmusgo nhemsley h3mosphere woonhock parthjadhav nanderoo funshoelias urholaukkarinen ayshiff marsango16 temelgunaydin qrasmont romatthe aiprojectsources grasslandnetwork pudnax tauseefmohammed2 luxonis akugresimon karinamorel joyeldumbo ashsimmonds alsuren kiconiaworks santitfg rodrygojose vltown supsource shouvikghosh2048 kpreid pvadlak23 pauljw28 hardik-satasiya kayjayi tbszlg lidi100

rerun's Issues

`re_renderer` support for rounded/pretty (2D) rectangles

egui 2d rectangles are rounded right now
should have similar visual quality when going through re_renderer

solve either by adding features to the line renderer (good long term, more effort shortterm) or by making it easy to generate line primitives for rectangles

Ingest new data forever without running out of memory (memory pruning)

Closed by #363

Unify 2D and 3D renderer

Currently the 2D view is using egui shapes to do all painting. This makes it difficult to do things like apply colormaps to depth images. We should apply such colormaps in a shader. We also want to be able to display images in 3D views, something we cannot currently do. The solution is to unify the 2D and 3D views.

This issue is about replacing epaint with re_renderer in the 2D SpaceView (except for text, which we will use epaint for).

2D data is just 3D data with Z=0. This means the visualizer view can just put all data into the same Scene3D and paint it using WGPU. If all data is flat, then the default camera is a 2D camera, otherwise the default is a 3D orbit camera.

#380
demo 2D view in standalone demo
make it easy to convert 2D versions of primitives into 3D primitives that re_renderer already knows
Add support for the 2D types to re_renderer
- #388
- #389
Figure out layering/Z-ordering/painters-algorithm for 2D stuff in re_renderer
#382

This is in preparation of #373

How to treat spaces

Spaces is how we know where to place objects (world space vs camera space, etc).

Currently spaces are treated like any other attribute (e.g. like color). They can be assigned to an object (e.g. a point) at any time, an can even change over time (just like an object can change color, it can move to a different space).

Spaces are also objects themselves, which means they can have attributes. Currently the only attribute is the up (the preferred 3D up axis).

This designs offer a lot of flexibility (probably too much), ant it has a couple of issues:

A) speed. since each point in a point cloud is allowed to have a different space it means the viewer need to spend a lot of CPU cycles partitioning all the data in different space buckets at runtime. It can't really be done ahead of time, because a point can move to a different space, or be assigned a space long after it was first logged.

B) space attributes (like the up axis) aren’t really like other attributes. They don’t really belong on a timeline. In fact, setting the up-axis is something you would do once, but currently you have to take pains into making sure it is logged with all the time sources, and as the first thing that happens in each timeline.

So, we need a better design.

One solution is to assign spaces per type path. Just like you need to declare the type of an object beforehand (per type path), you also need to assign it a space before logging to it. Spaces would no longer be objects themselves, but treated as a special case (i.e. the up axis would be stored outside any time source).

For instance: Before logging a rectangle to /car/bbox you need to set the space for the type path /car/bbox. Before logging a point to /points/42 you need to set the space for the type path /points/_.

That all type paths have the same space has a drawback: consider an array of many cameras. For each camera image you run some detection algorithm to detect a human. Now you want to log that detection. To what path do you log it?

for cam in camera:
    human = detect(cam.rgb);
    if human:
        # This looks reasonable, but it would dynamically assign different spaces to the same type path,
        # so it would cause a runtime error. But such an easy mistake to make!
        # rerun.log('/human', human.bbox, space=cam.id)

        # This would work:
        rerun.log(f'/camera/{cam.id}/human', human.bbox, space=cam.id)

View the same space multiple times, with different object selections

User story: log RGB and Depth images to the same space together with bounding boxes. Then in the viewer show two views, one with RGB+bounding boxes, one with Depth+bounding boxes.

We need some UI for adding new SpaceViews. One simple way is to add a clone-button to each SpaceView. Another is to add a Add space view… button in the blueprint panel, followed by a modal popup for picking which space to use.

We can also think about automatically creating multiple SpaceViews for the same Space in some situations, e.g. when there are multiple images in the same 2D space, to create on SpaceView for each image, where the other images are invisible, and all other objects (rects etc) are visible.

Thoughts on styling and extra data

One of the last core pieces that's currently without a clear design is how to handle styling and other view-options.

A couple of things that are becoming clear:

we want to be able to set styling options on the blueprint
it probably makes sense with some CSS like inheritance
it's very convenient to also be able add styling to the data at the point in the code where it's being logged
it's often difficult to draw a line between what is data and what is options or styling
- For example both color and label maps (AnnotationContext) show up all the time and are difficult to definitively designate as data or styling.

I propose that we should think about it as follows:

You are allowed to log anything you want to a data path. It doesn't matter if the right way of seeing color is as styling or data. If the user wants to log it to a data path, that's up to them.
You can also set color and other data/styling/options on the blueprint.
- The difference is that for the blueprint, we only allow timeless data
Each Space View Blueprint describes how to render a SpaceView .
- The Space View Blueprint in turn contains one or more Data Blueprints
- Each Data Blueprint is responsible for describing how to fill each of the fields needed to render a renderable element (need a better name here) in the view. A renderable element could for instance be a line strip or set of points.
- Each renderable element has a number of fields that determines how it's rendered. For a point, that could be position, radius, color and size mode (size is constant in screen space or world space).
  - Some of these fields might have defaults.
- The Data Blueprint can fill each of the fields by either querying one or more data paths, using the default field value(s), or using a value from the Blueprint.
  - Blueprint values inherit from higher up in the blueprint hierarchy.
  - We also support queries on data paths that resolve inheritance in a similar way on the data path hierarchy. (I'm not sure yet if we also want to support queries that resolve inheritance differently (for instance skipping it).
Our heuristics generate Data Blueprints that by default:
- Creates simple queries (data path + timestamp) for the main "data"
- For more ambiguous fields that represent data/styling/options:
  - First look for values in the blueprint
  - Then look for matching names in the data path
  - Last use default values / default heuristics

replace epaint with `re_renderer` in the 2D view.

Need better 3D anchor point

When steering the camera in a 3d view we show a point for the origin we're moving/rotating around. The point disappears after a period of inactivity.

This can be confusing when there are other points are in the scene.

We should show something more helpful that is harder to confuse with objects in the scene - probably three thin orthogonal crossing lines or something similar. Consider rendering it on top without depth (support would need to be added to re_renderer)

Viewer crashes when running from Python on Windows

Windows 10 Home 21H2
Python 3.11.0
rustc 1.64.0
rerun 36e28fe811507422826c1907b5eef53300d5dde6

Viewer runs with cargo run -p rerun --release but crashes with the following error when run with rerun.show() (or similar with python -m rerun_sdk):

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: OsError("GetClassInfoExW function failed: Class does not exist. (os error 1411)")', C:\Users\morot\.cargo\git\checkouts\egui-5e4507fa4153be06\3d36a20\crates\eframe\src\native\run.rs:344:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "C:\Users\morot\source\repos\rerun\examples\car\main.py", line 326, in <module>
    main()
  File "C:\Users\morot\source\repos\rerun\examples\car\main.py", line 322, in main
    rerun.show()
  File "\\?\C:\Users\morot\source\repos\rerun\rerun_py\rerun_sdk\__init__.py", line 114, in show
    rerun_rs.show()
pyo3_runtime.PanicException: called `Result::unwrap()` on an `Err` value: OsError("GetClassInfoExW function failed: Class does not exist. (os error 1411)")

Reported by @rasmusgo

`re_renderer` support for 2D images

SDK installation failed because of missing dependency

When running the SDK installation I got the error below. Running pip3 install maturin manually fixed it.

gd@gd:~/dev/rerun$ pip3 install "crates/re_sdk_python[tests,examples]"
Defaulting to user installation because normal site-packages is not writeable
Processing ./crates/re_sdk_python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [20 lines of output]
      Checking for Rust toolchain....
      Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-1hv3nvzr --interpreter /usr/bin/python3`
      Traceback (most recent call last):
        File "/home/gd/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/gd/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/gd/.local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-uuhdcwu8/overlay/local/lib/python3.10/dist-packages/maturin/__init__.py", line 174, in prepare_metadata_for_build_wheel
          output = subprocess.check_output(command)
        File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
        File "/usr/lib/python3.10/subprocess.py", line 501, in run
          with Popen(*popenargs, **kwargs) as process:
        File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'maturin'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Realistic benchmark

Given some rerun data (a .rrd file) we could have a benchmark that measures:

Store ingestion time
Scene construction time (i.e. store query time)
Full application ui time

This would be helpful as regression testing, and could be run manually on large changes, and/or on CI.

Data generation requires some though though. It is not trivial to run our python nyud or objectron code from within cargo bench.

Query caching for "sticky" scenes (plots, text logs...)

Currently, querying a sticky scene is O(n), and as time goes on it gets more and more laggy.
This is already an issue for plots and text logs today.

Ideally, one would want to cache the results of a query up to now(), and then only query the difference the next frame, and so on and so on.
Problem is: what about out of order data?

UI per data blueprint: show latest + X seconds/frames of history

This will allow us to remove the time range selection for the time panel (leaving just the loop selection). This simplifies the UI and the code by quite a lot.

Set up documentation generation system

We need to set up a system to generate documentation. This includes both high level usage and concepts documentation as well as the api-reference.

Background

For example:

textual uses mkdocs for creating a docs from markdown (for high level) and inserts auto generates the api docs into that same site so it gives a consistent look and feel.
Streamlit has a similar approach to textual but builds their high level docs as a next.js website
Dimforge projects seems to use mkforge for high level but then keeps all api docs on docs.rs
polars Seems to build their high level docs using mdBook combined with jupyter notebooks & nbconvert, and api docs for both Python (generated by Sphinx) and Rust hosted using github pages.

Approach

High level docs written in markdown in our re-purposed "rerun-design-guidelines" site (renamed to "documentation"): here
Rust API api docs published to crates.io
Python API api docs published to either ReadTheDocs or self-hosted on Netlify and linked to form our high level docs site

TODO

We need to:

SDK failed installation on a clean system (fixed)

On a fairly clean 22.04 install I had the following issue:

(venv) bpinaya@msi:~/Documents/gitrepos/rerun$ pip install ./rerun_py                                                                                                                                                                                         
Processing ./rerun_py                                                                                                                                                                                                                                         
  Installing build dependencies ... done                                                                                                                                                                                                                      
  Getting requirements to build wheel ... done                                                                                                                                                                                                                
  Preparing metadata (pyproject.toml) ... done                                                                                                                                                                                                                
Collecting numpy==1.23                                                                                                                                                                                                                                        
  Using cached numpy-1.23.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.1 MB)                                                                                                                                                                
Building wheels for collected packages: rerun_sdk                                                                                                                                                                                                             
  Building wheel for rerun_sdk (pyproject.toml) ... error                                                                                                                                                                                                     
  error: subprocess-exited-with-error                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                              
  × Building wheel for rerun_sdk (pyproject.toml) did not run successfully.                                                                                                                                                                                   
  │ exit code: 1                                                                                                                                                                                                                                              
  ╰─> [103 lines of output]                                                                                                                                                                                                                                   
      Running `maturin pep517 build-wheel -i /home/bpinaya/Documents/gitrepos/rerun/venv/bin/python3 --compatibility off`                                                                                                                                     
      📡 Using build options locked from pyproject.toml                                                                                                                                                                                                       
      warning: /home/bpinaya/Documents/gitrepos/rerun/Cargo.toml: unused manifest key: patch.crates-io.wgpu-core.ref                                                                                                                                          
      warning: /home/bpinaya/Documents/gitrepos/rerun/Cargo.toml: unused manifest key: patch.crates-io.wgpu.ref                                                                                                                                               
         Compiling ahash v0.8.1                                                                                                                                                                                                                               
         Compiling zstd-safe v5.0.2+zstd.1.5.2                                                                                                                                                                                                                
         Compiling zstd-sys v2.0.1+zstd.1.5.2                                                                                                                                                                                                                 
         Compiling bincode v1.3.3                                                                                                                                                                                                                             
         Compiling wasm-bindgen-shared v0.2.83                                                                                                                                                                                                                
         Compiling owned_ttf_parser v0.15.2                                                                                                                                                                                                                   
         Compiling servo-fontconfig-sys v5.1.0                                                                                                                                                                                                                
         Compiling glib-sys v0.15.10                                                                                                                                                                                                                          
         Compiling pyo3-build-config v0.17.1                                                                                                                                                                                                                  
         Compiling unicode-normalization v0.1.21                                                                                                                                                                                                              
      The following warnings were emitted during compilation:                                                                                                                                                                                                 
                                                                                                                                                                                                                                                              
      warning: `"pkg-config" "--libs" "--cflags" "glib-2.0" "glib-2.0 >= 2.48"` did not exit successfully: exit status: 1                                                                                                                                     
                                                                                                                                                                                                                                                              
      error: failed to run custom build command for `glib-sys v0.15.10`                                                                                                                                                                                       
                                                                                                                                                                                                                                                              
      Caused by:                                                                                                                                                                                                                                              
        process didn't exit successfully: `/home/bpinaya/Documents/gitrepos/rerun/target/release/build/glib-sys-318aed40c122c20e/build-script-build` (exit status: 1)                                                                                         
        --- stdout                                                                                                                                                                                                                                            
        cargo:rerun-if-env-changed=GLIB_2.0_NO_PKG_CONFIG                                                                                                                                                                                                     
        cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu                                                                                                                                                                                        
        cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu                                                                                                                                                                                        
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG                                                                                                                                                                                                            
        cargo:rerun-if-env-changed=PKG_CONFIG                                                                                                                                                                                                                 
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu                                                                                                                                                                                   
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu                                                                                                                                                                                   
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH                                                                                                                                                                                                       
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH                                                                                                                                                                                                            
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu                                                                                                                                                                                 
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu                                                                                                                                                                                 
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR                                                                                                                                                                                                     
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR                                                                                                                                                                                                          
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu                                                                                                                                                                            
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu                                                                                                                                                                            
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR                                                                                                                                                                                                
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR                                                                                                                                                                                                     
        cargo:warning=`"pkg-config" "--libs" "--cflags" "glib-2.0" "glib-2.0 >= 2.48"` did not exit successfully: exit status: 1                                                                                                                              
        error: could not find system library 'glib-2.0' required by the 'glib-sys' crate
                                                                                                                                                                                                                                                              
        --- stderr                                                                                                                                                                                                                                            
        Package glib-2.0 was not found in the pkg-config search path.                                                                                                                                                                                         
        Perhaps you should add the directory containing `glib-2.0.pc'                                                                                                                                                                                         
        to the PKG_CONFIG_PATH environment variable                                                                                                                                                                                                           
        No package 'glib-2.0' found                                                                                                                                                                                                                           
        Package glib-2.0 was not found in the pkg-config search path.                                                                                                                                                                                         
        Perhaps you should add the directory containing `glib-2.0.pc'                                                                                                                                                                                         
        to the PKG_CONFIG_PATH environment variable                                                                                                                                                                                                           
        No package 'glib-2.0' found
      
      warning: build failed, waiting for other jobs to finish...
      error: failed to run custom build command for `servo-fontconfig-sys v5.1.0`
      
      Caused by:
        process didn't exit successfully: `/home/bpinaya/Documents/gitrepos/rerun/target/release/build/servo-fontconfig-sys-f0a074f7e904d57f/build-script-build` (exit status: 101)
        --- stdout
        cargo:rerun-if-env-changed=FONTCONFIG_NO_PKG_CONFIG
        cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
        cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG
        cargo:rerun-if-env-changed=PKG_CONFIG
        cargo:rerun-if-env-changed=FONTCONFIG_STATIC
        cargo:rerun-if-env-changed=FONTCONFIG_DYNAMIC
        cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
        cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
        cargo:rerun-if-env-changed=PKG_CONFIG_PATH
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
        cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
        cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
        cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
      
        --- stderr
        thread 'main' panicked at '`"pkg-config" "--libs" "--cflags" "fontconfig" "fontconfig >= 2.11.1"` did not exit successfully: exit status: 1
        error: could not find system library 'fontconfig' required by the 'servo-fontconfig-sys' crate
      
        --- stderr
        Package fontconfig was not found in the pkg-config search path.
        Perhaps you should add the directory containing `fontconfig.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'fontconfig' found
        Package fontconfig was not found in the pkg-config search path.
        Perhaps you should add the directory containing `fontconfig.pc'
        to the PKG_CONFIG_PATH environment variable
        No package 'fontconfig' found
        ', /home/bpinaya/.cargo/registry/src/github.com-1ecc6299db9ec823/servo-fontconfig-sys-5.1.0/build.rs:34:17
        note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit status: 101": `"cargo" "rustc" "--release" "--message-format" "json" "--locked" "--lib"`
      🍹 Building a mixed python/rust project
      🔗 Found pyo3 bindings
      🐍 Found CPython 3.9 at /home/bpinaya/Documents/gitrepos/rerun/venv/bin/python3
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/home/bpinaya/Documents/gitrepos/rerun/venv/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]
   
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for rerun_sdk
Failed to build rerun_sdk
ERROR: Could not build wheels for rerun_sdk, which is required to install pyproject.toml-based projects
WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available.
You should consider upgrading via the '/home/bpinaya/Documents/gitrepos/rerun/venv/bin/python3 -m pip install --upgrade pip' command.

With the following versions:

lsb_realease -a

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

python --version : Python 3.9.13
pip --version: pip 22.2.2
rustc --version: rustc 1.65.0 (897e37553 2022-11-02)

Solved by googling around and installing the following dependencies:

sudo apt install libfontconfig1-dev libatk-bridge2.0 libgtk-3-dev libglib2.0-dev libfreetype6-dev libatk-bridge2.0

You can close this issue, it's just here if someone has the same problem.

Add support for the 2D types to `re_renderer` (images and rounded rectangles)

cargo run --all-features crashes without showing the UI

python -m rerun_sdk runs well, but cargo run --all-features (as well as cargo install-ed executable) crashes without showing the UI at all? Maybe worth noting that python -m rerun_sdk prints this error too, but continues working:
ERROR egui_winit::clipboard: Cannot initialize smithay clipboard without a display handle!
I'm on ArchLinux, Gnome, Wayland, AMD GPU with opensource drivers.
log.txt

 Output with `RUST_BACKTRACE=1` is:
warning: /home/azymohliad/Downloads/FOSS/rerun/Cargo.toml: unused manifest key: patch.crates-io.wgpu-core.ref
warning: /home/azymohliad/Downloads/FOSS/rerun/Cargo.toml: unused manifest key: patch.crates-io.wgpu.ref
    Finished release [optimized] target(s) in 0.18s
     Running `target/release/rerun`
2022-11-04T08:28:16.053707Z  INFO rerun: Hosting a SDK server over TCP at 127.0.0.1:9876
2022-11-04T08:28:16.100883Z ERROR egui_winit::clipboard: Cannot initialize smithay clipboard without a display handle!
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', crates/re_viewer/src/lib.rs:104:28
stack backtrace:
   0: rust_begin_unwind
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:142:14
   2: core::panicking::panic
             at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panicking.rs:48:5
   3: re_viewer::customize_eframe
   4: core::ops::function::FnOnce::call_once{{vtable.shim}}
   5: <eframe::native::run::glow_integration::GlowWinitApp as eframe::native::run::WinitApp>::on_event
   6: eframe::native::run::run_and_return::{{closure}}
   7: winit::platform_impl::platform::wayland::event_loop::EventLoop<T>::run_return
   8: eframe::native::run::run_and_return
   9: std::thread::local::LocalKey<T>::with
  10: eframe::native::run::glow_integration::run_glow
  11: eframe::run_native
  12: re_viewer::native::run_native_app
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  14: std::thread::local::LocalKey<T>::with
  15: tokio::park::thread::CachedParkThread::block_on
  16: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  17: tokio::runtime::Runtime::block_on
  18: rerun::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
2022-11-04T08:28:16.107499Z  WARN egui_glow::painter: You forgot to call destroy() on the egui glow painter. Resources will leak!

Start Rerun viewer in non-buffered mode and keep it running from Python

The Python SDK should support starting Rerun Viewer in a subprocess, so that you don't need to start (and install) the rerun binary separately.

This covers most peoples use case of Rerun:

rerun.start() # rerun viewer pops up without blocking the python process

rerun.log(…) # possibly from multiple subprocesses

Pre-built Python wheels for our users

As a path towards getting ready to publish on pypi, we should set up a build process that creates installable wheels for all major operating systems / processors etc.

As part of this we should also set up a bi-weekly release schedule with semantic versioning.

Sub-tasks:

Decide on / reserve PyPi package name
Github jobs for building wheels across all release targets
Document versioning semantics
Document steps involved in producing a release from tag -> test -> publish
Make release-based changelogs

Reset space views makes strange choices on the deep_sdf example

It does this:

But would make more sense like this:

Reported by @rasmusgo

Automatically project 3D -> 2D when transforms are present

It should be possible to automatically project 3D data to 2D when the correct transforms are set up (projection in particular). For instance, in the Objectron demo, both tracking points and 3D bounding box should be automatically projected to the image space.

Use Turbo color map for 2D depth images

Depth images currently get shown as gray scale in the 2D view. This makes it hard to see what's going on. We should use a better default colormap like Turbo.

Automatically project 3D -> 3D from child to parent space

At a minimum this should roll up Rigid-3D-transformed child-spaces into the parent space.

More advanced Parent -> Child or Child -> Parent -> Child transform chains can likely be deferred until later.

View configuration & UI for whether to include children
If configuration set, during query: recursive walk through child-spaces building up transform-chain
Code to apply 3D transforms to primitives. Transform-on-CPU vs Send transform to renderer?

It's too hard to select points in the presence of lines

hard_to_select.mov

It feels unreasonably difficult to select points in the presence of lines. See video.

From SyncLinear.com

Tracking issue: Feature parity Arrow-based SDK logging

Expand the Python and Rust SDK:s to cover more data types.

Some things to figure out:

How should we store large components like Tensors and Meshes?

Structured data in Arrow
Blobs in Arrow
In a separate store which the Arrow table references with MeshId/TensorID

#517
#518
the rest of the data types

Tracking Issue: Arrow/Polars based data engine

Primary goal: end-to-end point cloud logging to 3D scene, with a functional and fast query engine that has the desired interface.

Sub-issues:

Things we can work on in parallel now:

Arrow data store
Python SDK
Rust SDK

Things we can work on once the Arrow data store is done

Data UI
Table UI
Scene population

Arrow data store

For the arrow data store (and query engine), follow https://www.notion.so/rerunio/Arrow-Table-Design-cd77528c77ae4aa4a8c566e2ec29f84f. We can probably work in parallel on the main table and on the batching.

Handle ingestion backpressure

What should we do if the logging SDK produces data faster than the Rerun Viewer can ingest it?

Currently the Rerun Viewer will use up more and more memory, and eventually crash. Before crashing, latency is building up too, which sucks.

There are two potential solution:

Drop messages (in the Viewer, but maybe also in the SDK)
Back-pressure to slow down the SDK

Biped says that when demoing, they would be fine with data being dropped.

In either case we should detect that a backlog is building up and warn the user.

No way to see text log body that doesn't fit on screen

The text log view is scrollable but I still can't see the body text when I scroll left:

Skarminspelning.2022-11-04.kl.11.03.02.mov

I would also expect to be able to click on the body and see the whole log line in the selection view.

From SyncLinear.com

Better looking Rerun GUI: lives up to the design

Do another pass of the GUI to get it closer to the design in the Figma file.

Bar chart plotting of 1D Tensor

This should be the fallback visualization for 1D tensors logged by log_tensor. Can for example be used to show:

feature vectors
user computed histogram

most python api should be in plural

i.e. log_point should go away and be taken over by log_points (still the same api)

log_path should take 2d points as well

Arrow entity UI components

Being able to query "What does this entity contain?" and then show that in the UI. We can do this for the current data store, and we want to do this for the future data store. We can work on this as soon as we have implemented queries on the data store.

This is closely related to #404, but not exactly the same.

Python SDK overhaul

Overhaul the Python SDK in the direction of @jleibs suggestion:

# Duck-typable interfaces
class ComponentBatch:
    def as_arrow() -> ArrowArray
        ...

class EntityBatch:
    def as_components() -> Iterable[Component]
        ...

# Loggable Primitives

class Points(EntityBatch):
    def __init__(*args, **kwargs):
        ...

class Rects(EntityBatch):
    def __init__(*args, **kwargs):
        ...

# Typed logging helpers are trivial: top-level entry-point for new users
# Self-documenting usage of advanced API
def log_rects(path, coords, colors, labels, ids=rerun.AutoId):
    rects = Rects(coords, colors, labels, ids)
    log(path, rects)


def log_rect(path, coord, color, label, ids=rerun.Singular):
    rect = Rects([coord], [color], [label], ids)
    log(path,  rect)


# Magic Inference Logic
def auto_type(*args, **kwargs) -> Iterable[EntityBatch]
    ...

def log_auto(path, *args, **kwargs):
    auto = auto_type(*args, **kwargs)
    log(path, auto)

# Advanced API. Supports user-defined Entities and Components in the
# same as we would support user-defined types
def log(path, loggable: Union[EntityBatch, ComponentBatch]):

    # Just do the right thing based on what the type supports
    if hasattr(loggable, 'as_components'):
        arrow_batch = build_arrow_batch(loggable.as_components())
        rerun_sdk.log_arrow_batch(path, arrow_batch)
    elif hasattr(loggable, 'as_arrow'):
        arrow_batch = build_arrow_batch([loggable.as_arrow()])
        rerun_sdk.log_arrow_batch(path, arrow_batch)

# Low-level API. Build your own arrow-batch according to the spec
def log_direct(path, arrow_batch: ArrowBatch):
    rerun_sdk.log_arrow_batch(path, arrow_batch)

A note from @teh-cmc:

Clear SDK layers & boundaries
Language agnostic FFI (i.e. anything with an Arrow SDK can send data)

Re-center the view extents of the timeline when memory gets freed

Whenever a chunk of memory get's freed, we need to re-center the timeline view because it very quickly get's so far away that you can never find it again.

Configure image scaling method used by the viewer

Some images don't make sense to rescale with smooth interpolation so the image interpolation method used when resizing in the viewer should be configurable. We should for instance at least also offer nearest neighbor interpolation.

Minimum Rust logging SDK for Arrow

We should start with a very simple API for logging point clouds using the new Arrow messages. This will let us easily test and benchmark our arrow code.

Log points
Simple example
Works over network (--connect)
Works with show

Get everything to link dynamically on native debug builds

This would drastically reduce compile times.

Tensor view's texture scaling behaves strangely

Theory: Issue might be related to when using non-default dimension mapping.

Skarminspelning.2022-11-09.kl.21.45.06.mov

Add follow-latest or real-time mode

When replaying data in a long-running process, we want to be able to toggle a mode that always tracks the latest, or at least aligns the playback clock with real-time. This should be a toggle-button.

orthographic camera support in `re_renderer`

Unify 2D and 3D views

Blocked by #350

This means:

Populating the Scene3D with 2D-data (with Z=0)
Creating an orthographic camera for the 3D view to use in the case when all data is 2D, replacing the current 2D view ScrollArea with a proper 3D orthographic camera
UI to switch between orthographic and perspective camera
Removing the 2D view and the Scene2D

feature : viewpoint linking between spaces

When using multiple views of the same type (2D or 3D), I would find the data exploration easier if we could choose to link the view points between the spaces.

Another test issue

Want to test updated github action

make AnnotationContext a multi-object of ClassDescriptions instead of a field

Pro:

makes for a less confusing api: Right now one specifies the path an annotation context is added to whereas all (?) other things take a path that specifies where the logged object lives
allows to name your annotation context (right now it’s hardcoded to _annotation_context!)
makes it more straight forward fix ui issues from the keypoint api - we can then show keypoints only when a certain ClassDescription is displayed
“annotation context” goes away, it’s just “a bunch of ClassDescriptions”
paves the way for having multiple sets of class descriptors (I might have several kinds of class ids at once because I ran two classifies)

Con:

need to overhaul the way class descriptions are queried again, unclear to me how hard/easy
how hard is it to keep the table we have on annotation-context right now? need to query children when annotation context object is selected
annotations not being a property of a path might make it confusing when they apply
could do something else with precious time

Internal slack thread https://rerunio.slack.com/archives/C033K5VS2KD/p1669625884256649

Ctrl-C doesn't kill a running Python script using the Rerun SDK

When executing a script that calls into the Rerun SDK, it's not possible to kill it prematurely with <Ctrl+C>

egui `Table` features required for Rerun viewer

Just a meta issue to keep track of our needs Table-wise. Modify/add/comment at will.

Not intended to be a fully detailed spec for each and every thing that we need, just enough information to understand each feature unambiguously.

Roughly in decreasing order of importance.

Support for horizontal scrolling

Table should either be embeddable within an horizontal ScrollArea or (more likely) natively support horizontal scrolling (as it does for vertical scrolling).
While one can technically wrap a Table in a ScrollArea today, it leads to all kinds of serious clipping issues (text being clipped inappropriately, widgets disappearing...) and other layout bugs.

Needed pretty much everywhere.

Automatic optimal sizing for columns

One should be able to specify an automatic column size when instantiating a Table, e.g.:

Table::new().column(Size::Optimal).column(...).show(...);

Needed by text view, annotation contexts, and probably many more.

Ability to show/hide columns from the UI

E.g. right clicking on the table pops up a ComboBox with checkboxes for each column.

Reset column size from UI

We should be able to reset a Table's columns to their original size (which might be a fixed or an optimal size) from the UI (just a matter of clearing the internal resize state).

Needed pretty much anywhere we have resizable tables.

Ability to scroll to a given row

One should be able to easily scroll to a given row number, and specify where the row should then be positioned (at the top? centered? at the bottom?).
Today it is only possible to scroll a specific amount of points, which is very cumbersome and very leaky.

enum ScrollTarget {
    Top,
    Center,
    Bottom,
}
Table::new().column(...).column(...).start_row(42, ScrollTarget::Top).show(...);

Needed by text logs.

Ability to scroll past the end

Consider e.g.:

Table::new().column(...).column(...).start_row(LAST_ROW, ScrollTarget::Top).show(...);

which is something you want to do e.g. anytime you need to align the text logs to a given timestamp.