For example, where it returns 'Stream server error' - if it returned more information about that error, we could show it to users or even just log it in the console to greatly improve the debugging experience. Perhaps it could also return an error param that is the underlying error object?

Given the size of images (some are ~800KB for each channel in KITTI), I would like the ability to rescale images in the builder API. Often data from modern camera comes in HD (1080p or even 4K), but for visualization purposes, something like 720p or even 480p is usually enough, with > 2x-4x space savings. Sometimes encoding quality can also be sacrificed, especially on secondary streams.

So we might want to add for a Builder API for specifying max width/height/quality/binary size (possibly in addition to) an API for just annotating.

Need to find an appropriate module for doing the downscaling. In the browser one can use canvas but in node we might need something that wraps a native library.

Buildkite results are still internal only so we could set up these asap

Metadata builder needs an API to be able to add Declarative UI elements.

Specifically it should take an object and validate that object using our schema definition.

Pre-requisite:
#45

Current documentation is in the context of a conversion of kitti data in streetscape or the spec in XVIZ.

We do not have anything that gives a guided incremental minimal demonstration of XVIZ in action. Ideally this would be like a codepen with streetscape embedded along with the appropriate metadata & builder windows.

Question is if this is needed for Release or now.

Stream data from sensors can have a pose offset relative to the vehicle pose. In order to support this we can add a pose field to stream metadata that can be applied to the transform prior to rendering in streetscape.gl

See #17 and aurora-opensource/streetscape.gl#18

The current XVIZ (https://github.com/uber/xviz/blob/master/docs/overview/concepts.md)[object model] results in many questions.

We should describe some reasoning behind this (that the same object and be updated by different parts of a system and therefore difficult to define all the object and properties in one place.

Thus the approach taken to assemble later in the pipeline.

Related to #64

Scope out the options for management of persistent objects, where the boundaries are and alternatives (custom layers) and how they would or would not be able to utilize XVIZ

In order to support direct pass through to GPU's support for flattened arrays instead of nested arrays would be required.

Specifically this is in reference to the points type https://github.com/uber/xviz/blob/master/docs/protocol-schema/geometry-primitives.md#point-primitive

But may be considered for all types.

Current [https://github.com/uber/xviz/blob/master/docs/api-reference/xviz-configuration.md](xvizConfig documentation).

However it is difficult to understand how and when this should be used.

We should provide an advanced configuration documentation that goes into this further. As such there is an expectation that normal users should not have to provide this until much later in their adoption.

The builder API is useful for when there are many variations to parameters and multiple parameters. However, the validation required to keep track of internal state and validation as well as IDE support makes separate builders per type an attractive refactor.

Builder list breakdown

• primitives & futures
• variables
• time_series

The idea is to have the XVIZBuilder have top-level functions that takes stream name and returns the appropriate builder.

const builder = new XVIZBuilder();
builder.primitive("/world/object").polygon(...).id(1);
builder.variable("/world/object/speed").value(...).id(1);

• Remove .eslintrc overrides (mainly camelcase)
• fix linter setup
• update code to pass linter and prettier.

Declarative UI is a simple structure for defining UI elements that connect to stream data for rendering of controls, charts, and tables.

We need a builder to support the Declarative UI specification.

The specification for variables is unique compared to the other types as it does not support an id.

It seems like being able to specific an id would be desirable and consistent with the other types.

time_series represent single value data along with a timestamp, such as current velocity or acceleration.

From a builder API perspective, it means a timestamp() and value() is set only once for a frame.

Requirements

• verify schema
• verify documentation
• proper v2 output time_series

Stream data can be in different reference frames. In order to generalize this the proposal is to add a coordinate field to the stream metadata.

The value of this field will be one of the following:

• GEOGRAPHIC
• VEHICLE-RELATIVE
• CUSTOM - user-defined string

If CUSTOM the postProcessFrame() provided in the streetscape.gl xvizConfig() should return a transforms dictionary of matrices that will use the streams coordinate string as the lookup.

https://github.com/uber/xviz/blob/master/docs/protocol-schema/geometry-primitives.md defines types not currently supported by the [https://github.com/uber/xviz/blob/master/modules/builder/src/builders/xviz-builder.js](XVIZ Builder)

Specifically we must support

• text
• circle
• stadium

Add support for arbitrary constant data in the XVIZ Metadata builder. The intention is that this would be just arbitrary key/value settings that are included in the metadata.

field(key:string, value:any)

Primary Use-case

Add contextual key/values such as start time, end time, vehicle, versions, etc

• API
• Documentation
• Tests

To allow arbitrary stacking, merge containers and components fields into a single list children.

Loading external assets will be required eventually in XVIZ to support the general use-cases we expect to support.

Use-cases

1. Models and textures for primary vehicle
2. icons for object state, street signs, etc (along with way to "map" to these assets)
3. Images from cameras
4. Complex system states as image

Implementation

• how it would work with the "stream" concept, specifically temporal vs non-temporal
• how it would work with the GLB format (including streaming from that file)
  • URI Reference, JSON Pointer

unknown

• orientation of object

out of scope

• articulation

Formally defining the schema will allow front ends to validate there input data and for us to validate various builder implementations (of which there are multiple complete and under way).

styles

There are v1 values we can support:

• color
• thickness_m (negative means fill, but do not think we need to support that in v2?)
• font_scale

There is a list in the above link about the v2 properties, their valid context (such as polygon), default values, and support for an array of values.

NOTE the Metadata builder takes a raw object, while the xviz builder has an api for each property. They should be in sync and just use an object for definition of the style. We will run the schema validation over it locally.

• update schema
• add to builder
• validate with schema
• tests

xvizConfig represents junction points and data for how to handle XVIZ processing. This is an advanced feature and we should strive to minimize it's surface area.

With that aim the audit should
[ ] verify the reason for any field to be in this config
[ ] document every field with and explanation
[ ] document how a field effect processing and possibly streetscape along with context for why default would need to be changed
[ ] document the set/get functions and how they are intended to be used by a consuming application

There are related projects that we should highlight similarities and differences. This can also help inform our roadmap for the use-cases we want to address.

http://wiki.ros.org/rviz#User_Documentation
https://cesiumjs.org/tutorials/Visualizing-Spatial-Data/

Some data that we want to represent does not have a temporal relationship to the scene. It is persistent and fixed.

Another consideration will be management of this data over long periods of time and or distance. What control mechanisms are available to ensure that this data does not accumulate such that it creates a burden on the runtime.

Examples of such data would be map information, signage, or geo-spatial markers.

For the management issue, we could consider long data sets over large distance with mapping data that accumulates well beyond it's usefulness to the current area of focus. How can this data be safely purged?

Some systems, ROS, have multiple ways to specify such information. Such as a flag stating persistence or a pre-specified lifetime on the object.

There are different approaches to achieve this functionality, each with their own trade offs.

Options to specify this data

1. A separate message type
   A separate message type could be used (instead of state_updates it might be world_updates) where the pose timestamp is not used.

2. Stream metadata attributes
   We could add an attribute to stream metadata that marks the stream as perpetual, or with a duration.

3. Per-object attributes
   Allow per object attributes irrespective of the stream

Option 1 feels the best, as it is clear this data is different and distinct from normal state_updates.

Option 2 This would be a second choice, but the burden of labeling every stream as special feels a bit more overhead than I would prefer (very minor issue), but also the processing of validating streams against a list of stream metadata feels less clean than having a separate entity at a higher level (another minor issue maybe)

Option 3 Just increases the processing at run-time, included mainly to address that it is an option that was considered and capture the reasons why it is a bad fit.

Options to manage this data

1. No management for v2.0
2. Manage by bounding box center distance to current pose. Basically anything beyond a fixed distance is free to be purged.

A major problem is is that we have no robust mechanism to request the data that has been purged.

For the code generating the XVIZ data, if tagged with a known distance or duration the XVIZ conversion could re-emit by managing some state on the dynamic conversion side. Or alternatively, the data could simply be periodically sent based on some criteria or state during the conversion process. But w/o robust server-client state management (which is really not a core focus of XVIZ i would say) maybe any data that needs purged would have to use custom injection of data into the seen (as supported by streetscape.gl) rather than be part of the XVIZ pipeline?

It may be nice to allow XVIZ to be used to produce those layers.

We have the start of schema validation but we need:

• examples for every type
• debug validation in builders

Declarative UI is a simple structure for defining UI elements that connect to stream data for rendering of controls, charts, and tables.

Document of the format is required
Metadata Builder must support adding this data

See comment at bottom of this page: https://github.com/uber/xviz/blob/master/docs/protocol-schema/session-protocol.md

Current primitive specification describes the inline style as an array of objects.

A more natural mapping is to treat the style properties as keys of a map.
List of style properties shows the value types of number, arrays, strings, booleans that can all be represented directly as the value of the mapping.

"style": [
    {
        "type": "fillColor"
        "value": "#FF0000",
    },
    {
        "type": "strokeColor",
        "value": "#000080"
     }
]

"style": {
    "fillColor":  "#FF0000",
    "strokeColor": "#000080"
}

The question of time synchronization has come up in may conversations. We need to provide clear explicit explanations of how the vehicle-pose in the "frame" provides the main time driver, and how the "time window" is used to determine what is "rendered" at a given time.

Furthermore we need to address the more nuanced aspect of a "empty" stream and how it relates to the time window.

Futures_states represent primitives with a timestamp (generally predictive into a future time).

Definition appears incomplete as it does not specify the "type" of the primitive. But the (schema)[https://github.com/uber/xviz/blob/master/modules/schema/core/future_instances.schema.json] looks correct.

From a builder API perspective, it means a timestamp() is applied to any of the existing geometric primitive. In this case, the object should be validated as a future, with appropriate validation.

For the builder, what needs to happen is the following:

• if primitive and has timestamp, treat as future_states entry
• must insert the timestamp & primitive in the correct place in the parallel arrays
• if existing timestamp exists, must append to appropriate array of primitives

The current specification calls for a List of elements, with each element containing a name along with other data.

There are two issues this creates.

1. Processing this data requires a linear search find a particular stream
2. Creates the opportunity for duplicate entries for a single stream

This proposal recommends changing this to a map mapping the name to the underlying element. This approach removes the above issues and does not reduce any functionality.

This would change variables, future_states, and primitive but leave time_series the same as it is currently defined.

1 Linear search

The processing of XVIZ data in Streetscape.gl will process each update and looking for specific streams. The update data is passed around internally and a quick lookup rather than a linear scan makes sense.

This index could be created, but if there is no loss of functionality, simply defining the structure to be a map in the spec would avoid this additional processing on the front end. (which must happen for every update).

2 Duplicate entries

With a list there is the issue of how to handle duplicate entries for the same stream. Are they merged or does one take precedence? The spirit of the specification implies that the stream will be unique for every update, while it's contents may contain an array of elements.

Using a map in the specification guarantees there will only be 1 entry, and avoids the issue at the stream level.

Spec changes

Stream Set

This is a single cohesive set of streams which all happened at the same time and be associated with a single system frame. This is what an XVIZ extractor produces.

Future Instances

Future instances are used to provide lookahead world states for a given instant. These future instances can be used to keep track of what is predicted to happen in the seconds after a given instant. These are not for normal time ordered primitives.

Primitive State

/object/bounds -> [, ]

Variable states are for representing metrics data as a function of time, distance, or anything else. Variable states can be used to provide lookahead values at a given moment, similar to how future instances provide visual lookahead information.

An example of this would be planning data as a function of time over the next 10 seconds. For example, the following three streams would represent three variable states

/plan/time
/plan/velocity
/plan/jerk

Each stream would contain the same number of values and then the velocity and jerk streams could be plotted as a function of the time stream. For more details on how to plot see the UI Metadata section.

Example Data

{
  timestamp: 1001.0
  primitives: [
    {
      "name": "/object/polygon",
      "primitives": [
          {
            "type": "polygon"
            "classes": ["CAR"]
            "vertices": [[9, 15, 3], [20, 13, 3], [20, 5, 3]],
            "id": 1
          },
          {
            "type": "polygon"
            "classes": ["CAR"]
            "vertices": [[2, 18, 3], [5, 9, 3], [9, 4, 1]],
            "id": 2
          }
      ]
    }
  ],
  time_series: [
    {
      "timestamp": 1001.0
      "values": [
          ["/vehicle/torque/commanded", 5],
          ["/vehicle/torque/actual", 4.8]
          ["/vehicle/velocity/commanded", 5],
          ["/vehicle/velocity/actual", 4.8]
      ],
      "id": 0
    },
    {
      "timestamp": 1001.0
      "values": [
          ["/object/velocity", 5],
          ["/object/accel", 4.8]
      ],
      "id": 1
    },
    {,
      "timestamp": 1001.0
      "values": [
          ["/object/velocity", 2],
          ["/object/accel", 2.5]
      ],
      "id": 2
    }
  ],
  future_states: [
    {
      name: "/plan/position",
      timestamps: [1001.1, 1001.2],
      primitives: [
        [
          {
            "type": "polygon"
            "vertices": [[9, 15, 3], [20, 13, 3], [20, 5, 3]],
            "id": 1
          },
          {
            "type": "polygon"
            "vertices": [[2, 18, 3], [5, 9, 3], [9, 4, 1]],
            "id": 2
          }
        ],
        [
          {
            "type": "polygon"
            "vertices": [[14, 15, 3], [25, 13, 3], [25, 5, 3]],
            "id": 1
          },
          {
            "type": "polygon"
            "vertices": [[7, 18, 3], [10, 9, 3], [14, 4, 1]],
            "id": 2
          }
        ]
      ]
    }
  ],
  variables: [
    {
      name: "/plan/time",
      "values": [1001.3, 1002.3, 1003.3]
    }
    {
      name: "/plan/velocity",
      "values": [, 1002.3, 1003.3]
    }
  ],
}

{
  timestamp: 1001.0
  primitives: {
    "/object/polygon": [
      {
        "type": "polygon"
        "classes": ["CAR"]
        "vertices": [[9, 15, 3], [20, 13, 3], [20, 5, 3]],
        "id": 1
      },
      {
        "type": "polygon"
        "classes": ["CAR"]
        "vertices": [[2, 18, 3], [5, 9, 3], [9, 4, 1]],
        "id": 2
      }
    ]
  },
  time_series: [ // NO CHANGE
    {
      "timestamp": 1001.0
      "values": [
          ["/vehicle/torque/commanded", 5],
          ["/vehicle/torque/actual", 4.8]
          ["/vehicle/velocity/commanded", 5],
          ["/vehicle/velocity/actual", 4.8]
      ],
      "id": 0
    },
    {
      "timestamp": 1001.0
      "values": [
          ["/object/velocity", 5],
          ["/object/accel", 4.8]
      ],
      "id": 1
    },
    {,
      "timestamp": 1001.0
      "values": [
          ["/object/velocity", 2],
          ["/object/accel", 2.5]
      ],
      "id": 2
    }
  ],
  future_states: {
    "/plan/position": {
      timestamps: [1001.1, 1001.2],
      primitives: [
        [
          {
            "type": "polygon"
            "vertices": [[9, 15, 3], [20, 13, 3], [20, 5, 3]],
            "id": 1
          },
          {
            "type": "polygon"
            "vertices": [[2, 18, 3], [5, 9, 3], [9, 4, 1]],
            "id": 2
          }
        ],
        [
          {
            "type": "polygon"
            "vertices": [[14, 15, 3], [25, 13, 3], [25, 5, 3]],
            "id": 1
          },
          {
            "type": "polygon"
            "vertices": [[7, 18, 3], [10, 9, 3], [14, 4, 1]],
            "id": 2
          }
        ]
      ]
    }
  },
  variables: {
    "/plan/time": {
      "values": [1001.3, 1002.3, 1003.3]
    },
    "/plan/velocity": {
      "values": [, 1002.3, 1003.3]
    }
  }

Stream names are currently completely free-form but the intention has been to be very path-like. Lets make this intention part of the specification.

There are no semantics to this path. There is no requirement that /foo/bar must have some hierarchical relationship to /foo. However if you where to maintain such a relationship it may make some elements in streetscape.gl work.

One example is a stream browser with a sorted list of stream names.

@twojtasz

Related to aurora-opensource/streetscape.gl#49
Instead of specifying height in XVIZ styles, we want to use inline styling so that each tracklet can have its own height.

The solution should be applicable for all primitive types and all stylable props in the style spec.

The specification need documented.
See comment at bottom of this page: https://github.com/uber/xviz/blob/master/docs/protocol-schema/session-protocol.md

XVIZ Metadata support many elements that are not currently well defined or demonstrate-able w/o a dynamic XVIZ server to handle requests.

We should either trim elements or complete their implementation for beta

https://github.com/uber/xviz/blob/master/docs/protocol-schema/session-protocol.md#session-metadata-to-client

So we have docs that cover most of this. Some updates are required, but also some structure will likely help (ie pure spec vs a more descriptive explanation of the system, but beyond an overview. Maybe captured in a tutorial or walk-thru will be the right approach to weave the pieces together rather than more dry an direct spec style). Need to meet with tech writing.

Pose

• not a stream but an object
• crucial for everything time keeper, other data relative to pose

Message Types

• metadata
• update

Update

• streams
  • name
  • types
    • primitive & futures
      • point, polygon, polyline, circle, stadium, text, image
    • variable
    • time_series
    • tree_table

Objects

• objects are not represented directly, as information from an object can come from various sub-systems. primitives can be tagged with an ID and information across all streams (coord

Styling

source

• json files over any transport (http, websocket, local file)
• binary glTF files with XVIZ

Video?

• basically URL

• coordinate systems

  • geospatial, vehicle relative, other

• data model

  • time window in the xvizSettings
    • how does time window work with vehicle relative pose? who does the translation?
  • persistant data
  • objects and how data is grouped by ID
  • frame represents a window of time
  • how signal for a stream but no data (ie a null signal to indicate last data is no longer valid)

• data flow & xvizConfig

• Pose time vs per stream time?

• variable vs time-series

  • xviz builder will automatically do this so it can be confusing re stream metadata
  • best demonstrated with live example of plots vs time-series in streetscape

• what is out of scope for xviz

• metadata and/or constants

• data limits and considerations

• roadmap for tools, programs, and features

  • simple dynamic server
  • configurable extractors

Other
Annotations
UI panels
Dynamic control protocol ie Session

Support converter using rosbag.js based on voyage code.

Need example public rosbags first, may be able to use voyage data?

variables represent multiple valued data (vs time_series) such as a 10hz sampling of predicted velocity.

From a builder API perspective, it means supporting an array of values.

value(<array>)

Requirements

• support number, string, boolean
• documentation
• unit tests

The points type is currently limited to our default styling of a single color. In order to support additional data that often is shown with point clouds we should support a per-point color array.

https://github.com/uber/xviz/blob/master/docs/protocol-schema/geometry-primitives.md#point-primitive

{
    "type": "point",
    "points": [[9, 15, 3], [20, 13, 3], [20, 5, 3]]
    "colors": [[255, 255, 0]]
}

Same approach as for streetscape: aurora-opensource/streetscape.gl#39

glTF version 2 with the GLB container supports the draco compression extension.

The XVIZWriter has an option for this but it is currently disabled as it was not working as expected. We need to work with @loaders.gl/gltf to fix this path and enable the option.

We know that draco decompression can take ~500ms to decode, but the size savings are still significant and is a valid option to add to XVIZ handling.

Documentation:

• Clarify Point3D type vs. point primitive

Spec:

• Add Pose type ({x, y, z, pitch, roll, yaw})
• Add Matrix4 type (column-major Number[16])
• coordinate field in stream metadata (per-Stream Coordinate System)
  • geographic
  • map_relative
  • vehicle_relative
  • custom
• pose or transformMatrix field in stream metadata (per-Stream Coordinate System)
• Specify vehiclePose structure
  • origin (coordinate)
  • pose (pose)
  • mapPose (pose, optional)
  • customTransformMatrix (matrix4, optional)

Parser:

• Support packed color array in addition to per-object color in point cloud

Currently the pose often named vehicle_pose is not well defined but MUST contains some fields in order to operate with streetscape.

Proposal is to define pose as the following:

{
  x: number,
  y: number,
  z: number,
  longitude: number,
  latitude: number,
  altitude: number,
  pitch: number,
  roll: number,
  yaw: number,
  time: number
}

longitude, latitude, altitude are geospatial coordinates that anchor this pose
x, y, z are meter offsets from the specified geospatial coordinates
roll, yaw, pitch are the orientation of the object
time is the GPS timestamp this pose was captured and serves as the main time signal for synchronization across all streams for visualization.

Alternative definition:

{
  time: number,
  mapOrigin: {
    longitude: number,
    latitude: number,
    altitude: number
  },
  position: {
    x: number,
    y: number,
    z: number
  },
  orientation: {
    pitch: number,
    roll: number,
    yaw: number
  }
}