opencyphal / specification Goto Github PK
View Code? Open in Web Editor NEWThe Cyphal specification documents are maintained here.
Home Page: https://opencyphal.org/specification
License: Creative Commons Attribution 4.0 International
The Cyphal specification documents are maintained here.
Home Page: https://opencyphal.org/specification
License: Creative Commons Attribution 4.0 International
Here comes an early draft for an early review. This PDF is compiled directly from the current master: UAVCAN_Specification.pdf
I would like @kjetilkjeka and @thirtytwobits to have a critical look. I am going to review everything once again when the content is finished to fix the remaining styling and wording issues.
The DSDL chapter is rather different from what is published on the website right now: TAO is gone, and the serialization section is moved up.
The PDF is great but you can't provide links into it. We should also generate an HTML/web version from the same tex and publish to the uavcan.org domain.
Once we have the application parts of the specification completed be sure to add a paragraph on the high-level capabilities this part of the specification provides to section 1.2.
So I was writing the new spec doc and got stuck at the semantic and binary compatibility part. I'm leaving the lab right now with the intention of getting back to this first thing in the morning. @kjetilkjeka if you happened to have good definitions at hand, please share here.
This is what I ended up drafting so far:
\section{Data type compatibility and versioning}\label{sec:dsdl_versioning}
\subsection{Rationale}
As can be seen from the preceding sections,
the concept of \emph{data type} is a cornerstone feature of UAVCAN,
which sets it apart from many competing solutions.
In order to be able to interoperate successfully,
all nodes connected to the same bus must use compatible definitions of all employed data types.
This section is dedicated to the concepts of \emph{data type compatibility}
and \emph{data type versioning}.
A \emph{data type} is a named set of data structures defined in DSDL.
As has been explained above, in the case of message data types,
the set consists of just one data structure, whereas in the case of service data types
the set is a pair of request and response data structures.
Data type definitions may evolve over time as they are refined to better address the needs of their applications.
In order to formalize the data type evolution process with respect to the data type compatibility concerns,
UAVCAN introduces two concepts: \emph{semantic compatibility} and \emph{binary compatibility},
which are discussed below.
\subsection{Semantic compatibility}
\subsection{Binary compatibility}
While the two versions of the protocol can co-exist successfully on the same bus (thanks to the Toggle bit measures we've taken), supporting both versions in the same implementation concurrently can be challenging, especially after our big redesign of the standard data type definitions has landed (it's not even proposed yet, working on that).
With that in mind, I would like to revisit the question of migrating deployed nodes from v0 to v1. As I see it, we have two options:
Seeing that the set of fielded nodes is still relatively small and that the old and the new sets of standard data type definitions are likely to end up drastically different, I am inclined to prefer the second option. Its advantage is that it requires much less workload from the library maintainers (us). The disadvantage is that the memory footprints will rise considerably until v0 is phased out completely (although for some this could be a motivation for speedier migration to v1).
This is what the spec should be formatted like when built correctly:
This is what I'm getting after downloading build artifacts from GitLab:
Section headers are offset from the left for some reason, which breaks alignment and makes the doc look sloppy. I think it's important that we get consistent results on local machines and from the CI, otherwise, we might accidentally publish a bad doc.
Context: https://forum.uavcan.org/t/proposed-change-to-variable-length-arrays/740
Affected sections of the Specification:
Changes: the bit length of the implicit field shall be rounded up to the smallest of 8, 16, 32, 64.
A new directive reversing this behavior may be introduced in v1.x (x>0).
Consider replacing this in section 3.8.3.3:
A port identifier of a given kind (subject or service) shall only be used with one major version of one data type in a given UAVCAN network.
Allow data types to be different as long as they are semantically compatible and mutually bit-compatible. See the rationale here https://forum.uavcan.org/t/uavcan-v1-0-and-ardupilot/671/8?u=pavel.kirienko:
As mentioned in the post about meta-transport, highly hierarchical definitions are hard to view because the representation of namespace members is inherently flat which does not reflect the data structure.
Fixed port-IDs make top-level types very salient. It is also natural since a nested type does not need to have a fixed port-ID. This solution is unavailable for top-level types that don't have a fixed port-ID, such as in the linked case; an alternative is to simply add a special character pattern, such as an underscore, in place of the fixed port-ID. For example, a DSDL file Frame.1.0.uavcan
would become _.Frame.1.0.uavcan
. Example:
Alternatively, non-top-level types could be named starting with .
like .Error.1.0.uavcan
. That would make them hidden on Unix-like systems which is interesting.
These are very easy to implement but they are probably not the best possible solutions; other suggestions are welcome.
Multidimensional arrays are currently prohibited. They were avoided mostly due to the issues with byte alignment in variable-length arrays where the width of the length prefix is not a multiple of 8 bits. Since in UAVCAN v1 we require that implicit fields be always standard-size (8/16/32/64), it is now possible to allow multidimensional arrays without sacrificing data alignment.
One issue that has to be considered is how to introduce multidimensional arrays into the grammar while avoiding left recursion. This is because the DSDL syntax is defined through a PEG grammar which does not allow left recursion.
Paging @thirtytwobits
The LaTeX Workshop extension for VSCode supports chktex natively, and we should also run it in the CI.
use our new texer container to build in Buildkite with PR validation builds.
@pavel-kirienko I have a couple of questions regarding the redundant interfaces.
Context: https://forum.uavcan.org/t/uavcan-v1-0-and-ardupilot/671
@thirtytwobits: Pavel, what do you think of defining a formal mechanism for migrating a message definition to a different namespace while maintaining compatibility with the existing one? Ultimately names shouldn’t matter on the wire so the only thing that prevents this is a lookup that can tolerate duplicate definitions when the duplicates are compatible.
@pavel-kirienko: As I just posted above, nothing prevents one from migrating a data type from one namespace into another (unless you are using an experimental transport that uses the data type hash), but I don’t think the use cases where this would be actually needed are common.
@tridge [new inputs here, update your models]: the reason for moving it is social/political. The original messages are in the namespaces of startups that don’t necessarily want their name to be ensconced forever in the message used across the industry. When they develop the message to solve a problem they are not thinking ahead to when they want it to get into an upstream implementation, they are just trying to hit a deadline for a shipping product. Later when they start caring about working with others we want to move responsibility for the message into a longer term organisation namespace.
This issue is the result of the above exchange. We need to mention in the specification that it's legal to rename a data type, including its relocation into a different namespace. Right now it's semi-illegal.
Note to our future selves: this change invalidates the proposal for the data type hash that I introduced here: https://forum.uavcan.org/t/alternative-transport-protocols/324. New proposals are welcome.
We need to replace "message broadcasting" with "message publication", because "broadcasting" does not suit the purpose well; after all, we are not addressing all existing nodes in the network; rather, we are allowing nodes to opt-in into receiving our data.
I already changed this in all chapters of the specification except the DSDL chapter, which is currently being edited by @kjetilkjeka.
I've open this issue to discuss how changes to data types should be handled. This issue is a follow up on OpenCyphal/public_regulated_data_types#35, #5 and #6 but tries to isolate the compatibility discussion without blending it together with other review related items.
I will attempt to keep the text under the line updated to reflect the consensus of the discussion. The initial text is based on previous discussion and (where there are little discussion) my first instincts. Please criticize so we can improve upon it.
Compatibility is a relationship that message definition A
can have with message definition B
. If A
is compatible with B
and B
is compatible with A
we say they are mutually compatible. The different forms of compatibility is defined below.
A structure definition A
is bit-compatible with a structure definition B
if any valid encoded representation of B
is also a valid encoded representation of A
.
A data structure definition A
is semantically compatible with a data structure definition B
if an application that correctly uses A
exhibits a functionally equivalent behavior to an application that correctly uses B
, and A
is bit-compatible with B
.
We say that A
is code compatible with B
if valid use of code generated from B
also is valid use of code generated from A
with the same spec-adhering compiler.
For a compiler to be spec-adhering it must generate compatible code if:
B
must also exist in A
. They must also have the same type.B
must also exist in A
. They must also have the same type.We say that A
is ID compatible with B
If B
has a default ID, and A
has exact same default ID or B
does not have a default ID.
0
, no compatibility guarantees are given.0
and identical path the version with the highest version number will have the following compatibilities with the other version.
All definitions are immutable. This means that after a definition is published a new definition with the same namespace and name must have a higher version number. There are a few exceptions:
Thanks to the explicit syntax-semantics separation, type extensibility is less important for UAVCAN v1 than it is for some other protocols, such as v0 or MAVLink. However, it does not mean that it is not a major enabler for the long-term stability of the ecosystem.
We should explore the possible approaches to this problem and consider implementing one in a future minor revision of the protocol after v1.0, possibly v1.1.
I think we should be able to come up with a more rigorous and strongly-typed solution than naive field injection at the end. I have been thinking about the idea of polymorphism and publisher-subscriber type covariance for months, but the idea is still too vague for a constructive discussion, largely because I am still focused on more immediate issues. The discussion started by @tridge about the adoption of v1 in ArduPilot induced me to post a very brief and highly incomplete description of my ideas in that thread on the forum.
New proposals are welcome.
Scott you dunce. You messed it up!
Per https://forum.uavcan.org/t/serialization-example-on-page-32-section-3-7-5/882 the first
field is a truncated uint12 but the last four bits shown are the last for in the full 16-bit binary representation.
While in the bootloader, always use the mode SOFTWARE_UPDATE
, unless the bootloader is about to launch the application, in which case INITIALIZATION
is also acceptable. Usage of the OPERATIONAL
mode should be discouraged because it is reserved to indicate that the node is in a normal working state. Kocherga can be shown as an example: https://github.com/Zubax/kocherga#uavcan
Consider mentioning in the Specification that the transfer-ID timeout can be auto-detected based on the incoming messaging frequency.
https://forum.uavcan.org/t/alternative-transport-protocols/324/19?u=pavel.kirienko
This change does not require modifying any of the existing specified behavior; rather, it's an implementation advice. Corner cases have to be analyzed first.
Domain-Specific Namespaces are sets of pre-defined messages/data-types for various types of Unmanned Autonomous Vehicles. To start with the standard should define the following DSNs:
@hamishwillee @mrpollo we are optimistically aiming to finalize the draft next week. What are our chances of getting assistance from Dronecode to get the document proofread and corrected?
Semantic compatibility is hard to determine automatically. Consider replacing semantic compatibility with a stricter alternative similar to polymorphic compatibility. This will make type punning impossible.
At the moment, UAVCAN does not support sparse matrices at the protocol level, but rather delegates the issue to the user via matrix packing recommendations.
Native support for sparse matrices will remove the work of ensuring correct packing/unpacking from applications, which will likely have positive effect on simplicity and reliability.
The feature can be probably added in a backward-compatible way. It has not been included in the first revision of the specification due to the following:
There is an experimental branch in the pyuavcan repository, that contains a simple modification to the DSDL parser that enables support for matrices. It is based on two new syntax constructs:
type[R,C] field_name
for a dense matrix, where R,C
stands for the number of rows and columns;type[<=R,C] field_name
for a sparse matrix, R,C
as above.We will need native support for strings in UAVCAN v1.1 or later to avoid ambiguity like:
uint8[<=100] text
Changes:
Item 4 was controversial from the google group start of this proposal but I still think there is value to allowing for a different data serialization layer to sit on top of some UAVCAN implementations (e.g. using libuavcan but not DSDL). I'd like to at least propose it for the record here in our specification repository.
This is an attempt of gathering references to everything concerned by the new protocol. The list will be edited to be kept updated. To keep this issue helpfull instead of just becoming yet another place to discuss things, please keep specialized discussion in the relevant issue.
@deprecated
(#10)We need to set up a CI to automatically build the spec doc and upload the resulting PDF somewhere. Ideally, for pull requests, the resulting PDF should be attached to a comment after every build. Travis supposedly can be configured to do that, see https://damien.pobel.fr/post/github-api-from-travisci/. Although I'm not sure if it's possible to upload an attachment using that API.
Would somebody like to volunteer?
To help system architects that are evaluating a change from v0 to v1 of the protocol we should provide a section of the document that details the differences and describes strategies for migrating existing components.
Make it more readable (closer to what was on the website) while only using code.
Affected sections:
3.7.1.2 Bit and byte ordering
3.7.3.1 General principles
3.7.5 Composite types
3.8.3.4 Data type version selection (the bit flags in the example should be ordered the other way)
4.2.3 Examples
Split out the deprecation discussion into this thread (why do we always seem to wander off topic)
The state of the discussion when split up
Pavel:
I have added a @deprecated keyword to the current draft a couple of weeks ago; it's supposed to be translated into language-specific deprecation markers, e.g. [[deprecated]] in C++ or DeprecationWarning in Python.
Kjetil:
How do we express deprecation?
Let's address this concern before moving on to the bigger concerns.
I have added a @deprecated keyword to the current draft a couple of weeks ago; it's supposed to be translated into language-specific deprecation markers, e.g. [[deprecated]] in C++ or DeprecationWarning in Python.
That's nice. I was playing around with a similar idea myself. What i was thinking about was adding
@lifecycle <pre-release|released|deprecated|removed>
, this way we can add a new type without releasing it, yank a buggy type and deprecate old types. This is how i vision it would work.
- Pre-realease - do not generate code for this definition unless some dangerous keyword is presented.
- Released - As normal (as it currently works)
- Deprecated - Issue a warning that the type is deprecated and will be removed in the future.
- Removed - Issue an error that this type has been through the deprecation process and is no longer usable.
Even if we initially only supported
deprecated
It would allow us to add other lifecycle items (like removed) later on without making a new directive. Both our solutions would of course require the directive related todeprecated
to be an exception to the immutability guarantees together with comments and whitespaces (which is of course totally fine).
Pavel:
I like the flexibility argument, but I have two questions:
- What's the point of the
removed
option if the definition can be just physically removed to the same effect?pre-release
goes against the versioning guarantees that once something is published it can't be removed or changed without a proper deprecation process. If you want tentative volatile definitions, use version 0. I think we can make this work but that would be at the cost of extra complexity in the specification, so I am against it.If the two options mentioned above are removed, we're back to just
deprecated
andreleased
. I think the meaning of the keyword@lifecycle
might be slightly less transparent than@deprecated
, i.e. somebody who is not familiar with the UAVCAN specification might fail to understand what it means;@deprecated
seems more approachable. Also, most published data type definitions will be in therelease
stage, meaning that it would make sense to assumerelease
by default if not specified, so@lifecycle release
becomes redundant.I suggest keeping
@deprecated
.
In Cyphal v1.1 or later we will need native support for enumeration types to improve type safety. This could be implemented through an additional directive like (this is not a formal proposal):
@enumeration
uint11 FIRST = auto
uint11 SECOND = auto
Figure 4.3 Shows the CAN ID bit layout.
There's is a typo in the first table table stating "Service, not message" whereas it's a Message, thus it should note "Message, not service"
The spec draft is ready except for the chapter 5 (Application layer). The work on the application layer is blocked by the lack of the new standard data type set, so that seems to be the next thing to do.
The document has been restructured rather considerably compared to the v0 spec published on the website. The new structure provides dedicated chapters for the transport layer, the application layer, and the physical layer. Also, per Kjetil's suggestion, CAN-specific information has been extracted into two well-isolated sections (one for the transport layer and the other for the physical layer), allowing for future support of other transports.
The CAN FD physical layer compatibility recommendations are missing. They are going to remain this way until we've ran some tests with CAN FD in the lab here; nevertheless, relevant discussions and suggestions are highly welcome.
The current draft could probably use some refinement in the following areas (help needed):
Here is the latest draft (built from the current master): UAVCAN_Specification.pdf
I think as we are slowly approaching completion of the document I should stop cowboying and begin using a more disciplined approach to editing with feature branches, especially if you're going to edit the document alongside.
About a year ago there was a thread on the ArduPilot forum about sensor lags and related issues. From that thread I learned that certain concepts need to be spelled out in the specification:
@jschall once proposed to add those to the set of native DSDL types:
Can we use a floating point standard that matches IEEE 754 for 16, 32, 64 and 128-bit floats, but also provides for arbitrary bit lengths of floating point fields? I've got a spot where I'd like to use a 24-bit float, but there isn't really any support for it.
Actually what I really want to use here is fixed point Q16.8 - because I want uniform precision for this field. This keeps coming up, again and again and again, and I'd really like built-in support for it.
I can't make it a 32-bit float without spilling into another CAN frame
To which I said:
while I do like the idea in general, it seems like too much work right now, and besides there are questions on how to represent fixed-point numbers in generated DSDL classes. In Python there is support for fixed-point arithmetics in the standard library; but what about libuavcan. We'll have to either cast fixed-point numbers to floats, which pretty much defeats the purpose, or we'll need to code up custom fixed-point arithmetic classes (I know that many modern MCU support fixed-point in hardware, but the solution has to be generic). Overall I'm not sure if this feature is worth the extra complexity, at least right now.
This issue is to attract relevant discussion and feedback.
In this thread over there there was also a related discussion on how to represent the elements of rotation quaternions in the most efficient and usable way; fixed-point types could be applied there greatly: OpenCyphal/public_regulated_data_types#34
Consider modifying the CAN ID format to permit coexistence of v0/v1 on the same bus: https://forum.uavcan.org/t/uavcan-v1-0-and-ardupilot/671/22?u=pavel.kirienko
See #60
Under the definition provided in "Alternative Transport Protocols", data type hash makes the network too rigid and incompatible with polymorphic types. I have removed this experimental implementation from PyUAVCAN following the discussions on the forum and #64.
Currently, the legal statement says this, which is not exactly correct:
UAVCAN is an interface standard open to everyone. No copyrights are reserved and no licensing is necessary for its implementation, distribution, or use. In no event shall the authors of the standard be liable for any damage arising, directly or indirectly, from its use.
A better wording should be found. The objectives are the following:
Use gh-pages to automatically publish the specification after a build. See https://github.com/UAVCAN/libuavcan/blob/uavcan-v1.0/test/ci/postbuild-upload-docs.sh for an example.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.