Giter VIP home page Giter VIP logo

hit9 / bitproto Goto Github PK

View Code? Open in Web Editor NEW
121.0 7.0 17.0 1.36 MB

The bit level data interchange format for serializing data structures (long term maintenance).

Home Page: https://bitproto.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 8.18% C 90.26% Go 0.68% Makefile 0.54% Assembly 0.28% Vim Script 0.06%
bitproto protocol data-interchange data-exchange embedded serialization marshalling serialization-library

bitproto's People

Contributors

hit9 avatar waszil avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

bitproto's Issues

Auto-generate rst documentation for serialization structures

Add a script, which will allow to generate .rst documentation for the given bit proto schema.

Rationale: Schemas often represents messages being exchanged between the devices and messages needs documentation in human readable form. Bitproto already supports generation of the serialization/deserialization source code. Having auto generation of the documentation should add additional great value to the bitproto.

Ambiguity about the functionality of Encode/Decode

For byte-aligned field, Encode/Decode overwrite the dest. For non-byte-aligned field, Encode/Decode is just a BitOR

Message definition:

message Fipr {
    uint1 api_type = 1
    uint5 engine_id = 2
    uint13 pre_install_data = 3
}
message Flit {
    uint32 dw0 = 1
    uint32 dw1 = 2
    uint32 dw2 = 3
    uint32 dw3 = 4
}

Test code:

TEST(BitProto, 0) {
    Fipr fipr{};
    uint8_t buf[256] = {};

    fipr.engine_id = 1;
    EncodeFipr(&fipr, buf);
    fipr.engine_id = 0;
    DecodeFipr(&fipr, buf);
    EXPECT_EQ(fipr.engine_id, 1);

    fipr.engine_id = 2;
    DecodeFipr(&fipr, buf);
    EXPECT_EQ(fipr.engine_id, 1); // FAILED, It's 3

    fipr.engine_id = 8;
    EncodeFipr(&fipr, buf);
    fipr.engine_id = 0;
    DecodeFipr(&fipr, buf);
    EXPECT_EQ(fipr.engine_id, 8); // FAILED, It's 9
}

TEST(BitProto, 1) {
    Flit flit{};
    uint8_t buf[256] = {};

    flit.dw0 = 1;
    EncodeFlit(&flit, buf);
    flit.dw0 = 0;
    DecodeFlit(&flit, buf);
    EXPECT_EQ(flit.dw0, 1);

    flit.dw0 = 2;
    EncodeFlit(&flit, buf);
    flit.dw0 = 0;
    DecodeFlit(&flit, buf);
    EXPECT_EQ(flit.dw0, 2);

    flit.dw0 = 4;
    EncodeFlit(&flit, buf);
    flit.dw0 = 0;
    DecodeFlit(&flit, buf);
    EXPECT_EQ(flit.dw0, 4);
}

The implementation of BpCopyBufferBits in lib/c/bitproto.c make this happen.

What do you think? Is this ambiguity reasonable?

Image on the Readme.md does not respresent what the library is doing

The description is:
message Data {
uint3 the = 1
uint3 bit = 2
uint5 level = 3
uint4 data = 4
uint11 interchange = 6
uint6 format = 7
}

data-encoding-sample

As far as I understood, from playing with the python generated code, the 3 bits of 'the' won't be the 3 MSB of the first bytes, they would be the 3 LSB for the first byte. So the image should be:

data-encoding-sample_drawn

Is that correct?

Simple example fails Python 3.10.8

The simple Pen example latest Ubuntu with Python 3.10.8 virtualenv (pen.py is main.py) fails as follows:

import-im6.q16: attempt to perform an operation not allowed by the security policy PS' @ error/constitute.c/IsCoderAuthorized/421. ./pen.py: line 4: syntax error near unexpected token ('
./pen.py: line 4: `p = bp.Pen(color=bp.COLOR_RED, produced_at=1611515729966)'

Other test examples throw up a similar error. Would very much appreciate a fix. No doubt something simple.
In any case thanks for all the great work.

Invalid enum field name on imported proto on python generation

Test on bitproto v1.1.0:

// b.bitproto
proto b;

enum X : uint1 {
    OK = 0;
}
// a.bitproto
proto a

import "b.bitproto"


message A {
    b.X x = 1;
}

Generates for python:

# b_bp.py

# Code generated by bitproto. DO NOT EDIT.


import json
from dataclasses import dataclass, field
from typing import ClassVar, Dict, List, Union
from enum import IntEnum, unique

from bitprotolib import bp


@unique
class X(IntEnum): # 1bit
    OK = 0


# Aliases for backwards compatibility
OK: X = X.OK


_X_VALUE_TO_NAME_MAP: Dict[X, str] = {
    X.OK: "OK",
}

def bp_processor_X() -> bp.Processor:
    return bp.EnumProcessor(bp.Uint(1))
# a_bp.py


import json
from dataclasses import dataclass, field
from typing import ClassVar, Dict, List, Union
from enum import IntEnum, unique

from bitprotolib import bp

import b_bp as b


@dataclass
class A(bp.MessageBase):
    # Number of bytes to serialize class A
    BYTES_LENGTH: ClassVar[int] = 1

    x: Union[int, b.X] = b.X.b.OK          # !!!!!!!!!!!!! ERROR here
    # This field is a proxy to hold integer value of enum field 'x'
    _enum_field_proxy__x: int = field(init=False, repr=False) # 1bit

    def __post_init__(self):
        # initialize handling of enum field 'x' as `enum.IntEnum`
        if not isinstance(getattr(A, "x", False), property):
            self._enum_field_proxy__x = self.x
            A.x = property(A._get_x, A._set_x)  # type: ignore

    @staticmethod
    def dict_factory(kv_pairs):
        return {k: v for k, v in kv_pairs if not k.startswith('_enum_field_proxy__')}

    def _get_x(self) -> b.X:
        """property getter for enum proxy field"""
        return b.X(self._enum_field_proxy__x)

    def _set_x(self, val):
        """property setter for enum proxy field"""
        self._enum_field_proxy__x = val

    def bp_processor(self) -> bp.Processor:
        field_processors: List[bp.Processor] = [
            bp.MessageFieldProcessor(1, b.bp_processor_X()),
        ]
        return bp.MessageProcessor(False, 1, field_processors)

    def bp_set_byte(self, di: bp.DataIndexer, lshift: int, b: bp.byte) -> None:
        if di.field_number == 1:
            self.x |= (b.X(b) << lshift)
        return

    def bp_get_byte(self, di: bp.DataIndexer, rshift: int) -> bp.byte:
        if di.field_number == 1:
            return (self.x >> rshift) & 255
        return bp.byte(0)  # Won't reached

    def bp_get_accessor(self, di: bp.DataIndexer) -> bp.Accessor:
        return bp.NilAccessor() # Won't reached

    def encode(self) -> bytearray:
        """
        Encode this object to bytearray.
        """
        s = bytearray(self.BYTES_LENGTH)
        ctx = bp.ProcessContext(True, s)
        self.bp_processor().process(ctx, bp.NIL_DATA_INDEXER, self)
        return ctx.s

    def decode(self, s: bytearray) -> None:
        """
        Decode given bytearray s to this object.
        :param s: A bytearray with length at least `BYTES_LENGTH`.
        """
        assert len(s) >= self.BYTES_LENGTH, bp.NotEnoughBytes()
        ctx = bp.ProcessContext(False, s)
        self.bp_processor().process(ctx, bp.NIL_DATA_INDEXER, self)

    def bp_process_int(self, di: bp.DataIndexer) -> None:
        return

Feature: add a "packed" or `-p` flag

Problem

Currently, bitproto's serialized buffer is not packed.

Here's an example:
Here's my simple schema:

proto mytest

message Data {
    uint20 preamble = 1
    uint15 start = 2
    uint64 data = 3
    uint15 crc = 4
}

We compile using:

$ bitproto c .\myschema.bitproto -O

Using the newly generated file, we can do the following tests:

uint8_t buffer[BYTES_LENGTH_DATA] = {0};

struct Data data = {
    .preamble = 0x123,
    .start = 32767, //< 2^15 -1
    .data = 0,
    .crc = 0,
};

EncodeData(&data, buffer);
printf("0x%02X\n", buffer[0]);
printf("0x%02X\n", buffer[1]);
printf("0x%02X\n", buffer[2]);
printf("0x%02X\n", buffer[3]);

When executing this test code, we print

$ make && ./test
0x23
0x01
0xF0

Therefore, this structure is padded.
If the structure would be packed (no padding), I would expect:

0x48
0xFF
0xFF

Here's a visualization of the packed structure.
image

Solution

Adding a -p packed flag.

Why is this valuable

This would enable bitproto's users to have better control over how their data is serialized.

[Bug report] Parsing error when the message name is contained in the message fields

I think the parser will fail if the message name is contained in the message field type.

Steps to reproduce
Using the following schema:

proto mytest

enum payload_data_type_e: uint8 {
    PAYLOAD_DATA_TYPE_UNKNOWN = 1
    PAYLOAD_DATA_TYPE_DETECTOR = 2
}

message payload_data_t {
    payload_data_type_e type = 1 
}

When compiling

$ bitproto -O c .\mytest.bitproto 
error:  .\myschema.bitproto:L9  { => Some grammar error occurred during bitproto grammar parsing.

My guess is that this error is due to a kind of name collision in the lexer.

Representing enums as enum.IntEnum

Hi, great library! I have one question: do you plan to implement enum type representation with enum.IntEnum? It would be nice to be able to use enum classes instead of simple constants.

Example proto:

proto enums;

enum MyEnum : uint2 {
    MY_ENUM_UNKNOWN = 0;
    MY_ENUM_ONE = 1;
    MY_ENUM_TWO = 2;
    MY_ENUM_THREE = 3;
}

message EnumContainer {
    MyEnum my_enum = 2;
}

If the generated python code would be something like this:

import json
from dataclasses import dataclass, field
from typing import ClassVar, Dict, List
from enum import IntEnum, unique

from bitprotolib import bp


@unique
class MyEnum(IntEnum):
    MY_ENUM_UNKNOWN = 0
    MY_ENUM_ONE = 1
    MY_ENUM_TWO = 2
    MY_ENUM_THREE = 3


_MYENUM_VALUE_TO_NAME_MAP: Dict[MyEnum, str] = {
    MyEnum.MY_ENUM_UNKNOWN: "MY_ENUM_UNKNOWN",
    MyEnum.MY_ENUM_ONE: "MY_ENUM_ONE",
    MyEnum.MY_ENUM_TWO: "MY_ENUM_TWO",
    MyEnum.MY_ENUM_THREE: "MY_ENUM_THREE",
}


def bp_processor_MyEnum() -> bp.Processor:
    return bp.EnumProcessor(bp.Uint(2))


@dataclass
class EnumContainer(bp.MessageBase):
    # Number of bytes to serialize class EnumContainer
    BYTES_LENGTH: ClassVar[int] = 1
 
    # based on https://stackoverflow.com/a/61709025/1169220
    my_enum: MyEnum = MyEnum.MY_ENUM_UNKNOWN
    _my_enum: int = field(init=False, repr=False)

    def __post_init__(self):
        if not isinstance(getattr(EnumContainer, "my_enum", False), property):
            self._my_enum = self.my_enum
            EnumContainer.my_enum = property(EnumContainer._get_my_enum, EnumContainer._set_my_enum)

    def _get_my_enum(self):
        return MyEnum(self._my_enum)

    def _set_my_enum(self, val):
        self._my_enum = val

...

Then it could be used like this:

import enums_bp as bp

enum_container = bp.EnumContainer(my_enum=bp.MyEnum.MY_ENUM_ONE)
s = enum_container.encode()
enum_container_new = bp.EnumContainer()
enum_container_new.decode(s)
assert enum_container_new.my_enum == enum_container.my_enum
assert enum_container_new.encode() == s
assert isinstance(enum_container.my_enum, bp.MyEnum)
assert isinstance(enum_container_new.my_enum, bp.MyEnum)
assert enum_container.my_enum is bp.MyEnum.MY_ENUM_ONE
assert enum_container_new.my_enum is bp.MyEnum.MY_ENUM_ONE

What do you think?

bitproto.c compilation errors using TI's CCS.

I made a simple library project in Code Composer Studio (v12.4) using the default TI toolchain.
Unfortunately bitproto.c (v1.1.1) does not compile, with the following errors:

"../src/bitproto.c", line 50: error #29: expected an expression
"../src/bitproto.c", line 50: error #20: identifier "k" is undefined
"../src/bitproto.c", line 155: error #29: expected an expression
"../src/bitproto.c", line 155: error #20: identifier "k" is undefined
"../src/bitproto.c", line 165: error #29: expected an expression
"../src/bitproto.c", line 165: error #20: identifier "k" is undefined
"../src/bitproto.c", line 402: remark #1532-D: (ULP 5.3) Detected vsprintf() operation(s). Recommend moving them to RAM during run time or not using as these are processing/power intensive
"../src/bitproto.c", line 402: remark #2553-D: (ULP 14.1) Array index (involving "ctx") of type "int". Recommend using "unsigned int"
"../src/bitproto.c", line 414: error #29: expected an expression
"../src/bitproto.c", line 414: error #20: identifier "k" is undefined
"../src/bitproto.c", line 527: error #29: expected an expression
"../src/bitproto.c", line 527: error #20: identifier "k" is undefined
10 errors detected in the compilation of "../src/bitproto.c".

The BpMessageDescriptor structure defiined as a macro, doesn't seems to work with this compiler.

Floating point support

Hi,

Is it a possible goal to add floating point support to bitproto? If it seems feasible, I can try to work on it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.