opensourcerouting / c-capnproto Goto Github PK

View Code? Open in Web Editor NEW

This project forked from jmckaskill/c-capnproto

118.0 118.0 39.0 1.74 MB

C library/compiler for the Cap'n Proto serialization/RPC protocol

License: MIT License

C++ 10.06% C 80.56% Cap'n Proto 8.31% Makefile 0.42% M4 0.21% Meson 0.44%

c-capnproto's People

Contributors

Stargazers

Watchers

c-capnproto's Issues

Add CI for Windows and macOS

Field Accessors Bug ( I think)!

Hi ,
I was trying to use the plugin and I encountered what I think is a bug, please let me know if my understanding is wrong about how the generated API works.
The schema definitions that I used is the following:

struct Leaf{
        value @0 : Int32;
}
struct Inner {
        left @0  :TreeNode;
        right @1 : TreeNode;
}
struct TreeNode {
  nodeType @0 :Int32;
  leaf  @1 : Leaf;
  inner @2 :Inner;

}

For the following code I expect 1 to be printed but 0 is printed instead ;It looks like there is something wrong in the method Tree_Node_set_leaf(...) unless I misunderstand how it should work!

   struct capn c;
   struct capn_segment *cs;
   capn_ptr cr;
        
    capn_init_malloc(&c);
    cr = capn_root(&c);
    cs = cr.seg;

   TreeNode_ptr n = new_TreeNode(cs);
   TreeNode_set_nodeType(n, 1);
   Leaf_ptr l = new_Leaf(cs);
   Leaf_set_value(l , 1);
   TreeNode_set_leaf(n, l);
    cout<<Leaf_get_value(TreeNode_get_leaf(n))<<endl;

create() doesn't set user member

I could be mis-reading this, but it looks like https://github.com/opensourcerouting/c-capnproto/blob/master/lib/capn-malloc.c#L38 should instead be:

s->user = u;

Trying to write a NULL struct

I somehow can't leave it uninitialized, or I get,

*** ERROR DECODING PREVIOUS MESSAGE ***
The following error occurred while decoding the message above.
This probably means the input data is invalid/corrupted.
Exception description: expected boundsCheck(segment, ptr, ref->structRef.wordSize()); Message contained out-of-bounds struct pointer.
Code location: capnp/layout.c++:2159
*** END ERROR ***

From the capnp FAQ:

Pointer fields are a bit different. They start out “null”, and you can check for nullness using the hasFoo() accessor. You could use a null pointer to mean “not present”. Note, though, that calling getFoo() on a null pointer returns the default value, which is indistinguishable from a legitimate value, so checking hasFoo() is in fact the only way to detect nullness.

What do I need to do to not set a pointer?
Specifically, I sometimes do not set the worker field here: https://gitlab.com/dxpb/dxpb/blob/1b8775bd089ca3c2df1c53030ddb5adec7fa4313/src/log.capnp#L45-48

Support Meson buildsystem

Meson is a very popular build system for C-only projects nowadays. It's used by many big projects like Mesa, ffmpeg, etc.

There are two implementations of it:

Main Meson implementation in Python
Slender implementation in pure C - Muon

Documentation / Security Warning

I strongly believe that there should be a security warning in the README regarding malicious input.
There is currently no code to verify buffers/structures, unless I missed something completely?
I think a small notice like #29 should suffice.

Any program will crash horrendously should it encounter malicious/corrupted input. If anyone is interested, I adjusted the included tests/examples to be used with american fuzzy lop here. You can test with make fuzz-mem and make fuzz-fp.

capnpc-c exit with code 2 if there is an unused struct in included file

$ cat infra.capnp
@0xab01f6ae1ef59a57;
struct S1 {
s1 @0 :Int64;
}
struct S2 {
s2 @0 :Int64;
}

$ cat msg.capnp
@0xed2b3b98b3eb515b;
using import "/infra.capnp".S1;
struct M {
m1 @0: S1;
}

$ capnp -I ~/ws/capn compile -o /home/assafb/ws/capn/capnpc-c:/home/assafb/ws/capn/ ~/ws/capn/msg.capnp
cant find node with id 0x85f0eda94e40ad1a
/home/assafb/iguazio/workspace/engine/zeek/build/vendor/c-capnproto/capnpc-c: plugin failed: exit code 2

capn_getv returns -1

I have problem with reading build-in list types. This code simply creates new Message structure, calls capn_setv, writes message to capn segment and reads it. After this operations calling capn_getv on body fails.

d.capnp:

@0xec7cea5be31ae639;

struct Message {
    body @0 :List(UInt8);
}

d.c:

#include <capnp_c.h>
#include <string.h>

#include "d.capnp.h"

struct msg_capn_t {
    uint8_t segment_bytes[128];  
    struct capn_segment s; 
    struct capn c;
    struct Message msg;
    Message_ptr ptr;
};

int main(void){
    struct msg_capn_t ms;
    memset(&ms, 0, sizeof(ms));
    ms.s.data = (char*)ms.segment_bytes;
    ms.s.cap = sizeof(ms.segment_bytes);
    capn_append_segment(&ms.c, &ms.s);
    capn_root(&ms.c);
    
    // Begin capnp root
    ms.ptr = new_Message(&ms.s);
    capn_setp(capn_root(&ms.c), 0, ms.ptr.p);
    
    {   // Begin capnp child
        uint8_t buffer[2] = {99, 76};
        ms.msg.body = capn_new_list8(&ms.s, 2);
        capn_setv8(ms.msg.body, 0, buffer, 2);
    }
    write_Message(&ms.msg, ms.ptr);
    read_Message(&ms.msg, ms.ptr);
    {   // Begin capnp child
        uint8_t buffer[2] = {0, 0};
        int c = capn_getv8(ms.msg.body,0, buffer, 2);
        if (c == 2){
            printf("success\n");
            return 0;
        } else {
            printf("fail capn_getv returned %d\n",c);
            return 1;
        }
    }
}

I have found one solution - manually changing generated file d.capnp.c from:

void read_Message(struct Message *s, Message_ptr p) {
	capn_resolve(&p.p);
	s->body.p = capn_getp(p.p, 0, 0);
}

void read_Message(struct Message *s, Message_ptr p) {
	capn_resolve(&p.p);
	s->body.p = capn_getp(p.p, 0, 1);
}

This can also be changed in c-capnproto/compiler/capnpc-c.c:604
str_addf(func, "%s = capn_getp(%s, %d, 1);\n", pvar, ptr, f->f.slot.offset);
but I am not sure if it is correct solution, or bug is somewhere else (for example in my code).

../tests/example-test.cpp:89: Failure Expected: (f) != ((void*)0), actual: NULL vs NULL

[ RUN      ] Examples.RoundTripPerson
../tests/example-test.cpp:89: Failure
Expected: (f) != ((void*)0), actual: NULL vs NULL
[  FAILED  ] Examples.RoundTripPerson (0 ms)
[ RUN      ] Examples.PersonWithAccessors
[       OK ] Examples.PersonWithAccessors (0 ms)
[----------] 2 tests from Examples (0 ms total)

[----------] 1 test from Schema
[ RUN      ] Schema.ReadSimple
[       OK ] Schema.ReadSimple (0 ms)
[----------] 1 test from Schema (0 ms total)

[----------] Global test environment tear-down
[==========] 24 tests from 4 test suites ran. (0 ms total)
[  PASSED  ] 23 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] Examples.RoundTripPerson

 1 FAILED TEST
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――


Summary of Failures:

1/1 capn-test FAIL            0.40s   exit status 1

Ok:                 0
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

It's Clang on macOS ARM64 (M1):

ℤ clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

ℤ sw_vers
ProductName:		macOS
ProductVersion:		13.3.1
BuildVersion:		22E261

Solaris/Sparc 8-byte alignment

I am currently working on a project where I will be using this on Solaris/Sparc 32-bit and I wanted to understand why there is a requirement for 8-byte alignment on the data, len, and cap fields in the struct capn_segment. As of right now, the call to ALIGNED in struct ALIGNED_(8) capn_segment fails. Thank you in advance!

hard to detect bug when `capn_setp` not called.

If capn_setp not been called. data serialized shell not contain it. or an assert shell emit at least.

It's really hard to debug it.

eg.

file a.capnp

@0xb5c2cbd0a1ee9f74;

struct Person{
    name @0 :Text;
}

#include <unistd.h>
#include<sys/types.h>
#include<fcntl.h>
#include <capnp_c.h>

#include "a.capnp.h"


void x(){

};

int main() {
    struct capn capn;
    struct capn_segment *seg;
    capn_init_malloc(&capn);
    capn_ptr root = capn_root(&capn);
    Person_ptr personobj=new_Person(root.seg);
    printf("root->type %d\n",root.type);
    struct capn_text ustr={
        3,"aaa", NULL
    };  
    Person_set_name(personobj, ustr);
    int fd=open("aaa.data",O_CREAT|O_RDWR);
    capn_write_fd(&capn, write , fd, 0); 
    close(fd);
}

this will generate file aaa.data, content:

[root@7e6235426e5d bbb]# od -x aaa.data 
0000000 0000 0000 0003 0000 0000 0000 0000 0000
0000020 0001 0000 0022 0000 6161 0061 0000 0000
0000040
[root@7e6235426e5d bbb]# capnp decode a.capnp Person < aaa.data 
()

[root@7e6235426e5d test-go-capnp]# od -x good.data
0000000 0000 0000 0003 0000 0000 0000 0000 0001
0000020 0001 0000 0022 0000 6161 0061 0000 0000
0000040
[root@7e6235426e5d test-go-capnp]# capnp decode a.capnp Person < good.data 
(name = "aaa")

Really Need Documentation

Document Required. digging code makes me crezy.

ASAN and UBSAN warnings

../lib/capn.c:484:17: runtime error: left shift of negative value -1
[       OK ] WireFormat.StructRoundTrip_OneSegment (6 ms)
../lib/capn.c:704:19: runtime error: member access within null pointer of type 'struct copy'
../lib/capn-malloc.c:219:12: runtime error: store to misaligned address 0x7ffcfc967a0a for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0a: note: pointer points here
 00 00  f8 ff ff ff ff ff ff ff  90 7a 96 fc fc 7f 00 00  1e 00 00 00 00 00 00 00  f8 00 00 00 00 00
              ^ 
../lib/capn-malloc.c:220:22: runtime error: store to misaligned address 0x7ffcfc967a0e for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0e: note: pointer points here
 00 00 00 00 ff ff  90 7a 96 fc fc 7f 00 00  1e 00 00 00 00 00 00 00  f8 00 00 00 00 00 00 00  00 00
             ^ 
../lib/capn-malloc.c:225:17: runtime error: store to misaligned address 0x7ffcfc967a0e for type 'uint32_t', which requires 4 byte alignment
0x7ffcfc967a0e: note: pointer points here
 00 00 00 00 00 00  00 00 96 fc fc 7f 00 00  1e 00 00 00 00 00 00 00  f8 00 00 00 00 00 00 00  00 00
             ^

Show in docs the Example Usage in C

It would be great if you could show in the docs how the Example Usage in https://capnproto.org/cxx.html translates into C using this plugin.

generated files include c.capnp.h but this header is not installed

I see this included even in the file tests/addressbook.capnp.h

https://github.com/opensourcerouting/c-capnproto/blob/master/tests/addressbook.capnp.h#L10

#include "c.capnp.h"

This header file is not installed by make install

Why is this file included in the resulting schema header? If it is necessary shouldn't c.capnp.h be installed along with other headers?

compiler/test.capnp.c:3159:42: warning: bitwise comparison always evaluates to true

[19/44] Compiling C object test.capnp.c.p/compiler_test.capnp.c.o
../compiler/test.capnp.c:3159:42: warning: bitwise comparison always evaluates to true [-Wtautological-bitwise-compare]
        s->secondBit = (capn_read8(p.p, 0) & 2) != 1;
                       ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
../compiler/test.capnp.c:3190:39: warning: bitwise comparison always evaluates to true [-Wtautological-bitwise-compare]
        secondBit = (capn_read8(p.p, 0) & 2) != 1;
                    ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~

capn_size() return value should be int64_t

Per #47 - the expression headersz + datasz could conceivably require int64_t.

Won't compile

configure.ac:18: error: required file 'ac/ltmain.sh' not found
configure.ac:14: error: required file 'config.h.in' not found

Small-size data serialization problem

According to our experiments, if the size of the raw message is less than 200 bytes, cap'proto gets slower than protobuf. The reason could be the overhead taken by the malloc process invovled in the deserializing each message?

generated code broken for Data type with empty string as default

The generated .c file for the below capnp does not compile due to undeclared capn_buf:

$ cat Test.capnp.c                 
#include "Test.capnp.h"            
/* AUTO GENERATED - DO NOT EDIT */ 
#ifdef __GNUC__                    
# define capnp_unused __attribute__((unused))                                                                                                 
# define capnp_use(x) (void) x;    
#else                              
# define capnp_unused              
# define capnp_use(x)              
#endif                             
                                                                       
static const capn_ptr capn_null = {CAPN_NULL};                         
static capn_data capn_val1 = {{2,0,0,0,1,0,0,(char*)&capn_buf[0],(struct capn_segment*)&capn_seg}};                                           
...

File Test.capnp:

@0xa4ba9dce9642ee58;

struct Record {
   info     @0: Data = "";
}

struct RootStruct {
   myList   @0: List(Record);
}

Can't retrieve list length of native field type

I have a struct like this:

struct Event {
    args        @0  :List(Text);
}

Which generates a struct like this:

struct Event {
    capn_ptr args;
};

Which seems wrong because if I try to get the length of the list using capn_len I cannot compile my program because capn_len tries to access a member named p and struct capn_ptr doesn't have a member named p:

In file included from bufs/main.capnp.h:4:0,
                 from event.c:3:
event.c: In function 'event_tocapn':
../deps/build/usr/local/include/capnp_c.h:182:31: error: 'capn_ptr {aka struct capn_ptr}' has no member named 'p'
 #define capn_len(list) ((list).p.type == CAPN_FAR_POINTER ? (capn_resolve(&(list).p), (list).p.len) : (list).p.len)
                               ^
event.c:173:27: note: in expansion of macro 'capn_len'
     printf("Len is %d\n", capn_len(ie->args));
                           ^~~~~~~~

...

If I use a custom defined type as the list members (such as a struct containing only a Text field) everything is fine.

From looking at other examples, it looks like this field should be a capn_data struct or one of the capn_list structs?

Nested unions generate incorrect code

For this schema:

struct A {
    union {
        zero @0 :Void;
        one :union {
            first @1 :Void;
            second @2 :UInt16;
        }
    }
}

I get this C code:

struct A {
	enum A_which which;
	capnp_nowarn union {
		enum A_one_which one_which;
		capnp_nowarn union {
			uint16_t second;
		} one;
	};
};

The one_which member is in a union alongside one, which means assigning to one.second will overwrite one_which.

packed header is not 32 bit aligned->can cause memory fault

In the function capn_write_mem_packed in capn-malloc.c the uint32_t pointer "header" is calculated.
The function gets a 32 bit aligend pointer to the payload buffer passed.
"header" has an offset to the payload buffer pointer of header size + 2 bytes. Depending on header size it can happen that "header" is not aligned to 32 bit.
Subsequently header_render() performs 32 bit write access to the unaligned "header" pointer.

Processor which can't perform unaligned 32 bit access went to hard fault/memory fault.

As a walk around it is possible to write the data to be written in a variable and copy the variable to the pointer with memcpy.

Incorrect assumption that a C enum is always 2 octets wipes enum values sometimes.

I have a Cap'n Proto schema that looks like:

struct UI {
    union {
        thing1 @0 :Text;
        thing2 @1 :AnEnumType;
        thing3 @2 :UInt16;
    }
}

The enum type is represented as a C enum. But the generated code treats it as interchangeable with the UInt16 type:

void read_UI(struct UI *s capnp_unused, UI_ptr p) {
	capn_resolve(&p.p);
	capnp_use(s);
	s->which = (enum UI_which)(int) capn_read16(p.p, 0);
	switch (s->which) {
        // ...
	case UI_thing2:
	case UI_thing3:
		s->thing3 = capn_read16(p.p, 2);

The size of a C enum is not specified by the standard; compilers can choose whatever is appropriate. On my platform this is 4 octets (32 bits), so the LSBs of the enum are wiped. This changes with the positioning of the enum in the Cap'n Proto definition; in an even numbered position, it's fine.

Deserialization returns all 0's

Not sure if this is a bug in the library, or user error (but I fail to see how, do tell if you find an issue) I have created

enum MessageCategory 
{
    none @0;
    system @1;
    log @2;
    debug @3;
    file @4;
    cmd @5;
    max @6;
}

struct Message
 {
    category @0 :MessageCategory;
    type @1 :UInt32;
    containedMessage @2 :Data;
}

Built fine with c-capnproto compiler, but upon serialization/de-serialization something breaks.

int main()
{
	uint8_t data[] = { 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41, 0x41 };
	uint8_t buf[0x1000];
	int32_t bufDataLen = 0;
	{
		struct capn c;
		capn_init_malloc(&c);
		capn_ptr cr = capn_root(&c);
		struct capn_segment* cs = cr.seg;

		const char helloWorld[] = "Hello fucking world";

		capn_list8 list8 = capn_new_list8(cs, sizeof(helloWorld));
		capn_setv8(list8, 0, helloWorld, sizeof(helloWorld));
		
		struct Message m;
		m.category = MessageCategory_file;
		m.type = 2;
		m.containedMessage.p = list8.p;

		memset(buf, 0xFE, sizeof(buf));

		bufDataLen = capn_write_mem(&c, buf, sizeof(buf), false);

		capn_free(&c);
	}
	FILE* file = fopen("dump.bin", "wb");
	fwrite(&bufDataLen, sizeof(bufDataLen), 1, file);
	fwrite(buf, sizeof(uint8_t), bufDataLen, file);
	fclose(file);
	file = NULL;
	{
		struct capn rc;
		int init_mem_ret = capn_init_mem(&rc, buf, sizeof(buf), false);
		if (init_mem_ret != 0)
		{
			return -1;
		}
		Message_ptr root;
		struct Message msg;
		root.p = capn_getp(capn_root(&rc), 0, true);
		read_Message(&msg, root);
		capn_free(&rc);

		printf("got msg (%d) (%d)\n", msg.category, msg.type);
	}
	return 0;
}

dump.bin contains some tags, and my string "Hello fucking world" just fine. The de-serialization code runs also with no issue. The Message msg; is full of uninitialized values, and after the read_Message call, it's all zeroed out (instead of intended values). Not what they intended to be from the buffer.

capn_write_mem() return value is int

capn_write_mem()'s return value is int.

The return value is actually down-cast from size_t, since inside capn_write_mem(), the actual header and data sizes are size_t.
the int type of the return value basically limits the size of a capnproto file to 2GB.

Example to add "Data" type data

Hi,

Thanks for the C implementation of capnproto.

I have hit a wall with regard to using a "Data" type for serializing. De-serialization works well but I am not able to serialize and send data using any method.

Can you please give a simple example to serialize the following structure
struct DataMessage {
packetdata @0 :Data;
}

Thank you very much in advance

Packed input/output not working

capn_write_mem() and capn_init_mem() take a packed parameter, but this parameter is not tested in the tests and when I tried to use it, it does not work correctly (actually writing works fine, reading does not).

The problem is in init_fp() the code needs to first read in the packed data, unpack it, then use that unpacked data as input. Enabling the packed option in capn-stream-test.cpp will demonstrate my point.

Build C source issue with Ubuntu

When I try to build capnp file to c source, I always get an error. Any suggestions would be greatly appreciated.

sudo capnp compile -o c addressbook.capnp
addressbook.capnp:27:18-28: error: Import failed: /c.capnp
addressbook.capnp:28:2-3: error: Not defined: C

capn_get64() always returns zero

Hello!

I want to perform element-wise reading on a capn_list64. I have a problem with capn_get64(), which always returns a zero value, whereas capn_getv64() works fine. What could be the reason?

I test as follows:

uint64_t sample_get = capn_get64(samples, sample_index);

uint64_t sample_getv;
capn_getv64(samples, sample_index, &sample_getv, 1);

In the above example, sample_get = 0, but sample_getv = 4625979026566705573, which is the number that I stored in the first place.

Thanks in advance!

Get expected size before writing in memory

From what I see, in each example or project using c-capnproto, each time capn_write_mem is used, the target buffer is mostly everytime allocated on stack with small length (typically 4k or 8k).

If my object is (let assume) random amount of megabytes, how can I allocate a buffer large enough ?
I don't see any function able to tell me the expected size of the output.

Is there any way possible to expect the final size to allocate it ?

Project is UNMAINTAINED

PRs aren't being reacted upon, last commit two years ago. Would be nice to make it work on modern systems and make a new release.

opensourcerouting / c-capnproto Goto Github PK

c-capnproto's People

Contributors

Stargazers

Watchers

Forkers

c-capnproto's Issues

Recommend Projects

Recommend Topics

Recommend Org