confluentinc / avro Goto Github PK

Is this project open to outside contributions? I haven't been successful in my attempts to get .NET updates and improvements into the official Avro project (see PR #307 in the official Avro project). You already have a NuGet package which seems to have some traction (40k downloads at the time of this post). I would love to transfer some of my contributions to this project if you are open to them.

Hello,

It doesn't seem like the C# assemblies in the Confluent.Apache.Avro nuget are strong name signed even though there is an snk file at the root of the project. Can we please have these dlls strong name signed?

Thanks,

Todd

Hi , I have to use reserved C# keyword "in" as enum symbol (in for inch):

{
"name": "DimensionUnits",
"type": {
  "type": "enum",
  "name": "DimensionUnitOfMeasure",
  "symbols": [
    "mm",
    **"in"**
  ]
}

I can achieve that in avrogen.exe by adding @ before symbol name in CodeGen.cs: (line 309)

foreach (string symbol in enumschema.Symbols)
{
    var alwaysAllowedSymbol = symbol;
    if (CodeGenUtil.Instance.ReservedKeywords.Contains(symbol))
        alwaysAllowedSymbol = '@' + symbol;
    CodeMemberField field = new CodeMemberField(typeof(int), alwaysAllowedSymbol);
    ctd.Members.Add(field);
}

Can we add it to official source code? Thanks. If not, could you provide me with explanation (maybe I could figure it out).

Hello, for my master thesis project, i'm generating mock data using kafka-connect and avro schema, but the timestamp field in kafka is generated in this way/Unix epoch i suppose: 8203758334795633000 with 19 digits, and some values are negative.
I tried to convert it using transforms=TimestampConverter but then i get this : "174673355-10-23 23:41:30.000228" , how can I generate normal timestamps I can deal with ?

This how i defined timestamp in avro schema:
{ "name": "Timestamp", "type": "long", "logicalType": "timestamp-millis" }

Hi, I am trying to parse a long value into optional long type, but it gives an exception stating can't find a match for System.Double. If I include double in the type, then it works fine. Please find below details -

AVRO schema -
{ "name": "xyz", "type": ["long", "null"], "default": null, "logicalType": "timestamp-millis" }

Exception -
Avro.AvroException: Cannot find a match for System.Double in {"type":["long","null"]} in field ********* at Avro.Generic.DefaultWriter.WriteRecord(RecordSchema schema, Object value, Encoder encoder) at Confluent.SchemaRegistry.Serdes.GenericSerializerImpl.Serialize(String topic, GenericRecord data, Boolean isKey) at Confluent.SchemaRegistry.Serdes.AvroSerializer1.SerializeAsync(T value, SerializationContext context)
at Confluent.Kafka.Producer2.ProduceAsync(TopicPartition topicPartition, Message2 message)
--- End of inner exception stack trace ---`

I will really appreciate any inputs or workaround with this regard.

Ur tool does not add default values to fields. For "Map" for example their is an error if u not filled some property

I've submitted a PR to apache/avro that supports logical types in .NET:
apache#492

Currently, in order to use the Avro serializer, a dependency on Confluent.Apache.Avro is required because of the required dependency on Confluent.SchemaRegistry.Serdes. The two versions can't be used side by side because they populate the same namespace.

I was wondering if the above PR could be considered for inclusion?

With Confluent.Apache.Avro v1.7.7.7

Given the following schema:

{
    "type": "record",
    "namespace": "com.example.foo",
    "name": "bar",
    "doc": "An example record doc field, which will be ignored by the DatumFileWriter.",
    "fields": [
        ...
    ]
}

If you create a GenericRecord using the above schema and populate the fields, then use the following code to persist it to an .avro file; the doc comment from the top level record is lost.

using var datumFileWriter = DataFileWriter<GenericRecord>.OpenWriter(datumWriter, writeStream);
                datumFileWriter.Append(record);
                datumFileWriter.Flush();

e.g. Result (Note, schema was extracted from within .avro file's contents and fields trimmed for brevity):

{"type":"record","name":"bar","namespace":"com.example.foo","fields":[...]}

According to the Avro spec (https://avro.apache.org/docs/current/spec.html#schema_record), record type should support doc comments; incidentally the python library we use for processing avro files works as expected and includes doc comments for records.

Can the avrogen.exe take a configuration from the command line or from avrogen.exe.config so that we can customise the Property name generated from tool?

C# naming convention for Property is PascalCasing, as the schema is shared across multiple languages it has been decided that the casing will be camelCasing in the schema.

There is no support for Union types in avrocodegen tool. This prevents interoperability with other languages like python.

How can I get access to the actual field values when we have a generic record with a nested record as one of the fields?

{
fields: [
"someNestedRecord" : {
"someName": "someValue
}
]
}

Even the ToString() on generic record does not seem to be able to handle this?

The GenericRecord class allows access to values by key, but it's not iterable, so you need to know the keys in advance. The Java implementation is the same.

But this seems like something you'd often want! Is there something I'm missing?