Giter VIP home page Giter VIP logo

Comments (2)

mattkirwan avatar mattkirwan commented on May 24, 2024 1

Hi mtth,

Perfect! That got me nudged onto the correct path and I've solved with the following (should anyone else google themselves here):

'use strict';
let avsc = require('avsc');

function toMessageBuffer(val, type, schemaId, length) {
  let buf = new Buffer(length || 1024);
  buf[0] = 0; // Magic byte.
  buf.writeInt32BE(schemaId, 1);

  let pos = type.encode(val, buf, 5);
  if (pos < 0) {
    // The buffer was too short, we need to resize.
    // return getMessageBuffer(type, val, schemaId, length - pos);
  }
  return buf.slice(0, pos);
}

// This could be any source - each key being the Unique Schema ID
let schema = {
    '8': {},
    '9': {},
    '10': {},
    '11': {},
    '12': {
        type: "record",
        name: "User",
        fields: [
            {
                name: "firstname",
                type: "string"
            },
            {
                name: "surname",
                type: "string"
            }
        ],
    },
    '13': {},
    '14': {},
}

// Our message
let message = {
    'firstname': 'John',
    'surname': 'Smith'
}

// Create the type from the unique schema
let type = avsc.parse(schema[12]);

// Encode a new message - the 3rd param is the unique schema id
let bufferedMessage = toMessageBuffer(message, type, 12, 1024);

/****************
 *  ~KAFKA~  *
 ****************/

console.log("=====Received Message=====");
let readerType = avsc.parse(schema[bufferedMessage[4]]); // Let's parse this shit with our unique schema ID

// Decode our message
let receivedMessage = readerType.decode(bufferedMessage, 5); // It lives at 5 just after our schema ID
console.log(readerType.toString(receivedMessage.value));

Again, thank you for your help. Things are a little clearer now in regards to Avro and where this package fits in.

Kind regards,

Matt

from avsc.

mtth avatar mtth commented on May 24, 2024

Hey, no problem.

As you can see the message buffer that is sent doesn't (seemingly) contain any Avro header/meta information (such as schema id/string etc...). I would expect these things through from my limited understanding of the Avro docs.

I'm guessing you're referring to this section of Avro's documentation. The first paragraph can be a bit misleading, Avro actually only specifies that the schema should be included with the records when writing container files (and RPC, but that's another story). And this is a good thing since a schema can easily be larger than the encoded data it represents (so the overhead of including it inside each Kafka message would be prohibitive).

So it's left up to the client to decide how best to share schemas between readers and writers. If you can assume that your schema never changes (the simplest case), you don't have to include it at all in the messages. Otherwise a common approach is to prefix each message by a schema ID from which the full schema can be retrieved (this is what happens if you use Confluent's schema registry, see #22 for more information on this).

Does this help clarify?

from avsc.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.