Comments (2)
Hi mtth,
Perfect! That got me nudged onto the correct path and I've solved with the following (should anyone else google themselves here):
'use strict';
let avsc = require('avsc');
function toMessageBuffer(val, type, schemaId, length) {
let buf = new Buffer(length || 1024);
buf[0] = 0; // Magic byte.
buf.writeInt32BE(schemaId, 1);
let pos = type.encode(val, buf, 5);
if (pos < 0) {
// The buffer was too short, we need to resize.
// return getMessageBuffer(type, val, schemaId, length - pos);
}
return buf.slice(0, pos);
}
// This could be any source - each key being the Unique Schema ID
let schema = {
'8': {},
'9': {},
'10': {},
'11': {},
'12': {
type: "record",
name: "User",
fields: [
{
name: "firstname",
type: "string"
},
{
name: "surname",
type: "string"
}
],
},
'13': {},
'14': {},
}
// Our message
let message = {
'firstname': 'John',
'surname': 'Smith'
}
// Create the type from the unique schema
let type = avsc.parse(schema[12]);
// Encode a new message - the 3rd param is the unique schema id
let bufferedMessage = toMessageBuffer(message, type, 12, 1024);
/****************
* ~KAFKA~ *
****************/
console.log("=====Received Message=====");
let readerType = avsc.parse(schema[bufferedMessage[4]]); // Let's parse this shit with our unique schema ID
// Decode our message
let receivedMessage = readerType.decode(bufferedMessage, 5); // It lives at 5 just after our schema ID
console.log(readerType.toString(receivedMessage.value));
Again, thank you for your help. Things are a little clearer now in regards to Avro and where this package fits in.
Kind regards,
Matt
from avsc.
Hey, no problem.
As you can see the message buffer that is sent doesn't (seemingly) contain any Avro header/meta information (such as schema id/string etc...). I would expect these things through from my limited understanding of the Avro docs.
I'm guessing you're referring to this section of Avro's documentation. The first paragraph can be a bit misleading, Avro actually only specifies that the schema should be included with the records when writing container files (and RPC, but that's another story). And this is a good thing since a schema can easily be larger than the encoded data it represents (so the overhead of including it inside each Kafka message would be prohibitive).
So it's left up to the client to decide how best to share schemas between readers and writers. If you can assume that your schema never changes (the simplest case), you don't have to include it at all in the messages. Otherwise a common approach is to prefix each message by a schema ID from which the full schema can be retrieved (this is what happens if you use Confluent's schema registry, see #22 for more information on this).
Does this help clarify?
from avsc.
Related Issues (20)
- finish event fires too early HOT 3
- Array with null items supported? HOT 1
- Avro union - remove type information in resulting json HOT 1
- Bun support HOT 1
- Update `snappy` examples in wiki for `snappy` 7.x.x (async) HOT 1
- How to convert decoded avro data into JSON? HOT 1
- IDL not exporting types for array of union HOT 3
- Support ?-syntax for optional fields in avdl HOT 2
- can schema support dynamic keys? HOT 1
- Extending a schema causes a "truncated buffer" error when using fromBuffer HOT 5
- Invalid Avro header does not raise error event HOT 1
- "new SlowBuffer" is deprecated since Node v.6 --> cannot use it with VITE5 and VUE3 HOT 2
- Doesn't handle trailing 0s from buffer. HOT 1
- Unwrapping unions when deserialising HOT 4
- Unable to consume messages produced by Java application with AVRO schema HOT 9
- Using type.isValid() with a union of records HOT 3
- long encoding/decoding is not reversible for some large but safe js ints HOT 1
- Convert String Representations into Logical Type HOT 1
- Which version of the Avro specification does the latest version of avsc (5.7.7) implement? HOT 1
- Not being able to use records that use the "bytes" type field HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avsc.