Error type.fromBuffer for string of json data about avsc HOT 13 CLOSED

+1

from avsc.

Are you sure your encoded data is valid? Could you attach that as well? (buffer in your above examples.)

from avsc.

I'm sure my encoded data is valid. Because of:

users_mention: '[{"user_id": "2", "user_name": "test2"}]',

It's working.

users_mention: '[{"user_id": "2", "user_name": "test2"}, {"user_id": "3", "user_name": "test3"}]',

It's not working. Although users_mention field is also a string.

Beside, I also have another consumer (wrote by Python) which is also subscribe to the same topic in Apache Kafka, and it is working properly for 2 above case. So, My encoded data is valid.

I will attach 2 files for 2 above data case.

• file buffer_pass : with users_mention: '[{"user_id": "2", "user_name": "test2"}]'
• file buffer_error : with users_mention: '[{"user_id": "2", "user_name": "test2"}, {"user_id": "3", "user_name": "test3"}]'

Thanks so much !!!

from avsc.

Sorry I do not have write permission in this repository, so I can not upload buffer file to you. :(

from avsc.

Can you copy the hex-encoded bytes for your record here?

I can't reproduce this on my side with the data you gave so far: when I serialize your JSON record to bytes and back again it works fine.

Edit: A full stack trace would be helpful as well.

from avsc.

Yes, Are you want to this data?

• bytes with users_mention: '[{"user_id": "2", "user_name": "test2"}]':

'\x0ementionP[{"user_id": "2", "user_name": "test2"}]\x06\'1\'H08b541da-8508-11e5-8afe-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\x0eclothes\x0cupdateH4c3448cd-c74b-43fa-85d2-b6693bfb0c21'

• bytes with users_mention: '[{"user_id": "2", "user_name": "test2"}, {"user_id": "3", "user_name": "test3"}]':

'\x0emention\xa0\x01[{"user_id": "2", "user_name": "test2"}, {"user_id": "3", "user_name": "test3"}]\x06\'1\'H93064150-8508-11e5-9896-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\x0eclothes\x0cupdateH4c3448cd-c74b-43fa-85d2-b6693bfb0c21'

from avsc.

Yes, perfect, thanks!

Two more quick questions:

• Could you copy the full stacktrace of your error?
• Which version of avsc are you using?

from avsc.

Yes,

• 1. This is the full stacktrace of my error:

RangeError: out of range index
     at RangeError (native)
     at Tap.readString (/home/ngoccuong/working/source_code/websocket/node_modules/avsc/lib/tap.js:225:19)
     at StringType._read (/home/ngoccuong/working/source_code/websocket/node_modules/avsc/lib/types.js:434:58)
     at RecordType.readMention [as _read] (eval at <anonymous> (/home/ngoccuong/working/source_code/websocket/node_modules/avsc/lib/types.js:1159:10), <anonymous>:4:39)
     at readObj (/home/ngoccuong/working/source_code/websocket/node_modules/avsc/lib/types.js:1481:17)
     at RecordType.Type.fromBuffer (/home/ngoccuong/working/source_code/websocket/node_modules/avsc/lib/types.js:159:13)
     at Object.avroDecode (/home/ngoccuong/working/source_code/websocket/services/avro.js:29:24)
     at /home/ngoccuong/working/source_code/websocket/websockets/mentions.js:89:32

• 2. I am using avsc version 2.0.1, nodejs version 4.0.0 and npm version 2.14.2

Thank you

from avsc.

So I just ran the following code (on the same version of avsc you have, and the current one), using the two strings above:

var avsc = require('avsc');

var type = avsc.parse({
  "namespace": "mention.avro",
  "type": "record",
  "name": "Mention",
  "fields": [
    {"name": "type",  "type": "string"},
    {"name": "users_mention",  "type": "string"},
    {"name": "from_user",  "type": "string"},
    {"name": "rq",  "type": "string"},
    {"name": "object_id",  "type": "string"},
    {"name": "object_type",  "type": "string"},
    {"name": "action", "type": "string"},
    {"name": "comment_id", "type": "string"}
  ]
});

// The first string.
var s1 = '\x0ementionP[{"user_id": "2", "user_name": "test2"}]\x06\'1\'H08b541da-8508-11e5-8afe-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\x0eclothes\x0cupdateH4c3448cd-c74b-43fa-85d2-b6693bfb0c21';
var b1 = new Buffer(s1, 'binary')
var r1 = type.fromBuffer(b1); // The first record, OK.

// The second string.
var s2 = '\x0emention\xa0\x01[{"user_id": "2", "user_name": "test2"}, {"user_id": "3", "user_name": "test3"}]\x06\'1\'H93064150-8508-11e5-9896-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\x0eclothes\x0cupdateH4c3448cd-c74b-43fa-85d2-b6693bfb0c21';
var b2 = new Buffer(s2, 'binary');
var r2 = type.fromBuffer(b2); // The second record, still OK.

Both records are deserialized fine.

What is the type of message.value in your snippet above? Is it a binary string? If so, you need to pass 'binary' as second argument to Buffer's constructor (otherwise it'll expect a UTF-8 encoded string).

from avsc.

@cuongbangoc @thanhson1085 - did the suggestion above fix your issue? If not, could you post the output of JSON.stringify(message)?

from avsc.

Sorry, Today I come back to my work. :).

I used to use binary parameter in Buffer's constructor, but It is still not working.

This is my JSON.stringify(message) for case is working:

{"topic":"mention_notify","value":"\u0000\u0000\u0000\u0000\u0001\u000ementionP[{\"user_id\": \"2\", \"user_name\": \"test2\"}]\n'166'Ha7a7e8d4-8751-11e5-81be-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\u000eclothes\fupdateH05c79127-6001-43ec-91c5-032783e973d3","offset":262,"partition":0,"key":-1}

This is my JSON.stringify(message) for case is error:

{"topic":"mention_notify","value":"\u0000\u0000\u0000\u0000\u0001\u000emention�\u0001[{\"user_id\": \"2\", \"user_name\": \"test2\"}, {\"user_id\": \"3\", \"user_name\": \"test3\"}]\n'166'H6a4bc174-8750-11e5-95d2-56847afe9799H65c6074e-b83c-4379-837b-315c190acd35\u000eclothes\fupdateH05c79127-6001-43ec-91c5-032783e973d3","offset":261,"partition":0,"key":-1}

Note: Because I have many schema's version, so I used the first 5 bytes header (\u0000\u0000\u0000\u0000\u0001) in message.value to store schema_id. So before you can decode message.value, You need to slice bytes array of message.value.

Currently, I am decoding by:

var type = avsc.parse(schema);
buffer = buffer.slice(5); // cut header byte
var obj = type.fromBuffer(buffer);

from avsc.

Thanks for the extra details.

Your second message is corrupt. If you look at the bytes, the 9-th character (�) is different from what it should be (it represents unicode code-point \ufffd, encoded as \xef\xbf\xbd which is different from \xa0 in the bytes you copied above).

This is not a problem with avsc since the bytes are corrupted upstream. Maybe some encoding issue in the Kafka library you are using.

from avsc.

Ok, thank you very much for your quickly supporting. :). I have seen my message is corrupt.

from avsc.