MQTT-Explorer 0.4.0-beta4 SparkplugB decode some strings as "{}" about mqtt-explorer HOT 13 CLOSED

Thank you, I consider this rather critical.
Maybe we can use part of this implementation #493 to make the decoding customizable.

I believe currently we run a decoder chain which is kind of broken

tryToDecodeAsJson
else tryToDecodeAsSparkplugB
else ....

from mqtt-explorer.

I lack deeper understanding of protobuf decoding, maybe there is a way to inspect the payload to determine if it's sparkplugb.

As someone that has knowledge about Sparkplug but not about JS/TS, I would suggest to inspect the topic structure to detemine if the message is sparkplugb. In my opinion, relying only on the message payload is a bad idea. The sparkplug specifikation can be found here: https://www.eclipse.org/tahu/spec/sparkplug_spec.pdf. The chapter 4.1 is describing the topic structure:

All MQTT clients using the Sparkplug specification MUST use the following topic namespace structure:

namespace/group_id/message_type/edge_node_id/[device_id]

The namespace element must be spBv1.0.

from mqtt-explorer.

Just for the record, in our application we use custom protobuf messages and the wrongly decoded messages we are seeing are not empty but with random invalid data, as shown below.

Hope @jst-mbs proposal and @bj00rn implementation fixes this issue, thanks!

Payload 26 bytes
b'*\x18\x1a\x16\x08\x0e\xf2\x01\x11\x08\x01\x12\r\x08\x03\x10\x01\x18\x02 \x02HUP\xf0.'
Decoded Sparkplug(?)
{"body":{"0":26,"1":22,"2":8,"3":14,"4":242,"5":1,"6":17,"7":8,"8":1,"9":18,"10":13,"11":8,"12":3,"13":16,"14":1,"15":24,"16":2,"17":32,"18":2,"19":72,"20":85,"21":80,"22":240,"23":46}}

Payload 36 bytes
b'\x12\x0fwmod-din-v0.08b*\x11f/wmod-din-v0.08b'
Decoded Sparkplug(?)
{"metrics":[{"isHistorical":true,"isNull":true,"doubleValue":3.0644533518218016e-75,"booleanValue":true}],"body":{"0":102,"1":47,"2":119,"3":109,"4":111,"5":100,"6":45,"7":100,"8":105,"9":110,"10":45,"11":118,"12":48,"13":46,"14":48,"15":56,"16":98}}

from mqtt-explorer.

Just for the record, in our application we use custom protobuf messages and the wrongly decoded messages we are seeing are not empty but with random invalid data, as shown below.

Hope @jst-mbs proposal and @bj00rn implementation fixes this issue, thanks!

Payload 26 bytes b'*\x18\x1a\x16\x08\x0e\xf2\x01\x11\x08\x01\x12\r\x08\x03\x10\x01\x18\x02 \x02HUP\xf0.' Decoded Sparkplug(?) {"body":{"0":26,"1":22,"2":8,"3":14,"4":242,"5":1,"6":17,"7":8,"8":1,"9":18,"10":13,"11":8,"12":3,"13":16,"14":1,"15":24,"16":2,"17":32,"18":2,"19":72,"20":85,"21":80,"22":240,"23":46}}

Payload 36 bytes b'\x12\x0fwmod-din-v0.08b*\x11f/wmod-din-v0.08b' Decoded Sparkplug(?) {"metrics":[{"isHistorical":true,"isNull":true,"doubleValue":3.0644533518218016e-75,"booleanValue":true}],"body":{"0":102,"1":47,"2":119,"3":109,"4":111,"5":100,"6":45,"7":100,"8":105,"9":110,"10":45,"11":118,"12":48,"13":46,"14":48,"15":56,"16":98}}

Thanks, there is clearly a bug with sparkplug decoding. Hopefully we'll get a new beta sorted soon that you can try.

from mqtt-explorer.

5-character strings starting with "5" are what I noticed. So "50000" or "50.00" but not "5000"

from mqtt-explorer.

True, did some more tests and all the strings that you mentioned turned into {} in MQTT-Explorer. I can't count 😞

from mqtt-explorer.

@thomasnordquist this is a release stopper i guess, sparkplugb decoding "detection" seems to be flaky..

The decoder choice logic is pretty naive at the moment it seems, it simply tries to decode the payload as sparkplug and if the result is !== undefined we return it. Any suggestion on how to improve the logic is welcome.

A naive solution might be, to add a check to see if Object.keys(message).length === 0 here, and return undefined here? I lack deeper understanding of protobuf decoding, maybe there is a way to inspect the payload to determine if it's sparkplugb.

from mqtt-explorer.

Tried to run some more tests, here's what i found:
Strings starting with any of the following characters: 'e', 'E', 'u', 'U', 'm', 'M', 5, with a length that is a multiple of 5, will be decoded as {}.

from mqtt-explorer.

Thank you, I consider this rather critical. Maybe we can use part of this implementation #493 to make the decoding customizable.

I believe currently we run a decoder chain which is kind of broken

tryToDecodeAsJson else tryToDecodeAsSparkplugB else ....

Thank you, I consider this rather critical. Maybe we can use part of this implementation #493 to make the decoding customizable.

I believe currently we run a decoder chain which is kind of broken

tryToDecodeAsJson else tryToDecodeAsSparkplugB else ....

Would be awesome to prepare for image support to while we are at it.
#734

from mqtt-explorer.

I think @jst-mbs has a pretty good point.
spBv1.0 will always be in the topic of the message. It is at least a good way to double check if decoding might need to be used.
Though in some cases people send uncompressed messages to the spBv1.0 namespace. For example i do not know if all STATE messages in SparkplugB are send protobuf compressed.

from mqtt-explorer.

Though in some cases people send uncompressed messages to the spBv1.0 namespace. For example i do not know if all STATE messages in SparkplugB are send protobuf compressed.

You are right: The state messages of the 'Host application' is JSON encoded, see chapter '4.2 Sparkplug Host Application'.

from mqtt-explorer.

Sidenote, there is a package sparkplug-payload that can do decoding/encoding. Might be worth looking into instead of rolling own protobuf decoding.

from mqtt-explorer.

I think @jst-mbs has a pretty good point. spBv1.0 will always be in the topic of the message. It is at least a good way to double check if decoding might need to be used. Though in some cases people send uncompressed messages to the spBv1.0 namespace. For example i do not know if all STATE messages in SparkplugB are send protobuf compressed.

@tobehn @jst-mbs have a look at #793, Will (topic.match(/spBv1\.0\/[^/]+\/(DDATA|NDATA|NCMD|DCMD|NBIRTH|DBIRTH|NDEATH|DDEATH\/[^/]+\/)/u)) regexp suffice to identify encoded sparkplug payloads?

from mqtt-explorer.