davalapar / what-the-pack Goto Github PK

https://appspector.com/blog/how-to-improve-messagepack-javascript-parsing-speed-by-2-6-times

Additional Optional Initialization Options

• ignoreUndefined : boolean ignores object properties with undefined value
• ignoreEmptyObjects : boolean ignores empty objects
• ignoreEmptyArrays : boolean ignores empty arrays

References

• kawanet/msgpack-lite#71

Overview

• Transforms class instances into lower-level data structures upon encode, then back to class instances upon decode
• On encoding, class instances are identified / recognized by using for-loop + instanceof
• Internally automates the assigning of application-specific extension types (0 to 127)
• Two variants:
  • class instances - when a class instance is passed directly to our encoder function
  • nested class instances - when a class instance is nested within an object or array passed to our encoder function
• Uses an internal boolean, to check if custom extension types are used
• Tests

const { extend } = MessagePack.initialize(2 ** 24);

class Vector {
  constructor(x, y) {
    this.x = x;
    this.y = y;
  }
}

// transforms class instance to msgpack-compatible data
const encodeVector = (vector) => {
  return [vector.x, vector.y];
};

// transforms data into new class instance
const decodeVector = (data) => {
  return new Vector(data[0], data[1]);
};

extend(Vector, encodeVector, decodeVector);

// `extend` parameters:
// - class (we use instanceof for recognition) : `Object`
// - encodeTransformFunction : `Function`
// - decodeTransformFunction : `Function`

Algo

• For each classes
  • if value instanceof class
    • apply encodeTransformFunction
      • return encoded

Another use case:

• A User class instance in client-side can be automatically encoded only using its id property
• In the server-side its received id property can be automatically transformed back into a User class instance
• User -> id -> encode -> transport -> decode -> id -> User

References

• darrachequesne/notepack#18
• https://github.com/kawanet/msgpack-lite#custom-extension-types-codecs
• https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types

Breakdown

• Uses fixext and ext format

fixext 1 stores an integer and a byte array whose length is 1 byte
+--------+--------+--------+
|  0xd4  |  type  |  data  |
+--------+--------+--------+

fixext 2 stores an integer and a byte array whose length is 2 bytes
+--------+--------+--------+--------+
|  0xd5  |  type  |       data      |
+--------+--------+--------+--------+

fixext 4 stores an integer and a byte array whose length is 4 bytes
+--------+--------+--------+--------+--------+--------+
|  0xd6  |  type  |                data               |
+--------+--------+--------+--------+--------+--------+

fixext 8 stores an integer and a byte array whose length is 8 bytes
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
|  0xd7  |  type  |                                  data                                 |
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+

fixext 16 stores an integer and a byte array whose length is 16 bytes
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
|  0xd8  |  type  |                                  data                                  
+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+
+--------+--------+--------+--------+--------+--------+--------+--------+
                              data (cont.)                              |
+--------+--------+--------+--------+--------+--------+--------+--------+

ext 8 stores an integer and a byte array whose length is upto (2^8)-1 bytes:
+--------+--------+--------+========+
|  0xc7  |XXXXXXXX|  type  |  data  |
+--------+--------+--------+========+

ext 16 stores an integer and a byte array whose length is upto (2^16)-1 bytes:
+--------+--------+--------+--------+========+
|  0xc8  |YYYYYYYY|YYYYYYYY|  type  |  data  |
+--------+--------+--------+--------+========+

ext 32 stores an integer and a byte array whose length is upto (2^32)-1 bytes:
+--------+--------+--------+--------+--------+--------+========+
|  0xc9  |ZZZZZZZZ|ZZZZZZZZ|ZZZZZZZZ|ZZZZZZZZ|  type  |  data  |
+--------+--------+--------+--------+--------+--------+========+

where
* XXXXXXXX is a 8-bit unsigned integer which represents N
* YYYYYYYY_YYYYYYYY is a 16-bit big-endian unsigned integer which represents N
* ZZZZZZZZ_ZZZZZZZZ_ZZZZZZZZ_ZZZZZZZZ is a big-endian 32-bit unsigned integer which represents N
* N is a length of data
* type is a signed 8-bit signed integer
* type < 0 is reserved for future extension including 2-byte type information

• https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family

For implementation correctness, the byteOffset must be considered.

https://stackoverflow.com/a/54646864

• kawanet/msgpack-lite#87 (comment)
• https://github.com/ygoe/msgpack.js

Usage of Buffer isn't really necessary for browsers.

We just want simpler functions that let us encode and decode data to/from UInt8Array instances.

Looking at https://github.com/feross/buffer/blob/master/index.js the implemetation actually has internal checks and assertions which could be skipped, which could speed up our encoding and decoding process in the browser.

If this ever succeeds, we can even use these UInt8Array functions on the node implementation for speed, since UInt8Arrays can be converted to Buffer instances anyways.

Notes:

• This would also require stricter type checks, such as isPlainObject implementation from Lodash.
• fetch requests accepts UInt8Array instances directly as ArrayBufferView, XMLHttpRequest also accepts UInt8Array as BufferSource
  • https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch#Body
  • https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/send
• fetch response body has .arrayBuffer() method which returns an ArrayBuffer
• ArrayBuffer to UInt8Array : new UInt8Array(arrayBuffer);
• UInt8Array to ArrayBuffer : ( new UInt8Array() ).buffer
  • byteOffset must also always be considered: https://stackoverflow.com/a/54646864

References

• https://github.com/feross/buffer/blob/master/index.js
• https://github.com/feross/ieee754/blob/master/index.js

The string "WALL·E – Typeset in the Future" decodes as "W" and causes further decoding problems if it is contained by an object with other data in it.

This appears to be due to the fixstr condition in index.js, which checks if the number of characters is less than 32, rather than the number of bytes as defined by the MessagePack spec and implemented in decode(). "WALL·E – Typeset in the Future" is 30 characters long but uses 33 bytes in utf8.

internalEncode and internalDecode must be wrapped in ONE try-catch block to catch encoding and decoding errors. Doing the try-catch block within (instead outside) will nest it recursively and will be costly.

Overview

• Document usage in README
• Provide simple wrapper functions that automatically encodes parameters, and decodes responses

const { encode, decode } = //...
const response = await fetch('https://example.com', {
  method: 'POST',
  body: JSON.stringify({foo: true}),
  headers: { 'content-type': 'application/octet-stream' }
});
if (!response.ok) {
  throw new HTTPError('Fetch error:', response.statusText);
}
const parsed = decode(await response.arrayBuffer());
console.log(parsed);
//=> `{ asd: '123'}`

const headers = { 'content-type': 'application/octet-stream' };
const parsed = decode(await ky.post('https://example.com', {
  headers,
  body: encode({ some: 'data' }),
}).arrayBuffer());

References

• https://www.npmjs.com/package/ky
• https://developer.mozilla.org/en-US/docs/Web/API/Body/arrayBuffer

When used with websocket clients with .binaryType = 'arraybuffer', messages received will be instances of ArrayBuffer. .decode function on the other hand always expects a Buffer instance, hence what should be called is .decode(Buffer.from(MessageEvent.data)).

Could be user error, but using register to try to use dictionary results in the same output as not using it. I'm using exactly as per docs example.