socketio / engine.io-parser Goto Github PK

It appears since 2.0.1, multi-byte characters are breaking in browser clients.

In practice, we have an application that happened to send a JSON message containing a no-break space (U+00A0) somewhere, and the browser client received the message with the final } cut-off.

With a normal install of [email protected], zuul tests also fail there. When the engine.io-parser dep is then replaced with 2.0.0 (by simply swapping the directory in node_modules), tests succeed again.

All tests seem to work fine in Node.js.

(macOS 10.12.3, Chrome 56.0.2924.87. Same results in Firefox 52.0.)

I'm attempting to bundle the socket.io client with webpack 5 and yarn 2. I believe these have more strict dependency checks which would explain why this issue hasn't been opened for webpack 4 and/or npm users.

Webpack is giving the error

ERROR in ../../../.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/decodePacket.browser.js 7:18-47
Module not found: Error: engine.io-parser tried to access base64-arraybuffer, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

Required package: base64-arraybuffer (via "base64-arraybuffer")
Required by: engine.io-parser@npm:4.0.1 (via /home/alex/GitProjects/tools/webpack-hot-module-replacement/.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/)
 @ ../../../.yarn/cache/engine.io-parser-npm-4.0.1-6bdb879e8a-3b71ef8b5a.zip/node_modules/engine.io-parser/lib/index.js 2:21-46
 @ ../../../.yarn/cache/engine.io-client-npm-4.0.2-9adb255344-af3880585b.zip/node_modules/engine.io-client/lib/index.js 14:24-51
 @ ../../../.yarn/cache/socket.io-client-npm-3.0.3-512b232ec1-8bb891554f.zip/node_modules/socket.io-client/build/manager.js 4:12-39
 @ ../../../.yarn/cache/socket.io-client-npm-3.0.3-512b232ec1-8bb891554f.zip/node_modules/socket.io-client/build/index.js 5:18-38 69:16-36
 @ ../../@skoville/webpack-hmr/client/universal/default-socket-io-accessor~/build/socket-io-accessor.js 13:27-54
 @ ../../@skoville/webpack-hmr/client/universal/default-socket-io-accessor~/build/package.js 13:13-44
 @ ../../@skoville/webpack-hmr/client/web/default~/build/entry.js 13:66-142

seems like this is accurate since it's being required here

is there any reason why this isn't included in dependencies and is instead only in devDependencies?

In case there's a native component to this library which causes it to fail compiling for some users, consider following the approach that was eventually taken here websockets/ws#1220, where we would move base64-arraybuffer from devDependencies to optionalDependencies.

I can open a PR if this approach (move to optional dependencies) works for you.

Can you add a license to this please?

Readme refers to engine.io-server which does not exist.

We should come up with some baseline contributing guidelines for this repo (and all the other socket.io related repos) So that when people create issues or PRs they get some guidance on what to do and where to post issues.

This issue can be a starting discussion on what to include.

If the packet that should be decoded isn't a string (in decodePacket), the parser runs into the following problem in browsers:

Excerpt from exports.decodePacket:

var asArray = new Uint8Array(data);
var type = asArray[0];
var rest = sliceBuffer(data, 1);
...
return { type: packetslist[type], data: rest };

The problem is that 'type' will contain an useless number that refers to nothing contained in packetlist[type]. Making the returned object look like:
{ type: undefined, data: ArrayBuffer }... which makes everything that follows fail.
This only happens when using XHR.

exports.decodePacket is called from exports.decodePayloadAsBinary.

This issue is originated from nwjs/nw.js#5034. When using Socket.io in latest NW.js, binary data cannot be handles properly.

The root cause is that engine.io-parser doesn't handle ArrayBuffer properly. See lib/index.js:L148-L157.

  if (binaryType === 'arraybuffer') { // <-- [1]
    var type = data[0]; // <-- [2]
    var intArray = new Uint8Array(data.length - 1);
    for (var i = 1; i < data.length; i++) {
      intArray[i - 1] = data[i];
    }
    return { type: packetslist[type], data: intArray.buffer };
  }
  var type = data[0]; // <-- [3] 
  return { type: packetslist[type], data: data.slice(1) };

1. In Node.js or NW.js, binaryType is always null. I assume it's a bug as well.
2. ArrayBuffer cannot be directly read using array styled syntax data[0]. One should construct a TypedArray form the buffer, such as Uint8Array, and read the data inside.
3. I guess here it assumes data could only be a Node Buffer. However, in NW.js data can be of type ArrayBuffer from browser builtin WebSocket.

/node_modules/engine.io-parser/build/esm/index.d.ts (3,57): Relative import paths need explicit file extensions in EcmaScript imports when '--moduleResolution' is 'node12' or 'nodenext'. Did you mean './commons.js'?

Extensions ARE REQUIRED in esm type modules.

maybe i'm missing something, why does browser.js export protocol 2, but index.js exports protocol 3?

        it('should encode/decode empty payloads', function () {
          encPayload([], function(data) {
            decPayload(data,
              function (packet, index, total) {
                expect(packet.type).to.eql('open');
                var isLast = index + 1 == total;
                expect(isLast).to.eql(true);
              });
          });
        });

This test did not pass because it did not execute the assertion

In an effort to get this noticed a bit more, I am creating an issue to request the release of a new version to npm that includes the merged PR #55 so that Socket.io will work with React Native.

A few of us have been trying in vain to get an ETA on a new release but have not heard anything since April so this is a gentle effort to get a bit more love on this issue. Excuse my forwardness with this issue in advance! 😄

CC @rauchg

Hello!

I've encountered a pretty weird issue for Android Honeycomb.

The problem comes from the fact that there is specific handling for "android" devices as per this article.

It assumes that "android" can handle sending things as ArrayBuffer in here however this only works on android >= 4.

The check that is done for "Android" here works for 2.* because those versions doesn't identify as "Android" but the 3.* versions does.
The problem then arises here because Uint8Array isn't available until Android 4.

I'm currently testing on a Samsung Galaxy Tab 7.7 on Android 3.2 and will try to find some way to either fix this or work around it.

I am working with binary data in the browser and – as far as I can see – I can receive binary data which comes as an object { base64: true, data: base64Data }.

BUT there is no way to do the same to send binary data. Sending binary data strictly requires an ArrayBuffer which is not available in older browsers.

I propose to introduce the same concept for sending binary data as for receiving binary data.
Sending an object { base64: true, data: base64Data } should send the data with binary flag to the wire.
This would allow older browsers to work binary. No need to modify server code nor client code. Only the
engine.io-parser needs a small modification.

I am working on a patch, if interested I will provide a pull request.

My bundle throw this error :
Uncaught Error: Cannot find module "engine.io-parser/keys" from "engine.io-parser"

I use Brunch.io and Babel.

Some extract from my bundle :

require.register('engine.io-parser', function(exports,req,module){
      var require = __makeRequire((function(n) { return req(n.replace('./', 'engine.io-parser//')); }), {}, 'engine.io-parser');
      (function(exports,require,module) {
        /**
 * Module dependencies.
 */

var keys = require('./keys');

if I replace here : require('./keys') by require('./lib/keys') that's work !

Or I need to replace this register by require.register('engine.io-parser/keys',...

require.register('engine.io-parser/lib/keys', function(exports,req,module){
      var require = __makeRequire((req), {}, 'engine.io-parser');
      (function(exports,require,module) {

/**
 * Gets the keys for an object.
 *
 * @return {Array} keys
 * @api private
 */

module.exports = Object.keys || function keys (obj){
  var arr = [];
  var has = Object.prototype.hasOwnProperty;

  for (var i in obj) {
    if (has.call(obj, i)) {
      arr.push(i);
    }
  }
  return arr;
};

      })(exports,require,module);
    });

How I can avoid this problem ?

I would submit a PR, but none of the dependencies seem to support component. Is component support planned for engine.io?

Currently, the encodePacket function always returns the result in its callback argument. I understand that's likely done to prevent blocking the thread in some situations.

I'd like to propose exporting a new function that'd be Packet -> string.

function encodePacketInline(
  packet: Packet,
  supportsBinary: boolean
): string

The name is illustrational. I'm open to suggestions.

I'd like to use such a function in my custom message parser:

client.on('message', (event) => {
  server.send(
    encodePacketInline({
      type: 'message',
      data: encoder.encode({ type: 2, data: ['message', 'payload'], nsp: '/' })
    })
  )
})

This is intentionally simplified example of my setup. In reality, I have a transport.parser instance that automatically wraps outgoing events to the server in transport.parser.encode(data) before providing the payload to the MessageEvent. This is also code that the consumer of my library is intended to write (their own parser).

How?

The logic for encoding is already synchronous:

I simply propose moving it to a new standalone public function.

Describe the bug
An error is thrown while running tsc in my typescript project using engine.io-parser on version v5.0.3.

Run cd backend && tsc
  cd backend && tsc
  shell: /usr/bin/bash -e {0}
Error: ../node_modules/engine.io-parser/build/esm/commons.d.ts(7,81): error TS2304: Cannot find name 'Blob'.
Error: Process completed with exit code 2.

To Reproduce

Please fill the following code example:

not applicable

engine.io-parser version: v5.0.3

Expected behavior
tsc runs without any issues as it does on version v5.0.0

Platform:

• Software: tsc
• OS: ubuntu

Additional context
The logs of the workflows can be found here https://github.com/jojomatik/blockcluster/runs/6233313889 (v5.0.0) and here https://github.com/jojomatik/blockcluster/runs/6233323368 (v5.0.3). The respective commits are linked from within the workflows.

It might be possible that the problem has to be solved somewhere else in the socket.io dependency chain but the error message lead me here. Here is the link to the commit that fails jojomatik/blockcluster@5cecdce. It shows all upgraded dependencies of that might cause the issue.

Hi, im getting an error when trying to install the version 5.2.1.

Looks like that version is not published in the npm repository yet. The latest published is 5.2.0.

relevant to socket.io issue socketio/socket.io-client#1166

published webmodules/blob#14, but not getting any response (+last commit to blob was in 2015)

@darrachequesne @rauchg @nkzawa

utf-8 strings worked just fine before. Now they are broken.
Characters that are represented as multiple bytes now get split into individual characters.

http://updates.html5rocks.com/2014/08/Easier-ArrayBuffer---String-conversion-with-the-Encoding-API

Not sure if this is relevant to the parser or our needs, but I figured I would bring it up to at least make sure everyone was aware of it. Maybe we are already using it, in that case...awesome.

I just stumbled upon a weird error:
I am currently working on a typescript project which is using socket.io in its backend. (nodejs)

When trying to build it, I got this error:

node_modules/engine.io/build/engine.io.d.ts:3:25 - error TS7016: Could not find a declaration file for module 'engine.io-parser'. 'C:/Users/Lenovo/Code/wehub_development/wehub/node_modules/engine.io-parser/build/cjs/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/engine.io-parser` if it exists or add a new declaration (.d.ts) file containing `declare module 'engine.io-parser';`

3 import * as parser from "engine.io-parser";
                          ~~~~~~~~~~~~~~~~~~

node_modules/engine.io/build/transport.d.ts:4:24 - error TS7016: Could not find a declaration file for module 'engine.io-parser'. 'C:/Users/Lenovo/Code/wehub_development/wehub/node_modules/engine.io-parser/build/cjs/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/engine.io-parser` if it exists or add a new declaration (.d.ts) file containing `declare module 'engine.io-parser';`

4 import { Packet } from "engine.io-parser";
                         ~~~~~~~~~~~~~~~~~~


Found 2 errors in 2 files.

Errors  Files
     1  node_modules/engine.io/build/engine.io.d.ts:3
     1  node_modules/engine.io/build/transport.d.ts:4

When i tried copying the .d.ts files in your package from the esm folder into the cjs folder, it suddenly worked without complaint.

Could it be possible that you forgot to add the .d.ts files to the cjs export of your package or are my tsconfig configurations incorrect in some way?

Thank you! :)

engine.io-parser includes various fixes since the last release 1.1.0...master (I need the PhantomJS fixes) Is there a chance to get an updated version published to npm?

The 5.0.5 release hosted at https://www.npmjs.com/package/engine.io-parser appears to be missing the changes from 351ba82
in build/esm/encodePacket.browser.js:37
callback("b" + content);
however, build/cjs/encodePacket.browser.js:39 as the correct line:
callback("b" + (content || ""));

parser.encodePacket return 8192 bytes on binary message in browser

I'm a little confused at line 106 of lib/browser.js:

  if (global.ArrayBuffer && data instanceof ArrayBuffer) {
    return encodeArrayBuffer(packet, supportsBinary, callback);
**  } else if (Blob && data instanceof global.Blob) { **
    return encodeBlob(packet, supportsBinary, callback);
  }

The Blob comes as a local variable from require('blob') while global.Blob is not. Is that on purpose?

lib/index.js requires "after" module. But where can I find it?

thanks.

There have been a couple of commits and one in particular is quiet important - so i'd like to ask if you could publish a new version on npm soon?

Apologies if this is the wrong repo for this.

I have noticed that when I pass binary data to socket.io's emit function, that data is transformed in place. For example, File objects become ArrayBuffer objects. It makes sense why this needs to happen, but would it be possible to generate a modified copy of the data for transmission instead of mutating it in place?

As it is now, I find myself defensively deep cloning anything I pass to socket.io which is actually kind of hard to do well (JSON encoding and parsing leads to bugs, structured clone would be great but isn't exposed by browsers, circular references create issues, etc).

Thanks for the great software, btw.

A few questions:

• why are there so many public methods? Shouldn't encodePayload and decodePayload be enough?

Currently, it seems that:
- encodePayload/decodePayload is used for polling transport
- encodePacket/decodePacket is used for websocket transport

Quoting socketio/engine.io-client#140 (comment):

encodePacket is more efficient in this case so we can leverage the WS framing

Is that really more efficient? Shouldn't we rather concatenate two packets instead of sending them one by one? (I think that's why, currently, sending a buffer with socket.io results in two frames: socketio/socket.io#2444)

• according to the following comment, I understand that utf8 is needed for mixing binary/string content

The reason we are using utf8.js is that when encoding mixed binary and string contents, we need to support multibyte strings. It is easiest to encode the multibyte characters so that we can do a transformation to a binary representation one byte at a time.

Yet it seems that, since encodePayload always call encodePacket with utf8encode to true, the strings transmitted over polling transport are UTF8-encoded twice (socketio/engine.io#315):

https://github.com/socketio/engine.io-parser/blob/1.3.2/lib/index.js#L229

exports.encodePayload = function (packets, supportsBinary, callback) {
  ...
  if (supportsBinary) {
    return exports.encodePayloadAsBinary(packets, callback);
    // which call encodePacket with utf8encode = true => OK
  }
  ...
  function encodeOne(packet, doneCallback) {
    exports.encodePacket(packet, supportsBinary, true, function(message) {
      doneCallback(null, setLengthHeader(message));
    });
    // since the payload is not encoded as binary, shouldn't utf8encode be set to false here?
  }
}

cc @calzoneman @rase- @rauchg

same Readme style as engine.io-client