This error randomly occurs for 1-2 times a day for one particular instagram account.<b

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

"Expected flags 2 but got X" error crashes node,about nerixyz/instagram_mqtt

Comments (51)

Nerixyz commented on August 11, 2024 1

Can confirm Error: Unknown type: 10 here as well even when running the live example

This should be fixed in the latest commit.

from instagram_mqtt.

Nerixyz commented on August 11, 2024

There's a new version, it may fix this. But I guess this is some concurrency error. Unsubscribe-packets are not sent by the MQTT broker, so there has to be something wrong with the parsing beforehand.

from instagram_mqtt.

Ektaros commented on August 11, 2024

@Nerixyz, got one instance of this problem on 0.2.3 version. Not sure if all the others to come will be the same.
Now there is no "Expected flags 2 but got X" error instead I got this one

"error": {
    "name": "UnexpectedPacketError",
    "message": "PublishReceivedPacket",
    "stack": "UnexpectedPacketError: PublishReceivedPacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:334:40)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:25)"
  }

And again followed by "Z_DATA_ERROR" and node crash.

from instagram_mqtt.

Nerixyz commented on August 11, 2024

And again followed by "Z_DATA_ERROR" and node crash.

Does this happen immediately or after a "request"?
And could you show the portion of your code before the error?

from instagram_mqtt.

Ektaros commented on August 11, 2024

Right after i get any kind of error in realtime.on('error') I call realtime.disconnect() so i guess after this request where is a "Z_DATA_ERROR" (all logs unfortunatly have thie same timestamp)
Yeah sure here it is:

this.ig.realtime.on('error', error => {
        if (!this.disconnectExpected) this.resetRealtime(channel);
        Log.error(error);
      });
async resetRealtime(channel) {
    if (!this.resetRealtimePlanned) {
      this.resetRealtimePlanned = true;
      await this.stopRealtime();
      await timeout(60 * 1000 * 2 * (1 + Math.random()));
      await this.initRealtime(channel, false);
      this.resetRealtimePlanned = false;
    }
  }
async stopRealtime() {
    if (this.realtimeIsActive) {
      await this.ig.realtime.disconnect();
      this.realtimeIsActive = false;
    }
  }

from instagram_mqtt.

Ektaros commented on August 11, 2024

also using 0.2.3 I faced this error and if got looped for some reason and spammed over 50k events in several minetes so i moved back to 0.2.1 it seem more stable for now.
Then the error first occured I called disconnect (as always). Got .on('disconnect') event and after that it started spamming
here is the error I was getting

 "error": {
    "name": "InvalidDirectionError",
    "message": "Invalid direction: read",
    "stack": "InvalidDirectionError: Invalid direction: read
    at UnsubscribeRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/unsubscribe.request.packet.ts:30:15)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:100:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:99:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:288:29)"
  }

from instagram_mqtt.

Nerixyz commented on August 11, 2024

I've updated the library to only tryunzip the content. You can test it using npm i git+https://[email protected]/Nerixyz/instagram_mqtt.git

from instagram_mqtt.

Ektaros commented on August 11, 2024

Thanks a lot but i could not test this today. Planning to do it at most in 2 days.

from instagram_mqtt.

Ektaros commented on August 11, 2024

So i finally got my hands on it. Could not test whether it fixed Z_DATA_ERROR cause every time there is a direct messege i get this error in realtime.on(warning) handler:

error: {
    name: 'Error',
    message: 'Unknown type: 10',
    stack: 'Error: Unknown type: 10\n' +
      '    at getReadFunction (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\thrift\\thrift.reading.ts:123:19)\n' +
      '    at Object.thriftRead (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\thrift\\thrift.reading.ts:25:23)\n' +
      '    at SafeSubscriber._next (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\realtime\\realtime.client.ts:179:53)\n' +
      '    at processTicksAndRejections (internal/process/task_queues.js:93:5)'
  }

So cant really test it in production at this stage (the error that crashes node happens only with one particular channel)

from instagram_mqtt.

Nerixyz commented on August 11, 2024

So i finally got my hands on it. Could not test whether it fixed Z_DATA_ERROR cause every time there is a direct messege i get this error in realtime.on(warning) handler:

error: {
    name: 'Error',
    message: 'Unknown type: 10',
    stack: 'Error: Unknown type: 10\n' +
      '    at getReadFunction (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\thrift\\thrift.reading.ts:123:19)\n' +
      '    at Object.thriftRead (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\thrift\\thrift.reading.ts:25:23)\n' +
      '    at SafeSubscriber._next (C:\\Users\\mi\\Documents\\repos\\channels-manager-instagram\\node_modules\\instagram_mqtt\\src\\realtime\\realtime.client.ts:179:53)\n' +
      '    at processTicksAndRejections (internal/process/task_queues.js:93:5)'
  }

So cant really test it in production at this stage (the error that crashes node happens only with one particular channel)

This is probably some out of sync error. For now, you can create a new instance of the realtime client whenever you disconnect.
For example:

await ig.realtime.disconnect()
ig.realtime = new RealtimeClient(ig);

from instagram_mqtt.

Ektaros commented on August 11, 2024

Oh I actually never thought about that. This approach made code overall a little bit more stable.
But unfortunately it didn't fix nor the issue above with "Unknown type: 10" on current github perository version nor z_data_error.

from instagram_mqtt.

Ristovski commented on August 11, 2024

Can confirm Error: Unknown type: 10 here as well even when running the live example

from instagram_mqtt.

Ektaros commented on August 11, 2024

@Nerixyz got a proper call stack of that error

"error": {
    "errno": -3,
    "code": "Z_DATA_ERROR",
    "name": "Error",
    "message": "invalid distance too far back",
    "stack": "Error: invalid distance too far back\n    at Zlib.zlibOnError [as onerror] (zlib.js:170:17)
    at processChunkSync (zlib.js:416:12)\n    at zlibBufferSync (zlib.js:156:12)
    at Object.syncBufferWrapper [as unzipSync] (zlib.js:755:14)
    at transformer (/usr/backend/node_modules/instagram_mqtt/src/realtime/realtime.client.ts:204:72)
    at MapSubscriber.project (/usr/backend/node_modules/instagram_mqtt/src/mqtt/mqtt.client.ts:185:62)
    at MapSubscriber._next (/usr/backend/node_modules/rxjs/src/internal/operators/map.ts:84:29)
    at MapSubscriber.Subscriber.next (/usr/backend/node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at FilterSubscriber._next (/usr/backend/node_modules/rxjs/src/internal/operators/filter.ts:101:24)
    at FilterSubscriber.Subscriber.next (/usr/backend/node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at Subject.next (/usr/backend/node_modules/rxjs/src/internal/Subject.ts:70:17)
    at /usr/backend/node_modules/instagram_mqtt/src/mqtt/mqtt.client.ts:269:46"

So this call is responsible for the error instagram_mqtt/src/realtime/realtime.client.ts:204:72)
And in was not changed in last commit so mb this will help.

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Forgot to update unzipSync, sould be up now.

from instagram_mqtt.

Ektaros commented on August 11, 2024

@Nerixyz thanks for the update but I am still back with new errors using 0.2.6 :)
They were from the same instagram account which is causing Z_data_error (yes it still does but another error crashed node i believe)
So there are the errors what ive got:
First in realtime.on('error')

"name": "Error",
    "message": "Invalid length",
    "stack": "Error: Invalid length
    at PingRequestPacket.readRemainingLength (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:92:23)
    at PingRequestPacket.read (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:57:14)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)"

then followed with this error in realtime.on('warning')

"name": "UnexpectedPacketError",
    "message": "PingResponsePacket",
    "stack": "UnexpectedPacketError: PingResponsePacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:337:32)
   at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:25)"

and this is the one that I believe crashed node cause it was caught by uncaughtException handler

"name": "SyntaxError",
    "message": "Unexpected token x in JSON at position 0",
    "stack": "SyntaxError: Unexpected token x in JSON at position 0\n    at JSON.parse (<anonymous>)
    at IrisParser.parseMessage (/usr/backend/node_modules/instagram_mqtt/src/realtime/parsers/iris.parser.ts:7:21)
    at transformer (/usr/backend/node_modules/instagram_mqtt/src/realtime/realtime.client.ts:203:45)
    at MapSubscriber.project (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:214:62)
    at MapSubscriber._next (/usr/backend/node_modules/rxjs/src/internal/operators/map.ts:84:29)
    at MapSubscriber.Subscriber.next (/usr/backend/node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at MapSubscriber._next (/usr/backend/node_modules/rxjs/src/internal/operators/map.ts:89:22)
    at MapSubscriber.Subscriber.next (/usr/backend/node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at FilterSubscriber._next (/usr/backend/node_modules/rxjs/src/internal/operators/filter.ts:101:24)
    at FilterSubscriber.Subscriber.next (/usr/backend/node_modules/rxjs/src/internal/Subscriber.ts:99:12)
    at Subject.next (/usr/backend/node_modules/rxjs/src/internal/Subject.ts:70:17)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:324:46"

z_data_error was also present but as an unhandledRejection not as uncaughtException as it was in the previous versions.
Hope this helps (unknown type 10 is gone btw)

from instagram_mqtt.

Nerixyz commented on August 11, 2024

@Ektaros the Json error shouldn't happen. Maybe I didn't publish the latest version.

from instagram_mqtt.

Ektaros commented on August 11, 2024

@Nerixyz I am sure that im using 0.2.6 version with isJson() but there are stiil crashes
Last one I got was this followed by Unexpected token x in JSON at position 0"

 "name": "UnexpectedPacketError",
    "message": "PublishReceivedPacket",
    "stack": "UnexpectedPacketError: PublishReceivedPacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:337:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:25)"

from instagram_mqtt.

Nerixyz commented on August 11, 2024

@Nerixyz I am sure that im using 0.2.6 version with isJson() but there are stiil crashes
Last one I got was this followed by Unexpected token x in JSON at position 0"

 "name": "UnexpectedPacketError",
    "message": "PublishReceivedPacket",
    "stack": "UnexpectedPacketError: PublishReceivedPacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:337:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:25)"

For unexpected packets I need the debug log (DEBUG=*). The unexpected token error shouldn't show as the data is always checked if it's a zipped payload.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Currently working on a workaround that will allow to debug particular instagram accounts. Hope to get some info soon.
There is also this type of error started apperaing at realtime('error') handler in the latest version. It too leads into JSON.parse() error and crashes node.

"name": "InvalidDirectionError",
    "message": "Invalid direction: read",
    "stack": "InvalidDirectionError: Invalid direction: read
    at SubscribeRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/subscribe.request.packet.ts:41:15)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)"

from instagram_mqtt.

Nerixyz commented on August 11, 2024

There is also this type of error started apperaing at realtime('error') handler in the latest version. It too leads into JSON.parse() error and crashes node.

There's no JSON.parse in your stacktrace. InvalidDirection errors are errors concerning MQTT, the packet is sent in the wrong direction.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Yes there is none. These errors preced "Unexpected token x in JSON at position 0" that makes node crash. JSON.parse error happens shortly after ~6k (I counted the logs) InvalidDirectionError events.
Then I get "InvalidDirectionError" it is comming repeatedly for a 2-3 minutes. Mb there is something to do right then I get the first one, to prevent this mqtt client from running anymore?

from instagram_mqtt.

Ektaros commented on August 11, 2024

Ill try to provide more info soon

from instagram_mqtt.

Ektaros commented on August 11, 2024

I noticed that the json error in question (u can find the stacktrace 8 messages above) is comming from IrisParser and isJson() is utilized only in GraphqlParser. Maybe IrisParser should use something same?

from instagram_mqtt.

Nerixyz commented on August 11, 2024

I noticed that the json error in question (u can find the stacktrace 8 messages above) is comming from IrisParser and isJson() is utilized only in GraphqlParser. Maybe IrisParser should use something same?

isJson() is obsolete. The parser should receive valid input. An x at the start (pos 0) indicates, the payload is compessed. However, the parser shouldn't receive compressed data. You can unzip the data if you want to. But your whole parser is out of sync.

Currently working on a workaround that will allow to debug particular instagram accounts.

You can also tell me what the account is doing/receiving.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Yeah Im working on that.

from instagram_mqtt.

Ektaros commented on August 11, 2024

@Nerixyz I got debug logs (Hooray!) they are huge cause InvalidDirectionError is comming continiosly and spams a huge ammont of logs
so here is some before this 10k error sequence

2020-03-05T15:35:29.966Z ig:request Requesting POST /api/v1/push/register/
2020-03-05T15:35:30.650Z ig:mqtt:realtime Connecting to realtime-broker...
2020-03-05T15:35:30.650Z ig:mqtt:realtime Overriding: 
2020-03-05T15:35:30.659Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Creating client
2020-03-05T15:35:30.660Z mqtt:client Connecting...
2020-03-05T15:35:30.676Z mqtt:client:packet Sent MQTToTConnectRequestPacket
2020-03-05T15:35:30.725Z mqtt:client:packet Received ConnectResponsePacket
2020-03-05T15:35:30.725Z mqtt:client Connected!
2020-03-05T15:35:30.726Z mqtt:client Starting keep-alive-ping {delay: 19.5}
2020-03-05T15:35:30.726Z ig:mqtt:realtime Connected. Checking initial subs.
2020-03-05T15:35:30.726Z ig:mqtt:realtime Subscribing with Skywalker to ig/u/v1/1577702934
2020-03-05T15:35:30.727Z ig:mqtt:realtime Iris Sub to: seqId: 88269, snapshot: 1583422530561
2020-03-05T15:35:30.728Z mqtt:client:packet Sent PublishRequestPacket id: 9 topic: 88
2020-03-05T15:35:30.729Z mqtt:client:packet Sent PublishRequestPacket id: 10 topic: 134
2020-03-05T15:35:30.765Z mqtt:client:packet Received PublishAckPacket id: 9
2020-03-05T15:35:30.801Z mqtt:client:packet Received PublishAckPacket id: 10
2020-03-05T15:35:30.910Z ig:request Requesting undefined /api/v1/direct_v2/pending_inbox/
2020-03-05T15:35:30.925Z mqtt:client:packet Received PublishRequestPacket topic: 135
2020-03-05T15:35:31.410Z ig:request Requesting POST /api/v1/direct_v2/threads/340282366841710300949128129498340013004/approve/
2020-03-05T15:35:31.415Z ig:request Requesting POST /api/v1/direct_v2/threads/340282366841710300949128130083369826061/approve/
2020-03-05T15:35:31.656Z ig:request Requesting undefined /api/v1/direct_v2/inbox/
2020-03-05T15:35:32.095Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid length
	Stack: Error: Invalid length
    at UnsubscribeResponsePacket.readRemainingLength (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:92:23)
    at UnsubscribeResponsePacket.read (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:57:14)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)
2020-03-05T15:35:32.130Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid direction: read
	Stack: InvalidDirectionError: Invalid direction: read
    at UnsubscribeRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/unsubscribe.request.packet.ts:30:15)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)

and after the "InvalidDirectionError" events chain

2020-03-05T15:36:49.066Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid direction: read
	Stack: InvalidDirectionError: Invalid direction: read
    at UnsubscribeRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/unsubscribe.request.packet.ts:30:15)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)
    at runNextTicks (internal/process/task_queues.js:63:3)
    at processImmediate (internal/timers.js:412:9)
    at process.topLevelDomainCallback (domain.js:130:23)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)
2020-03-05T15:36:49.157Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Disconnected.
2020-03-05T15:36:49.158Z mqtt:client:packet Sent DisconnectRequestPacket
2020-03-05T15:36:49.175Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: This socket has been ended by the other party (removed the stack)
2020-03-05T15:36:49.178Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid direction: read (removed the stack it is the same as the one above)
2020-03-05T15:36:49.180Z mqtt:client:packet Received PublishRequestPacket id: 8746 topic: 146
2020-03-05T15:36:49.180Z mqtt:client:packet Sent PublishAckPacket id: 8746
2020-03-05T15:36:49.181Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Error: PingRequestPacket
	Stack: UnexpectedPacketError: PingRequestPacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:337:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:25)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:94:5)
    at runNextTicks (internal/process/task_queues.js:63:3)
    at processImmediate (internal/timers.js:412:9)
    at process.topLevelDomainCallback (domain.js:130:23)
2020-03-05T15:36:49.192Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: This socket has been ended by the other party (removed the stack)
uncaughtException SyntaxError: Unexpected token x in JSON at position 0 (removed the stack could be seen above)

Last one crashes node and no more logs. I saved the logs preceding that I send in the first part so if needed I can provide.

from instagram_mqtt.

Nerixyz commented on August 11, 2024

2020-03-05T15:35:32.095Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid length
Stack: Error: Invalid length
at UnsubscribeResponsePacket.readRemainingLength (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:92:23)
at UnsubscribeResponsePacket.read (/usr/backend/node_modules/mqtts/src/mqtt.packet.ts:57:14)
at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)

This is the actual error. The packet is too long.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Is this request the cause?
2020-03-05T15:35:31.656Z ig:request Requesting undefined /api/v1/direct_v2/inbox/

from instagram_mqtt.

Nerixyz commented on August 11, 2024

This is the actual error. The packet is too long.

As of MQTT 3.1.1, the packet length is computed like this:

       multiplier = 1
       value = 0
       do
            encodedByte = 'next byte from stream'
            value += (encodedByte AND 127) * multiplier
            multiplier *= 128
            if (multiplier > 128*128*128)
               throw Error(Malformed Remaining Length)
       while ((encodedByte AND 128) != 0)

In mqtts/mqtt.packet you can try to swap

do {
            encodedByte = stream.readByte();

            this.remainingPacketLength += (encodedByte & 0x7f) * multiplier;
       +->     if (multiplier > Math.pow(0x80, 4)) {
       +->         throw new Error('Invalid length');
       +->     }
       |
       + >    multiplier *= 0x80;

        } while ((encodedByte & 0x80) !== 0);

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Is this request the cause?
2020-03-05T15:35:31.656Z ig:request Requesting undefined /api/v1/direct_v2/inbox/

This isn't related.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Could this cause some unexpected issues or it is up to me to find out?

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Could this cause some unexpected issues or it is up to me to find out?

Doing this won't hurt you unless you're receiving malformed packets.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Umm ok I'll try it

from instagram_mqtt.

Ektaros commented on August 11, 2024

I ran this function before starting but there are still invalid direction errors. Cant yet confirm if there is the Invalid Length error cause do not have logs for now but everything else seems the same

function overrideInvalidPacketLength() {
  MqttPacket.prototype.readRemainingLength = function (stream) {
    this.remainingPacketLength = 0;
    let multiplier = 1;
    let encodedByte;
    do {
        encodedByte = stream.readByte();
        this.remainingPacketLength += (encodedByte & 0x7f) * multiplier;
        if (multiplier > Math.pow(0x80, 4)) {
            throw new Error('Invalid length');
        }
        multiplier *= 0x80;
    } while ((encodedByte & 0x80) !== 0);
 }
}

from instagram_mqtt.

Ektaros commented on August 11, 2024

There is also another case of JSON.parse() error (this one has no Invalid direction read errors) finally got the logs for it. What do you think about this one?

2020-03-06T09:17:53.668Z mqtt:client:ping PingPong @ 1583486273668
2020-03-06T09:18:10.128Z mqtt:client:ping PingPong @ 1583486290128
2020-03-06T09:18:20.118Z mqtt:client:packet Received PublishRequestPacket id: 7857 topic: 146
2020-03-06T09:18:20.119Z mqtt:client:packet Sent PublishAckPacket id: 7857
2020-03-06T09:18:20.119Z mqtt:client:packet Received DisconnectRequestPacket
2020-03-06T09:18:20.121Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Disconnected.
2020-03-06T09:18:20.122Z mqtt:client:packet Sent DisconnectRequestPacket
2020-03-06T09:18:20.146Z mqtt:client:packet Received DisconnectRequestPacket
uncaughtException SyntaxError: Unexpected token x in JSON at position 0
unhandledRejection OperationalError: invalid distance too far back
    at Zlib.zlibOnError [as onerror] (zlib.js:180:17) {
  cause: Error: invalid distance too far back
      at Zlib.zlibOnError [as onerror] (zlib.js:180:17) {
    errno: -3,
    code: 'Z_DATA_ERROR'
  },
  isOperational: true,
  errno: -3,
  code: 'Z_DATA_ERROR'
}

from instagram_mqtt.

Nerixyz commented on August 11, 2024

If you're subscribing to the client config updates, don't do that it disconnects your client.

from instagram_mqtt.

Ektaros commented on August 11, 2024

I have only 2 subs:
SkywalkerSubscriptions.directSub and fbns
What does irisData in realtime connect actually do? Mb there is something wrong with that?

from instagram_mqtt.

Ektaros commented on August 11, 2024

And about the Invalid direction read. Invalid length error was actually fixed by shifting the limit and it is no more. But repeating Invalid direction errors are still comming here are the logs before the chain of errros:

2020-03-06T10:35:56.189Z ig:request Requesting undefined /api/v1/direct_v2/pending_inbox/
2020-03-06T10:35:56.505Z ig:request Requesting POST /api/v1/direct_v2/threads/340282366841710300949128205452651604079/approve/
2020-03-06T10:35:56.690Z ig:request Requesting undefined /api/v1/direct_v2/inbox/
2020-03-06T10:36:05.036Z mqtt:client:ping PingPong @ 1583490965036
2020-03-06T10:36:05.036Z mqtt:client:ping PingPong @ 1583490965036
2020-03-06T10:36:05.523Z mqtt:client:ping PingPong @ 1583490965523
2020-03-06T10:36:23.838Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid direction: read
	Stack: InvalidDirectionError: Invalid direction: read
    at UnsubscribeRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/unsubscribe.request.packet.ts:30:15)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:106:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:105:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:289:29)
2020-03-06T10:36:23.866Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid direction: read
...

from instagram_mqtt.

Ektaros commented on August 11, 2024

Got another chain of events leading to crash. Here it is:

2020-03-06T12:50:12.584Z mqtt:client:packet Received PublishRequestPacket id: 16092 topic: 146
2020-03-06T12:50:12.584Z mqtt:client:packet Sent PublishAckPacket id: 16092
2020-03-06T12:50:16.183Z mqtt:client:packet Received PublishRequestPacket id: 911 topic: 146
2020-03-06T12:50:16.183Z mqtt:client:packet Sent PublishAckPacket id: 911
2020-03-06T12:50:16.652Z mqtt:client:ping PingPong @ 1583499016652
2020-03-06T12:50:16.652Z mqtt:client:ping PingPong @ 1583499016652
2020-03-06T12:50:31.899Z mqtt:client:packet Received PublishRequestPacket id: 9633 topic: 146
2020-03-06T12:50:31.899Z mqtt:client:packet Sent PublishAckPacket id: 9633
2020-03-06T12:50:31.899Z mqtt:client:packet Received UnsubscribeResponsePacket id: 3194
2020-03-06T12:50:31.901Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Error: UnsubscribeResponsePacket
	Stack: UnexpectedPacketError: UnsubscribeResponsePacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:337:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:291:25)
unhandledRejection OperationalError: invalid distance too far back
    at Zlib.zlibOnError [as onerror] (zlib.js:180:17) {
  cause: Error: invalid distance too far back
      at Zlib.zlibOnError [as onerror] (zlib.js:180:17) {
    errno: -3,
    code: 'Z_DATA_ERROR'
  },
  isOperational: true,
  errno: -3,
  code: 'Z_DATA_ERROR' 
} (removed rejected promise)
uncaughtException SyntaxError: Unexpected token x in JSON at position 0

Is there any additional info I can provide?

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Is there any additional info I can provide?

In the new version of mqtts, the parser should be able to recover on its own. You'll also see the the stream that's in the parser.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Using new version of mqtts there seems to be none of the invalid length error. But code still crashed regularly with json error. Im planing to rewrite parser like this and then there is event: error in on('iris') just create new RealtimeClient.

parseMessage(topic, payload) {
        try {
            let data = JSON.parse(payload.toString('UTF-8')).map((x) => ({ topic, data: x }))
            return data
        } catch (error) {
            return [{ data: { event: 'error', data: [{ message: 'parser failed' }] }}]
        }
    }

I can save those broken packets in some form and provide them to you to help improve the lib. But not sure that is the best way to save them. What do you think?

from instagram_mqtt.

Ektaros commented on August 11, 2024

Also got the logs before crash on 0.1.9 of mqtts

2020-03-11T09:58:11.904Z mqtt:client:packet Received PublishRequestPacket id: 1302 topic: 88
2020-03-11T09:58:11.904Z mqtt:client:packet Sent PublishAckPacket id: 1302
2020-03-11T09:58:14.218Z mqtt:client:ping PingPong @ 1583920694218
2020-03-11T09:58:14.250Z mqtt:client:ping PingPong @ 1583920694250
2020-03-11T09:58:15.016Z mqtt:client:packet Received PublishRequestPacket id: 1303 topic: 146
2020-03-11T09:58:15.017Z mqtt:client:packet Sent PublishAckPacket id: 1303
2020-03-11T09:58:16.608Z mqtt:client:parser End of stream: EndOfStreamError: 
    at PublishRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/publish.request.packet.ts:78:68)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:111:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:110:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:29)
2020-03-11T09:58:17.155Z mqtt:client:ping PingPong @ 1583920697155
2020-03-11T09:58:29.678Z mqtt:client:parser End of stream: EndOfStreamError: 
    at PublishRequestPacket.read (/usr/backend/node_modules/mqtts/src/packets/publish.request.packet.ts:78:68)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:111:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:110:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:29)
2020-03-11T09:58:29.680Z mqtt:client:parser No packet found for 15;
                         @1/183
                         parsed: 3
2020-03-11T09:58:29.687Z mqtt:client:parser Error in parser (type: 9): 
                        Error: Invalid return code: 167
    at SubscribeResponsePacket.assertValidReturnCode (/usr/backend/node_modules/mqtts/src/packets/subscribe.response.packet.ts:59:19)
    at SubscribeResponsePacket.read (/usr/backend/node_modules/mqtts/src/packets/subscribe.response.packet.ts:39:18)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:111:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:110:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:29); 
                        exiting; 
                        resetting;
                        stream: +5BAWxenB/tQzs396PKQDW+H96PHu8fRvzOhSR6sQ5ii0FUB2M4efrZTio+wdsKfyHKoYz2Tvzqn5rr2iZ29baXtPJMrSirJODt7Oob5Y0fMoSEgP+VVOAYJuUTs7P32N3CoKPLklTuzsWdTgDKYDDb6ag7VVu3v51BpsF/jUOmx2XYq5/bae+u1PWawbY8qett246J7ZvVTl+X0Y7zD3vQOtaS0nEH/Ytnv/wcAAP//AwAIixUc
2020-03-11T09:58:29.688Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Warning: Invalid return code: 167
	Stack: Error: Invalid return code: 167
    at SubscribeResponsePacket.assertValidReturnCode (/usr/backend/node_modules/mqtts/src/packets/subscribe.response.packet.ts:59:19)
    at SubscribeResponsePacket.read (/usr/backend/node_modules/mqtts/src/packets/subscribe.response.packet.ts:39:18)
    at /usr/backend/node_modules/mqtts/src/mqtt.parser.ts:111:28
    at tryCatcher (/usr/backend/node_modules/bluebird/js/release/util.js:16:23)
    at Function.Promise.attempt.Promise.try (/usr/backend/node_modules/bluebird/js/release/method.js:39:29)
    at MqttParser.parse (/usr/backend/node_modules/mqtts/src/mqtt.parser.ts:110:35)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:290:29)
2020-03-11T09:58:29.714Z mqtt:client:packet Received PublishRequestPacket id: 2379 topic: 146
2020-03-11T09:58:29.714Z mqtt:client:packet Sent PublishAckPacket id: 2379
2020-03-11T09:58:29.715Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Error: PingRequestPacket
	Stack: UnexpectedPacketError: PingRequestPacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:338:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:292:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:292:25)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
2020-03-11T09:58:29.719Z mqtt:client:packet Received PublishCompletePacket id: 40382
2020-03-11T09:58:29.719Z ig:mqtt:mqttot edge-mqtt.facebook.com:443: Error: PublishCompletePacket
	Stack: UnexpectedPacketError: PublishCompletePacket
    at MQTToTClient.handlePacket (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:338:32)
    at /usr/backend/node_modules/mqtts/src/mqtt.client.ts:292:43
    at Array.forEach (<anonymous>)
    at MQTToTClient.parseData (/usr/backend/node_modules/mqtts/src/mqtt.client.ts:292:25)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Followed by Z_data_error and Invalid JSON

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Could you please log all messages using ig.realtime.client.transport.socket.on('data', data => console.log(data.toString('hex')); And paste the log here or on pastebin.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Ok done that waiting for errors :)

from instagram_mqtt.

Nerixyz commented on August 11, 2024

@Ektaros I can't just remote debug your code. I don't know your code, I don't know what you're doing to the account.

from instagram_mqtt.

Ektaros commented on August 11, 2024

I am not asking u for that :) Added the line above. Now I am waiting for errors to occur. Ill paste them here then they appear.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Here are the logs before the error https://pastebin.com/kDecsmit
Catching in with the code above and creating new RealtimeClient worked for me, no more crashes I hope :)

from instagram_mqtt.

Nerixyz commented on August 11, 2024

Here are the logs before the error https://pastebin.com/kDecsmit
Catching in with the code above and creating new RealtimeClient worked for me, no more crashes I hope :)

Thanks, seems like this was a problem with the parser. It sould be updated in mqtts 0.1.11 and instagram_mqtt 0.2.8.

from instagram_mqtt.

Ektaros commented on August 11, 2024

Are they out? I see there is a 0.1.11 commit for mqtts and 0.2.8 version of instagram_mqtt in npm but no commit here.

from instagram_mqtt.

Nerixyz commented on August 11, 2024

It's not pushed yet but it's on npm and just a bump of mqtts

from instagram_mqtt.

Ektaros commented on August 11, 2024

Ive been on new version of mqtts for a few days now. Everything seems alright. No more parser errors or crashes. Thanks for the updates!

from instagram_mqtt.

"Expected flags 2 but got X" error crashes node about instagram_mqtt HOT 51 CLOSED

Comments (51)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent