Giter VIP home page Giter VIP logo

Comments (9)

jaxer avatar jaxer commented on August 20, 2024

By quickly googling and looking through the code this mqtt_cpp issue came up: redboltz/mqtt_cpp#540

It seems fixed long ago though. Although it is not clear to me if "acquire_unique_packet_id" should be used without explicit release.

Commenters seem to have conflicting opinion on that.

In cppagent i see it being used, i.e. in here:

m_clientId = derived().getClient()->acquire_unique_packet_id();

from cppagent_dev.

wsobel avatar wsobel commented on August 20, 2024

from cppagent_dev.

jaxer avatar jaxer commented on August 20, 2024

Got another occurrence of that. What's interesting is that its actually happening pretty quickly (after about 40 minutes of running). And first error is not about packet_id but something else that gets followed by exhaust of packet_ids. See part of log below, thats about the time packet_id's errors started:

2022-12-07T18:19:57.058450Z (0x00007f7007fff640) [info] : MQTT ConnAck: MQTT Connected
2022-12-07T18:23:49.052597Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:23:49.052655Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:23:49.052750Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:44:56.078750Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:44:56.078925Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:45:06.086513Z (0x00007f7007fff640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 36957090.
2022-12-07T18:45:06.086700Z (0x00007f7007fff640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:06.086815Z (0x00007f7007fff640) [error] : Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.094591Z (0x00007f7007fff640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2022-12-07T18:45:16.094705Z (0x00007f7007fff640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.094834Z (0x00007f7007fff640) [error] : Error occurred processing data: The data stream needs to restart
2022-12-07T18:45:16.100658Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2022-12-07T18:45:16.100730Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2022-12-07T18:45:16.100742Z (0x00007f7007fff640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2022-12-07T18:45:16.102495Z (0x00007f7007fff640) [error] : Error getting request header: beast.http end of stream
2022-12-07T18:45:16.102631Z (0x00007f7007fff640) [error] : header: Retry last failed request
2022-12-07T18:45:27.983920Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:45:27.984007Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:45:38.466486Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:45:38.466529Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:45:38.467687Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:50:47.482037Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:50:47.482258Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:50:57.982034Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:50:57.982074Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:50:57.983228Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:07.467340Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:07.467464Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:17.890404Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:17.890451Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:17.891746Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:24.106991Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:24.107442Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:34.354029Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:34.354072Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:34.355301Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:37.018117Z (0x00007f7007fff640) [error] : Error getting response: beast.http body limit exceeded
2022-12-07T18:52:37.018363Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:52:47.558852Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:52:47.558895Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:52:47.561044Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:52:53.273330Z (0x00007f7007fff640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2022-12-07T18:52:53.273459Z (0x00007f7007fff640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2022-12-07T18:52:53.276823Z (0x00007f7007fff640) [error] : Error getting response: system Bad file descriptor
2022-12-07T18:52:53.276846Z (0x00007f7007fff640) [error] : read: Retry last failed request
2022-12-07T18:53:03.520595Z (0x00007f7007fff640) [warning] Observation: Error occurred converting VALUE: (): cannot convert string 'HIGH' to double
2022-12-07T18:53:03.520638Z (0x00007f7007fff640) [warning] : Error while parsing XML: (VALUE): cannot convert string 'HIGH' to double
2022-12-07T18:53:03.521804Z (0x00007f7007fff640) [warning] : Device: D00660: Cannot data item for id submissionTime or name:SubmissionTime
2022-12-07T18:53:05.218692Z (0x00007f7007fff640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2022-12-07T18:53:05.218796Z (0x00007f7007fff640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2022-12-07T18:53:05.218848Z (0x00007f7007fff640) [error] : Error getting response: system Bad file descriptor
2022-12-07T18:53:05.218861Z (0x00007f7007fff640) [error] : read: Retry last failed request

Makes me wonder if actual reason is in Error getting response: beast.http body limit exceeded which triggers resubmit of same data and saturates some queue?

from cppagent_dev.

jaxer avatar jaxer commented on August 20, 2024

Another observation is that when i am starting agent then CPU is around 20%, memory on 37%. And it keeps around those values during normal operation (for 40~ minutes). After that CPU spikes to 300-400% (and stays there until restart) but memory stays the same. Its about time packet_id errors starting to come into agent.log.

Hard to say which comes first.

Restarting agent process resets everything back to normal.

UPDATE: Now i am seeing those packet_id errors with CPU very low

from cppagent_dev.

jaxer avatar jaxer commented on August 20, 2024

Unfortunately #242 didn't appear to have solved the issue. I've rebuilt agent and running under similar conditions and getting same results.
Here is a part of log between connection until first failure (after that it just repeats same 4 lines over and over):

2023-01-06T12:39:45.619365Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:dev_asset_rem
2023-01-06T12:39:45.619422Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:lightTowerState
2023-01-06T12:39:45.619922Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id submissionTime or name:SubmissionTime
2023-01-06T12:39:45.621971Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T12:39:45.622076Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T12:39:45.786349Z (0x00007f1da2fd0640) [info] : MqttClientImpl::connect: connected
**2023-01-06T12:39:46.361625Z (0x00007f1da2fd0640) [info] : MQTT ConnAck: MQTT Connected**
2023-01-06T12:55:44.156401Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http partial message
2023-01-06T12:55:44.156524Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:04:50.443161Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:04:50.443381Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:05:00.450302Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28665110.
2023-01-06T13:05:00.450490Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:00.450562Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.458148Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:05:10.458293Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.458452Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:05:10.463744Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:05:10.463931Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:05:10.463965Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:05:10.466672Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:05:10.466796Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:25:23.828245Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:25:23.828464Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:30:15.731961Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:30:15.732126Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:30:25.739372Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28693009.
2023-01-06T13:30:25.739459Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:25.739573Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.747288Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:30:35.747375Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.747486Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:30:35.754383Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:30:35.754477Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:30:35.754490Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:30:35.757392Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:30:35.757471Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:43:40.363907Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:43:40.364084Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:50:47.868802Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:50:47.868950Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:51:06.317723Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id and no name:lightTowerState
2023-01-06T13:51:09.329672Z (0x00007f1da2fd0640) [warning] : Device: lffs9487: Cannot data item for id submissionTime or name:SubmissionTime
2023-01-06T13:53:39.749963Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:53:39.750133Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:54:51.062896Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:54:51.063012Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:19.529316Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:19.529426Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:33.425201Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:33.425436Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:41.031630Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:41.031797Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:47.220647Z (0x00007f1da2fd0640) [error] : Error getting response: beast.http body limit exceeded
2023-01-06T13:55:47.220796Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:55:51.036617Z (0x00007f1da2fd0640) [error] : Received protocol error: OUT_OF_RANGE 'from' must be greater than or equal to 28720907.
2023-01-06T13:55:51.036749Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:55:51.036835Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:55:59.766996Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:55:59.767379Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:55:59.767414Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:55:59.767456Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:55:59.767498Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:01.041568Z (0x00007f1da2fd0640) [error] : Received protocol error: UNSUPPORTED The following path is invalid: /H350/assets
2023-01-06T13:56:01.041961Z (0x00007f1da2fd0640) [warning] : AgentAdapter - Error occurred processing data: The data stream needs to restart
2023-01-06T13:56:01.042099Z (0x00007f1da2fd0640) [error] : Error occurred processing data: The data stream needs to restart
2023-01-06T13:56:01.044670Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:activeAlarms
2023-01-06T13:56:01.044831Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_chg
2023-01-06T13:56:01.044878Z (0x00007f1da2fd0640) [warning] : Device: h350tln1sim: Cannot data item for id and no name:h350tln1sim_asset_rem
2023-01-06T13:56:01.049061Z (0x00007f1da2fd0640) [error] : Error getting request header: beast.http end of stream
2023-01-06T13:56:01.049282Z (0x00007f1da2fd0640) [error] : header: Retry last failed request
2023-01-06T13:56:11.692088Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:11.692467Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:11.692548Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:11.693061Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:11.693120Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:23.666464Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:23.667046Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:23.667084Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:23.667135Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:23.667171Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:23.669000Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:23.669212Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:23.669305Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:23.669324Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:35.560654Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:35.560964Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:35.561012Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:35.564180Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:35.564295Z (0x00007f1da2fd0640) [error] : read: Retry last failed request
2023-01-06T13:56:47.509094Z (0x00007f1da2fd0640) [error] : AgentAdapter - Error occurred processing data: packet_id exhausted error
2023-01-06T13:56:47.509411Z (0x00007f1da2fd0640) [error] : Exception occurred in AgentAdapter::processData: Retry last failed request
2023-01-06T13:56:47.509453Z (0x00007f1da2fd0640) [error] : Stream body but no request: Retry last failed request
2023-01-06T13:56:47.509868Z (0x00007f1da2fd0640) [error] : Error getting response: system Bad file descriptor
2023-01-06T13:56:47.509918Z (0x00007f1da2fd0640) [error] : read: Retry last failed request

from cppagent_dev.

MRIIOT avatar MRIIOT commented on August 20, 2024

@jaxer what broker are you running this against? Can you share its configuration?
Also, are you sending with QoS 0, 1, 2?

from cppagent_dev.

jaxer avatar jaxer commented on August 20, 2024

@MRIIOT I was trying AWS IoT Core and MQTT broker inside AWS Greengrass (based on Moquette)

I was not changing any configuration of those so i guess QoS should be 0

from cppagent_dev.

wsobel avatar wsobel commented on August 20, 2024

I remember running into this issue before. The problem had to do with the data rates to the broker and overloading the broker before it could free up another another free packet id. If we reduce the QoS we may lose data. The guys from SpB mentioned this as a problem they had, which is why the started combining multiple observations into a singe message.

There are a few solutions. We could do something similar to what SpB did, but then we have issues with retention and we need a way to get the current state (SpB does this put publishing a "REBIRTH" from the subscriber)

If this is what is happening and greengrass can't offload fast enough, then we may need to do something.

Another option is to publish a current to one topic periodically and then deltas to another topic. The streams will sync every so often and an application can figure it out by checking sequence.

What do you think?

from cppagent_dev.

MRIIOT avatar MRIIOT commented on August 20, 2024

Can you try it against a local Mosquitto instance, log everything, and share the broker logs?

from cppagent_dev.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.