Performance loss on 3.9.0 about plotjuggler HOT 17 CLOSED

perfect. I will come back to you with a solution!

from plotjuggler.

Solved. The problem was in https://github.com/PlotJuggler/plotjuggler-ros-plugins

It was in fact CPU bounded: a relatively expensive operation was done over and over again, when it shouldn't.

See commit here PlotJuggler/plotjuggler-ros-plugins@35b9de9

Please try and let me know if you see the issue fixed also on your side

from plotjuggler.

Which part of "give me a way to reproduce the issue" was unclear?

from plotjuggler.

I will try to design a workload that I can share and demonstrates my point but the cockiness/arrogance was uncalled for.

from plotjuggler.

@parakhm95 you are probably right, but I work on this project in my spare time and I really care about giving to user like you a good software. I am thankful that you are taking the time to report a bug, but you are also deliberately ignoring the instructions.

There is a template here that you MUST have read and decided to ignore: https://github.com/facontidavide/PlotJuggler/blob/main/.github/ISSUE_TEMPLATE.md

You would not be wrong if you call those instructions passive-aggressive. But with 100+ issues piling up, it could be overwhelming to track down which one are important and which one are not.

from plotjuggler.

Just starting the investigation now, PlotJuggler v3.9.0, here are dense plots from replaying the rosbag through commandline :

and the same rosbag loaded offline to plot :

.

Here are they zoomed in to 60.0 to 61.0 secs time interval, first, plotted from rosbag play :

and second, loaded offline :

.

@facontidavide : RosBag : Next step, build Plotjuggler from master on 3.9.0?

from plotjuggler.

I have tried quickly with RQT publisher, but everything looks in order at 1 KHz

from plotjuggler.

@facontidavide I will break down my response into two for addressing the bug/issue and this little run-in.

First of all, the community is well aware of and thankful for your contributions and plotjuggler plays an integral part in the lives of robotics community in general. The tool has been invaluable for me in the last 5-6 years in my work and I cannot overstate its importance. But, we are also aware that you have a full-time job and if this is stressful, I would suggest that a call for developers to assist you is a good way to share this burden with the open-source community.

The reason I ignored those instructions is because my issue is live plotting and that is not covered by the scope of the template presented. The reason was to also publicly start a discourse about this. And now, here are more details about the problem and my plan to recreate it and check it.

I have some simulation environments that are very taxing and my realtime simulation rate for them usually drops below 1.0 on my work machine. I have been developing in those two particular environments for 2 years and even when the entire system was bogged down in the past, I have easily been able to live plot my ROS topics at 100Hz. Last week I noticed that a data stream at 100Hz and another at 250Hz were missing dots on the plot (vs time) and only when I slowed down the simulation deliberately to prevent overloading, I got higher density of points being plotted. This was the first time I have seen this on Plotjuggler. Trusting PlotJuggler as much as I do, I chased down a time-dilation issue on my side which was never to be found because it turned out that the rate and time were fine. My next suspicion was my graphics driver so I checked that as well. But I noticed that my co-worker was also confused by something similar two evenings ago where the plot density of points from a 100 Hz topic was all over the place. I revisited my old tried and tested simulation environment and I noticed hiccups that I haven't seen before. This is difficult to replicate for debugging because I would have to provide access to a demanding environment which is proprietary/closed-source, however, I will consider it.

My plan is as follows:

• Figure out which environment can be easily obtained in the open-source community, first guess would be PX4's SITL simulation where I first noticed this. I will then consider adding you to my repository I have been developing.
• Once established, I will progressively downgrade the plotjuggler package either through snap or by checking out tags on the master and building from source.
• And then I will write back with my findings.

What are your thoughts on this?

from plotjuggler.

thanks. At the end of the day, I am just trying to help, but debugging is often a needle in a haystack and having a way to reproduce the issue would be great.

If reproducing your environment is hard, maybe there is a shortcut. Could you try doing profiling yourself?
Try using https://github.com/KDAB/hotspot , that is what I personally use to find performance bottlenecks.

If you have never used it before, you will find it extremely useful also for your own projects 😄

Once you have the results, you can send them to me and I can evaluate if there is any "smoking gun" that points to the issue

from plotjuggler.

Anyway, we are talking about ROS topics here, right? can it be reproduced with rosbag play ?

from plotjuggler.

I fully understand. I will let you know as I have more information during my testing.

I will also try out the hotspot tool yes. About the rosbag play, I can try using my rosbags but I suspect that this is related to the cpu load so I would have to find a way to maybe adjust that stress while playing. Let's see.

from plotjuggler.

And what happens if you do "rosbag play" and subscribe to the streaming?
Can the issue be reproduced in that way?

from plotjuggler.

@facontidavide I already did it, sorry I broke the markdown formatting, read my comment again. Sent you the rosbag as well. What should be the next direction of investigation? Apparently this is not CPU load dependent at all.

from plotjuggler.

Thank you and please let us know what you find, I am curious (as an Engineer and a not-by-choice software developer).

from plotjuggler.

Just curious... @matemat13 who are you and why do you feel compelled to add emoji to all the comments? 😄
Nothing against it, of course!

from plotjuggler.

Just curious... @matemat13 who are you and why do you feel compelled to add emoji to all the comments? 😄 Nothing against it, of course!

I'm Parakh's colleague. We both run into this issue, so I wanted to support it by adding some interaction to show that it's not just Parakh who cares about it without writing redundant "me too" comments. And I didn't like your original reaction, so I felt compelled to show it as well.

from plotjuggler.

But thank you for resolving this issue so fast, we all appreciate your hard work on this tool.

from plotjuggler.