Profiler: Increase speed about ardupilog HOT 6 CLOSED

Here's my short-term vision: I'd like to see if we can improve the log-processing speed to be fast like Mission Planner is, taking ~10-20sec to read/process a full log. (Maybe 1 sec of log-processing time per 1 min of actual log-time would be fast-but-reasonable?)

If we can get the log processing this fast, I think we can remove the input asking the user how many log lines to process.

While I don't know if this speed up is possible, my idea is to try a "more memory intensive" approach: perhaps read the entire file into memory up front, this allows message-counting, pre-allocating space, maybe streamlining the expensive "switch" structure, and so forth.

I guess the thing for me to do is start a speed-up branch and play with ideas to see what helps?

from ardupilog.

That sounds like a good idea.

What I think would work is to use the same find call to count the number of msgs and store the locations of the header bytes in the file.
Then re-read the file only at those points (fseek) and find out what msg types are there.
Count them, allocate matrices and then decode the file.

from ardupilog.

Okay, I've been experimenting with increasing the speed, here's what I've found so far:

1. Pre-allocating the LogMsgGroup fields (as empty) doesn't notably increase the speed. If you want to see this, compare the increase_speed branch (PR'ed to master) with the preallocate_msggroup branch (which I don't intend to PR). (As a side note, it's difficult to know how much to pre-allocate for each MsgGroup, but it doesn't seem to matter since there's no real speed-up)

2. Using a local copy of the FMT info (which msgTypeNum is which Name/Length) increases speed. So I've added that in pull request: #15

At this point (with 1 unhelpful and 2 pull-requested) my profiling suggests we are about as fast as possible processing the log sequentially. (using a for-loop to go line-by-line through the log, or the local copy of the log data) I conclude this because if we comment the time-consuming line obj.(msgName).storeMsg(lineNum, msgData); then the other time-consumers are very simple/quick instructions that are just called a lot... once (or twice!) for every log line. No way to change that, except...

My next idea: Some sort of "batch" processing. I'm now going to experiment with attempting to read all (or most) of the messages of a certain type, then "vectorizing" the processing-and-store functionality. I'll report back as I learn things.

from ardupilog.

Picking up from #17:

In a relatively small log portion (10k lines), the preallocation branch not only doesn't help, but worsens things:

With preallocation: <84sec
Without preallocation: <25sec

from ardupilog.

EDIT with the batch version in pull request #20 I'm abandoning the pre-allocation tests mentioned below. No need.

Just wanted to record/share this somewhere...

I wanted to run a test on a large log file between pre-allocated vs grow-with-dataset code, so I downloaded that large UAVCanberra log you mentioned somewhere. I counted the message numbers using existing tools.

With really good guesses at the pre-allocation amounts, converting that large log took 24 hours on my 5-year-old desktop using the pre-allocation version. (Note: I cheated to get the pre-allocation amounts, but that's not important)

I haven't yet run the grow-with-dataset version, I'll edit this post when I get that test complete.

from ardupilog.

I think this has been dealt with by #21.
We are getting times comparable to 1min for a 150MB log, which I believe is better than Mission Planner.

from ardupilog.