Upon request we should be able to produce a bare bones MATLAB struct with all of the basic information.

Example:
myData:
GPS:
TimeUS: 1x44459 double
Lat: 1x44459 double
Lng:
AHRS:
BARO:

We don't like how Mission Planner does things currently, so we proposed our definition.

A good reason why this is needed is to help users share data with other people who don't have the Ardupilog source but want access to the log inside MATLAB.

Possible wanted features would be

• A nice plotting API
• A set of predefined plots
• Non-volatile user-defined plots

We could take ideas from:
http://ardupilot.org/dev/docs/using-mavexplorer-for-log-analysis.html
http://ardupilot.github.io/MAVProxy/html/analysis/mavexp.html

The DF definition changes from time to time. Even though we are always able to parse it (because it is self-describing) meta-analysis, like plotting, isn't applicable through versions.

We need some way to keep track of what is applicable for each DF version and what is not, so as to make decisions such as:

• What is the unit multiplier for a field, so that it is correctly plotted?
• What is the name of the field carrying the board voltage information so that I can plot it?

This is a difficult task and proper software architecture is required to make it scalable.

This will be primarily used to display a progress bar.

log.filterMsgs(msgFilter) works well in deleting empty message timeseries.

But log = Ardupilog(path-to-log, msgFilter) fills out all the empty messages.

Hello everyone,

I'm facing an issue with MATLAB: when I run my program, I get the following error: "Requested 4294967295x1 (32.0GB) array exceeds maximum array size preference (7.9GB)". It seems that the log file I'm trying to process is too large for my machine's capacity, which has 8 GB of memory.

I could potentially bypass this problem by using a PC with 32 GB of memory, but I'm wondering if there's an alternative solution to process this large log file on my current 8 GB memory PC.

For reference when doing log analysis.

#47 created another property to store an index of the LogMsgGroups, so that it can iterate over them during slicing.

It would be better to avoid property inflation and use 'isa()' to parse LogMsgGroup properties upon request.

Hunter,

Do you think it would be of any merit to filter the parsed messages before storing them, against a provided message list?

I think the complexity to do it during parsing would be substantial, whereas a simple array deletion post-parsing is trivial.
Plus, I don't expect any time savings if we choose to do it the 1st way.

It is desirable that functionality is implemented for:

• Cropping an Ardupilog object to a smaller time window
• Cropping one or more windows out of the log and keeping the rest

Arguments for cropping could be time or line indexed. For example:

• getSlice([50 5000], ‘LineNo’)
• getSlice([0 5.37],’TimeUS’)
• getSlice([0 105.1],’TimeS’)
• getSlice([t1 t2],’TimeUTC’)
• double value = getSlice(5.4,’TimeS’) % For interpolation
• getSlice(times,’type’,’endpoints’,’format’,’UTC’)
• double interps = getSlice(times,’type’,’interp’,’format’,’TimeS’)

@hunt0r you ok with that? I think this is deprecated now.

Unit information is not embedded in the DF log currently.
Still, we might need those to effectively plot data ( #37 ).

A solution would be to upkeep a separate function or class which would hold the unit information for every field of every message possible.
This is a lot of manual work, but no other way exists currently. Even more work of one wants to make it work across versions.

Peter Barker mentioned here that he is has an addition of units in the DF in the works: ArduPilot/ardupilot#5864

The Matlab script worked flawlessly for a while, but now they switched to Chibios based operating system and possibly changed the log description, so ardupilog gives an error.
A file to test is attached here:
https://wetransfer.com/downloads/72b68488cf91076b94ba4750dc2f48be20190916175750/52c55b5efd8eb26dbc0d4b010883e7c220190916175750/fe0f05

Very easy to fix: Each LogMsgGroup doesn't know it's own text-string name (e.g. "GPS2" or "PARM"). We should add this as a private property, with type_num and the others.

In LogMsgGroup, a set of format characters specify the data format (e.g. 'b' = int8, 'I' = uint32, etc.). We have an "otherwise" case in our switch statement set to warn if a (future) log character is presently unsupported by Ardupilog. At present, the warning message suggests we'll capture the data as uint8, but we don't.

I'm not sure this is the best behavior, anyways. Since data types are of different lengths, if we default to some length and it's wrong, we'll corrupt the remainder of the LogMsgGroup data.

Furthermore, in the formatLength function, we threw an error if we didn't know the type... so I dunno if we would even get to this point in the code.

I think the very best design would implement a "known character types" table, and then replace the switch statements with some sort of "lookup from character types table" functionality. That would put all our character-types information in a single spot. Does this sound like a good design to you?

In the meantime, I'm happy to leave this open as a low-priority issue on GitHub.

Hello

I've been using ardupilog for a few years and very impressed. BUT Just tried to convert a 2 hour .BIN file (from ArduRover in boat mode) recorded yesterday and the script fails with these warning and errors:

Warning: incompatible data on msg type=0/  �ð (>>>00<<< >>>00<<<)incompatible data on msg type=0/�ð (>>> 
> In LogMsgGroup/storeFormat (line 44)
In LogMsgGroup (line 32)
In Ardupilog/createLogMsgGroups (line 323)
In Ardupilog/readLog (line 135)
In Ardupilog (line 68)
In pixhawkBinToLogViaCLASS_corruptFileAnalysis (line 44) 
Error using Ardupilog/addprop
'  �ð' is not a valid dynamic property name. Property names must begin with a letter and be composed of letters, numbers, and underscores.

Error in Ardupilog/createLogMsgGroups (line 329)
                    addprop(obj, newName);

Error in Ardupilog/readLog (line 135)
            obj.createLogMsgGroups(data');

Error in Ardupilog (line 68)
            readLog(obj);

Error in pixhawkBinToLogViaCLASS_corruptFileAnalysis (line 44)
        log = Ardupilog(fNameOP); 

[I've added the hex output in the 1st line to check that the type_num and type_name are both zero (null, 0x00 etc) ];

this is about 9,000,000 doubles along the file (72,000,000 bytes along): total file length 407,115,390 Bytes

My Matlab coding is not expert, so I am trying to get the code to just ignore the 'glitch' and catch up with the next valid record, but the structure of the script defeats my sledgehammer attempts. Frustrating is that the code stops at the error , and fails to write what it has thus far.

Thoughts and debugs most welcome

Phil

Our log-slicing capability assumes the TimeUS are non-decreasing as the Line Number increases. This is a good assumption... but a corrupt log means our slicing fails to perform as intended, and does not alert the user.

Of course we can't be robust to all kinds of corrupt data... but if we saw this issue happen more, we might consider doing something about it?

[Details]
Someone posted a log with a bugged value (probably some sort of corrupted-data). The file can be found on this forum post: https://discuss.ardupilot.org/t/arduplane-skywalker-help/33869?u=hunt0r
Look at the TimeUS values for the ARSP log message group. The problematic array-index is 6691. So the corresponding ARSP.LineNo(6691) = 652620. Check out the jump in TimeUS values around this index:

l.ARSP.TimeUS(6689:1:6693)
ans =
1.0e+18 *
0.000000001463860
0.000000001463959
4.213092985657280 <---- clearly wrong
0.000000001464060
0.000000001464160

The ARSP.TimeUS(6691) is 4.2131e+18. Compare that with ARSP.TimeUS(6690) = 1.4640e+09 and ARSP.TimeUS(6690) = 1.4641e+09 to see that it's ridiculously high. Or checkout the jump in DatenumUTC which is derived from TimeUS:

datestr(l.ARSP.DatenumUTC(6689:1:6693))
ans =
09-Oct-2018 16:37:34
09-Oct-2018 16:37:34
17-May-5526 23:20:47 <--- clearly wrong
09-Oct-2018 16:37:34
09-Oct-2018 16:37:34

When we slice based on TimeS, the corrupt value is guaranteed to be the "first" value over the limit, no matter what the slicing limit is, and it often becomes one of the endpoints of the slice, without the user understanding why. Then plotting sliced-data versus time looks very corrupted, because the corrupt value makes the plot limits silly.

Heartbeat message sends a numeric mode designator but Mission Planner knows the translation. Where does it pull the data from? What is the "M" variable type?

@hunt0r Here's a proposal for the release blog. All suggestions welcome.

Hello everyone!
1.5 year ago, me and @hunt0r needed to parse some ArduPlane logs. We are both on our track to get a PhD in UAV control systems and Matlab is a very common tool in our trade.

Sadly, at the time the only tool able to convert from DataFlash logs (.bin) to a format readable by Matlab was Mission Planner and it didn't do a good job at that. It would work only on Windows, take a very long time to convert and would ignore all text fields.

So, we decided to create our own converter, ardupilog. It is Matlab-based, meaning that you need Matlab to run it.
It can read all DataFlash logs, since they are self-defining, and create a versatile log object.
You can even "flatten" the log into a classic struct, if you want the information to be independent of our library.

Example Usage

Reading a log

Add the ardupilog source code to your path.
Then,

log = Ardupilog()

will open a file browser, where you can select the log file you want to decode.
The variable struct log will be generated with the included message types as fields.
Its contents can be normally processed with Matlab commands:
<< Placeholder for images of the resulting struct and a plot >>

Filtering messages

You can optionally filter the log file for specific message types:

log = Ardupilog('<path-to-log', <msgFilter>)

msgFilter can be:

• Either a vector of integers, representing the message IDs you want to convert.
• Or a cell array of strings. Each string is the literal name of the message type.
  << Placeholder image with filtered log >>

Slicing

Typically, only a small portion of the flight log is of interest. Ardupilog supports slicing logs to a specific start-end interval. For example, to keep log messages from boot time 1000sec to 2500sec:

sliced_log = log.getSlice([1000, 2500], 'TimeS')

<< Placeholder image with sliced log >>

Exporting to plain struct

Creating a more basic struct file, free of the ardupilog dependency, is possible with:

log_struct = log.getStruct();

log_struct does not need the ardupilog source code accompanying it to be shared.
<< Placeholder image with plain struct >>

More information can be found on the repo's README.

With the filter capability (as added in #26) it makes sense to me to delete the (empty) LogMsgGroup if it was filtered out by the user. Do you agree?

I began working on it myself, but found it's a little more complicated than I thought. Just calling "delete(obj.(msgName))" deletes the object, but doesn't get rid of the property. I haven't yet figured out an easy way to clear it completely...

(Also, we'd want to move the call to obj.findInfo() from line 65 to be inside readLog(), perhaps around line 120ish right before % Iterate over all the discovered msgs)

I found that matlab has an object called a timeseries. Unfortunately, I can't figure out what the advantages of using this over "usual" data vectors is... but since our data is fundamentally connected to time, maybe we should use these?

Arguments which make sense are timeUS, timeS and timeUTC. Line numbers not so much.
Argument handling could be common with #35

I just tried to convert a large .BIN file (just under 4M messages). It was the log file from CanberraUAV's 2016 OBC run.

After an hour and a half, it had reached halfway, but it seemed like the progress rate would slow down progressively.

I fear that dynamic size allocation gets slower, as array size grows, which is understandable.

We should investigate the possibility to pre-allocate message arrays, to increase storage speed.

This idea got discussed alongside PR #41 which was discussed, and closed. I couldn't tell if you ever responded to this one directly? (My apologies if you already did) So I'm re-posting it as an issue.

[Code Design: or "What should we do with and without message filtering"]
I agree there's a design-choice to be made here, with pros-and-cons for any choice.

With the log parsing speed so fast, I propose the following general design:

1. Read the entire file into memory (a single call to fread).
2. Find all FMT messages.
3. Process them (at least) to extract the lengths and ID's of other message types which may exist.
   3.5) At this point, there's probably no significant increase in time to also get also their names, fields, format strings, etc... but if we know the message will be filtered, we could theoretically skip this.
   3.75) I think at this point, it also makes sense to search for the MSG type, and extract the firmware version, etc. (I think this is the findInfo() method, presently)
4. For FMT message defining an id/length, locate all valid messages of this type.

At this intermediate point, we have everything we need: the position of all valid messages in the log. We could also check for any regions of the log which are NOT part of a valid message, as well as any overlap. (if that's even possible) We also have a count of the total number of log lines, the number of lines of each type, etc.

5. For each message that was not filtered out, create a LogMsgGroup (if we didn't already do it in step3) and store the data. (Adding LineNo info wherever appropriate) Note that "storing the data" may get more complex in the near future, as we determine units, interpolability, etc. If we did create the LogMsgGroup back in line 3, I think at this point we agree that we should delete it for messages which are filtered out.

A con of this design is that we spend time finding (and counting) messages that we intend to filter, and could have skipped. Note that we still don't spend time processing-and-storing those messages, so the message filtering is still valuable for increasing speed, just not quite as fast as possible.

I hope the pros of this design outweigh the cons. Here are a few pros I see: having true LineNo info, having a count of the total log messages, having the ability to see if the log has lots of undefined/invalid messages (not presently implemented)

What do you think?

Entering

log = Ardupilog('mylog.bin')

will error out.
The problem is probably due to

log = Ardupilog('./mylog.bin')

works.

From recent logs I am getting incorrect datenums.

E.g.

>> log.bootTimeUTC

ans =

1980-01-05 23:59:50

>> datestr(max(log.GPS.DatenumUTC))

ans =

06-Jan-1980 07:30:00

Hi,

When I run the code and select a .BIN log file, I get this error:

'RSSI' is already defined as a property.

Error in Ardupilog/createLogMsgGroups (line 322)
                    addprop(obj, newName);

I tried to upload the .BIN file on Flight Log Review website and it works fine. So I assume the file itself is okay.

Any help would be appreciated!

At the beginning of ArduPilog, if a filename is specified as the .BIN file, the InputParser should detect that as p.Results.path and call fileparts(which(p.Results.path)). (Line 47 of ArduPilog.m, I believe)

Unfortunately, which() doesn't seem to work as desired with files specified as 'path/to/file.ext'

I can demonstrate this at the command line. Having a valid file and typing:
which('valid/path/to/valid_file.ext')
returns:
'valid/path/to/valid_file.ext' not found. (<- this is incorrect)

While exist('valid/path/to/valid_file.ext') returns 2, indicating that a file is found and exists.

If I addpath('valid/path/to/') and then which('valid_file.ext') the function works as desired, returning:
valid/path/to/valid_file.ext (<- as a MATLAB string)

I'm going to hack a quick workaround for now, but we should decide in the future some better fix.

I ran the profiler, the most expensive operation is to continually re-size the obj.(field_name_string) with each new addition. We should investigate pre-allocating to increase speed.

Another expensive operation was the switch() on the format type character, I'm not sure how to speed this up.

Finally, the findMsgStart was smaller, but still not a trivial part of the processing time. Maybe we could streamline this?

Naturally, this is a completely new and separate format, justified only by its popularity and constant use with all things Ardupilot.

I think the best way to approach it is to create another generator Python script (see https://github.com/ArduPilot/pymavlink/tree/master/generator) which will produce MATLAB code (scrirpt/function/class) based on the up-to-date MAVLink definition.
Then, the provided log could be parsed using that dynamic code.

Looks like the commit referenced below for LogStructures.h set the SBRH and SBRM fields to integers 1,2,3,4,... and MATLAB complains with the warning linked below.

For a first work-around, we could change SBRM 1 to "m1", 2 to "m2", and so on... probably SBRH similarly with 1 = "h1", 2 = "h2", 3 = "h3", and so on.

[The MATLAB warning]

Warning: '1' is not a valid dynamic property name. Property names must begin
with a letter and be composed of letters, numbers, and underscores. This will
become an error in a future release.

In LogMsgGroup/storeFormat (line 35)
In LogMsgGroup (line 24)
In Ardupilog/createLogMsgGroups (line 304)
In Ardupilog/readLog (line 128)
In Ardupilog (line 62)

[The git commit where this probably changed]
commit 0b5dad33f26b96ffaed212d75674c30dfa07f422
Author: ebethon [email protected]
Date: Mon Jun 26 16:41:24 2017 +0100

DataFlash: Rename SBR1/SBR2 to SBRH/SBRM and change format

Add (sender_id, msg_len) in SBRM message
Add index/pages in SBRH/SBRM messages
Change format to integers to avoid GCS string parsing
Decrease data size in SBRH

Generally in software engineering, the existence of automated (unit) tests is favourable.
These are test cases with known result, for which the algorithms should have a good known output. If not, something got broken.

The good thing is that they can be run automatically and if all tests pass, you know that your changes and new features didn't break the code.

The bad thing is that your testing suite is as good as the test cases you write.

I'm leaving this here for future reference and food for thought.
https://www.mathworks.com/help/matlab/matlab-unit-test-framework.html
https://www.mathworks.com/videos/matlab-unit-testing-framework-74975.html

This is extracted from comments in the code.

As extracted from code comments.

According to this report: https://discuss.ardupilot.org/t/ardupilog-a-dataflash-log-to-matlab-converter/33812/11?u=georacer

Reportedly it's from (Arduplane?) 3.9.7.
testlog.zip

Log file attached.

This is extracted from comments in the code.