Giter VIP home page Giter VIP logo

orangebox's Introduction

Orangebox

PyPI version Documentation Status

A Cleanflight/Betaflight blackbox log parser written in Python 3.

Orangebox has no dependencies other than the Python standard library, although it might be worthy to investigate how using numpy could possibly bring some performance gain in the future. If so, it shouldn't be a usability barrier since any user wanting to make something out of the decoded data will more than likely have numpy installed transitively as well as it's also a dependency of libraries like matplotlib and pandas (among others).

This parser was roughly modeled after the one in Blackbox Log Viewer hence produces the same output.

Documentation

You can browse the full documentation online on Read the Docs, here's a quick example:

from orangebox import Parser

# Load a file
parser = Parser.load("btfl_all.bbl")
# or optionally select a log by index (1 is the default)
# parser = Parser.load("btfl_all.bbl", 1)

# Print headers
print("headers:", parser.headers)

# Print the names of fields
print("field names:", parser.field_names)

# Select a specific log within the file by index
print("log count:", parser.reader.log_count)
parser.set_log_index(2)

# Print field values frame by frame
for frame in parser.frames():
    print("first frame:", frame.data)
    break

# Complete list of events only available once all frames have been parsed
print("events:", parser.events)

# Selecting another log changes the header and frame data produced by the Parser
# and also clears any previous results and state
parser.set_log_index(1)

Contributing

  • Contributions are very welcome!
  • Please follow the PEP8 Style Guido.
  • More info in the docs.

Changelog

0.3.1

  • Add bb2gpx utility for converting GPS data into GPX

0.3.0

  • Add support for GPS frames (thanks to @tblaha!)

0.2.0

  • Improved Reader class can now handle multiple logs in a single file
  • Add bbsplit command-line script for splitting flashchip logs (thanks to @ysoldak!)
  • Improved logging
  • Added HTML documentation
  • Fix parsing stats

0.1.1-beta

  • Add bb2csv command-line script for converting logs into CSV

0.1.0-beta

  • First release (with a lot of missing parts)

Known issues

  • No explicit validation of raw data against corruption, missing headers or whatsoever, but it's highly likely that a Python exception will be raised in these cases anyway
  • Tested only on logs generated by Betaflight
  • Not all event frames are parsed (see TODO comments)
  • Some decoders are missing (see TODO comments)

Acknowledgement

Original blackbox data encoder and decoder was written by Nicholas Sherlock.

License

This project is licensed under GPLv3.

orangebox's People

Contributors

atomgomba avatar tblaha avatar ysoldak avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

orangebox's Issues

bb2csv errors

When I try and run bb2csv against my logfile I get the following error:

:~$ bb2csv -v BTFL_BLACKBOX_LOG_20200827_190605.BBL
Traceback (most recent call last):
File "/usr/local/bin/bb2csv", line 4, in
import('pkg_resources').run_script('orangebox==0.2.0', 'bb2csv')
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 667, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 1470, in run_script
exec(script_code, namespace, namespace)
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/EGG-INFO/scripts/bb2csv", line 57, in
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/EGG-INFO/scripts/bb2csv", line 29, in main
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/orangebox/parser.py", line 77, in load
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/orangebox/reader.py", line 59, in init
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/orangebox/reader.py", line 82, in set_log_index
File "/usr/local/lib/python3.8/dist-packages/orangebox-0.2.0-py3.8.egg/orangebox/reader.py", line 146, in _build_field_defs
RuntimeError: No predictor found for 10

The log decodes in blackbox_decode fine and produces the csv, gps csv & gpx files

Platform is Ubuntu 20.04 and install was from source.

log attached and renamed as .txt for attachment (sorry if there is a better way I cant see it).
BTFL_BLACKBOX_LOG_20200827_190605.txt

iNAV log error - No predictor found for 10

Hi, I appreciate your lib and like to use it to load it for plots, qgis or kdenlive visualisation.
Unfortunatly the blackbox logs created by INAV 3 are .TXT files which cause the error message

~/.virtualenvs/orangebox/lib/python3.8/site-packages/orangebox/reader.py in _build_field_defs(self)
211 if prop == "predictor":
212 if framedef_value not in predictors:
--> 213 raise RuntimeError("No predictor found for {:d}".format(framedef_value))
214 else:
215 field_defs[frame_type][i].predictorfun = predictors[framedef_value]

RuntimeError: No predictor found for 10

To my understanding, as another fork of cleanflight, the blackbox structures should be compatible?
The rendering using the official blackbox-viewer by betaflight works fine so far.

tag8_8svb decoder fails when there are 8 fields

Thanks for this amazing project!

I came across this bug when actually using exactly 8 group for the TAG8_8SVB encoding. This occurs in a betaflight fork (github.com/tblaha/indiflight).
If exactly 8 fields are found in a TAG8_8SVB encoding group (the maximum), then group_count to 0, if 8 groups are found. This gets the parser get stuck in an endless loop, because next(data) is never called.

@map_to(6, decoder_map)
def _tag8_8svb(data: Iterator[int], ctx: Optional[Context] = None) -> DecodedValue:
# count adjacent fields with same encoding
group_count = 0
fdeflen = ctx.field_def_counts[ctx.frame_type]
for i in range(ctx.field_index + 1, ctx.field_index + 8):
if i == fdeflen:
break
if ctx.field_defs[ctx.frame_type][i].encoding != 6:
group_count = i - ctx.field_index
break
if group_count == 1:
# single field
return _signed_vb(data, ctx)
else:
# multiple fields
header = next(data)
values = ()
for _ in range(group_count):
values += (_signed_vb(data, ctx) if header & 0x01 else 0,)
header >>= 1
return values

Setting group_count = 8 in line 57 seems to fix the issue in my logfiles.

AFAIK blackbox-log-viewer also has a similar bugs, will also raise an issue there.


Sidenote: TAG8_8SVB itself is already broken by definition if you use less than 8 fields and the next field is also encoded with TAG8_8SVB. If I have a TAG8_8SVB group with less than 8 values, followed by another TAG8_8SVB group, there is no way to know where the boundary is, as the number of values is not written in the logs. We only know which ones are zero or not, so if the first TAG8_8SVB has 4 values, then the last 4 are declared 0 in its header. The parser simply checks up to 8 fields for TAG8_8SVB encoding (and in this case will then also accept up to 4 fields from the next TAG8_8SVB group, which are then zero'd even though they may not be).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.