tormoder / fit Goto Github PK
View Code? Open in Web Editor NEWA Go package for decoding and encoding Garmin FIT files
License: MIT License
A Go package for decoding and encoding Garmin FIT files
License: MIT License
The majority of FIT files have type Activity. Consider special handling of them. Benchmark.
A quick question - I'm trying to parse a monitoring file (monitoringB with heart rate data from a forerunner 35). The heart_rate fields fail to parse. The FIT profile shows 37 possible fields in the monitoring message (mesg_num 55), but the MonitoringMsg (messages.go) struct only has 12 of these fields defined or available. These 12 looks fairly similar to the fields shown in the FIT File Types Description document (2.2), which also doesn't show the whole list of available fields stated in the message defined in the profile. Any idea what's going on there? There seems to be a discrepancy between the profile and the document, and the code roughly aligns with the document.
In some real life scenarios sensors may connect / disconnect / reconnect during the recording of an activity. As the result, the activity file may have several "blocks" of records with different definition messages that have different set of fields. A common use case is when a power meter was connected to a Garmin bike computer after the activity recording has been started.
In the current fit file model we combine all messages of the same type in one collection (per type), with message fields initialized with "invalid" values for all the fields that don't have a valid data. When this file is writing back by the writer
it is quite possible to have such message with missing fields as the first message in the collection, which then will be used as a template to build a definition message for all records of this type. All fields of the following records that don't have a matching definition field will be excluded from the final result.
Currently, after decoding you basically get a flat activity file.
This closly represents the fit file protocol.
But an application might be more interested in a tree-structure of the activity, e.g.
type StructuredActivityFile {
Sessions []StructuredSession
}
type StructuredSession{
Lap []*StructuredLap
}
type StructuredLap {
LapMsg // Copy or reference of original lap for summary information
Records []*RecordMsg // Contains only data records which were during this lap.
// maybe also others, such as events, ...
}
This could be calculated in a post-processing step as a method of ActivityFile by
func (a *ActivityFile) StructuredActivity() StructuredActivity {...}
I guess it's only neccessary to compare timestamps to build the tree.
What do you think of this?
Or is the scope of the package to represent the fit protocol closly, leaving all higher level functionality for other packages?
I'm trying to use this library to generate fit files of the type "course".
I'm able to generate them, but they always come out as invalid.
After a lot of investigation, it turns out that the main issue is that for a "course" file, the SDK doesn't allow you to define event messages (it only allows this for "activity" files). From what I can read in the Garmin Fit SDK and the cookbook samples, the timer events are required.
Am I overlooking something or does the SDK currently doesn't allow you to do this?
I get the following error message when importing a fit file:
parsing definition message: validating Session failed: field 6: size (4) is greater than size of profile base type BaseSint32 (4)
which to me seems like stating 4>4
This is the file causing the problem.
Currently there is precisely one DeviceInfoMsg
, and it lives in the File
.
It looks like having multiple DeviceInfoMsg
s is valid per the spec, and indeed a number of the testdata references include multiple DeviceInfoMsg
s. This can be useful to track battery voltage over time, or for multiple devices.
The question is, how to support this? The simplest thing would be to just delete DeviceInfo
from File
and add a slice of them to the relevant file types (according to SDK 21.32.00 that's ActivityFile
, WeightFile
, BloodPressureFile
, MontoringAFile
, MonitoringBFile
). However, that would break anyone that's relying on File.DeviceInfo
.
So, I made a change which duplicates DeviceInfoMsg
s into both File.DeviceInfo
and a slice in the individual file types (usedbytes/fit@17b8c4f) - however this causes a problem when a file is decoded and re-encoded. The last DeviceInfoMsg
ends up in both the slice and File.DeviceInfo
, which means it ends up duplicated when writing out. This could be avoided by just not writing out File.DeviceInfo
, but that might be "surprising" for users. Checking File.DeviceInfo
against each entry in the slice and avoiding duplicates would be possible but a bit ugly, because the writer is using reflection and we'd need a special case just for DeviceInfo
.
So, the simplest looks like just removing File.DeviceInfo
, but I'm wary of what breakage that might cause, next easiest would be skip File.DeviceInfo
on write-out. Any thoughts/opinions?
Received following error parsing a wahoo .fit file.
header decoded: size: 14 | protover: 32 | profver: 2027 | dsize: 337952 | dtype: .FIT | crc: 0x4965
definition message parsed: local: 0 | global: FileId | arch: BigEndian | fields: 5
parsed file_id message: {Activity WahooFitness 31 3107002583 2021-12-11 10:14:48 +0000 UTC 65535 }
definition message
parsed: local: 0 | global: DeveloperDataId | arch: BigEndian | fields: 2
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: FieldDescription | arch: BigEndian | fields: 5
definition message parsed: local: 0 | global: Event | arch: BigEndian | fields: 4
parsing time: seconds from device power on
parsing data message: missing data definition message for local message number 7
Tried with the java FitCSVTool directly and that worked, but the Local Number seems to be 0 everywhere ( which looks like the reason the decoding actually fails. )
In code: (reader.go:213)
d.defmsgs[dm.localMsgType] = dm
In this case, dm.localMsgType is always 0.
Tried with other wahoo .fit files, but all have the same outcome.
First of all, I want to thank you for this project. It is very well done and super useful. Thank you very much. I might even be able to break off a couple cycles if that helps you to keep maintaining it.
I am having an issue with a FIT file that I downloaded from garmin. Running your "Example_test.go" with my file when I parse the fit file I get...
parsing data message: missing data definition message for local message number 10
It was a fit downloaded directly from garmin. I don't fully understand your fit decoder yet to be able to debug it. Have you seen this before? I included the fit file if it helps.
Broken.zip
Trying to decode my fit file fails with the following error:
parsing definition message: validating Session failed: field 10: type BaseSint32 is not compatible with profile type BaseUint32
Logging output:
header decoded: size: 14 | protover: 16 | profver: 2078 | dsize: 120284 | dtype: .FIT | crc: 0x30ec
definition message parsed: local: 0 | global: FileId | arch: LittleEndian | fields: 7
parsed file_id message: {Activity Garmin 2691 3982257377 2019-06-02 09:10:36 +0000 UTC 65535 }
definition message parsed: local: 1 | global: FileCreator | arch: LittleEndian | fields: 3
definition message parsed: local: 2 | global: Event | arch: LittleEndian | fields: 7
definition message parsed: local: 3 | global: DeviceInfo | arch: LittleEndian | fields: 27
definition message parsed: local: 4 | global: MesgNum(22) | arch: LittleEndian | fields: 10
definition message parsed: local: 5 | global: MesgNum(141) | arch: LittleEndian | fields: 7
definition message parsed: local: 6 | global: DeviceSettings | arch: LittleEndian | fields: 72
definition message parsed: local: 7 | global: UserProfile | arch: LittleEndian | fields: 28
definition message parsed: local: 8 | global: MesgNum(147) | arch: LittleEndian | fields: 54
definition message parsed: local: 9 | global: MesgNum(79) | arch: LittleEndian | fields: 17
definition message parsed: local: 10 | global: Sport | arch: LittleEndian | fields: 10
definition message parsed: local: 11 | global: MesgNum(13) | arch: LittleEndian | fields: 43
definition message parsed: local: 12 | global: ZonesTarget | arch: LittleEndian | fields: 11
definition message parsed: local: 13 | global: Record | arch: LittleEndian | fields: 9
definition message parsed: local: 14 | global: MesgNum(104) | arch: LittleEndian | fields: 5
definition message parsed: local: 15 | global: Lap | arch: LittleEndian | fields: 80
definition message parsed: local: 0 | global: MesgNum(216) | arch: LittleEndian | fields: 13
definition message parsed: local: 1 | global: MesgNum(233) | arch: LittleEndian | fields: 1
definition message parsed: local: 5 | global: MesgNum(140) | arch: LittleEndian | fields: 32
illegal definition message: local: 6 | global: Session | arch: LittleEndian | fields: 93
The files created by Encode
are somehow invalid for some platforms. For example, if I upload the newly created file to Strava, it says "The upload contains bad data.".
I tried to debug them by having a valid FIT file (e.g., https://github.com/tormoder/fit/blob/master/testdata/me/activity-small-fenix2-run.fit) and running it through the following simple program:
package main
import (
"bytes"
"encoding/binary"
"log"
"os"
"github.com/tormoder/fit"
)
func main() {
testData, err := os.ReadFile("activity-small-fenix2-run.fit")
if err != nil {
log.Fatalln(err)
}
file, err := fit.Decode(bytes.NewReader(testData))
if err != nil {
log.Fatalln(err)
}
f, err := os.Create("new.fit")
if err != nil {
log.Fatalln(err)
}
defer f.Close()
err = fit.Encode(f, file, binary.LittleEndian)
if err != nil {
log.Fatalln(err)
}
}
When uploading the new.fit
file to https://www.fitfileviewer.com/, it shows empty data (whereas the original activity-small-fenix2-run.fit
file showed valid content):
Weirdly enough, https://www.gpxsee.org/ shows the new.fit
file correctly.
Analyzing both files with https://runalyze.com/tool/fit-viewer, the main differences seem to be that new.fit
has a lot of enhanced_speed
, enhanced_max_speed
, enhanced_altitude
fields on its laps and records (whereas activity-small-fenix2-run.fit
doesn't have any enhanced_*
fields) and that new.fit
always uses 0
as its local message type index.
Update: I did another test run where I removed the code that generates the enhanced_*
field generation code from this library and even without those, neither Strava nor FitFileViewer would accept the file. So I'm guessing that those aren't the problem.
This would remove the Python dependecy for profile code generation.
See test file testdata/python-fitparse/compressed-speed-distance.fit
.
Compressed distance is not accumulated correctly. Possibly a roll over/overflow issue.
gofmt
changes over time, meaning that the comparison against the "golden" files in TestGenerator
breaks easily, even when the code is logically equivalent.
The official recommendation is to use a specific version of gofmt
to avoid such problems:
Note that these kinds of minor updates to gofmt are expected from time to time. In general, we recommend against building systems that check that source code matches the output of a specific version of gofmt. For example, a continuous integration test that fails if any code already checked into a repository is not “properly formatted” is inherently fragile and not recommended.
The alternative would be to read in the golden data, format it with the "current" version of gofmt
via go/format.Source()
, and compare that against the generated code. That's slightly complicated by the fact that all the files are concatenated, but should be easy to solve.
For example, when running fitgen
on SDK 21.38, it fails with
panic: genMsgs: could not find type for ref field name "HeartRateSourceType"
For some reason this field in not enabled in the profile. Find out why and possibly enable some kind of special handling of such fields inside fitgen
.
Currently new SDKs are generated locally by patching fitgen
to enbale the missing field(s).
From SDK 16.30 (page 15):
"The FIT protocol allows for multiple FIT files to be chained together in a single FIT file. Each FIT file in the chain must be a properly formatted FIT file (header, data records, CRC)."
See #61 for some context.
Spec:
"if date_time is < 0x10000000 then it is system time (seconds from device power on)"
Use any coordinates (if present in any messages) to find local time zone.
Can you decode messages in a loop, instead of decoding the whole file in on call?
An application is a corrupted fit file, where you want to extract all record messages that are available.
Required for Fit 2.0 (#20).
When I call 'fit.Decode' method, I get the following error messages:
parsing definition message: validating Session failed: field 8: unknown base type 0X%!X(PANIC=runtime error: index out of range)
or
parsing data message: missing data definition message for local message number 10
or
parsing definition message: validating Session failed: field 9: type BaseSint32 is not compatible with profile type BaseUint32
When I call 'fit.Decode' method, I get the following error messages:
header decoded: size: 14 | protover: 32 | profver: 2085 | dsize: 114956 | dtype: .FIT | crc: 0x773b
definition message parsed: local: 1 | global: FileId | arch: LittleEndian | fields: 7
error parsing file id message: error reading data message: missing data definition message for local message number 5
But I can decode it with FitCSVTool.jar and turn it into csv file.
It's not obvious how the length of Arrays should be handled when encoding. In the profile xls, some arrays are tagged with "[N]", presumably indicating that they have a variable length. In some of these cases, the "EXAMPLE" column includes a number which appears to be the intended length. In other cases, the "EXAMPLE" column only has '1'.
The decoder always reads the number of elements which are encoded in the file. The encoder could also do this - but the length would need to be known at the point of generating the definition message, which means if the length is different between messages in the same file, we'd need to make a new definition message (see also, #36) or truncate/pad.
The current behaviour is to use the value from the "EXAMPLE" column, and always encode exactly that number of elements, truncating or padding as necessary. This causes the DecodeEncodeDecode test to fail in a number of cases, because this length is '1' in the xls, whereas the input file has many elements.
Think about ways of verifying decoded data against other SDKs (most likely Java).
Implement at CSV-tool?
Hi- thank you for this project- works very well for all my Garmin files, but fails for leyzene.
The error is:
error: parsing definition message: validating Session failed: field 139: size 4 for BaseUint32 as base type in definition message is greater than size 2 for BaseUint16 as base type from profile
With the standard logger enabled I get the following:
header decoded: size: 14 | protover: 16 | profver: 2016 | dsize: 27297 | dtype: .FIT | crc: 0x5140
definition message parsed: local: 0 | global: FileId | arch: LittleEndian | fields: 7
parsed file_id message: {Activity Lezyne 4 2585936027 2018-03-27 07:14:43 +0000 UTC 65535 }
definition message parsed: local: 1 | global: FileCreator | arch: LittleEndian | fields: 2
definition message parsed: local: 9 | global: TimestampCorrelation | arch: LittleEndian | fields: 2
definition message parsed: local: 3 | global: DeviceInfo | arch: LittleEndian | fields: 11
definition message parsed: local: 5 | global: Record | arch: LittleEndian | fields: 9
definition message parsed: local: 2 | global: Event | arch: LittleEndian | fields: 9
definition message parsed: local: 6 | global: Lap | arch: LittleEndian | fields: 56
illegal definition message: local: 7 | global: Session | arch: LittleEndian | fields: 48
I've also tried the version from @beyoung but get the same result.
If you point me in the right direction I think I'd be able to fix it.
Thanks in advance.
error decoding header: not supported: protocol version 2.x not supported by sdk protocol version 1.0
Would be great thx if fit supports version 2.x. _.
To make it easier to contribute we should probably find a way to get rid of "use of internal package not allowed" - error messages after clone. Currently I need to sed s/tormoder/torgil/g all code.
Example:
"../../go/src/github.com/torgil/fit/reader.go:14:2: use of internal package not allowed"
changing the line "github.com/tormoder/fit/internal/types" to "github.com/torgil/fit/internal/types" removes the error for me but clutters the workspace and will make for constant merge-confilcts in every pull/rebase-iteration.
Include definition messages in addition to byte payload of unknown fields. Maybe make this a decode option.
Ref.
golang.org/x/xerrors
and
https://go.googlesource.com/proposal/+/master/design/29934-error-values.md.
Reminder.
In Fit files which consist of slices of messages (e.g. ActivityFile.Records
), the writer implementation constructs the definition message based on the first element in that slice, and omits any fields which are invalid in that first element.
This means that if later messages include data in fields which were invalid in the first message, they will not be encoded because they weren't in the original definition.
I can see a few ways to improve this, with various tradeoffs:
With 2., there would need to be considerations made about whether to use more than one local message ID. The FIT SDK suggests that local message IDs should be kept to a minimum to ensure maximum compatibility for decoders. That might be something else to add as an encode option.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.