openbikesensor / openbikesensor-scripts Goto Github PK

Sometimes I leave my device recording while I go into a store or similar.

It would be nice if we had some way of detecting this, because the GPS jitters around in the place I stopped (due to inaccuracy), and that accumulates a lot of incorrect mileage on the track that I don't like in the statistics.

Probably some moving window distance traveled/time formula could detect this, find the center of the points inside that timeframe, and clip everything from the entry to the exit of a radius around that point. This can become incredibly smart, but somethin stupid might do the trick for starters.

Most use cases will do well without this information, and from a privacy perspective, that is desirable.

I'd say we should support three options for both of these pieces of information:

• do not include (default)
• include, but anonymize (hashing)
• include verbatim

I suggest the following changes to the command line:

--user-id=[full|hashed|none]
--measurement-id=[full|hashed|none]
--hash-seed=[...]

The "hashed" option will require to provide an explicit hash seed (do not hardcode a default!). The default should be "none".

While we're at it: Do not use MD5. SHA256 or 512 is almost always the better choice. Also, I do not understand what [0:-1:2] is used for, why not just take the whole digest? And if you only want half length, at least use [0::2].

Apparently sometimes the geo coordinates are invalid but that is not properly detected. Example:

Those should be filtered out.

The file starts like this, and after a few lines switches to proper coordinates

Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea
25.08.2020;14:21:53;0.000000;0.000000;0.000;0.0000;255;255;0;0

The annotation process looks at the oneway tag to determine if a road is available in one direction only, mapping all travel directions to the same segment.

There is also the tag oneway:bicycle to tag roads that are oneway for cars, and bidirectional for bikes. We should use that, if available, to ignore oneway tags. See: https://wiki.openstreetmap.org/wiki/Key:oneway:bicycle

We might also want to consider not caring at all about those tags, as bikers tend to ignore oneway streets anyway, or there might be a dedicated bike lane in reverse direction. Either way, there are probably few overtaking events in the reverse direction (as cars aren't travelling there), but for biker's road usage statistics, this is a valuable information to have and we should not discard it, or assume travel in the "forward" direction even if GPS indicates the reverse.

The metadata line is read from the CSV file, but except for the version identifier, every field is optional. The parser should be able to read files with just a version identifier, and provide sensible defaults for all other fields (e.g. MaximumMeasurementsPerLine is required right now).

Bei Straßen mit Tramgleis in der Mitte gibt es in der openStreetMap zwei Straßen: Einmal die tatsächliche und einmal das Straßenbahngleis. Nun werden die Events vom face-Script auf beide Straßen aufgeteilt, obwohl sie ja tatsächlich nur auf der richtigen Straße stattfinden.

Die Tram-Strecke hat auch den Straßennamen unknown.

Right now we have to manually specify the districts to download from the overpass API for road segment matching.

This won't be feasible if the Portal is to automatically annotate all uploaded tracks.

I therefore suggest a mechanism to automatically detect boundaries (geometric or administrative) to download the data from, based on the geocoordinates of the input data.

Took me some time to figure out why samples were wrongly matched to crossroads although the bike continued on the main road.

Not sure why direction filtering has been disabled in AnnotateMeasurements on calling
self.roads = Roads(map_source, d_max=point_way_tolerance, d_phi_max=90, cache_dir=cache_dir)

The direction of 90 in fact disables direction filtering resulting in lots of wrong matches to crossroads. Get much better results with the default 40 degrees.

Obviously, we're querying for only German road segments here.

Make sure that this also works if you live somewhere else. Might be solved through #7 though, if that uses non-semantic/geometric regions for downloading.

The legal minimum distance for overtaking depends on whether the road is located within or outside a built-up area.
The OpenStreetMap way tag "zone:traffic" basically provides this information. However, it is not used for annotation in many regions. Currently, the distance limit is chosen as 1,5m by default, which is thus wrong often.

The proposed solution is to introduce a hand-crafted or trained binary classifier which predicts from existing way properties such as speed limits whether this is an urban or rural road and annotates ways accordingly. This prediction is always outvoted by the "zone:traffic" tag.

I got a couple of tracks which cause errors in obs-face.

The error message is:

Traceback (most recent call last):
  File "/usr/local/bin/obs-face", line 11, in <module>
    load_entry_point('obs', 'console_scripts', 'obs-face')()
  File "/home/webuser/htdocs/portal/api/scripts/obs/bin/obs_face.py", line 368, in main
    measurements, statistics = process_datasets(datasets, args.path_annotated, map_source,
  File "/home/webuser/htdocs/portal/api/scripts/obs/bin/obs_face.py", line 150, in process_datasets
    dummy_process.dequeue_and_process()
  File "/home/webuser/htdocs/portal/api/scripts/obs/bin/obs_face.py", line 209, in dequeue_and_process
    measurements = self.annotate(dataset)
  File "/home/webuser/htdocs/portal/api/scripts/obs/bin/obs_face.py", line 240, in annotate
    measurements = self.annotator.annotate(measurements)
  File "/home/webuser/htdocs/portal/api/scripts/obs/face/annotate/AnnotateMeasurements.py", line 45, in annotate
    self.ensure_map_coverage(measurements)
  File "/home/webuser/htdocs/portal/api/scripts/obs/face/annotate/AnnotateMeasurements.py", line 59, in ensure_map_coverage
    self.map_source.ensure_coverage(lat, lon, extend=self.point_way_tolerance)
  File "/home/webuser/htdocs/portal/api/scripts/obs/face/osm/DataSource.py", line 41, in ensure_coverage
    tiles = self.tile_source.get_required_tiles(lat, lon, self.tile_zoom, extend=extend)
  File "/home/webuser/htdocs/portal/api/scripts/obs/face/osm/TileSource.py", line 116, in get_required_tiles
    s_lat, s_lon = LocalMap.get_scale_at(lat[i], lon[i])
  File "/home/webuser/htdocs/portal/api/scripts/obs/face/mapping/LocalMap.py", line 319, in get_scale_at
    s_lon = s_lat / math.cos(math.radians(lat))
TypeError: must be real number, not NoneType

I've attached a sample track which causes this error. I suspected the missing timestamps indicate a bad recording. The script should detect bad data and discard it instead of crashing.

obs_track.csv

Apparently some files start counting at unix epoch 0 and already have a timestamp, then jump to actual GPS time once that is available.

Filter out timestamps that are not feasible, e.g.

• No OBS recorded data in 1970. That's probably epoch related. Maybe filter out everything prior to 2000.
• If a jump happens in the date/time file that does not match the internal time counter of the recording device, assume a correction in the system time. Maybe calculate the amount the clock has jumped and retroactively correct the previous timestamps by that difference.

Not sure what the smartest way is to handle this. But we can't trust a 01.01.1970 00:04:23 ;)