luka1199 / geo-heatmap Goto Github PK

The line that opens heatmap.html in the default browser doesn't work when running the program in the Windows Subsystem For Linux (WSL). This is because the default browser in WSL is w3m, a text-only browser.

This is the line that causes the issue: https://github.com/luka1199/geo-heatmap/blob/master/geo_heatmap.py#L83

It would be nice if the output didn't automatically open in a browser, but instead notifies over standard output what file was created.

Here's a screenshot of what the result looks like:

Looking at the generated map has brought back a lot of good memories from trips. It may not make sense in this project, but have you considered adding photos from Google Photos onto the map using their locations? Exploring cities you've traveled to and seeing the photos you took could be a fun experience.

Getting this in the console.

Loading data from Location History.json...
|##################################################################################################|100% Time:  0:00:01
Traceback (most recent call last):
  File "geo_heatmap.py", line 85, in <module>
    generator.run(data_file, output_file)
  File "geo_heatmap.py", line 73, in run
    self.loadData(data_file)
  File "geo_heatmap.py", line 30, in loadData
    lat = round(loc["latitudeE7"] / 1e7, 6)
KeyError: 'latitudeE7'

Describe the bug
I have both GPX and KML files for the same activity from Garmin connect (attached)

To Reproduce
Steps to reproduce the behavior:
Executing python geo_heatmap.py activity_5972548926.klm opens the browser in coordinates 0:0

Executing python geo_heatmap.py activity_5972548926.gpx opens the browser with heatmap in the center

Software version - commit 7c12365
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Chrome version 87.0.4280.66 (Official Build) (64-bit)
Win 10

10X

When I ran the script and pointed to the .zip archive with the location data, I received the following error:

Loading data from ../data/takeout-20191130T173344Z-001.zip...
Reading location data file from zip archive: 'Takeout/Locatiegeschiedenis/Locatiegeschiedenis.json'
Traceback (most recent call last):
File "./geo_heatmap2.py", line 179, in
generator.run(data_file, output_file)
File "./geo_heatmap2.py", line 135, in run
self.load_zip_data(data_file)
File "./geo_heatmap2.py", line 87, in load_zip_data
self.loadJSONData(read_file)
File "./geo_heatmap2.py", line 33, in loadJSONData
data = json.load(json_file)
File "/usr/lib/python3.5/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'

I think this is because geo_heatmap.py assumes a folder structure in the archive that uses English terms. If your Google services language is set to another language, these paths don't work out.

I was able to run geo_heatmap.py by first unzipping the archive and directly pointing to the .json file.

File "geo-heatmap/geo_heatmap.py", line 104, in <module> generator.run(data_file, output_file) File "geo-heatmap/geo_heatmap.py", line 77, in run self.loadData(data_file) File "geo-heatmap/geo_heatmap.py", line 28, in loadData data = json.load(json_file) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 296, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

It would be nice to be able to call the script with multiple files and merge the timestamps all together.

Does it mean I need a more powerful computer?

The location data contains a timestamp, it would be cool if we could generate a day-by-day (or week-by-week or more options) animation of the heatmap, although I'm not sure how this would fit into the OpenStreetMap, but at the very least it would be interesting to see it if it could stitch them together as a gif.

I ran the python script on a Pixelbook and received this error:

/usr/bin/env: ‘python3\r’: No such file or directory

I fixed this by removing the carriage returns:
cat geo_heatmap.py | tr -d "\r" > geo_heatmap_noCR.py

I downloaded my location data today and tried to run the script but I'm getting the following exception:

python geo_heatmap.py takeout-20230101T215713Z-001.zip

(1/3) Loading data from takeout-20230101T215713Z-001.zip
Traceback (most recent call last):
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 327, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 252, in run
    self.loadZIPData(data_file, date_range)
  File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 173, in loadZIPData
    (data_path,) = fnmatch.filter(
ValueError: not enough values to unpack (expected 1, got 0)

The Takeout folder structure doesn't seem to match the one the script is expecting:

Are there optimization possibilities for loading data from a KML file, because for me loading is very slow compared to loading with a JSON file?

Is there anyway to have it process 2 .json files. or could someone help me with the structure so I can combine the .json files from 2 separate google accounts?

Followed the steps, had to install folium and progreessbar, using python 3.7 on an anaconda env

python geo_heatmap.py Standortverlauf.json
Loading data from Standortverlauf.json...
Traceback (most recent call last):
  File "geo_heatmap.py", line 111, in <module>
    generator.run(data_file, output_file)
  File "geo_heatmap.py", line 83, in run
    self.loadData(data_file)
  File "geo_heatmap.py", line 34, in loadData
    widgets=w) as pb:
TypeError: __init__() got an unexpected keyword argument 'max_value'

• Amount of location points
• Amount of countries with location points

I get the following error message when running this script:

|##############################################################################################################################|100% ETA:  00:00:00
Traceback (most recent call last):
  File "geo_heatmap.py", line 222, in <module>
    generator.run(data_file, output_file, date_range)
  File "geo_heatmap.py", line 177, in run
    self.loadKMLData(data_file, date_range)
  File "geo_heatmap.py", line 93, in loadKMLData
    if self.timestampInRange(loc['timestampMs'], date_range):
TypeError: list indices must be integers or slices, not str

Input KML file:

<kml xmlns='http://www.opengis.net/kml/2.2' xmlns:gx='http://www.google.com/kml/ext/2.2'>
 <Document>
  <Placemark>
   <open>1</open>
   <gx:Track>
    <altitudeMode>clampToGround</altitudeMode>
    <when>2013-12-22T11:38:47Z</when>
    <gx:coord>-1.1 52.1 0</gx:coord>
   </gx:Track>
  </Placemark>
 </Document>
</kml>

Any idea what might cause this?

Hello, excellent project, I tested it with the data that I was able to download from Google takeout and it works perfectly.

I would like to be able to use your project to try to graph the route of a GPS tracker, from which I can only obtain LAT, LONG and Date/Time.

I tried to replicate the format of the "Records.json" file but still can't get it to work.

Here below I am going to leave an example of the json file that I want to make work with this project, could you look at it and tell me what is wrong?
Greetings

{ "locations": [ { "latitudeE7": -3472408, "longitudeE7": -5851444, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:01.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:05.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:52:05.269Z" } ] }

It would be interesting to be able to use it with csv or json files with a simple format that contains LAT, LONG and date/time

I leave it as a suggestion, but I would appreciate help with this problem

Sorry for my English.

Frederic from Argentina

Probably worth mentioning that you have stored your own location history in a few of the commits. I'd recommend following these steps: https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository

This can be personal information and would be best not to end up in the wrong hands!

Progressbar won't be useable with this and it would be way slower.

Suggested by https://www.reddit.com/r/dataisbeautiful/comments/e2zv4w/i_wrote_a_python_script_that_generates_a_geo/f924sza?utm_source=share&utm_medium=web2x

https://www.openstreetmap.org/copyright/

WIP. All data will be processed instantly without including a SQL or any such database to store any form of data.

once the json file has finished streaming, i immediately get the following error:
Value 2528098 is out of range, should be between 0 and 2528097.328244275
at pb.update(i) line 86

Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError

C:\Users\pgroy\Python\Python38-32\geo-heatmap>python geo_heatmap.py LocationHistory.json
Loading data from LocationHistory.json...
Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError

Describe the bug
Folium throws an exception due to a missing field:

(2/3) Generating heatmap
Traceback (most recent call last):
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 302, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 245, in run
    m = self.generateMap(settings)
  File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 192, in generateMap
    m = folium.Map(location=self.max_coordinates,
  File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/folium.py", line 288, in __init__
    tile_layer = TileLayer(tiles=tiles, attr=attr,
  File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/raster_layers.py", line 111, in __init__
    raise ValueError('Custom tiles must have an attribution.')
ValueError: Custom tiles must have an attribution.

Desktop (please complete the following information):

• OS: Windows

$ uname -a
Linux kihikihi 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ python3 --version
Python 3.8.10

$ python3 geo_heatmap.py ~/Projects/Trips/Takeout/Location\ History/Semantic\ Location\ History/2022/2022_AUGUST.json

(1/3) Loading data from /home/jsibert/Projects/Trips/Takeout/Location History/Semantic Location History/2022/2022_AUGUST.json
Traceback (most recent call last):
File "geo_heatmap.py", line 327, in
generator.run(data_file, output_file, date_range, stream_data, settings)
File "geo_heatmap.py", line 258, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 57, in loadJSONData
first_element = data["locations"][0]
KeyError: 'locations'

Describe the bug
Downloaded location data file contains timestamp field instead of timestampMs field.

$ python3 geo_heatmap.py ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json -s

(1/3) Loading data from /home/theo/Downloads/takeout-20220203T164644Z-001/Takeout/Historique des positions/Records.json

Traceback (most recent call last):
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 303, in <module>
    generator.run(data_file, output_file, date_range, stream_data, settings)
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 232, in run
    self.streamJSONData(json_file, date_range)
  File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 75, in streamJSONData
    if timestampInRange(loc["timestampMs"], date_range):
KeyError: 'timestampMs'

To Reproduce
Steps to reproduce the behavior:

1. Get your location data at https://takeout.google.com/
2. Run the script

Expected behavior
Normal generation of heatmap.html file.

Screenshots
Content of Records.json file

$ head -20  ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json 
{
  "locations": [{
    "latitudeE7": 413522827,
    "longitudeE7": 6672174,
    "accuracy": 20,
    "source": "WIFI",
    "deviceTag": 1611787950,
    "timestamp": "2013-12-30T20:30:26.396Z"
  }, {
    "latitudeE7": 417153171,
    "longitudeE7": 75560909,
    "accuracy": 736,
    "source": "CELL",
    "deviceTag": 1611787950,
    "timestamp": "2014-01-01T07:44:41.819Z"
  }, {
    "latitudeE7": 417153171,
    "longitudeE7": 75560909,
    "accuracy": 736,`
    "source": "CELL",

The file does not contain any timestampMs field :

$ grep -ri timestampMs ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/
$

Desktop:

• OS: Ubuntu 21.04

Hello, I currently have Firefox dev, Firefox, and Chrome on my computer and I was given the error:

Traceback (most recent call last):
  File "geo_heatmap.py", line 251, in <module>
    if not isTextBasedBrowser(webbrowser.get()):
  File "/usr/lib/python3.6/webbrowser.py", line 51, in get
    raise Error("could not locate runnable browser")
webbrowser.Error: could not locate runnable browser

I this might be a problem because I am running Windows Subsystem for Linux, making the installing paths of my applications a little weird. There is an option in webbrowser that allows for registration of a browser and it accepts a hard-coded path for the browser, but I don't see this as working generally enough for everbody.

As suggested here: https://www.reddit.com/r/dataisbeautiful/comments/e2zv4w/i_wrote_a_python_script_that_generates_a_geo/f91pjr9?utm_source=share&utm_medium=web2x

Desired solution
I wish to import items / locations I exported from a custom Google MyMaps. I created the pins by uploading a csv with addresses in the format of street city country to Google MyMaps that were correctly parsed and are shown on the Google MyMaps Map. Then, I downloaded the points through this dialogue, trying KML and KMZ format:

I wish I could pass the export files along to be executed by the Python script to have a heat map generated. I also used this converter to convert the data to gpx: https://mygeodata.cloud/conversion.

Errors encountered
When trying to pass the respective export file, I got the respective errors:

1. Converted .gpx file and kml file: Execution was successful, but the map was empty
2. kmz file: NotImplementedError: Unsupported file extension for 'International_Addresses.kmz'

Is there any workaround to get the script working for single points in Google MyMaps?
Many thanks for your help!