luka1199 / geo-heatmap Goto Github PK
View Code? Open in Web Editor NEW:world_map: Generate an interactive geo heatmap from your Google location data
License: MIT License
:world_map: Generate an interactive geo heatmap from your Google location data
License: MIT License
The line that opens heatmap.html in the default browser doesn't work when running the program in the Windows Subsystem For Linux (WSL). This is because the default browser in WSL is w3m, a text-only browser.
This is the line that causes the issue: https://github.com/luka1199/geo-heatmap/blob/master/geo_heatmap.py#L83
It would be nice if the output didn't automatically open in a browser, but instead notifies over standard output what file was created.
Looking at the generated map has brought back a lot of good memories from trips. It may not make sense in this project, but have you considered adding photos from Google Photos onto the map using their locations? Exploring cities you've traveled to and seeing the photos you took could be a fun experience.
Getting this in the console.
Loading data from Location History.json...
|##################################################################################################|100% Time: 0:00:01
Traceback (most recent call last):
File "geo_heatmap.py", line 85, in <module>
generator.run(data_file, output_file)
File "geo_heatmap.py", line 73, in run
self.loadData(data_file)
File "geo_heatmap.py", line 30, in loadData
lat = round(loc["latitudeE7"] / 1e7, 6)
KeyError: 'latitudeE7'
Describe the bug
I have both GPX and KML files for the same activity from Garmin connect (attached)
To Reproduce
Steps to reproduce the behavior:
Executing python geo_heatmap.py activity_5972548926.klm opens the browser in coordinates 0:0
Executing python geo_heatmap.py activity_5972548926.gpx opens the browser with heatmap in the center
Software version - commit 7c12365
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)] on win32
Chrome version 87.0.4280.66 (Official Build) (64-bit)
Win 10
10X
When I ran the script and pointed to the .zip archive with the location data, I received the following error:
Loading data from ../data/takeout-20191130T173344Z-001.zip...
Reading location data file from zip archive: 'Takeout/Locatiegeschiedenis/Locatiegeschiedenis.json'
Traceback (most recent call last):
File "./geo_heatmap2.py", line 179, in
generator.run(data_file, output_file)
File "./geo_heatmap2.py", line 135, in run
self.load_zip_data(data_file)
File "./geo_heatmap2.py", line 87, in load_zip_data
self.loadJSONData(read_file)
File "./geo_heatmap2.py", line 33, in loadJSONData
data = json.load(json_file)
File "/usr/lib/python3.5/json/init.py", line 268, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "/usr/lib/python3.5/json/init.py", line 312, in loads
s.class.name))
TypeError: the JSON object must be str, not 'bytes'
I think this is because geo_heatmap.py assumes a folder structure in the archive that uses English terms. If your Google services language is set to another language, these paths don't work out.
I was able to run geo_heatmap.py by first unzipping the archive and directly pointing to the .json file.
File "geo-heatmap/geo_heatmap.py", line 104, in <module> generator.run(data_file, output_file) File "geo-heatmap/geo_heatmap.py", line 77, in run self.loadData(data_file) File "geo-heatmap/geo_heatmap.py", line 28, in loadData data = json.load(json_file) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 296, in load parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
It would be nice to be able to call the script with multiple files and merge the timestamps all together.
Does it mean I need a more powerful computer?
The location data contains a timestamp, it would be cool if we could generate a day-by-day (or week-by-week or more options) animation of the heatmap, although I'm not sure how this would fit into the OpenStreetMap, but at the very least it would be interesting to see it if it could stitch them together as a gif.
I ran the python script on a Pixelbook and received this error:
/usr/bin/env: βpython3\rβ: No such file or directory
I fixed this by removing the carriage returns:
cat geo_heatmap.py | tr -d "\r" > geo_heatmap_noCR.py
I downloaded my location data today and tried to run the script but I'm getting the following exception:
python geo_heatmap.py takeout-20230101T215713Z-001.zip
(1/3) Loading data from takeout-20230101T215713Z-001.zip
Traceback (most recent call last):
File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 327, in <module>
generator.run(data_file, output_file, date_range, stream_data, settings)
File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 252, in run
self.loadZIPData(data_file, date_range)
File "/Users/davidmigloz/repos/geo-heatmap/geo_heatmap.py", line 173, in loadZIPData
(data_path,) = fnmatch.filter(
ValueError: not enough values to unpack (expected 1, got 0)
The Takeout folder structure doesn't seem to match the one the script is expecting:
Line 175 in fc25368
Are there optimization possibilities for loading data from a KML file, because for me loading is very slow compared to loading with a JSON file?
Is there anyway to have it process 2 .json files. or could someone help me with the structure so I can combine the .json files from 2 separate google accounts?
Followed the steps, had to install folium
and progreessbar
, using python 3.7 on an anaconda env
python geo_heatmap.py Standortverlauf.json
Loading data from Standortverlauf.json...
Traceback (most recent call last):
File "geo_heatmap.py", line 111, in <module>
generator.run(data_file, output_file)
File "geo_heatmap.py", line 83, in run
self.loadData(data_file)
File "geo_heatmap.py", line 34, in loadData
widgets=w) as pb:
TypeError: __init__() got an unexpected keyword argument 'max_value'
I get the following error message when running this script:
|##############################################################################################################################|100% ETA: 00:00:00
Traceback (most recent call last):
File "geo_heatmap.py", line 222, in <module>
generator.run(data_file, output_file, date_range)
File "geo_heatmap.py", line 177, in run
self.loadKMLData(data_file, date_range)
File "geo_heatmap.py", line 93, in loadKMLData
if self.timestampInRange(loc['timestampMs'], date_range):
TypeError: list indices must be integers or slices, not str
Input KML file:
<kml xmlns='http://www.opengis.net/kml/2.2' xmlns:gx='http://www.google.com/kml/ext/2.2'>
<Document>
<Placemark>
<open>1</open>
<gx:Track>
<altitudeMode>clampToGround</altitudeMode>
<when>2013-12-22T11:38:47Z</when>
<gx:coord>-1.1 52.1 0</gx:coord>
</gx:Track>
</Placemark>
</Document>
</kml>
Any idea what might cause this?
Hello, excellent project, I tested it with the data that I was able to download from Google takeout and it works perfectly.
I would like to be able to use your project to try to graph the route of a GPS tracker, from which I can only obtain LAT, LONG and Date/Time.
I tried to replicate the format of the "Records.json" file but still can't get it to work.
Here below I am going to leave an example of the json file that I want to make work with this project, could you look at it and tell me what is wrong?
Greetings
{ "locations": [ { "latitudeE7": -3472408, "longitudeE7": -5851444, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:01.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:50:05.269Z" }, { "latitudeE7": -3472407, "longitudeE7": -5851443, "source": 110144, "deviceTag": 110144, "accuracy": 33, "timestamp": "2022-04-01T10:52:05.269Z" } ] }
It would be interesting to be able to use it with csv or json files with a simple format that contains LAT, LONG and date/time
I leave it as a suggestion, but I would appreciate help with this problem
Sorry for my English.
Frederic from Argentina
Probably worth mentioning that you have stored your own location history in a few of the commits. I'd recommend following these steps: https://help.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository
This can be personal information and would be best not to end up in the wrong hands!
Progressbar won't be useable with this and it would be way slower.
WIP. All data will be processed instantly without including a SQL or any such database to store any form of data.
once the json file has finished streaming, i immediately get the following error:
Value 2528098 is out of range, should be between 0 and 2528097.328244275
at pb.update(i) line 86
Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError
C:\Users\pgroy\Python\Python38-32\geo-heatmap>python geo_heatmap.py LocationHistory.json
Loading data from LocationHistory.json...
Traceback (most recent call last):
File "geo_heatmap.py", line 260, in
generator.run(data_file, output_file, date_range, tiles)
File "geo_heatmap.py", line 207, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 88, in loadJSONData
data = json.load(json_file)
File "C:\Users\pgroy\Python\Python38-32\lib\json_init_.py", line 293, in load
return loads(fp.read(),
File "C:\Users\pgroy\Python\Python38-32\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
MemoryError
Describe the bug
Folium throws an exception due to a missing field:
(2/3) Generating heatmap
Traceback (most recent call last):
File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 302, in <module>
generator.run(data_file, output_file, date_range, stream_data, settings)
File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 245, in run
m = self.generateMap(settings)
File "/home/savvasdalkitsis/geo-heatmap/geo_heatmap.py", line 192, in generateMap
m = folium.Map(location=self.max_coordinates,
File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/folium.py", line 288, in __init__
tile_layer = TileLayer(tiles=tiles, attr=attr,
File "/home/savvasdalkitsis/.local/lib/python3.9/site-packages/folium/raster_layers.py", line 111, in __init__
raise ValueError('Custom tiles must have an attribution.')
ValueError: Custom tiles must have an attribution.
Desktop (please complete the following information):
$ uname -a
Linux kihikihi 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ python3 --version
Python 3.8.10
$ python3 geo_heatmap.py ~/Projects/Trips/Takeout/Location\ History/Semantic\ Location\ History/2022/2022_AUGUST.json
(1/3) Loading data from /home/jsibert/Projects/Trips/Takeout/Location History/Semantic Location History/2022/2022_AUGUST.json
Traceback (most recent call last):
File "geo_heatmap.py", line 327, in
generator.run(data_file, output_file, date_range, stream_data, settings)
File "geo_heatmap.py", line 258, in run
self.loadJSONData(json_file, date_range)
File "geo_heatmap.py", line 57, in loadJSONData
first_element = data["locations"][0]
KeyError: 'locations'
Describe the bug
Downloaded location data file contains timestamp
field instead of timestampMs
field.
$ python3 geo_heatmap.py ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json -s
(1/3) Loading data from /home/theo/Downloads/takeout-20220203T164644Z-001/Takeout/Historique des positions/Records.json
Traceback (most recent call last):
File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 303, in <module>
generator.run(data_file, output_file, date_range, stream_data, settings)
File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 232, in run
self.streamJSONData(json_file, date_range)
File "/home/theo/git/geo-heatmap/geo_heatmap.py", line 75, in streamJSONData
if timestampInRange(loc["timestampMs"], date_range):
KeyError: 'timestampMs'
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Normal generation of heatmap.html
file.
Screenshots
Content of Records.json
file
$ head -20 ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/Records.json
{
"locations": [{
"latitudeE7": 413522827,
"longitudeE7": 6672174,
"accuracy": 20,
"source": "WIFI",
"deviceTag": 1611787950,
"timestamp": "2013-12-30T20:30:26.396Z"
}, {
"latitudeE7": 417153171,
"longitudeE7": 75560909,
"accuracy": 736,
"source": "CELL",
"deviceTag": 1611787950,
"timestamp": "2014-01-01T07:44:41.819Z"
}, {
"latitudeE7": 417153171,
"longitudeE7": 75560909,
"accuracy": 736,`
"source": "CELL",
The file does not contain any timestampMs
field :
$ grep -ri timestampMs ~/Downloads/takeout-20220203T164644Z-001/Takeout/Historique\ des\ positions/
$
Desktop:
Hello, I currently have Firefox dev, Firefox, and Chrome on my computer and I was given the error:
Traceback (most recent call last):
File "geo_heatmap.py", line 251, in <module>
if not isTextBasedBrowser(webbrowser.get()):
File "/usr/lib/python3.6/webbrowser.py", line 51, in get
raise Error("could not locate runnable browser")
webbrowser.Error: could not locate runnable browser
I this might be a problem because I am running Windows Subsystem for Linux, making the installing paths of my applications a little weird. There is an option in webbrowser that allows for registration of a browser and it accepts a hard-coded path for the browser, but I don't see this as working generally enough for everbody.
--min-date=yyyy-mm-dd
--max-date=yyyy-mm-dd
Desired solution
I wish to import items / locations I exported from a custom Google MyMaps. I created the pins by uploading a csv with addresses in the format of street city country to Google MyMaps that were correctly parsed and are shown on the Google MyMaps Map. Then, I downloaded the points through this dialogue, trying KML and KMZ format:
I wish I could pass the export files along to be executed by the Python script to have a heat map generated. I also used this converter to convert the data to gpx: https://mygeodata.cloud/conversion.
Errors encountered
When trying to pass the respective export file, I got the respective errors:
Is there any workaround to get the script working for single points in Google MyMaps?
Many thanks for your help!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.