Giter VIP home page Giter VIP logo

go-file-processing-daemon's People

Contributors

richeza avatar

Stargazers

 avatar

Watchers

 avatar

go-file-processing-daemon's Issues

[FEATURE] Add data from Google Location History

Feature request: Google Location history

Is your feature request related to a problem? Please describe.

No it's not.

Describe the solution you'd like

As talked about during the Demo Day and during meetings, it would be really nice to implement Google Location History as a way for contributors to open-source their own Google data, so that it could be used with their permission to have real-world use.

I have done some research, and the Google Location History files are very useful for the bikedataproject. The data dump comes in a highly compressed .zip-file (22.6MB for 453MB of JSON) which would be very nice for users to be able to upload large amounts of data in a very short amount of time.

Playing around with Go a little further, I discovered that the main JSON file is just a large file (split up per 1GB according to Google) containing location, timestamp, type of activities that are predicted & their confidence as a percentage. Using a loop over this data, we can extract all points where the activity ON_BICYCLE has a confidence of at least 50% (can be tweaked though) and create trips based on these points.

Some days contain wrongly classified activities, but this is often just 1-5 points and can be filtered out if there are not enough points in this trip. The Google Location service (dump) contains datapoints between every 1 second to 5 minutes, depending on the movement etc. This makes the trips not super accurate, but more a rough estimate of the trajectory.

We could make another file upload option for users. There could be 2 ways forwards:

  1. Let the user create their data dump, and upload the entire .zip-file. This would be beneficial for the user, as the upload will go very fast. A file between 1-100MB can be uploaded, and the bikedataproject service can then unzip the file in the backend and process the delivered files. The downside of this method is that we will need a lot of validation for malicious or fake files.
  2. Let the user create their data dump, extract the Locationhistory.json file, and make them upload just this. The upside to this is that it get's much easier for us. Less validation for malicious files, no unzipping procedure, etc. The downsides are though that this method will take much longer for the user: manual extraction + uploading times will increase drastically (500MB vs 22MB upload as an example).

My preference goes out to option 1, though can be discussed.

Describe alternatives you've considered (if applicable)

Not applicable.

Additional context

  1. Location history can be retrieved through https://takeout.google.com/settings/takeout

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.