Bson file about venmo-data HOT 2 CLOSED

This is a bit late, but I will try to help out.

1. download a torrent client (I used qbittorrent) and google how to open a torrent file using the downloaded client.

2. you will then have to follow the directions in the README file in the repo to get mongoDB and restore the test database.

3. you can now query the data using a python package that interacts with mongoDB (I use pymongo). There might be a better package for interacting with mongoDB directly using pandas, but I usually use numpy so I'm not sure.

4. Go ahead and query the database using the above tool (there should be a lot of documentation on tool's website) and load the query into a pandas df...

just note that depending on your system you may not be able to load all of the data at once (~3 Gigs of raw data without database indices). So you might have filter the data a bit to suit your needs or run your data analysis on partitions of the whole set and analyze metadata at the end. Additionally, if you convert all the data into a CSV, I believe pandas has a chunk method that allows you to read data in chunks:
https://datascience.stackexchange.com/questions/27767/opening-a-20gb-file-for-analysis-with-pandas

from venmo-data.

Not to grave dig an old issue, but if anyone is still having trouble accessing the data I've uploaded the original MongoDB version as well as a CSV copy and a SQLite database to this folder: https://drive.google.com/drive/folders/1m59TMADiwk3gArT-teUUOBlSKokq51ZL?usp=sharing

from venmo-data.