jcode89 / datadotpy Goto Github PK

Both versions are in three files, but TweetCollector.py Has too much functionality that should be split up. TweetCollector.py is the go to file to gather live stream, or at least that is the goal.

Is it better to use Spark, or write your own scripts to pull data and analyze it?

There are two new classes:

• Configuration - contains configuration information
• EnvironmentConfiguration - returns a populated Configuration instance from environment variables

These have not yet been integrated into DataDotPy

This will allow for easier testing and the ability to move between Twitter clients in the figure if desired.

Upon learning about this new module tqdm. I feel it would be nifty to be able to show the user the progress of the stream. It isn't crucial to the overall program, but a nice little feature that would make it friendlier to use.

This doesn't have to be done right away, but once we know the general format of the program won't be changing much, the README.md should be updated so the user understands how to use the program.

Create an EnvironmentConfiguration class that will populate the configuration data from environment variables.

Make a generic configuration class that contains configuration data like twitter API keys

This can later be leveraged to create a service to populate the twitter API info from environment variables or other sources.

Progress has been made in adding in some further analysis, however this has not been reflected properly in the code. Since I created a Flask app and made some modifications, I am going to properly update TweetCollector.py so that some of those same modifications can be utilized in the CLI version.

RethinkDB allows you to get notified of changes to your DB in real time. Research is needed to see if we can do real time analysis on the database.

Considering wrapping this entire program in Docker containers for easier portability.

Currently a number of classes store configuration data for use when querying twitter. Rather then doing this, configuration data should be passed directly to the method that queries twitter.