gwu-libraries / social-feed-manager Goto Github PK

"Old SFM" -- manage rules and streams from social data sources, starting with twitter.

License: MIT License

Python 62.89% CSS 11.03% JavaScript 17.28% HTML 5.46% Shell 1.50% Dockerfile 1.84%

social-feed-manager's Introduction

THIS IS THE "OLD" SFM, the original version of Social Feed Manager developed between 2012-2015.

IF YOU ARE LOOKING FOR THE "NEW" SFM, which began development in 2015, head to go.gwu.edu/sfm for full information (and start at https://github.com/gwu-libraries/sfm-ui if you want just the code)

social feed manager

A django application for managing multiple feeds of social media data.

Developed at the GWU Libraries in Washington, DC, USA.

This project was made possible in part by the Institute of Museum and Library Services, Sparks Grant LG-46-13-0257 and the National Historical Publications and Records Commission (NHPRC), Innovation Grant NAR14-DI-50017-14.

Current status

We're only addressing bugs in this version of SFM. However, we're actively developing the new version of SFM with similar and extended functionality.

Documentation

A growing set of documentation is available under the docs/ directory, available also online:

http://social-feed-manager.readthedocs.org/

We encourage you to read the Introduction before implementing this app. Maintaining growing feeds of social media data takes some planning and system adminstration experience. The Introduction should help you prepare for this.

Development

To work on sfm, clone/fork it in github. If you send pull requests from a fork, please make them as atomic as you can.

We are actively working on this, making changes in response to university researcher requests. If the features it supports seem fickle and spotty it's because we're still just getting started and still figuring out what we need it to do. Please get in touch and tell us what you think.

Installation and configuration

Installation and configuration is described here. Alternatively, instructions for installing and configuring with Docker are available here.

social-feed-manager's People

Contributors

Stargazers

Watchers

Forkers

studiozut patrickmj quoideneuf uc-data-services jackiekazil dmmd ranti melodykramer sharonhoward cazzerson oldcrow86 edsu arpithparikh juicyja kayiwa dgo-list dchud digideskio iitians

social-feed-manager's Issues

remove gnip pieces

add column headers on csv export

avoid negative reset times

from September 26, 2012 10:18:16 AM EDT:

-=> fetching json for http://twitter.com/ABCPolitics/statuses/250935942792224768
   using http://api.twitter.com/1/statuses/show/250935942792224768.json
rate limit class/limit/remaining: api/150/127
rate limit resets in -22477815.0167 minutes
updated http://twitter.com/ABCPolitics/statuses/250935942792224768
sleeping -10619441 seconds
[Errno 22] Invalid argument
-=> fetching json for http://twitter.com/PoliticalTicker/statuses/250936790633046017
   using http://api.twitter.com/1/statuses/show/250936790633046017.json
rate limit class/limit/remaining: api/150/126
rate limit resets in -22477815.0333 minutes
updated http://twitter.com/PoliticalTicker/statuses/250936790633046017
sleeping -10703722 seconds
[Errno 22] Invalid argument
-=> fetching json for http://twitter.com/PoliticalTicker/statuses/250936846060773376
   using http://api.twitter.com/1/statuses/show/250936846060773376.json
rate limit class/limit/remaining: api/150/125
rate limit resets in -22477815.0333 minutes
updated http://twitter.com/PoliticalTicker/statuses/250936846060773376
sleeping -10789352 seconds
[Errno 22] Invalid argument
-=> fetching json for http://twitter.com/ABC/statuses/250937806258573312
   using http://api.twitter.com/1/statuses/show/250937806258573312.json
rate limit class/limit/remaining: api/150/124
rate limit resets in -22477815.0333 minutes
updated http://twitter.com/ABC/statuses/250937806258573312
sleeping -10876363 seconds
[Errno 22] Invalid argument
-=> fetching json for http://twitter.com/FoxNews/statuses/250943507924148225
   using http://api.twitter.com/1/statuses/show/250943507924148225.json
rate limit class/limit/remaining: api/150/149
rate limit resets in 60.0 minutes
updated http://twitter.com/FoxNews/statuses/250943507924148225
sleeping 24 seconds

add a TwitterUserSet model and associate with a User

limiting to backend/admin functionality for now rather than user-driven in UI.

user_timeline should handle missing users cleanly

unshorten() no longer working

requests changed behavior with 1.0 release

add timeline of tweets per day to home page

add expanded URL to csv

add only last url for up to two included links in two columns

Consider running as a separate cron job.

handle '#' in hashtag rules

perhaps the best solution is to not create rules that have '#' in them; it might not matter whether the '#' is present in the rule or not to match.

manage rules, streams in the db

make it easier to build / track rulesets and feeds

consider approaches to post-processing items

possible tasks include:

expand urls on the way in
fetch media files
capture web pages
identify additional candidate tweets/users/etc. to capture

[Tue Apr 03 11:25:53 2012] [error] /home/dchud/projects/social-feed-manager/ENV/lib/python2.6/site-packages/django/template/defaulttags.py:1235: DeprecationWarning: The syntax for the url template tag is changing. Load the `url` tag from the `future` tag library to start using the new behavior.

see https://docs.djangoproject.com/en/1.4/ref/templates/builtins/#url for details

tweepy 2.0
filterstream
streamsample currently uses 1
trends api calls changed; remove?
user_timeline should be fine

improve indexing / search

swap in solr or equivalent, and add indexing of:

follows/followers counts, margin
hashtags
geo info
retweet status