Giter VIP home page Giter VIP logo

em-twitter's Introduction

EM-Twitter

Gem Version Build Status Dependency Status Code Climate

EM-Twitter is an EventMachine-based ruby client for the Twitter Streaming API.

Usage

require 'em-twitter'

options = {
  :path   => '/1/statuses/filter.json',
  :params => { :track => 'yankees' },
  :oauth  => {
    :consumer_key     => ENV['CONSUMER_KEY'],
    :consumer_secret  => ENV['CONSUMER_SECRET'],
    :token            => ENV['OAUTH_TOKEN'],
    :token_secret     => ENV['OAUTH_TOKEN_SECRET']
  }
}

EM.run do
  client = EM::Twitter::Client.connect(options)

  client.each do |result|
    puts result
  end
end

SSL

SSL is used by default (EventMachine defaults to verify_peer => false), and can be configured:

options = {
  :ssl => {
    :private_key_file => "path/to/key.pem",
    :cert_chain_file => "path/to/cert.pem",
    :verify_peer => true
  }
}

client = EM::Twitter.Client.connect(options)

Proxy Support

EM-Twitter includes proxy support via a configuration option:

options = {
  :proxy => {
    :username => 'myusername',
    :passowrd => 'mypassword',
    :uri => 'http://my-proxy:8080'
  }
}

client = EM::Twitter.Client.connect(options)

Error Handling

EM-Twitter supports the following callbacks for handling errors:

  • on_unauthorized
  • on_forbidden
  • on_not_found
  • on_not_acceptable
  • on_too_long
  • on_range_unacceptable
  • on_enhance_your_calm (aliased as on_rate_limited)

Errors callbacks are invoked on a Client like so:

client = EM::Twitter.Client.connect(options)
client.on_forbidden do
  puts 'oops'
end

Reconnections

EM-Twitter has two callbacks for reconnection handling:

client = EM::Twitter.Client.connect(options)
client.on_reconnect do |timeout, count|
  # called each time the client reconnects
end

client.on_max_reconnects do |timeout, count|
  # called when the client has exceeded either:
  # 1. the maximum number of reconnect attempts
  # 2. the maximum timeout limit for reconnections
end

Stream Processing

We recommend using TweetStream for a higher abstraction level interface.

REST

To access the Twitter REST API, we recommend the Twitter gem.

Todo

  • Gzip encoding support (see issue #1 for more information)
  • JSON Parser (see issue #2 for more information)

Inspiration

EM-Twitter is heavily inspired by Vladimir Kolesnikov's twitter-stream. I learned an incredible amount from studying his code and much of the reconnection handling in EM-Twitter is derived/borrowed from his code as are numerous other bits. Eloy Durán's ssalleyware was very helpful in adding SSL Certificate verification as was David Graham's vines.

Testing with EM can be a challenge, but was made incredibly easy through the use of Hayes Davis' awesome mockingbird gem.

Copyright

Copyright (c) 2011-2013 Steve Agalloco. See LICENSE for details.

em-twitter's People

Contributors

larrylv avatar rahuldess avatar rhenium avatar sferik avatar stve avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

em-twitter's Issues

Password authentication/xAuth support

This is not a high priority, but once all other issues are addressed, it would be nice to add support for password authentication and xAuth. This will allow em-twitter to be a feature-for-feature replacement of twitter-stream. I suspect a handful of current users of twitter-stream users depend on password authentication. Forcing them through the OAuth flow could be difficult or impossible.

But, again, not a high priority.

Enable stall warnings

Adding stall_warnings to the requests by default resulted in issues with TweetStream tweetstream/tweetstream#101 and as a result sferik/t

Ideally this is a feature that would be on by default but additional testing is needed to evaluate how well it's supported by Twitter.

Will it lose data potentially?

Hi,

I'm using em-twitter to track the /1.1/statuses/filter.json api.

An example code that includes a periodic timer would like below :

EM.run do
    client = EM::Twitter::Client.connect(options)
    client.each do |status|
        # handle_tweet ...
    end
    EM.add_periodic_timer(5) do
        # other job
    end
end

So the question is, since every 5 seconds event loop will jump to other job (e.g. an async connection to database), will this potentially lose some streaming data we are tracking at this point of time?

Thanks

Failed to connect to proxy

Hello

I am using em-twitter through tweetstream, but I can't get it working with a http proxy.

This is my stack trace, basically it just can't connect to the proxy:

/Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/tweetstream-2.5.0/lib/tweetstream/client.rb:436:in `block in connect': Failed to reconnect after 11 tries. (TweetStream::ReconnectError)
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/em-twitter-0.2.2/lib/em-twitter/connection.rb:296:in `call'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/em-twitter-0.2.2/lib/em-twitter/connection.rb:296:in `invoke_callback'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/em-twitter-0.2.2/lib/em-twitter/connection.rb:268:in `rescue in schedule_reconnect'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/em-twitter-0.2.2/lib/em-twitter/connection.rb:263:in `schedule_reconnect'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/em-twitter-0.2.2/lib/em-twitter/connection.rb:93:in `unbind'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/eventmachine-1.0.3/lib/eventmachine.rb:1440:in `event_callback'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/tweetstream-2.5.0/lib/tweetstream/client.rb:388:in `start'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/tweetstream-2.5.0/lib/tweetstream/client.rb:131:in `filter'
    from /Users/Marco/.rvm/gems/ruby-1.9.3-p448/gems/tweetstream-2.5.0/lib/tweetstream/client.rb:122:in `locations'
    from twitter_stream.rb:25:in `<main>'

I tried on 2 different machines, my local dev environment with Charles set up as http proxy and the production machine, which uses a proxy to go out to the internet.
On both machines, same result.

This is what I get in Charles, not sure if it's helpful:

screen shot 2013-10-20 at 16 42 20

Did anyone get it working?

EDIT: tried with SquidMan as well, same result unfortunately.

Events happening twice with Site Streams

When using Twitter Site Streams, I seem to be getting all events twice. I connect like this:

require 'em-twitter'

options = {
 :path => "/1.1/site.json",
 :params => {:follow => "2030131"},
 :host => "sitestream.twitter.com",
 :method => "POST",
 :on_inited => nil,
 :proxy => nil,
 :oauth => 
  {:consumer_key => mykey,
   :consumer_secret => mysecret,
   :token => mytoken,
   :token_secret => mytokensecret
   }
}

EM.run do
  client = EM::Twitter::Client.connect(options)

  client.each do |result|
    puts result
  end
end

And get results like this:

{"control":{"control_uri":"/1.1/site/c/1_277198_9f2c58a4474597af75cdfaeaaeea2331"}}
{"control":{"control_uri":"/1.1/site/c/1_277198_9f2c58a4474597af75cdfaeaaeea2331"}}
{"for_user":2030131,"message":{"friends":[807095,2097571,6594742,428333,759251,742143,19290942,814760,612473,7589572,23937508,89168924,69329527,12552,7111412,1652541,178311661]}}
{"for_user":2030131,"message":{"friends":[807095,2097571,6594742,428333,759251,742143,19290942,814760,612473,7589572,23937508,89168924,69329527,12552,7111412,1652541,178311661]}}

And the same for receiving messages etc.

Out of curiosity I tried replacing the path and params with these from the em-twitter example:

:path   => '/1/statuses/filter.json',
:params => { :track => 'yankees' },

and the same thing happened... until I removed this line:

:host => "sitestream.twitter.com",

Then I only received one event for each occurrence of 'yankees'.

It's possible this is some weird Twitter Site Streams thing, rather than em-twitter. But I'm not sure how to test any closer to Twitter, and am struggling to narrow it down any further. Any suggestions for isolating this more?

example code

How would I implement the example code in a controller?

NoMethodError: undefined method older_than?' for nil

From tweetstream/tweetstream#83

NoMethodError: undefined method older_than?' for nil:NilClass
from /var/lib/gems/1.9.1/gems/em-twitter-0.1.2/lib/em-twitter/connection.rb:113:installed?'
from /var/lib/gems/1.9.1/gems/em-twitter-0.1.2/lib/em-twitter/connection.rb:58:in block in post_init'
from /var/lib/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/em/timers.rb:56:incall'
from /var/lib/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/em/timers.rb:56:in fire'
from /var/lib/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:incall'
from /var/lib/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in run_machine'
from /var/lib/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:inrun'
from /var/lib/gems/1.9.1/gems/tweetstream-2.0.0/lib/tweetstream/client.rb:343:in start'
from /var/lib/gems/1.9.1/gems/tweetstream-2.0.0/lib/tweetstream/client.rb:117:infilter'
from /var/lib/gems/1.9.1/gems/tweetstream-2.0.0/lib/tweetstream/client.rb:84:in track'
from tweetstreamwrapper.rb:22:incrossYourFingers'
from (irb):32
from /usr/bin/irb:12:in `'

Test suite fails with Ruby 2.0

I'm starting to look at the tweetstream ruby2 issue raise here: tweetstream/tweetstream#117

I installed ruby-2.0.0-p247 (I'm using 195, so this is effectively a clean install), did bundle install, then rake spec.

I'm posting this since it's a failure, rather than a stall, and might be immediately useful.

I ran the test suite three times. The first time:

Finished in 1 minute 7.37 seconds
48 examples, 1 failure

The second time:

Finished in 25.07 seconds
23 examples, 1 failure

The third time the suite stalled.

$ rake spec
...............................................F

Failures:

  1) EM::Twitter::Connection reconnections reconnector setting after successful connect resets the network failure reconnector
     Failure/Error: Unable to find matching line from backtrace
       Exactly one instance should have received the following message(s) but didn't: reset
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-mocks-2.14.1/lib/rspec/mocks/any_instance/recorder.rb:84:in `verify'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-mocks-2.14.1/lib/rspec/mocks/space.rb:18:in `block in verify_all'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-mocks-2.14.1/lib/rspec/mocks/space.rb:17:in `each_value'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-mocks-2.14.1/lib/rspec/mocks/space.rb:17:in `verify_all'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-mocks-2.14.1/lib/rspec/mocks.rb:18:in `verify'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/mocking/with_rspec.rb:18:in `verify_mocks_for_rspec'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example.rb:313:in `verify_mocks'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example.rb:305:in `run_after_each'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example.rb:120:in `block in run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example.rb:254:in `with_around_each_hooks'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example.rb:111:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:400:in `block in run_examples'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:396:in `map'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:396:in `run_examples'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:381:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `block in run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `map'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `block in run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `map'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/example_group.rb:382:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/command_line.rb:28:in `map'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/command_line.rb:28:in `block in run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/reporter.rb:58:in `report'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/command_line.rb:25:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/runner.rb:80:in `run'
     # /usr/local/rvm/gems/ruby-2.0.0-p247/gems/rspec-core-2.14.1/lib/rspec/core/runner.rb:17:in `block in autorun'

Finished in 1 minute 7.37 seconds
48 examples, 1 failure

Failed examples:

rspec ./spec/em-twitter/connection_reconnect_spec.rb:34 # EM::Twitter::Connection reconnections reconnector setting after successful connect resets the network failure reconnector

Too many reconnects causing user level 420 rate limit responses

When running the the sample example from the repo with a common filter tag like '#2' causes the em-twitter gem to reconnect to the filter streaming api method... which causes twitter to send the following:

{"disconnect":{"code":7,"stream_name":"<user_name>","reason":"admin logout"}}

The client then reconnects and repeats the process. Doing this for just a little bit will cause Twitter to rate limit the user (respond with 420 code). The streaming api is intended to be a long-lived connection and doesn't require multiple reconnects.

Are any other em-twitter gem users seeing this issue?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.