twitter / hbc Goto Github PK

Hello
Im using code like here: https://github.com/sgykfjsm/scala_example/blob/fe75f540b15dfe4dc8a305e9d6dfed21a6bbebc0/src/main/scala/hbc/SitestreamExample.scala

my auth is for sure correct ( I use the same for user streams which works fine ) and here I get
20:17:34.231 WARN c.t.h.h.ClientBase: hosebird-client-0 Error connecting w/ status code - 403, reason - Forbidden

Regards

Hello,

Im not sure if i proper use it but after t4jClient.stop() when I will launch reconnect() connection doesnt alive. But when I launch connect() again I got errors that is already running..

Regards

BasicClient has a private ClientBase member and exposes stop() but not shutdown(). This causes the RateTracker executor threads to stay around. I think BasicClient should provide a shutdown as well as stop.

A little background. I have a condition where I need to disconnect the client and possibly reconnect at an unknown future time. Right now the only way I see to do this is to stop the BasicClient and create a new one. In stopping and creating a new BasicClient I notice an ever growing number of rateTracker threads. A BasicClient.disconnect() which would keep the processing thread around waiting for me to call reconnect() would also solve my problem because then I wouldn't need to keep creating new BasicClients.

I am a C# asp.net programmer. With no clue of Java.

It would be great if you give a sample of a working program so newbies like me can understand it. It would be great if you can make a guide for dummies on how to set it up and run it.
I also urge you to give some way so that .net users can also use this. I am looking for a way to feed tweets to a msmq.Pls guide me.
Thank you

Hi,
When I try to compile the project I get the following error:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbc-twitter4j: Compilation failure
[ERROR] /Users/gdfm/workspace/hbc/hbc-twitter4j/src/main/java/com/twitter/hbc/twitter4j/parser/JSONObjectParser.java:[29,38] <anonymous com.twitter.hbc.twitter4j.parser.JSONObjectParser$1> is not abstract and does not override abstract method compareTo(twitter4j.StatusDeletionNotice) in java.lang.Comparable

Hi,
is there a chance to check for which user are running stream for current IP adress ?
Thanks

In class UserstreamHandler I get
13:44:15.133 INFO c.t.h.t.BaseTwitter4jClient: Unhandled event: onFollow

but I have declared it.

Hello,

Im using UserStreamsHandler and it works very well.
But there seems to be one problem.
When stream for User A is stopped, and for example in a minute started again,
when new tweet is comming, 1st one is duplicated,
after this 1st msg on stream HBC closed it well, so next msg in stream are not duplicated.

Steps:
User A shutdown stream
minute later User A connect to stream again
new tweets is comming: duplicate
next tweets: not duplicate.

Regards

Hi, I'm trying to switch from twitter4j to hbc-twitter4j.
I found that in hbc-twitter4j StatusListener#onStatus, I can not get the rawjson by calling
DataObjectFactory.getRawJSON(status)

But in the original twitter4j, I can set the configuration to save the rawjson with each tweet object by:
ConfigurationBuilder.setJSONStoreEnabled(true);

In hbc-twitter4j, I can't find a way to set this and therefore will get "null" when calling DataObjectFactory.getRawJSON(status)

Trying the StatusesSampleEndpoint example, I get:

13/03/05 22:40:40 INFO httpclient.BasicClient: New connection executed: ExampleApplication, endpoint: /1.1/statuses/sample.json?delimited=length
13/03/05 22:40:40 INFO httpclient.ClientBase: ExampleApplication Establishing a connection
13/03/05 22:40:43 WARN httpclient.ClientBase: ExampleApplication Error connecting w/ status code - 503, reason - Service Unavailable

Using the OAuth tool on dev.twitter.com/apps, I can cut, paste, and successfully execute the curl command generated for https://stream.twitter.com/1.1/statuses/sample.json.

Also fail to connect using the tokens and secrets from the examples README.

There is a problem using endpoint.locations(List)
Calling Joiner.on(',').join(locations) gives me

com.twitter.hbc.core.endpoint.Location@3794d372

and not what it should be 42.692871,16.972741,46.933594,29.152161

Hi,

can You guys push 1.3.4 version to maven ? I would like to migrate to twitter4j 3.0.3 and newer version of hbc.
Thanks

Hi,
I am trying to run the test example. However, the below error occurs.

/hbc-master/hbc-twitter4j/src/main/java/com/twitter/hbc/twitter4j/parser/JSONObjectParser.java:[29,38] <anonymous com.twitter.hbc.twitter4j.parser.JSONObjectParser$1> is not abstract and does not override abstract method compareTo(twitter4j.StatusDeletionNotice) in java.lang.Comparable

yes.. i got track from user stream on HBC, plz confirm this code is right or wrong, because i am new user of hbc. This is working code, I got right result, with keyword search. plz just confirm my approach is right or wrong or it may give some problem in certain scenario. thank you.

public class UserStreamExample
{
public static void oauth(String consumerKey, String consumerSecret, String token, String secret) throws InterruptedException
{
BlockingQueue queue = new LinkedBlockingQueue(10000);

        UserstreamEndpoint endpoint = new UserstreamEndpoint();
        endpoint.addQueryParameter("track","farah");

        Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);

        // Create a new BasicClient. By default gzip is enabled.
        Client client = new ClientBuilder()
          .hosts(Constants.USERSTREAM_HOST)
          .endpoint(endpoint)
          .authentication(auth)
          .processor(new StringDelimitedProcessor(queue))
          .build();

        // Establish a connection
        client.connect();

        // Do whatever needs to be done with messages
        for (int msgRead = 0; msgRead < 1000; msgRead++) {
          String msg = queue.take();
          System.out.println(msg);
        }

        client.stop();
 }

 public static void main(String[] args) {
      try {
          UserStreamExample.oauth(args[0], args[1], args[2], args[3]);
      } catch (InterruptedException e) {
          System.out.println(e);
      }
  }

}

Since I upgraded to latest version, I've noticed the client shutting down the connection manager after a period of time (possible due to inactivity on the stream because it seems to happen after 2 or so days of inactivity).

Is this normal behavior?

2013-07-28 13:49:54,416 | hosebird-client-io-thread-0 | DEBUG | org.apache.http.impl.conn.PoolingClientConnectionManager | Connection manager is shutting down
2013-07-28 13:49:54,417 | hosebird-client-io-thread-0 | DEBUG | org.apache.http.impl.conn.DefaultClientConnection | Connection 0.0.0.0:51756<->199.16.156.81:443 closed
2013-07-28 13:49:54,418 | hosebird-client-io-thread-0 | DEBUG | org.apache.http.impl.conn.DefaultClientConnection | Connection 0.0.0.0:51756<->199.16.156.81:443 closed
2013-07-28 13:49:54,418 | hosebird-client-io-thread-0 | DEBUG | org.apache.http.impl.conn.PoolingClientConnectionManager | Connection manager shut down

I've been running the hosebird client for 1 month now and it's been working perfectly. Yesterday it missed a tweet and there was nothing in the log to suggest why. Due to this being a production issue I had to restart it immediately to get it working again.
I would like to be able to reproduce this issue, but have no idea how. I've now turned on debug and will monitor closely.
Have there been any other issues with the User Stream Endpoint missing a tweet?

BTW, I had 2 application running at the time, and 1 of them received the tweet.

We should add the Clirr Maven Plug-in to ensure we aren't breaking binary compat:
http://mojo.codehaus.org/clirr-maven-plugin/usage.html

My firehose client I am trying to replace with hbc has a feature that saves messages which aren't valid UTF-8 to files. This has proven useful because it has found and helped fix bugs in the Firehose. Right now the utf8 decoding happens roughly with this stack:

com.twitter.hbc.common.DelimitedStreamReader.readLine
...
com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage
com.twitter.hbc.httpclient.ClientBase.processConnectionData

readLine() calls new String(...), constructor which does this with invalid utf8:

This method always replaces malformed-input and unmappable-character sequences with this charset's default replacement string.

I believe this means the original data will be lost and worse the problem will likely be hidden. It would be nice if we could get at the invalid data to log it. One option would be returning byte[] instead of String. Another option would be to use a CharsetDecoder with CodingErrorAction.REPORT and perhaps add an event with the bad byte[]. The second option is a less invasive API change.

Thoughts?

Tried both 'FilterStreamExample' and 'SampleStreamExample'.

METHOD 1: mvn install && mvn exec:java -pl hbc-example -Dconsumer.key=XYZ -Dconsumer.secret=SECRET -Daccess.token=ABC -Daccess.token.secret=ABCSECRET

Error: Did not receive a message in 5 seconds
Did not receive a message in 5 seconds
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - sampl
eExampleClient Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - sampl
eExampleClient Establishing a connection

METHOD 2: Change the following in the example class:
From: Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);
To: Authentication auth = new BasicAuth("username", "password");

I want to use my applications consumerKey and consumerSecret as well as I want to give access to my application user to their own account details and tweets by their own keys like token and secret. is that possible?

my keys are like this ;
consumerKey : myAppsKey
consumerSecret : myAppsSecret
token : clientsToken (getting after oauth)
secret : clientsSecret (getting after oauth)

i tried with maven application but it gives me error like;

[INFO] Building Hosebird Client Examples 1.4.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example <<<
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example ---
[com.twitter.hbc.example.UserStreamExample.main()] INFO com.twitter.hbc.httpclie
nt.BasicClient - New connection executed: hosebird-client-0, endpoint: /1.1/user
.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 401, reason - Unauthorized
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 exit event - Retries exhausted
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Shutting down httpclient connection manager

I'd like to replace my firehose client with hbc, but I've hit a wall porting some existing functionality. Essentially my counter-part to RateTracker does persistence to ZooKeeper so backfill count can be calculated over process restarts.

My plan is to make RateTracker an interface, move the current functionality into an implementation BasicRateTracker, and allow RateTracker to be set with ClientBuilder. Does this seem feasible and something you'd like to merge?

This is with 1.3.0. I did not see this error until the past few days, but now it is happening fairly frequently.

A capture utility based on HBC will proceed for a while:

13/03/18 10:12:17 INFO capture.Capture: Processed 21302 tweets (43/sec)
13/03/18 10:12:47 INFO capture.Capture: Processed 22527 tweets (40/sec)
13/03/18 10:13:17 INFO capture.Capture: Processed 23780 tweets (41/sec)
13/03/18 10:13:47 INFO capture.Capture: Processed 25009 tweets (40/sec)
13/03/18 10:14:17 INFO capture.Capture: Processed 26240 tweets (41/sec)

Then:

13/03/18 10:14:39 WARN httpclient.ClientBase: ExampleApplication Unknown error processing connection: java.lang.NumberFormatException: For input string: "ownload\/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":313530233020104704,"in_reply_to_status_id_str":"313530233020104704","in_reply_to_user_id":489399228,"in_reply_to_user_id_str":"489399228","in_reply_to_screen_name":"FezTht70s","user" {"id":719393096,"id_str":"719393096","name":"Jack Dean","screen_name":"Jack_Dean123","location":"","url":null,"description":null,"protected":false,"followers_count":103,"friends_count":128,"listed_count":0,"created_at":"Fri Jul 27 04:10:15 +0000 2012","favourites_count":289,"utc_offset":null,"time_zone":null,"geo_enabled":false,"verified":false,"statuses_count":579,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"C0DEED","profile_background_image_url":"http:\/\/a0.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/si0.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2437219557\/image_normal.jpg","profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2437219557\/image_normal.jpg","profile_banner_url":"https:\/\/si0.twimg.com\/profile_banners\/719393096\/1360894264","profile_link_color":"0084B4","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweet_count":0,"entities":{"hashtags":[{"text":"20ThingsIDontLike","indices":[13,31]}],"urls":[],"user_mentions":[{"screen_name":"FezTht70s","name":"Fez","id":489399228,"id_str":"489399228","indices":[1,11]},{"screen_name":"TaylorrCrowley","name":"Taylor Crowley","id":572080692,"id_str":"572080692","indices":[110,125]}]},"favorited":false,"retweeted":false,"filter_level":"medium"}"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
    at java.lang.Integer.parseInt(Integer.java:449)
    at java.lang.Integer.parseInt(Integer.java:499)
    at com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:55)
    at com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:25)
    at com.twitter.hbc.core.processor.AbstractProcessor.process(AbstractProcessor.java:44)
    at com.twitter.hbc.httpclient.Connection.processResponse(Connection.java:51)
    at com.twitter.hbc.httpclient.ClientBase.processConnectionData(ClientBase.java:240)
    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
13/03/18 10:14:39 INFO httpclient.ClientBase: ExampleApplication Done processing, preparing to close connection
13/03/18 10:14:47 INFO capture.Capture: Processed 27188 tweets (31/sec)
13/03/18 10:15:17 INFO capture.Capture: Processed 27188 tweets (0/sec)

And at this point the processor is in some kind of zombie state, it won't reconnect:

13/03/18 10:15:17 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:15:47 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:16:17 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:16:47 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:17:17 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:17:47 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:18:17 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:18:47 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:19:17 INFO capture.Capture: Processed 27188 tweets (0/sec)
13/03/18 10:19:47 INFO capture.Capture: Processed 27188 tweets (0/sec)
[... forever ...]

Should I try using HEAD aka 1.3.1-SNAPSHOT?

I'm in the process of trying to migrate an application from using Twitter4J's site stream implementation over to hbc-twitter4j, and I've run into two somewhat critical but related bugs.

First, SitestreamEndpoint.java provides no way to supply the "follow" parameter to the connection endpoint, which is described by https://dev.twitter.com/docs/api/1.1/get/site as required. Not supplying that field results in the endpoint returning 400 BAD REQUEST on any connection attempt.

After hacking SitestreamEndpoint.java to accept and pass that parameter through, I ran into a second issue in JSONObjectParser.getSitestreamUser(...). All messages received from the site stream are passed through that method including the control stream message documented in https://dev.twitter.com/docs/streaming-apis/streams/site/control. The control stream message, however, does not contain a "for_user" field, so the method ends up throwing a JSONException. This results in the stream never being opened because the stream ID is never captured.

I hacked around this second problem as well to see what would happen and was eventually able to get my site stream connection running, but given the presence of these two bugs either of which completely prevent the Site Streams feature from working at all, I have significant concerns about deploying this code into a production environment.

Could anyone provide a simple code snippet on using user streams?

I'm trying to use it by simply changing the endpoint type passed to the client builder but I'm getting 404 error code all the time. Here you are the stack trace:

2013-04-29 09:32:02,795 INFO [com.twitter.hbc.httpclient.BasicClient] - New connection executed: Hosebird-Client-03, endpoint: /1.1/user.json?delimited=length&stall_warnings=true
2013-04-29 09:32:03,223 INFO [com.twitter.hbc.httpclient.ClientBase] - Hosebird-Client-03 Establishing a connection
2013-04-29 09:32:04,260 WARN [com.twitter.hbc.httpclient.ClientBase] - Hosebird-Client-03 Error connecting w/ status code - 404, reason - Not Found
2013-04-29 09:32:04,260 INFO [com.twitter.hbc.httpclient.ClientBase] - Hosebird-Client-03 exit event - Fatal error code: 404
2013-04-29 09:32:04,260 INFO [com.twitter.hbc.httpclient.ClientBase] - Hosebird-Client-03 Done processing, preparing to close connection
2013-04-29 09:32:04,266 INFO [com.twitter.hbc.httpclient.ClientBase] - Hosebird-Client-03 Shutting down httpclient connection manager

Thanks.

If you look at hbc-core's pom.xml: https://github.com/twitter/hbc/blob/master/hbc-core/pom.xml#L41

Pretty sure findbugs should be in the test scope as it's not a runtime dependency.

I picked up the SitestreamExample and ran it within some existing framework. (I have a working site streams impl using twitter4j today and am looking at hbc as a replacement.) I don't believe I have changed anything in the example but when it attempts to "addUser" to the site stream it throws the stack below.

Stepped through but don't know the code well enough to understand why. Anything obvious I'm missing?

Exception in thread "HosebirdStreamWorker" java.lang.IllegalStateException: Invalid use of BasicClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one.
at org.apache.http.impl.conn.BasicClientConnectionManager.getConnection(BasicClientConnectionManager.java:162)
at org.apache.http.impl.conn.BasicClientConnectionManager$1.getConnection(BasicClientConnectionManager.java:139)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:137)
at org.apache.http.impl.client.DecompressingHttpClient.execute(DecompressingHttpClient.java:108)
at com.twitter.hbc.SitestreamController.makeControlStreamRequest(SitestreamController.java:97)
at com.twitter.hbc.SitestreamController.getFriends(SitestreamController.java:82)
at com.twitter.hbc.SitestreamController.getFriends(SitestreamController.java:73)
at com.hubspot.broadcast.tracking.twitter.HosebirdStreamWorker.oauth(HosebirdStreamWorker.java:249)

Hello

heres my code:
https://gist.github.com/daaatz/5221554

But on connect I have:
hosebird-client-0 Error connecting w/ status code - 401
Is this problem w lib or my code ?

Thanks

i am new user, can anybody give me some guidelines how to use hose bird all API functionality? I run all examples given in hosebird Examples, i create new UserStreamExample and now i want to read mentions of user, i don not find any endpoint or event related to this, so if anyone tried already then plz suggest me some links or pointers or code snippets.

Hi,

guys do You know maybe how to repair issues which I have with new scala version ? 2.10.2, on 2.9.2 works fine not when i run connect() or disconnect() I get exceptions:

SLF4J: Failed toString() invocation on an object of type [com.twitter.hbc.httpclient.ClientBase]
java.lang.NoSuchMethodError: scala.Predef$.intWrapper(I)Lscala/runtime/RichInt;
at com.twitter.joauth.UrlEncoder$.apply(UrlEncoder.scala:42)
at com.twitter.joauth.UrlEncoder.apply(UrlEncoder.scala)
at com.twitter.hbc.core.endpoint.BaseEndpoint.addQueryParameter(BaseEndpoint.java:98)
at com.twitter.hbc.core.endpoint.DefaultStreamingEndpoint.addDefaultParams(DefaultStreamingEndpoint.java:48)
at com.twitter.hbc.core.endpoint.BaseEndpoint.getURI(BaseEndpoint.java:57)
at com.twitter.hbc.httpclient.ClientBase.toString(ClientBase.java:336)
at org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:304)
at org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:276)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:230)
at ch.qos.logback.classic.spi.LoggingEvent.(LoggingEvent.java:114)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:468)
at ch.qos.logback.classic.Logger.filterAndLog_1(Logger.java:442)
at ch.qos.logback.classic.Logger.info(Logger.java:632)
at com.twitter.hbc.httpclient.BasicClient.connect(BasicClient.java:111)
at com.twitter.hbc.twitter4j.v3.BaseTwitter4jClient.connect(BaseTwitter4jClient.java:58)
at com.twitter.hbc.twitter4j.v3.Twitter4jUserstreamClient.connect(Twitter4jUserstreamClient.java:30)

Thanks

It would be nice if I could hit ctrl-c to kill the app and drop the connection (nicely). When I do it now java just becomes a half-dead process that I need to kill from another window.

Not sure if I am missing something very obvious here by following the steps present in README.
When I followed the steps last week everything worked fine. I updated the code now and 'mvn compile && mvn exec:java -pl hbc-example' command is throwing the following error.

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hosebird Client Examples 1.3.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example <<<
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.909s
[INFO] Finished at: Mon Mar 04 18:14:16 EST 2013
[INFO] Final Memory: 8M/83M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hbc-example: Could not resolve dependencies for project com.twitter:hbc-example:jar:1.3.1-SNAPSHOT: The following artifacts could not be resolved: com.twitter:hbc-twitter4j:jar:1.3.1-SNAPSHOT, com.twitter:hbc-core:jar:1.3.1-SNAPSHOT: Failure to find com.twitter:hbc-twitter4j:jar:1.3.1-SNAPSHOT in https://oss.sonatype.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of sonatype-nexus-snapshots has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

I could see that https://oss.sonatype.org/content/repositories/snapshots/com/twitter has 1.2.1-SNAPSHOT related jar files [eg: hbc-core]. So, I modified all the poms to 1.2.1-SNAPSHOT version and then I was able to execute the above command successfully though.

how to implement this in GAE? I do not know Maven, so please give me some guidelines. thank you in advance.

Hey twitter guys! Thanks for the HoseBird! I never got around to getting the fallback stuff working on my own and was hoping that someone would release something like HoseBird to the public. I've been using curl since Oct 2010! :)

I'm trying to get the hbc-example to work. I'd like to connect to the streaming sample endpoint using BasicAuth. I see where I can use OAuth or BasicAuth, so I think that I've got that covered. I've not changed anything from the git repo yet. I'm sort of a noob when it comes to maven, so I may be doing something really simple and stupid here. Here's my build output. Not sure why this is failing.

build output here:
https://gist.github.com/scumola/5060747

CentOS 5.2, java 1.7, mvn 3.0.5

can anybody give me mapping about HoseBird API files used for what? means which HBC API is used for which stream, like for UserStream we have UserStreamEndpoint, for search we have StatusesFilterEndpoint, like this what other API do and how to use them?

Hi,
I want to read other streams like sent tweets, Direct Messages, mentions etc so which endpoints give me this tweets from hosebird api?

how to use UserstreamEndpoint methods like withFollowings(..) and allReplies(...), and where in client and how will be result. I tried but its not working. it is not giving error as well not any result.

i tried to create user stream using UserStreamEndPoint but i get error like following :

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hosebird Client Examples 1.4.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example <<<
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example ---
[com.twitter.hbc.example.UserStreamExample.main()] INFO com.twitter.hbc.httpclie
nt.BasicClient - New connection executed: hosebird-client-0, endpoint: /1.1/user
.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Error connecting w/ status code - 404, reason - Not Found
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 exit event - Fatal error code: 404
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Done processing, preparing to close connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb

ird-client-0 Shutting down httpclient connection manager

Please give me some pointer or links or examples so I can use all endpoints and create all kind of streams. I want to create all streams, I am novice user.

HBC uses Joauth which is built using scala 2.9.2. I was hoping to try out HBC in a scala 2.10 project...

So, there are only a few options:

1. Drop joauth for an alternative.
2. Cross-release joauth and let users drop the dependency (unfortunately sbt is the best build system to do this and I see it is currently using maven)
3. Shade joauth to hide the scala-ness.

Any chance you guys are interested in either of these options?

Hello everybody,

We have implemented the hosebird client (hbc-core-1.4.0.jar) to consume twitter firehose stream, everything works great 👍 , except the backfill param "count" which doesn't always work. That to be said, we set the backfill on startup, and we can see it works on first connection. However, we don't see the backfill param "count" anymore whenever it re-connects due to error during consuming.

We went through the HBC's source code few times, and it looks like backfill should always work with rate retrieved from rate tracker (we tried with the default rate tracker in ClientBuilder and our own instance of rate tracker as well).

Here are some logs around first connection, where we can see the backfill param "count"

2013-09-20 08:59:05,725 INFO TwitterStreamReader com.twitter.hbc.httpclient.BasicClient - New connection executed: HosebirdClient, endpoint: /1.1/statuses/firehose.json?count=150000&allow_restricted=true&delimited=length&partitions=0%2C1%2C2%2C3&stall_warnings=true
2013-09-20 08:59:06,080 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Establishing a connection
2013-09-20 08:59:06,085 INFO TwitterEventQueue TwitterEventQueue - [CONNECTION_ATTEMPT] - GET https://stream.twitter.com/1.1/statuses/firehose.json?count=150000&allow_restricted=true&delimited=length&partitions=0%2C1%2C2%2C3&stall_warnings=true HTTP/1.1
2013-09-20 08:59:06,741 DEBUG hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Connection successfully established
2013-09-20 08:59:06,742 INFO TwitterEventQueue TwitterEventQueue - [CONNECTED] - HTTP/1.1 200 OK
2013-09-20 08:59:06,742 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Processing connection data
2013-09-20 08:59:06,742 INFO TwitterEventQueue TwitterEventQueue - [PROCESSING] - Processing messages

Here are some logs around re-connection, where the backfill param "count" is missing

2013-09-20 11:28:27,083 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Disconnected during processing - will reconnect
2013-09-20 11:28:27,083 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Done processing, preparing to close connection
2013-09-20 11:28:27,083 INFO TwitterEventQueue TwitterEventQueue - [DISCONNECTED] - Read timed out
2013-09-20 11:28:27,093 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Establishing a connection
2013-09-20 11:28:27,093 INFO TwitterEventQueue TwitterEventQueue - [CONNECTION_ATTEMPT] - GET https://stream.twitter.com/1.1/statuses/firehose.json?allow_restricted=true&delimited=length&partitions=0%2C1%2C2%2C3&stall_warnings=true HTTP/1.1
2013-09-20 11:28:27,684 DEBUG hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Connection successfully established
2013-09-20 11:28:27,684 INFO TwitterEventQueue TwitterEventQueue - [CONNECTED] - HTTP/1.1 200 OK
2013-09-20 11:28:27,684 INFO hosebird-client-io-thread-0 com.twitter.hbc.httpclient.ClientBase - HosebirdClient Processing connection data
2013-09-20 11:28:27,685 INFO TwitterEventQueue TwitterEventQueue - [PROCESSING] - Processing messages

From source code we can see that it was IOExcepton causing the disconnection

catch (IOException ex) {
// connection issue? whatever. let's try connecting again
// we can't really diagnosis the actual disconnection reason without parsing (looking at disconnect message)
// but we can make a good guess at when we're stalling. TODO
logger.info("{} Disconnected during processing - will reconnect", name);
statsReporter.incrNumDisconnects();
addEvent(new Event(EventType.DISCONNECTED, ex));
}

According to a line of comment on ClientBase.run() method
"if IOException, time to restart the connection: handle http connection cleanup, do some backoff, set backfill"
So, we expect the backfill param would still be set before next connection, but it didn't.

Any help or direction will be much appreciated!

Thanks in advance!

Jack

After this much output, it is not showing anything, i waited for 30 minutes but its blank.

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Hosebird Client Examples 1.4.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example <<<
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) @ hbc-example ---
[com.twitter.hbc.example.UserStreamExample.main()] INFO com.twitter.hbc.httpclie
nt.BasicClient - New connection executed: hosebird-client-0, endpoint: /1.1/user
.json?delimited=length&stall_warnings=true
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Establishing a connection
[hosebird-client-io-thread-0] INFO com.twitter.hbc.httpclient.ClientBase - hoseb
ird-client-0 Processing connection data
{"friends":[15676118,372475592,108198542,19761445,179305666,9720292,1391679324,1
39344746,6844292,94652818,88344304,114387883,275126119,380927048,95481928,369471
121,19895282,133245413,376825877,163792579,122995784,44602678,6253282,304298721,
40885516,129722129,11069462,22256645,7588892,523503830,183709371,17093617,552095
2,132385468,20536157,819797,67272817,40215499,262202555,4933401,14230524,1333476
2,14602130,18100055,61505026,8633582]}

my code is like this :

BlockingQueue queue = new LinkedBlockingQueue(10000);
//BlockingQueue eventQueue = new LinkedBlockingQueue(1000);

        UserstreamEndpoint endpoint = new UserstreamEndpoint();

        Authentication auth = new OAuth1(consumerKey, consumerSecret, token, secret);
        // Authentication auth = new BasicAuth(username, password);

        // Create a new BasicClient. By default gzip is enabled.
        Client client = new ClientBuilder()
          .hosts(Constants.USERSTREAM_HOST)
          .endpoint(endpoint)
          .authentication(auth)
          .processor(new StringDelimitedProcessor(queue))
          .build();

        // Establish a connection
        client.connect();

        // Do whatever needs to be done with messages
        for (int msgRead = 0; msgRead < 1000; msgRead++) {
          String msg = queue.take();
          System.out.println(msg);
        }

        client.stop();

what is wrong in this code?

The current implementation of hbc-twitter4j is built against twitter4j 2.2.x, but this version is designed for Twitter API 1.0, currently deprecated and marked for removal in less than a month. twitter4j 3.0 is designed for API 1.1, and hbc-twitter4j should be updated to use it instead.

I'm piping my data from stdout into my app, so all of the debug messages from hbc are confusing my app. Would be nice if the debugging messages went to stderr.

My Code with output is here

https://gist.github.com/iamtechaddict/6397991

i'm using scala 2.9.2 and sbt 0.13.0

Hi, it looks like HBC does not support HTTP/HTTPS proxy. Any plan?

Getting a periodic NumberFormatException error in StringDelimitedProcessor. Using Basic Auth and most of the code in the sample streaming client.

[hosebird-client-io-thread-0] WARN com.twitter.hbc.httpclient.ClientBase - sampleExampleClient Unknown error processing connection: 

[a bunch of JSON here]

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:481)
at java.lang.Integer.parseInt(Integer.java:514)
at com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:55)
at com.twitter.hbc.core.processor.StringDelimitedProcessor.processNextMessage(StringDelimitedProcessor.java:25)
at com.twitter.hbc.core.processor.AbstractProcessor.process(AbstractProcessor.java:44)
at com.twitter.hbc.httpclient.Connection.processResponse(Connection.java:51)
at com.twitter.hbc.httpclient.ClientBase.processConnectionData(ClientBase.java:243)
at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:145)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

I'm getting this error

Exception in thread "pool-3-thread-1" java.lang.NoSuchMethodError: twitter4j.json.JSONObjectType.determine(Ltwitter4j/internal/org/json/JSONObject;)Ltwitter4j/json/JSONObjectType;
at com.twitter.hbc.twitter4j.BaseTwitter4jClient.processMessage(BaseTwitter4jClient.java:157)
at com.twitter.hbc.twitter4j.BaseTwitter4jClient.parseMessage(BaseTwitter4jClient.java:138)
at com.twitter.hbc.twitter4j.BaseTwitter4jClient$1.run(BaseTwitter4jClient.java:81)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Hi there,

I'm trying to use the client without much success. I had a look at the examples and wrote my own classes but I can't avoid this annoying issue: every time I call the connect method from the BasicClient class I'm getting this exception (it does not matter whether I use directly the BasicClient or I try to wrap it into the Twitter4jStatusClient class):

Actually, by simply copying some code example from the hbc-example module (e.g.: SampleStreamExample) the error is still there. Running the same code from the hbc-module gives no error and works properly; I am definetely missing something...

I ran the debugger and the error is caused by the underlying property from the ClientBase class, which when executing line 165 from this class, is null.

The complete stack is:

Any help will be much appreciated, thanks.

Hello,
is there a way to use thread pool to coordinate lot of user streams ( for example 1000 ) ?
I want to have for example in thread pool 500 thread and 1k users.
Because using code for now when there will be 1k users - 1k threads.

Regards