Loading ouput from twitter-intact-stream failed about birdspotter HOT 6 OPEN

It looks like twitter-intact-stream does some manipulation to the JSON object it receives from Tweepy, and isn't in the Tweet JSON Specification supplied by Twitter. I'd recommend using twarc to rehydrate a list of the IDs from the data.

Note, that not all the information birdspotter requires is in the JSON supplied by twitter-intact-stream.

from birdspotter.

Thanks for your reply. Would you please be a bit more specific on how to prepare the input for birdspotter? As per your answer, birdspotter cannot use the output of twitter-intact-stream directly and it has to go through twarc, doesn't it? If possibly, may you provide some examples?

from birdspotter.

On closer inspection; it looks like twitter-intact-stream does leave us in the correct format, assuming no post-processing has been done. I'll need to investigate further, as to why one or more of the json objects don't have a "text" or "full_text" field.

from birdspotter.

It looks like the twitter-intact-stream contains lines to indicated rate-limiting, such as

{"limit":{"track":283540,"timestamp_ms":"1483189188944"}}

Normally, birdspotter would ignore corrupted lines that aren't in valid json format, but this is valid json.

The work around at the moment would be to filter out lines that look like the above and then feed the result into birdspotter.

I think it would be better if birdspotter dealt with cases like this better. I'm going to put a temporary fix in so that it ignores objects without a text or full_text field.

I'll leave this open till that is implemented.

from birdspotter.

Rate limit messages are normal when using the search API, they give the number of lost tweets. You would expect them when using other Twitter API tools, so would be good if birdspotter would filter them our automatically.

from birdspotter.

db93307 in the development branch should fix this problem, at least temporarily. There should probably be a more robust check of the json object to verify it's validity, but we'll wait till there is more interest.

from birdspotter.