Tweet JSON format? about carmen-python HOT 12 CLOSED

Timeline is up to my advisor (Mark Dredze). I'll let him know there's interest!

Which Twitter endpoint did you use? If you have a list of IDs then you could also just use Hydrator https://github.com/DocNow/hydrator

from carmen-python.

Thank you @AADeLucia! I really appreciate your help.

from carmen-python.

Hi, thanks for checking out our tool! As of now the support is kind of limited, but we do have an update planned for the "future."

Carmen works purely on metadata right now. So you need the entire tweet JSON and not just the ID and text.

Also, ignore the warnings. They don't mean anything useful for you.

from carmen-python.

thanks for your reply - could you point me to a Twitter endpoint which returns tweet data in a format which works with your tool?

I'm very interested in staying involved with this project, do you have a timescale for the update? I'm happy to help if necessary

from carmen-python.

Hi, I am using Twitter Academic Research API v2 for tweet query. Attached is a sample file I get after query (in json format not txt) . When I use Carmen to retrieve location I get None. Does Carmen support json format of this output?

response.txt

from carmen-python.

Seems like you're missing a lot of metadata. And the file needs to be in JSONlines format, not just JSON. I would look into:

• JSONlines format
• Carmen paper

from carmen-python.

Thank you @AADeLucia for sending over the references. I will work on them.
A quick question, what do you mean that "I am missing a lot of metadata"? Should I change my query parameters?

from carmen-python.

Hi @AADeLucia
As you mentioned I provided josonline file from tweets and tried to add more meta data to it, which can be seen in the attached file (body_line.jsonl). But still I get "None" as an output of Carmen. Bellow is the code to load the jsonl file and apply Carmen for each line :

resolver = carmen.get_resolver()
resolver.load_locations()

with jsonlines.open('body_line.jsonl') as reader:
    for line in reader:
        print("Carmen location: ",resolver.resolve_tweet(line))

Can you please let me know what the problem is?

body_line.jsonl.txt

from carmen-python.

@nimahassanpour Carmen was written for the old Twitter API, not v2. your country info is at the top-level of the JSON, but Carmen expects that information to be under place. and usually the user-specified location is under user

https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/object-model/tweet

I haven't used the new API yet. So it just puts the requested fields in the top-level of the JSON instead of nested?
https://developer.twitter.com/en/docs/twitter-api/tweets/lookup/api-reference/get-tweets#tab2

from carmen-python.

Thank you @AADeLucia for your reply.
Do you have a sample input file that Carmen can work properly with?
I may be able to rearrange my jsonl file in a way that Carmen can process it.

from carmen-python.

Here's an example:
example.txt

And the full documentation is here:
https://developer.twitter.com/en/docs/twitter-api/v1/data-dictionary/object-model/tweet

from carmen-python.

Added API v2.0 support with Carmen 2.0 #4 #9

from carmen-python.