Giter VIP home page Giter VIP logo

twscrape's Introduction

twscrape

Twitter GraphQL and Search API implementation with SNScrape data models.

Install

pip install twscrape

Or development version:

pip install git+https://github.com/vladkens/twscrape.git

Features

  • Support both Search & GraphQL Twitter API
  • Async/Await functions (can run multiple scrapers in parallel at the same time)
  • Login flow (with receiving verification code from email)
  • Saving/restoring account sessions
  • Raw Twitter API responses & SNScrape models
  • Automatic account switching to smooth Twitter API rate limits

Usage

Since this project works through an authorized API, accounts need to be added. You can register and add an account yourself. You can also google sites that provide these things.

The email password is needed to get the code to log in to the account automatically (via imap protocol).

Data models:

import asyncio
from twscrape import AccountsPool, API, gather
from twscrape.logger import set_log_level

async def main():
    pool = AccountsPool()  # or AccountsPool("path-to.db") - default is `accounts.db` 
    await pool.add_account("user1", "pass1", "[email protected]", "email_pass1")
    await pool.add_account("user2", "pass2", "[email protected]", "email_pass2")

    # log in to all new accounts
    await pool.login_all()

    api = API(pool)

    # search api (latest tab)
    await gather(api.search("elon musk", limit=20))  # list[Tweet]

    # graphql api
    tweet_id, user_id, user_login = 20, 2244994945, "twitterdev"

    await api.tweet_details(tweet_id)  # Tweet
    await gather(api.retweeters(tweet_id, limit=20))  # list[User]
    await gather(api.favoriters(tweet_id, limit=20))  # list[User]

    await api.user_by_id(user_id)  # User
    await api.user_by_login(user_login)  # User
    await gather(api.followers(user_id, limit=20))  # list[User]
    await gather(api.following(user_id, limit=20))  # list[User]
    await gather(api.user_tweets(user_id, limit=20))  # list[Tweet]
    await gather(api.user_tweets_and_replies(user_id, limit=20))  # list[Tweet]

    # note 1: limit is optional, default is -1 (no limit)
    # note 2: all methods have `raw` version e.g.:

    async for tweet in api.search("elon musk"):
        print(tweet.id, tweet.user.username, tweet.rawContent)  # tweet is `Tweet` object

    async for rep in api.search_raw("elon musk"):
        print(rep.status_code, rep.json())  # rep is `httpx.Response` object

    # change log level, default info
    set_log_level("DEBUG")

    # Tweet & User model can be converted to regular dict or json, e.g.:
    doc = await api.user_by_id(user_id)  # User
    doc.dict()  # -> python dict
    doc.json()  # -> json string

if __name__ == "__main__":
    asyncio.run(main())

CLI

Get help on CLI commands

# show all commands
twscrape

# help on specific comand
twscrape search --help

Add accounts & login

First add accounts from file:

# twscrape add_accounts <file_path> <line_format>
# line_format should have "username", "password", "email", "email_password" tokens
# tokens delimeter should be same as an file
twscrape add_accounts accounts.txt username:password:email:email_password

The call login:

twscrape login_accounts

Accounts and their sessions will be saved, so they can be reused for future requests

Get list of accounts and their statuses

twscrape accounts

# Output:
# ───────────────────────────────────────────────────────────────────────────────────
# username  logged_in  active  last_used            total_req  error_msg
# ───────────────────────────────────────────────────────────────────────────────────
# user1     True       True    2023-05-20 03:20:40  100        None
# user2     True       True    2023-05-20 03:25:45  120        None
# user3     False      False   None                 120        Login error

Use different accounts file

Useful if using a different set of accounts for different actions

twscrape --db test-accounts.db <command>

Search commands

twscrape search "QUERY" --limit=20
twscrape tweet_details TWEET_ID
twscrape retweeters TWEET_ID --limit=20
twscrape favoriters TWEET_ID --limit=20
twscrape user_by_id USER_ID
twscrape user_by_login USERNAME
twscrape followers USER_ID --limit=20
twscrape following USER_ID --limit=20
twscrape user_tweets USER_ID --limit=20
twscrape user_tweets_and_replies USER_ID --limit=20

The default output is in the console (stdout), one document per line. So it can be redirected to the file.

twscrape search "elon mask lang:es" --limit=20 > data.txt

By default, parsed data is returned. The original tweet responses can be retrieved with --raw

twscrape search "elon mask lang:es" --limit=20 --raw

Limitations

API rate limits (per account):

  • Search API – 250 req / 15 min
  • GraphQL API – has individual rate limits per operation (in most cases this is 500 req / 15 min)

API data limits:

  • user_tweets & user_tweets_and_replies – can return ~3200 tweets maximum

See also

twscrape's People

Contributors

vladkens avatar variousred avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.