The holoscrape from dannylty

holoscrape's Introduction

Automated Extensible YouTube Livestream Chat Scraper and Visualiser.

Main Features

Automatically detect existing livestreams. Polling will be done periodically to the specified indexers. The indexers are in charge of generating live YouTube video-ids to be scraped.
Dispatch to tmux panes in real time. Display current streams in an interactive pane which will grow or shrink in size as live streams come and go.
Customisable processors. Write to a database, write to files, or create your own processor.
Customisable indexers. Don't follow Hololive? Write your own indexers instead to produce video-ids of your favourite streamers.

How To

Installing Requirements

sudo apt install tmux
pip3 install -r requirements.txt

Configuring

The in-built Holodex indexers require API keys to be supplied, and they are given by the env var HOLODEX_API_KEY.

Config reading defaults to config.json.

If write_to_db or write_to_local is false, their respective subconfigs can be omitted.

{
    "write_to_db": true, <-- Mandatory
    "db_host": <host>,
    "db_port": <port>,
    "db_user": "username",
    "db_password": "password",
    "db_database": "password",
    "db_table": "example_tab",
    "db_stream_table": "stream_tab",
    "db_nshards": 30,

    "write_to_local": true, <-- Mandatory
    "local_path": "/path/to/data/",

    "log_path": "/path/to/logs/", <-- Mandatory
}

An example database schema is given in init.sql that works with the example configs above. If you don't know your way around it, just turn write_to_db off.

Running (in tmux)

python3 main.py
'ctrl-b s' to change to the scraping window

holoscrape's People

Contributors

Stargazers

Watchers

holoscrape's Issues

Integrate holodex apikey

APIs require api key now. Need updates to indexers.

Refactor main.py config hardcoding

Find a way not to hardcode the indexers and writers?

    stream_indexers = (HolodexIndexer(), NijisanjiIndexer())
    writers = []
    if config_handler.write_to_db:
        writers.append(DatabaseWriter(config_handler, None))
    if config_handler.write_to_local:
        for folder in ['simple', 'metadata']:
            os.mkdir(os.path.join(config_handler.local_path, folder))
        writers.append(FilesystemWriter(config_handler, None))

Recommend Projects