Giter VIP home page Giter VIP logo

twitch-migration-tracker's Introduction

Twitch Migration Tracker

This project tracks users migrations between twitch channels and stores these migrations in a database. The visualizations are handled in a seperate repository called Twitch-Migration-Website

This is done by creating a thread running for each monitored stream and having that thread constantly grabbing the list of current viewers. This can be done by using the NAMES command through IRC which gives a list of all users in a channel, or by using Twitches API. Both of these methods will give the same list of names, since Twitch considers a user as watching if they are in the chat.

Every time a new list of viewers is gotten the thread determines who has joined and who has left the stream. Then if there is a user who has left one stream and joined another within some time limit, it is considered a migration and stored in the database.

Prerequisites

This project runs on python 3.6 and uses mongodb as a database

Getting Started

  1. Clone the repository
    git clone https://github.com/mroseman95/Twitch-Migration-Tracker
    
  2. Set up python virtualenv (optional)
    cd twitch-migration-tracker
    virtualenv -p python3 venv/
    source venv/bin/activate
    
  3. install the required pip libraries
    pip install -r requirements.txt
    
  4. set up the database config
    • install mongodb and make sure the service is running
    • modify config/db.cfg.template to match your setup and then remove the .template from the filename
  5. set up the api config
    • the api.cfg.template requires a client_id token which can gotten by registering the application as a connection under your twitch profile
    • again remove the .template from the filename
  6. set up the irc config
    • the irc.cfg.template asks for an oauth token also gotten from twitch
    • make sure you are logged into twitch and go to this site to get the oauth token
    • when done the config should look like
      oauth: oauth:abcdefghijklmnopqrstuvwxyz1234567890
  7. create the database by running python create_db.py back in the root directory
  8. running python update_streams.py will populate the database with which streams to monitor and running python watching.py will begin watching for any migrations amongst those monitored streams

Database Structure

There are 3 collections in the database

  1. monitored_streams
    {
        _id: unique id,
        list_category: main_list,
        streams: [
            {
                streamname: foo,
                last_updated: seconds since epoch
            },
            ...
        ]
    }
    
  2. stream_migrations
    {
        _id: unique id,
        from_stream: foo,
        to_stream: bar,
        migrations: [
            {
                username: user who changed streams,
                leave_time: seconds since epoch,
                join_time: seconds since epoch
            },
            {
                username: ...,
                leave_time: ...,
                join_time: ...
            },
            ...
        ]
    },
    {
        _id: unique id,
        from_stream: bar,
        to_stream: bar,
        migrations: [
            ...
        ]
    },
    ...
    
  3. stream_viewcount
    {
        _id: unique id,
        streamname: foo,
        user_count: number of users at this time
        time: seconds since epoch
    },
    {
        _id: unique id,
        streamname: bar,
        user_count: ...,
        time: ...
    },
    ...
    

Testing

There are currently no automated tests for this project, though this is a future goal

Authors

  • Steven Gates - Initial api code - Gatesyp
  • Matthew Roseman - All current code and design - Mroseman95

License

This project is licensed under the MIT License - see the LICENSE.md file for details

twitch-migration-tracker's People

Contributors

mattroseman avatar

Watchers

Steven Gates avatar

Forkers

kylevasulka

twitch-migration-tracker's Issues

Cycle through threads

instead of going through every stream in watching one at a time. Start a thread every time one joins back.

Assign nodes to streams

Connect to mongodb to get a list of streamers and the number of viewers for streams between two times. Then display nodes for each stream with size relative to viewcount.

dict changed size during iteration

Exception in thread sodakite-thread:main
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "watching.py", line 128, in update_stream
    stream.remove_stale_joiners(join_ttl)
  File "/home/matthew/Programming/Twitch-Migration/stream.py", line 76, in remove_stale_joiners
    for joiner in self.joining:
RuntimeError: dictionary changed size during iteration

This seems to only happen after one or two iterations

Stagger thread start

Stagger thread start so that it might be better optimized. While one thread is calling api the other is sending data to db.

Change how migrations are stored

Instead of storing the current time

con.db[con.migration_collection].insert_one(
                            {   
                                'username': user,
                                'from_stream': l.name,
                                'to_stream': j.name,
                                'time': time.time()
                            })

store the average time between leave and join, or possibly both times

Figure out optimal limits

There are a lot of limits, and some overlap. Figure out what the best options are. Maybe create a set of rules like the join_overlap_limit must be <= the join_ttl.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.