Giter VIP home page Giter VIP logo

robintrack's Introduction

Robintrack

https://robintrack.net/

Robintrack is a tool for viewing data about the popularity of various stocks on the Robinhood brokerage. It allows users to view information about which are the most (and least) popular stocks held by its users and view trends over time. When combined with price history and other forms of data, this can be a useful tool for analyzing market sentiment.

Components

Scraper

The scraper periodically scrapes the Robinhood API to pull popularity and price data for all of the stocks that it offers. A server/worker model is used, with RabbitMQ being used a work queue. Once scraped, the data is stored in MongoDB.

Backend

The backend is a Rails application that mainly serves as a shim to the MongoDB database. It serves the API used by the frontend.

Frontend

The frontend is a single-page React application that provides users with access to the site's data via data tables and visualizations.

Installation

You'll need a MongoDB instance running and a RabbitMQ instance for the scraper. Configuration for the backend is handled via environment variables; the MONGO_HOST and MONGO_PORT variables are read at runtime. The scraper takes the same environment variables for MongoDB and everything else via command line arguments. Use the --help flag for both the master and worker scripts to view available config options.

There also must be a Redis instance accessible to the instance with an address defined in the REDIS_HOST environment variable. This is used for caching API requests that do not change between runes of the scraper.

Docker

There is a Dockerfile provided for the backend. To use it, just run docker build -t robinhood-backend and then docker run -it --net host -p 4000:4000 -e SECRET_KEY_BASE=some_secret_key -e MONGO_HOST=example.com -e MONGO_PORT=27017 robinhood-backend. (--net host is only necessary if you have MongoDB or RabbitMQ running locally, and it doesn't work on Mac).

robintrack's People

Contributors

ameobea avatar dalexj avatar dependabot[bot] avatar michaelgathara avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

robintrack's Issues

Add endpoint for getting popularity index of a symbol

Given a symbol, return its popularity ranking among all other symbols. If all symbols are ordered by their popularity from least popular to most popular, this is the index in that list.

If two symbols have the same popularity, sort secondarily alphabetically by symbol.

Doesn’t work on weekends

When trying to view individual stock activity or popularity changes on the weekend, the robintrack service doesn’t work. It should be changed to let you be able to view the last available saved data(from Friday) rather than not work at all. Maybe it could also put up a alert/notification that “this data is not reflective of any activity today” or something. Because sometimes it’s useful to be able to use robintrack over the weekend

Add "Largest Popularity Increases" endpoint

Params:

  • Duration
    • Largest popularity increase in this period of time
  • Limit
  • Relative
    • If true, the change is calculated as a percentage of total popularity. If false, the absolute change amount is used instead.
  • Min Time Amount
    • Only used if relative is true. Only assets that have a popularity of this amount will be considered in relative popularity calculations. (If you have a stock with 5 popularity and it goes down to 4, that looks like a 20% popularity change. This filters those out.)

Maybe an endpoint like /popularity_changes/increase/[symbol]/[duration]/[limit], or duration and limit could be optional query params with defaults.

Add support for sending partial data in timeseries

A parameter should be added to the popularity history and quote history endpoints that, when provided, only sends one data point per day for data from over one year ago. The parameter can be called partialOldData or something like that.

Assuming the current date is June 12 2020 and the flag parameter is set, then the returned timeseries should be filtered to remove all but one update from each day before June 12 2019. The first data point from each day can be picked, discarding the rest.

The keys used in caching should be updated to reflect this new parameter.

Add ability to download popularity data for a given time frame

It would be nice if the popularity_history (and I suppose popularity_history_csv) API endpoint(s) supported the ability to return only data points in a given time frame (ex., "get me popularity data for TSLA between 2020-03-01 and 2020-03-06"). Currently, as far as I can tell from the source code, this endpoint only supports fetching the entire popularity history for a ticker (which for many tickers is years worth of several-times-a-day data). That makes it impossible to politely use your API to, say, keep daily tabs on a few tickers: if you want data for today (every day), you have to also fetch a couple of years worth of data (every day). Ouch.

The way I'd envision this is to add optional start_time and end_time query parameters to the endpoint(s), which (logically, at least) default to, say, the UNIX epoch (or first time when data was available for the ticker) and now, respectively, to maintain backwards compatibility with the current implementation (i.e., fetch all history). These parameters would both take some sort of encoding of a timestamp, say ISO-8601 or whatever is easy enough to get a parser for in Ruby. It should be easy enough (I think?) to plumb these further down into the get_history_for_symbol method and the MongoDB query that actually gets the data to service the endpoint call. This query gets modified to select based on the timestamp field being between start_time and end_time (inclusive).

I don't know what your bandwidth for working on Robintrack is these days, but if you think you wouldn't be able to get this feature out any time soon yourself, I'd be willing to (try to) cook up a pull request if you'd be willing to review it. (Assuming you want this feature yourself, that is, though I suspect it might help in your eventual implementation for #43.)

Add a `total_symbols` endpoint

Should simply return the number of unique symbols in the database. It should take an optional param that provides a number of hours ago in which there must have been a popularity update for it to be included in the count.

So, if the param is 5 and the symbol hasn't had a popularity update in the past 5 hours, it won't be included in the total.

Download price data

Would it be at all possible to make the scraped price data downloadable? Despite my best efforts, I can't find free price data to join with the popularities - every API I tried so far only has some variant of daily historical data, or realtime quotes. It would be a great help to have hourly data going back a year or more (the best RH API can do is interval=hour&span=month)

Add "Largest Popularity Changes" endpoint

Params:

  • Duration
    • Largest popularity change (either up or down) in this period of time
  • Limit
  • Relative
    • If true, the change is calculated as a percentage of total popularity. If false, the absolute change amount is used instead.
  • Min Time Amount
    • Only used if relative is true. Only assets that have a popularity of this amount will be considered in relative popularity calculations. (If you have a stock with 5 popularity and it goes down to 4, that looks like a 20% popularity change. This filters those out.)

Maybe an endpoint like /popularity_changes/change/[symbol]/[duration]/[limit], or duration and limit could be optional query params with defaults.

Handle changes in Robinhood ID

When events like splits happen, the internal Robinhood ID for assets changes while the ticker and underlying asset stays the same. In order to provide continuity in charts on the site, some kind of script or similar solution should be implemented to merge the data from the old and new ID into one.

Add "Largest Popularity Decreases" endpoint

Params:

  • Duration
    • Largest popularity decrease in this period of time
  • Limit
  • Relative
    • If true, the change is calculated as a percentage of total popularity. If false, the absolute change amount is used instead.
  • Min Time Amount
    • Only used if relative is true. Only assets that have a popularity of this amount will be considered in relative popularity calculations. (If you have a stock with 5 popularity and it goes down to 4, that looks like a 20% popularity change. This filters those out.)

Maybe an endpoint like /popularity_changes/decrease/[symbol]/[duration]/[limit], or duration and limit could be optional query params with defaults.

Create popularity distribution endpoint

This endpoint should return an array of bins indicating how many symbols have popularities within a given range. It should include one parameter, which is bin count. This should be a positive integer greater than 1.

You'll want to select the most recent popularity measures for all symbols in the index and figure out the range (it's going to be 0 to the popularity of the most popular symbol). You'll then want to divide that range into bin_count bins and for each bin sum up how many symbols have a popularity within its range.

In order to make it easier to render on the frontend, please include a min_popularity and max_popularity field along with the bin array itself.

Change symbol details to be inline on the leaderboard page

Rather than routing users to a different page to view graphs, the graph should be displayed inline on the leaderboard page. We can maintain a details page to view more advanced stats or more detailed info, and just have the chart display on the leaderboard page.

Add endpoint for getting symbol from popularity index

I want to implement a way to page through symbols in order of their popularity, so the page will have a "next most popular" and "next last popular" arrow that take you to the next symbols. So, I need a way to get that data from the API.

Technically, an endpoint that takes a symbol and returns the next and last most popular would be fine, but I figure that adding a more generic one that just returns the next and least most popular from an index would be more useful for other things.

Popularity Comparison View

Simple comparison chart that plots many different stocks side-by-side to show how their popularities have changed over time. Will probably require customized axes for the chart; we still want them to be displayed relative (maybe give users a chance to toggle them to absolute? Yeah, that would be good.)

Server seems to be down

Sorry for creating issue instead of using the feedback link. Feedback link did not work.

Seems that API is down and returning 503, hope this gets resolved soon.

Add an optional `startIndex` param to the most and least popular endpoints

I want to allow users to page through the popularity table for most and least popular symbols. So, I need an optional startIndex param that can be used to offset the search for either most or least popular.

In addition, I'd like the symbols to be secondarily sorted alphabetically by their symbols in the case that two or more symbols have the same popularity.

Add (Google) Analytics

I need to know what my users are doing. As a longer-term goal, I want to integrate the API clickstream into my fireworks visualizer or something similar so I can see EVERYTHING the users are doing.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.