trynmaps / metrics-mvp Goto Github PK
View Code? Open in Web Editor NEWPrototype of public transit data visualization system
Home Page: https://muni.opentransit.city
License: MIT License
Prototype of public transit data visualization system
Home Page: https://muni.opentransit.city
License: MIT License
While we're waiting on #89 to get merged, let's start making graphs (like those on www.busstat.nyc) that show how the scheduled travel times stack up to the actual travel times. Use fake data to make these graphs and let the PR sit until #89 is merged; then we can plug in the real data.
Should make these time ranges a constant
react/graphql/node stack
get to a state where json from relevant data will be ported to the output component.
for example, the control panel will have a bunch of options/filters that will be used to narrow a search.
given a search, return the json associated with the constraints.
Need to draw lines through circles (stops) on map
Nothing to see here folks
Currently, you have to re-run yarn build each time you make a change to React (because on starting the server, you run yarn build to make a static folder). Not ideal. Can we make yarn auto-rebuild each time you change react?
Working on it :)
Currently, in metrics-api.py
, the /metrics
endpoint is an unholy giant function. Can we separate that into separate smaller functions (or at least pull out some code into helper functions)?
The files are large so it might help to ignore them. But if our server needs it, it might be more convenient to keep tracking.
Add support for showing different route info like:
-Speed
See bit.ly/opentransit-mvp.
As Jose mentioned in #89 , we need to archive GTFS data because it updates regularly.
For example, between May 24 and June 11, the F outbound direction id changed from F____O_E00 to F____O_F00, with a new terminal ("Castro" instead of "17th and Noe") and new last stop id (33311). So attempts to show anything about F outbound trips will not find any data . Even if we add logic to fall back on older direction ids, we still have the problem of the stop list being out of sync also.
Muni timepoint data (time vehicle passed by a stop) can be found here: s3://muni-timepoint-avl-data/muni_timepoint_data_fall_2018.zip
Each month (total of 3) is about 100 MB uncompressed CSV.
It's the same data Muni uses for some of their metrics here: https://www.sfmta.com/performance-metrics
Need to add agency
and any test dates (i.e. date(2019, 4, 8)
) to constants.py
React 16.8 should be coming out soon which should include Hooks.
I think it would be worth having the team write new components with hooks
especially since we're building a SPA without too much state.
Noticed that metrics-mvp doesn't specify an open source license yet. Other projects in the trynmaps organization use Apache 2.0 or MIT license. Either of those seems fine to me. Anyone have a preference?
it runs locally but we arent sure if it works on heroku yet
Handle state with Redux and Redux-First Router
ability to have more than one date and add time ranges
Same chart in React-Vis vs react-d3-components, trying to match NYC bus stats as closely as possible just as a test:
In creating a new "phases of the day" chart, I came across these issues with react-d3-components:
I looked for articles about popular React charting packages:
Of these, React-Vis looked good visually. It offered more customization options (and correspondingly more complex component hierarchy). It uses D3-inspired scales.
I tried reimplementing a chart in React-Vis. Worked pretty well. Issues:
look up the actual timetable stop ID via the route config since it may not always be "1" + Nextbus stop tag
See https://github.com/trynmaps/metrics-mvp/pull/57/files/e4bea26eee9511c887baccb90cf6f3f4fa3401cb#diff-bb76665c29abb732e3d6a1d469444ca1
The mockup shows a bar chart (and the same data in a line chart with time of day on the x axis) of how wait time and travel time vary across different parts (am rush/midday/pm rush, etc.) of a day. Since we're currently showing data for one time range of one day, the next incremental step is to make a chart of multiple time ranges for one day. It is a slightly different workflow from what we have now, though, so we may need UI design changes to support this.
install react router
Borrow @ckrajewski's stop-selection-on-a-map and draw the speed in the lines between stops.
Currently we show the grade for a route given the to and from stops. But people keep asking us, "which is the best/worst route?" So we need to prototype out a way to give a grade for a route without having to specify the to and from stops.
I imagine this would be based on just the average speed and wait time for the route, across all stops, across all trips of the day.
Let's put this as a card in the MVP (so that in addition to the card about the grade for the specific stops, we also have a card about the grade for the route overall).
Don't just dump users on the main page. Have some explanation of who we are or some system-level, high-level metrics (like www.busstat.nyc and la.railstats.org).
should we enable cors?
this involves:
Maybe that will help them play together more nicely.
When pulling stop information for a route from nextbus, there will sometimes be stops with no associated direction. For example, on route KT, stop 7058 has no associated direction:
http://webservices.nextbus.com/service/publicJSONFeed?command=routeConfig&a=sf-muni&r=KT&t=0&terse
Searching for 7058 finds the stop info (name, lat/lon, etc) for stop 7058 and 17058 (the same stop, but for different directions) but those stops don't show up in the list of stops for each direction. This will happen for stops with only one direction as well - stop 4509 on route L is an example:
http://webservices.nextbus.com/service/publicJSONFeed?command=routeConfig&a=sf-muni&r=L&t=0&terse
I don't know the exact number of stops where this happens, but I'd estimate that 30% of routes have at least one stop with no associated direction; I can find the exact number later.
The 311.org API only provides timetables for a portion of the stops on each route. Without a timetable to compare actual arrival times to, it will be more difficult to evaluate how close buses are keeping on schedule at a particular stop. If we can't get this information from another source, here are a few possible workarounds:
for each route, SFMTA gives projected service frequencies, as well as the first and last trip for each day of the week (ex/ https://www.sfmta.com/routes/10-townsend). For any stop whose timetable isn't on 311, we can just use this information to generate a projected timetable. This would be simple, but I suspect it wouldn't be very accurate.
given the timetables that we already have access to, we could interpolate the timetables for the other stops by adding the projected time between stops to the stop with a known timetable. Since stops are different distances from each other, this feels like it would produce more accurate timetables, but I don't know how we would obtain the projected time between stops - using the data we have now doesn't make sense since what we want to do is compare that data to what is projected, as opposed to comparing the data to itself.
For most lines (e.g., the 12 or 27), the travel time from the first stop to any other stop (e.g., the second stop) is not accurate. The histograms show a wide spread of travel times when in actuality the spread should be narrow (as it is from the second stop to the third stop).
i noticed a few things with the procfile and at first glance there might be some bugs introduced here
the app is running on a flask dev server serving the static files - maybe we can use nginx to do this for us? we can also feed heroku nginx configs to fine tune the location rules. also, we can choose ports to listen on etc.
npm start: i don't know if this is ideal on a production server. i could be mistaken but isn't this script used for development? as in part of your workflow: change a file and the hot reload job reflects the change. an alternative could be to npm build -> let the web server know where the files are hosted (i think like /app/www/<static_folder>)
This will make development much easier. Explain the desired inputs and outputs.
In #111, we replaced the favicon with the OpenTransit logo. But it looks really bad when scaled down to 16x16. We should make a proper 16x16 logo with a pixel art maker. (Same with 32x32.)
@jtanquil What repo is that json in? Or is it from the API?
/(home): splashscreen with "basic details" like route, state, stop
/route/{route}: optional advanced filtering with graphics
/route/myroute?/{token}: specifically tailored route with graphics, filtering
Gitpod currently defaults to using Python 2.7 but a lot of our Python code uses Python 3. How do we make Gitpod default to using Python 3?
we can also get rid of the jquery then
As Jesse suggested in #132 : For production, we should probably have nginx in front of flask to gzip and cache responses to reduce bandwidth and reduce load on the flask backend. Don't need to implement that now though. Vickie H also suggests Envoy (https://www.loggly.com/blog/benchmarking-5-popular-load-balancers-nginx-haproxy-envoy-traefik-and-alb/)
Also, if the changes for #132 have no issues and can become permanent, then there is some refactoring and cleanup as described there. Can clone this issue as desired for that.
This will make it much easier to build APIs going forward. Right now the APIs are mixed in with app initialization code and some business logic. Ideally we would move the business logic (namely, everything that /metrics
uses) and the business logic code out somewhere else.
This data (like cached point-to-point travel times) could be useful in this MVP.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.