trynmaps / tryn-api Goto Github PK

Implementing https://github.com/trynmaps/orion/wiki/Point-Reliability

• includes documentation and some testing as well

This has been long overdue, but given that we're only showing all the vehicles at a given time, having all this data in a Cassandra database is pretty expensive ($200/month) with little benefit.

Using S3 is just as good for supporting the web-app, and as for doing geo-query analysis, Cassandra would've required setting up ElasticSearch or Spark on it, so we might as well make a new database from scratch optimized for geo-analysis. As for trip analysis, reading trips from S3 should be good enough (can be directly used in Jupyter - or use Spark notebook), with aggregated data eventually being stored in a more flexible DB like Postgres.

This ticket tracks modifying the API to use S3 such that it remains unchanged (ie. existing queries will continue to work).

Using the API to track schedule adherence for Marin Tiburon ferry bus route.

• also tracks GTFS-RT for GRT in Orion
• the map now works for other cities, so adding Waterloo and others is easy now

• it's already dockerized, but Lambda + API Gateway would work just fine
• contact me on Slack for AWS access

• make this in an endpoint: https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-catalogue/#bb967f18-8d90-defc-2946-db3543648bd6

• currently the web-app uses geojsons to draw and list routes directly. Instead, we want to rely on the API. This ticket tracks passing in the geojsons for each agency.

Actual: fetches stops regardless of whether the field's included in the query
Expected: shouldn't do so as it's an unnecessary network call

Right now, a query looks like this:

{
  trynState(agency: "muni", startTime: "1515272617534", endTime: "1515273457194") {
    agency
    startTime
    states {
      time
      routes {
        name
        stops {
          sid
          name
          lat
          lon
        }
        vehicles {
          vid
          lat
          lon
          heading
        }
      }
    }
  }
}

The route name along with all the stop info will be duplicated. The reason stops can't be moved upwards is because a stop belongs to a route. At the same time, the route name is also duplicated, as would any future metadata.

The proposed schema layout is as such:

{
  trynState(agency: "muni", startTime: "1515272617534", endTime: "1515273457194") {
    agency
    startTime
    routes {
      rid
      stops {
        sid
        name
        lat
        lon
      }
      routeStates {
        vtime
        vehicles {
          vid
          lat
          lon
          heading
        }
      }
    }
  }
}

• will make getting data easier

Goal: make this part of the Vehicle type so that the client will know for certain which direction a bus is moving towards

• start by supporting only the day of the muni trips file.
• it loads the file from S3, then returns the trips for the provided route (a route must be provided) for the hardcoded date.
• frontend recieves the trip. It determines the distances locally using turf.js

We currently only return stops for Muni, but not for the other agencies, including the TTC (easy to add as it uses Nextbus), Marin (have to make separate API call), and GRT (on the roadmap).

However, a better solution to this seems to have a separate API called tryn-stops that does this for us by either making an API call, or by looking it up in the GTFS file (it should have all the stops by route in memory).

This would involve:

• having a GraphQL/Rest API
• reading GTFS files and storing the data we need in memory (https://github.com/remix/partridge)
• making API calls to Nextbus or fetching remote GTFS files periodically or on start.
• It can be in Node, Python, Go, Scala, or any other language you're comfortable with.

goal: request an agency's gtfs from a certain day using the API
Why? Needed for accurate stops, routes, frequency, and schedule adherence

• Might not work well for highway buses for points where the bus is normally moving
• That said, we're focusing on city-buses, which also have many stops. In practice, the PointReliability feature will really be used for key stops where the bus is almost always stopped (or very close to the stop) for at least 20 seconds.

• while this is a good starter ticket, it requires prior use of Cassandra and might be more complicated than I think it is

I was looking at the schema and saw TrynState and trynstate. I don't know what other things might break if they're changed though.

and ensure it's been moved over to orion, which used it for the s3 migration

Currently, tryn-api ignores the endTime and only answers the startTime with one state.

Expected behaviour is to return multiple states instead of just one as it currently does. This is blocking the SpeedViz project - https://github.com/trynmaps/orion/wiki/Speed-Visualizations-(SpeedViz)-Design-Proposal

See https://github.com/trynmaps/orion/wiki/Point-Reliability

2-car Muni trains (on all lines except J) have both their cars show up in Nextbus. This can be observed on the web-app map.

For the MVP, this causes the median/average observed headways between cars to be much lower than what they really were.

Issues:

1. routes can have a mix of one and two car trains (and in a few years, three).
2. Muni Metro trains can bunch fairly closely, often staying that way for the entire trip.
3. each car sends its location separately, meaning that coupled cars could sometimes appear as being a block apart.

Notes:

1. Muni trains don't get decoupled when in-service, meaning that if two cars are coupled, then they stay coupled for the entire time they're in-service (unlike the previous Boeing ones did back in the day, where K/L/M trains and N/J trains were often coupled together at West Portal and Duboce as to run more efficiently in the subway; they lost that ability with the current Breda LRVs, and don't have plans for doing this with the new Siemens ones).
2. None of the other agencies available via the API have this issue, so it'll be Muni-specific.

Possible Approaches:

1. Resolve this in the MVP right before returning the state.
2. Correct and clean the vehicles in the S3 store. It would require a better way of storing them (like hourly chunks per agency+route or 15-minute chunks per agency), which would require tryn-api and opentransit-map changes.
3. Clean the vehicles as they are returned by the API. Not recommended since the MVP requests vehicles in 12-minute chunks, which is an issue since two trains could be closely behind one another for the entirety of a 12-minute span.

1. Add config file that points to relevant files, stored in S3 (manually updated)
2. Return those based on the connectivity fields in the graphql query