alcedok / capataz Goto Github PK

capataz allows users to explore, analyze and visualize both near-real-time and historical data, composed of geospatial location, distance traveled, number of passengers and time.

The platform was developed in 3 weeks as part of Insight's Data Engineering Fellowship in NYC. The project allowed me to explore several open-source Big Data technologies, their paradigms and limitations.

Features:

• Near-real-time data ingestion and processing
• Near-real-time queries on users and vehicles
• Batch processing on historical data
• Trip-time prediction based on historical data
• User-interface for data exploration and visualization
• Distributed and scalable

capataz (Spanish), translates to overseer/controller:

(n) a person or thing that directs or regulates something.

capataz was inspired by the rise of connected transportation and the imminent deployment of self-driving vehicles. It is a proof-of-concept platform for users that need to monitor, explore and vizualize vast amounts of streaming data. Such users may be city officials, urban-planners, emergency and mass-transportation services, and upcoming automated services such as delivery and carpooling. Moreover capataz was designed with the intention of being used by Data Scientist in order to incorporate predictive models for both real-time and batch streams of data. As it stands capataz is able to process and filter a simulated real-time stream of data which includes geospatial location, distance traveled, number of passengers and time. Moreover, it is able to process and filter large batches of data while simultaneously training a predictive model. The processed data can then be queried, visualized and explored through a user-interface.

As a proof-of-concept a Decision Tree Regressor was chosen in order to predict the time-delta (in mins), given a start and goal location at a given hour of the day. This information would be valuable for ridesharing and carpooling services that need to optimize their fleet logistics.

The data used for both real-time simulation and batch processing comes from NYC Taxi and Limousine Commission data, which encompasses multiple .csv's totaling 170 GBs.

Ingestion:

Apache Kafka serves as the primary messaging system between technologies within the pipeline.

• Kafka receives messages from:

  • a .csv that simulates real-time stream (currently ~100 messages per second)
  • a completed Spark Streaming process
  • a completed Spark Batch process

• Kafka sends messages to:

  • a Spark Streaming process
  • a new .csv file on HDFS
  • Elasticsearch, which are then indexed for querying

Stream:

Apache Spark Streaming serves as the near-real-time analytics framework by performing micro-batches on incoming messages from Kafka.

• Spark Stream performs the following operations:
  • Create a Direct Stream from Kafka, listening for incoming json messages
  • parse json messages into Dataframes
  • Filter invalid GPS coordinates and empty/null entries
  • Send messages back to Kafka, which will be then consumed by Elasticsearch

Batch:

Apache Spark SQL + ML is used for processing batch historical data coming from multiple large (2.5GB) .cvs's stored in HDFS.

• Spark SQL+ML performs the following operations: *

Datastore+Search Engine:

Elasticsearch serves as the datastore and search engine, where users, cars and predictions are indexed for later querying.

Frontend:

Kibana is used as the user-interface for Elasticsearch

TODO

TODO