This project demonstrates a real-time interactive dashboard for mimicking application metric data, leveraging powerful big data tools such as Apache Kafka and Apache Spark. It serves as an end-to-end template for building data engineering projects that utilize Apache Kafka, Apache Spark, and PostgreSQL database, all containerized with Docker.

• Real-Time Data Processing: Uses Apache Kafka for message queuing and Apache Spark for stream processing.
• Interactive Dashboard: Utilizes Plotly Dash to create dynamic and real-time visualizations of the processed data.
• Modular Design: The project is structured to be easily extensible for various use cases involving real-time data streams.
• Ease of Setup: Can be launched with a simple Docker Compose command.

This project integrates several technologies into a cohesive pipeline:

• Apache Kafka: Handles incoming data streams and acts as a buffer for log data.
• Apache Spark: Processes the data in real-time, performing computations and aggregations.
• PostgreSQL: Stores the processed data for persistence and visualization.
• Plotly Dash: Provides a web-based dashboard for displaying interactive visualizations of the data metrics.