Unlock the potential of the Internet of Things (IoT) with this cutting-edge Event Streaming Architecture. Leveraging the power of Apache Kafka and MQTT, this project is designed to handle IoT data at scale. 🚀

Ingest real-time data from IoT sensors across diverse locations, analyze metrics like temperature, humidity, pressure, and luminosity, and store them efficiently in a high-performance MongoDB database. 🌡️💧📈

The technology stack includes Spring Boot, Kafka Streams, Micrometer, and Grafana for real-time monitoring, making this architecture a powerhouse for IoT solutions. 💻📈🔍

With Docker containerization, deployment is a breeze. Explore this project, visualize IoT data, and gain insights into your sensor networks. 🐳🚀📊

Check out the detailed Medium article for a comprehensive overview. 📖

Thank you for visiting the Smart IoT Event Streaming GitHub repository! Empower your IoT endeavors and scale with confidence. 🌐📈💡

Project developed to practice what I have learned in the Udemy course Apache Kafka Series - Kafka Connect Hands-on Learning and Apache Kafka Series - Kafka Streams for Data Processing.

The main goals of this architecture are the following:

• Ingest and store real-time data from IoT sensors located in various locations.
• Analyze and make aggregations through rotating time windows to know average temperature by sensor or place, in addition to humidity, pressure and luminosity.
• Store data after processing for subsequent monitoring in a NOSQL database with good performance for frequent writes for relatively homogeneous document sizes such as MongoDB.
• Visualization of the metrics of the sensors in real time and of the performance of the components of the architecture through Grafana and the consolidated documents in MongoDB through Mongo-Express.

• Spring Boot 2.3.3 / Apache Maven 3.6.3.
• Spring Boot Starter Actuator.
• Kafka Streams.
• Spring Kafka.
• Micrometer Registry Prometheus.
• Eclipse Paho MQTT Client.
• Kafka Connect.
• Kafka Rest Proxy
• lombok.
• Jackson.
• NodeExporter (Exporter for machine metrics).
• Prometheus.
• Grafana.
• Eclipse Mosquitto.
• MongoDB.
• Mongo DB Express (Web-based MongoDB admin interface, written with Node.js and express).
• Cadvisor (Analyzes resource usage and performance characteristics of running containers).
• kafka-exporter (Kafka exporter for Prometheus).

The available tasks are detailed below (rake --task)

To start the platform make sure you have Ruby installed, go to the root directory of the project and run the rake deploy task, this task will carry out a series of preliminary checks, discard images and volumes that are no longer necessary and also proceed to download all the images and the initialization of the containers.