Store MQTT messages in S3
This sample uses an OpenFaaS MQTT-connector along with a Python function to receive JSON messages from MQTT and to store them in an S3 bucket.
The use-case could be ingestion of IoT sensor data from edge devices. It is a starting point, and easy to modify for your own uses.
Install arkade, the open-source Kubernetes marketplace:
# Move arkade into your $PATH
curl -sLS https://dl.get-arkade.dev | sh
# Or have arkade move itself into /usr/local/bin/
curl -sLS https://dl.get-arkade.dev | sudo shMake sure have Minio and OpenFaaS installed to your Kubernetes cluster:
# Follow the post-install instructions to log in and start
# port-forwarding.
arkade install openfaas
arkade install minioGet CLIs for the Minio client and OpenFaaS:
arkade get mc
arkade get faas-cliThen set up secret for Minio:
faas-cli secret create secret-key --from-file ./secret-key.txt
faas-cli secret create access-key --from-file ./access-key.txtSetup mc to access Minio:
kubectl port-forward svc/minio 9000:9000 &
mc alias set minio http://127.0.0.1:9000 $(cat access-key.txt) $(cat secret-key.txt)
Added `minio` successfully.Then make a bucket for the sensor data:
mc mb minio/sensor-data
# Show that it's empty
mc ls minio/sensor-dataThen install the mqtt-connector, which will invoke your functions:
arkade install mqtt-connector \
--topics openfaas-sensor-data \
--broker-host tcp://test.mosquitto.org:1883 \
--client-id mqtt-connector-1
kubectl logs deploy/mqtt-connector -n openfaas -fNext, deploy the functions:
faas-cli deployThen publish an event to the "openfaas-sensor-data" topic on the iot.eclipse.org test server.
You can run the sender/send.py file to publish a message to the topic. Our function will store the message in an S3 bucket.
First run:
pip3 install paho-mqttThen run sender/send.py
$ python3 send.py '{"sensor_id": 1, "temperature_c": 50}'
Connecting to test.mosquitto.org:1883
Connected with result code 0
Message "{"sensor_id": 1, "temperature_c": 53}" published to "openfaas-sensor-data"
$ python3 send.py '{"sensor_id": 1, "temperature_c": 53}'Check the logs of the function to see that it was invoked:
faas-cli logs mqtt-s3Now check the contents of your S3 bucket with mc:
mc ls minio/sensor-data
You should see a number of .json files created for each message you create with the sender.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent