Creates an AWS Lambda function which receives data from a AWS SQS queue and stores it in an SQL database.
This project uses Pydantic, SQLAlchemy, SQLModel, psycopg2. AWS is configured with the CLI and CloudFormation/SAM template.
- github.com/e-mit/aws-lambda-get creates an AWS Lambda which executes a periodic HTTP GET request of a REST API and puts the resulting data into a SQS queue.
- github.com/e-mit/aws-create-db creates and configures an AWS Relational Database Service (RDS) instance running PostgreSQL.
- github.com/e-mit/aws-ec2-grafana for configuring and deploying Grafana on an EC2 instance to display a public data dashboard
Tests and linting checks run via GitHub actions after each push. Tests can be run locally (no interaction with AWS), or with AWS (cloud tests).
- Optional: provide a PostgreSQL database server for testing, else the tests will use SQLite only.
- The following environment variables must be set before running the tests: TEST_DB_PORT, TEST_DB_USER, TEST_DB_NAME, TEST_DB_HOST, TEST_DB_PASSWORD, TEST_DB_DIALECT_DRIVER
- The database named
TEST_DB_NAME
must already exist. - Only
TEST_DB_DIALECT_DRIVER='postgresql+psycopg2'
is supported.
- Run all tests and linting with
test.sh
TODO
- Provide values for the environment variables listed in
config.sh
- Execute script
setup.sh
. This will create the resources and start the lambda. A file "id.txt" is created which stores a random number used for uniqueness. - Change log level using:
./create.sh loglevel <log level string e.g. DEBUG>
- Stop the lambda and delete all resources using:
./create.sh clean
After deploying the stack, the lambda code and the packaged Python library dependencies can be updated independently, and rapidly, using the following commands:
- Lambda function update:
./create.sh update_function
- Python packages update:
./create.sh update_layer