aws-batch-python-runner's Introduction

aws-batch-runner

This repo represents a Proof of Concept for using AWS Batch to run a python-based cli on a as-needed basis.

Includes gb-util which simulates "ingestion" of books from the "recent" feed on Project Gutenberg

Prerequisite Installations

Terraform
Python: tested with 3.8
pyenv: highly recommended for working with multiple versions of Python.

Deploying the Infrastructure

pushd infrastructure
terraform init
terraform import aws_cloudwatch_log_group.aws_batch_job_log_group /aws/batch/job
popd

Running the cli

pushd src
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

# for development
pip install -r requirements-dev.txt

python gb-util.py --help

Docker

# build the image
docker build ./src -f ./src/Dockerfile -t gb-util:latest

# run it
docker run --rm gb-util:latest --help

Deploy a new version

TODO: make this less error-prone

# 1. Review current version `app_image` var in `./infrastructure/variables.tf`
# 2. increment using semantic versioning: `export VERSION=x.x.x
# 3. build
docker build ./src -f ./src/Dockerfile -t gb-util:latest
# 4. tag
docker tag gb-util:latest krcourville/gb-util:${VERSION}
# 5. push
docker push krcourville/gb-util:${VERSION}
# 6.update Batch Job Definition:
pushd infrastructure
export TF_VAR_image_version=$VERSION
terraform apply
popd
# 7.commit changes
git add... git commit...etc..
# 8. tag the repo:
git tag $VERSION && git push --tags