This is an ETL pipeline to pull bitcoin exchange data from CoinCap API and load it into our data warehouse. For more details check out the blog at https://startdataengineering.com/post/data-engineering-project-to-impress-hiring-managers/
We use python to pull, transform and load data. Our warehouse is postgres. We also spin up a Metabase instance for our presentation layer.
All of the components are running as docker containers.
- Docker and Docker Compose v1.27.0 or later.
- AWS account.
- AWS CLI installed and configured.
- git.
We have a Makefile
with common commands. These are executed in the running container.
cd bitcoinmonitor
make up # starts all the containers
make ci # runs formatting, lint check, type check and python test
If the CI step passes you can go to http://localhost:3000 to checkout your Metabase instance.
You can connect to the warehouse with the following credentials
Host: warehouse
Database name: finance
The remaining configs are available in the env file.
Refer to this doc for creating a Metabase dashboard.
In production we will run the instances as containers. We have helper scripts in deploy_helpers for this.
You will need to have an ubuntu x_86
EC2 instance with a custom TCP inbound rule with port 3000
open to the IP 0.0.0.0/0
. These can be set when you create an AWS EC2 instance in the configure security group
section. A t2.micro
(free-tier eligible) instance would be sufficient.
You can setup a prod instance as shown below.
cd bitcoinmonitor
chmod 755 ./deploy_helpers/send_code_to_prod.sh
chmod 400 pem-full-file-location
./deploy_helpers/send_code_to_prod.sh pem-full-file-location EC2-Public-IPv4-DNS
# the above command will take you to your ubuntu instance.
# If you are having trouble connecting use method 2 from https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-fix-permission-denied-errors/
# install docker on your Ubuntu EC2 instance
chmod 755 install_docker.sh
./install_docker.sh
# verify that docker and docker compose installed
docker --version
docker-compose --version
# start the containers
unzip bitcoinmonitor.gzip && cd bitcoinmonitor/
docker-compose --env-file env up --build -d
You can spin down your local instance with.
make down
Do not forget to turn off your EC2 instance.