This is a data engineering pipeline that extracts, loads and transforms multiple sets of economic and financial data from the Alpha Vantage API.

The output of the pipeline could be used by data/business analysts, investment managers and other finance professionals, data and finance journalists alike, depending on the specific use case.

The data allows data analysts to investigate correlations and derive hypotheses on causation between those data points. These data points would also show the financial and economic changes coinciding with black swan events such as the outbreak of the COVID-19 pandemic in 2019 and Russia’s invasion of Ukraine in 2022.

The pipeline currently contains the following datasets:

• Foreign Exchange (FX) daily exchange rates for different countries' currencies compared against the US Dollar. The current pipeline defaults to exchange rates against the AUD, EUR, JPY, RUB and GBP respectively.

• Digital & Crypto Currencies daily exchange rates for different digital currencies rates compared against the US Dollar. The current pipeline defaults to exchange rates against BTC, ETH and DOGE respectively.

• US treasury yield data for various maturity timelines. The current pipeline shows the maturity timelines for 3 months, 2 years, 5 years, 7 years and 10 years respectively.

• Object-oriented Python extraction using requests and pandas with integrated logging, testing, and yaml configuration for user input
• Upsert load to Postgres with sqlalchemy and database transformations using modularized sql with jinja templating
• Dockerizing and serving ELT pipeline on AWS with use of ECS, ECR, S3, RDS, and IAM

The pipeline can be forked and run locally as well as run via Docker.

Pass in the secret variables to run the script. The script requres an Alpha Vantage API key (api_key, obtained via filling in the form on this page ) and the credentials of the Postgres database to load the data into (target_db_user, target_db_password, target_db_server_name and target_db_database_name)