Automated Apache airflow ETL pipeline, create connections between Airflow and AWS by creating credentials, copying s3 data, leveraging connections and hooks, build s3 data to the redshift DAG.
Login into the Airflow through webserver UI page
After logging in, add the needed connections through the Admin > Connections menu, namely the aws_credentials and redshift connections.
On the asame UI you need to set the following variables
key=value
s3_final_bucket = udacity-dend
s3_log_data_prefix = log_data
s3_song_data_prefix=song_data
metadata_path=log_json_path.json
or you can insert the missing credentials in and run the script ./set_connections.sh