aws-containers / airflow-on-fargate Goto Github PK

View Code? Open in Web Editor NEW

55.0 55.0 35.0 39 KB

License: MIT No Attribution

Dockerfile 2.73% Shell 1.23% Python 21.36% TypeScript 74.68%

airflow-on-fargate's People

Contributors

Stargazers

Watchers

airflow-on-fargate's Issues

Create SQS queue with CloudFormation

Hello,

It looks like you allow Celery to create SQS queue outside of CloudFormation stack. As for me it doesn't look good since you lose control towards the resource. Why don't you create the queue with CloudFormation and pass the url of the queue to Celery? I didn't find the information how to do that (like which parameters to use). Maybe you could show an example how to do that?

Thanks.

Sync airflow dags with S3

I am fairly new to AWS and Airflow, I managed to deploy airflow using this repo and I was wondering how I sync my dags from s3 or github to the running containers?

Deploy to existing VPC

This CDK code will try to create new VPC. It also should support existing VPC.

Python version for AWS CDK 2.0?

Are there any python forks for this? And on AWS CDK 2.x as well as Airflow:latest.

Thanks - Sandeep

Why not choose Aurora Serverless instead of RDS?

It's cool to run the workload on Serverless Fargate, but why not use Aurora Serverless to replace the RDS? so the total solution would be serverless, is that a good idea?

CDK Deploy Error - invalid signature

Our team was trying to setup Airflow through the following steps via CDK:

https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/

However, during the cdk deploy step, and after answering "yes" to the prompt, we are encountering the following error:

#5 [2/7] RUN apt-get update && apt-get install -y python3-pip     libcurl4-gnutls-dev     librtmp-dev     python3-dev     libpq-dev
#5 sha256:787088182aea3ef729295dade2fab650429db33d33b8265ed2abda11ad49abff
#5 0.327 + apt-get update
#5 0.568 Get:1 http://security.debian.org/debian-security buster/updates InRelease [34.8 kB]
#5 0.580 Get:2 http://deb.debian.org/debian buster InRelease [122 kB]
#5 0.633 Get:3 http://deb.debian.org/debian buster-updates InRelease [56.6 kB]
#5 0.715 Get:4 http://repo.mysql.com/apt/debian stretch InRelease [21.6 kB]
#5 0.769 Get:5 http://security.debian.org/debian-security buster/updates/main amd64 Packages [508 kB]
#5 0.916 Get:6 http://deb.debian.org/debian buster/main amd64 Packages [7909 kB]
#5 1.216 Get:7 http://deb.debian.org/debian buster-updates/main amd64 Packages [8788 B]
#5 1.227 Err:4 http://repo.mysql.com/apt/debian stretch InRelease
#5 1.227   The following signatures were invalid: EXPKEYSIG 8C718D3B5072E1F5 MySQL Release Engineering <[email protected]>
#5 2.403 Reading package lists...
#5 3.271 W: GPG error: http://repo.mysql.com/apt/debian stretch InRelease: The following signatures were invalid: EXPKEYSIG 8C718D3B5072E1F5 MySQL Release Engineering <[email protected]>
#5 3.271 E: The repository 'http://repo.mysql.com/apt/debian stretch InRelease' is not signed.
#5 ERROR: executor failed running [/bin/bash -o pipefail -e -u -x -c apt-get update && apt-get install -y python3-pip     libcurl4-gnutls-dev     librtmp-dev     python3-dev     libpq-dev]: exit code: 100
------
executor failed running [/bin/bash -o pipefail -e -u -x -c apt-get update && apt-get install -y python3-pip     libcurl4-gnutls-dev     librtmp-dev     python3-dev     libpq-dev]: exit code: 100

 ❌  FarFlow failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at publishAssets (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:391:74837)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async deployStack (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:393:390)
    at async Object.deployStack2 [as deployStack] (/Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:400:149977)
    at async /Users/<REDACTED>/.nvm/versions/node/v16.14.2/lib/node_modules/aws-cdk/lib/index.js:400:135508

Do you have any insight as to how to solve this problem? Thanks!

future plans for this repo / tagging / airflow versions / etc

Hi - even with managed airflow available on AWS, I still see a need for this repo. I have needs to customize around airflow that I can't do with managed airflow. I would like to start tagging releases in the repo with upgraded airflow versions. I would also like to do some other upgrades like pscopg2 instead of pygresql. Let me know if I can be a maintainer, I am happy to submit PRs without being a maintainer, but would want tagging as we advance this repo so people can always use older versions of the repo as they need to. (older airflow versions as an example)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.