qubole / afctl Goto Github PK

There is no way to stop local running Airflow setup as of now

global:
git:
upsteam:
name:
url:

Hi.

Could please someone explain what was the idea of commons folder usage? Properly, I mean how to import?.

Airflow has the init func prepare_syspath() that ensures that certain subfolders of AIRFLOW_HOME are on the classpath.
That allows to import modules from folders like DAGS_FOLDER/third_party, PLUGINS_FOLDER/third_party.

For instance i have the next structure:
── module /
│ ├── commons
│ └── dags

What have to be specified for DAGS_FOLDER to be able to import modules from commons into DAG-files.
Do I have to extend sys.path manually? If I keep this by default I will get a module import error.

I'd like to make it clear how to import modules from commons folders. Why commons dirs are on that project's structure level?

Ex. I want to use boto3 but when i run "pip install boto3" or after adding in requirements.txt and doing "pip install -r requirements.txt",
Then still logs says boto3 import error.

So i have to install this in all containers (worker, web server, scheduler, etc) manually to use boto3 in airflow.