sungchun12 / airflow-toolkit Goto Github PK

Hi,

Thanks for your help last time. I am trying to run Toolkit # 3 now but facing the below error:

Broken DAG: [/home/airflow/gcs/dags/airflow_utils.py] 'DEPLOYMENT_SETUP'

Also I have a question in regards to composer private IP:

a. I wish to pull from github instead of using google source repo for which there are 2 options. Either setup a Cloud NAT gateway OR disable private IP. Will it not defeat the purpose of having secure composer env in latter case ?

title explains it all

Hi,

Thank you for this awesome toolkit, really appreciate your hard work.

I am trying to run "Toolkit #1: Local Desktop Kubernetes Airflow Deployment" on my machine and facing the below error (in bold):
Changes done:

• changed dbt version to 0.18.1
• have tried running "dbt_bigquery_example" on CLI and it's working fine, looks like this is some airflow issue
• my local python virtual env is 3.7.9 (if it helps)

Could you please help ?

Error logs:

[2020-11-02 13:13:03,349] {logging_mixin.py:112} INFO - [2020-11-02 13:13:03,349] {pod_launcher.py:125} INFO - b"Cloning into 'composer-template'...\n"
[2020-11-02 13:13:07,742] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,742] {pod_launcher.py:125} INFO - b'+ cd composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ export PROJECT_ID=xxxxx\n'
[2020-11-02 13:13:07,743] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,743] {pod_launcher.py:125} INFO - b'+ PROJECT_ID=xxxxx\n'
[2020-11-02 13:13:07,745] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,745] {pod_launcher.py:125} INFO - b'++ pwd\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ export DBT_PROFILES_DIR=/dbt/composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ DBT_PROFILES_DIR=/dbt/composer-template/dbt_bigquery_example\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ export DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-02 13:13:07,746] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ DBT_GOOGLE_BIGQUERY_KEYFILE=/dbt/account.json\n'
[2020-11-02 13:13:07,747] {logging_mixin.py:112} INFO - [2020-11-02 13:13:07,746] {pod_launcher.py:125} INFO - b'+ dbt run --target service_account_runs\n'
[2020-11-02 13:13:08,502] {logging_mixin.py:112} INFO - [2020-11-02 13:13:08,501] {pod_launcher.py:125} INFO - b'Running with dbt=0.18.1\n'
[2020-11-02 13:13:10,056] {logging_mixin.py:112} INFO - [2020-11-02 13:13:10,056] {pod_launcher.py:125} INFO - b'Found 8 models, 21 tests, 1 snapshot, 0 analyses, 155 macros, 0 operations, 3 seed files, 3 sources\n'
[2020-11-02 13:13:10,061] {logging_mixin.py:112} INFO - [2020-11-02 13:13:10,061] {pod_launcher.py:125} INFO - b'\n'
[2020-11-02 13:13:11,040] {logging_mixin.py:112} INFO - [2020-11-02 13:13:11,040] {pod_launcher.py:125} INFO - b'Encountered an error:\n'
[2020-11-02 13:13:11,041] {logging_mixin.py:112} INFO - [2020-11-02 13:13:11,041] {pod_launcher.py:125} INFO - b'Expecting value: line 1 column 1 (char 0)\n'
[2020-11-02 13:13:12,059] {logging_mixin.py:112} INFO - [2020-11-02 13:13:12,059] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Running
[2020-11-02 13:13:12,060] {logging_mixin.py:112} INFO - [2020-11-02 13:13:12,060] {pod_launcher.py:135} INFO - Pod dbt-run-8cec9cee has state running
[2020-11-02 13:13:14,068] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,067] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Failed
[2020-11-02 13:13:14,068] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,068] {pod_launcher.py:234} INFO - Event with job id dbt-run-8cec9cee Failed
[2020-11-02 13:13:14,072] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,072] {pod_launcher.py:142} INFO - Event: dbt-run-8cec9cee had an event of type Failed
[2020-11-02 13:13:14,072] {logging_mixin.py:112} INFO - [2020-11-02 13:13:14,072] {pod_launcher.py:234} INFO - Event with job id dbt-run-8cec9cee Failed