saltstack-formulas / airflow-formula Goto Github PK

View Code? Open in Web Editor NEW

3.0 3.0 2.0 331 KB

Apache Airflow flexible formula

License: Other

Ruby 7.77% SaltStack 16.75% Shell 0.87% JavaScript 2.87% Jinja 71.74%

airflow-formula's People

Contributors

Stargazers

Watchers

Forkers

myii noelmcloughlin

airflow-formula's Issues

[FEATURE] Integrate bits of airflow ecosystem

Describe the solution you'd like

Some of these should be installed as default, others are for pillar.example.

AIRFLOW TODO

pips: pylint-airflow dag-factory, airflow-code-editor black afctl gusty jupyter tableauserverclient azure-storage-blob [DONE https://github.com//pull/34 ]
environment variable: AIRFLOW_HOME

K8S FORMULA

Operators: airflow-operator ( saltstack-formulas/kubernetes-formula#39)
k8s: the-keda-autoscaler ( saltstack-formulas/kubernetes-formula#40 )

MAYBE

scheduler-failover-controller: git+git://github.com/teamclairvoyant/airflow-scheduler-failover-controller.git@master

WILL NOT DO

dag: airflow-dag-dependencies # NEEDS V2 FIXES
- dags: https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/db-cleanup # NEEDS V2 FIXES
dags: airflow-maintenance # NO RELEASE FILE
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/backup-configs/airflow-backup-configs.py
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/clear-missing-dags/airflow-clear-missing-dags.py
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/delete-broken-dags/airflow-delete-broken-dags.py
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/kill-halted-tasks/airflow-kill-halted-tasks.py
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/log-cleanup/airflow-log-cleanup.py
wget https://raw.githubusercontent.com/teamclairvoyant/airflow-maintenance-dags/master/db-cleanup/airflow-db-cleanup.py

Also there is a warning about JupyterNotebookCommand in logs.

Jan 15 02:40:06 localhost.localdomain airflow[27300]: [2021-01-15 02:40:06,889] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.qubole.hooks.qubole.QuboleHook' from 'apache-airflow-providers-qubole' package: cannot import name 'JupyterNotebookCommand'

[FEATURE] add flask multi password backends

Describe the solution you'd like

Allow multiple authentication back ends for flask, useful for heterogeneous environments. Looks easy to do.
https://flask-multipass.readthedocs.io/en/latest/quickstart/#installation

Additional context

[FEATURE] Move CI/CD to Gitlab

Working CI/CD is needed using the Kitchen Test.
Once CI is moved to Gitlab we can do Kitchen Tests there.

[FEATURE] How to upgrade using IaC?

Is your feature request related to a problem?

This is just installer

Describe the solution you'd like

Upgrade

Describe alternatives you've considered

Review upgrade documentation:

[FEATURE] activate agrcomplete for ariflow user

Describe the solution you'd like

When creating the user, install argcomplete package and activate the feature for airflow user
See https://airflow.apache.org/docs/apache-airflow/stable/usage-cli.html#set-up-bash-zsh-completion

 su - airflow
 pip install argcomplete
 register-python-argcomplete airflow >> ~/.bashrc

[BUG] [Archlinux] track upstream bug and workaround

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: 1.14.4
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.2
       libgit2: Not Installed
      M2Crypto: 0.36.0
          Mako: 1.1.3
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.0 (default, Oct  7 2020, 23:09:01)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 19.0.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: arch rolling n/a
        locale: utf-8
       machine: x86_64
       release: 5.9.11-arch2-1
        system: Linux
       version: Arch Linux rolling n/a

Pillar / config used

postgres:
  users:
    airflow:
      ensure: present
      password: airflow
      createdb: true
      inherit: true
      createroles: true
      replication: true
  databases:
    airflow:
      owner: airflow
  acls:
    - ['local', 'airflow', 'airflow', 'md5']
    - ['local', 'all', 'all', 'peer']
    - ['host', 'all', 'all', '127.0.0.1/32', 'md5']
    - ['host', 'all', 'all', '::1/128', 'md5']
    - ['local', 'replication', 'all', 'peer']
    - ['host', 'replication', 'all', '127.0.0.1/32', 'md5']
    - ['host', 'replication', 'all', '::1/128', 'md5']

airflow:
  config:
    airflow:
      content:
        executor: LocalExecutor
  service:
    airflow:
      enabled:
        - airflow-flower
        - airflow-scheduler
        - airflow-webserver
        - airflow-worker
  pkg:
    airflow:
      version: 1.10.13
      use_upstream: pip
      extras:
        - postgres
        - google
  dir:
    airflow:
      config: /home/airflow/airflow
  linux:
    altpriority: 0   # zero disables alternatives

  # Just for testing purposes
  winner: pillar
  added_in_pillar: pillar_value

Bug details

Describe the bug

          ID: airflow-service-install-database
    Function: cmd.run
        Name: /home/airflow/airflow/bin/airflow initdb
      Result: False
     Comment: Command "/home/airflow/airflow/bin/airflow initdb" run
     Started: 16:55:40.274152
    Duration: 979.898 ms
     Changes:
              ----------
              pid:
                  6757
              retcode:
                  1
              stderr:
                  Traceback (most recent call last):
                    File "/home/airflow/airflow/bin/airflow", line 26, in <module>
                      from airflow.bin.cli import CLIFactory
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/bin/cli.py", line 94, in <module>
                      api_module = import_module(conf.get('cli', 'api_client'))  # type: Any
                    File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
                      return _bootstrap._gcd_import(name[level:], package, level)
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/api/client/local_client.py", line 24, in <module>
                      from airflow.api.common.experimental import delete_dag
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/api/common/experimental/delete_dag.py", line 26, in <module>
                      from airflow.models.serialized_dag import SerializedDagModel
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/models/serialized_dag.py", line 35, in <module>
                      from airflow.serialization.serialized_objects import SerializedDAG
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 28, in <module>
                      import cattr
                    File "/home/airflow/airflow/lib/python3.9/site-packages/cattr/__init__.py", line 1, in <module>
                      from .converters import Converter, GenConverter, UnstructureStrategy
                    File "/home/airflow/airflow/lib/python3.9/site-packages/cattr/converters.py", line 17, in <module>
                      from attr import fields, resolve_types
                  ImportError: cannot import name 'resolve_types' from 'attr' (/home/airflow/airflow/lib/python3.9/site-packages/attr/__init__.py)
              stdout:

Steps to reproduce the bug

Use Salt Py3: saltstack-formulas/salter#81 (comment)
clone this repo
cp ./airflow-formula/pillar-example /srv/pillar/
Run highstate with redis, postgres, airflow
And finally this: apache/airflow#11965

Expected behaviour

Attempts to fix the bug

Additional context

[FEATURE] Update config file for airflow 2.0

Is your feature request related to a problem?

Describe the solution you'd like

The files\default\airflow.cfg.jinja file needs to be updated for airflow 2.n according to testing

          ID: airflow-service-security-managed
    Function: cmd.run
        Name: /tmp/airflow/132495.py
      Result: False
     Comment: Command "/tmp/airflow/132495.py" run
     Started: 23:42:13.663366
    Duration: 2825.465 ms
     Changes:
              ----------
              pid:
                  32130
              retcode:
                  1
              stderr:
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:269: FutureWarning: The hostname_callable setting in [core]
has the old default value of 'socket:getfqdn'. This value has been changed to 'socket.getfqdn' in the running config, but please update your config befor
e Apache Airflow 2.1.
                    FutureWarning,
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:269: FutureWarning: The navbar_color setting in [webserver]
has the old default value of '#007A87'. This value has been changed to '#fff' in the running config, but please update your config before Apache Airflow
2.1.
                    FutureWarning,
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324: DeprecationWarning: The log_format option in [core] has
 been moved to the log_format option in [logging] - the old setting has been used, but please update your config.
                    option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324: DeprecationWarning: The simple_log_format option in [co
re] has been moved to the simple_log_format option in [logging] - the old setting has been used, but please update your config.
                    option = self._get_option_from_config_file(deprecated_key, deprecated_section, key, kwargs, section)
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The logging_config_class option in [
core] has been moved to the logging_config_class option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The logging_level option in [core] h
as been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The fab_logging_level option in [cor
e] has been moved to the fab_logging_level option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_log_format option in [co
re] has been moved to the colored_log_format option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_console_log option in [c
ore] has been moved to the colored_console_log option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The colored_formatter_class option i
n [core] has been moved to the colored_formatter_class option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The base_log_folder option in [core]
 has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The dag_processor_manager_log_locati
on option in [core] has been moved to the dag_processor_manager_log_location option in [logging] - the old setting has been used, but please update your
config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The log_filename_template option in
[core] has been moved to the log_filename_template option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The log_processor_filename_template
option in [core] has been moved to the log_processor_filename_template option in [logging] - the old setting has been used, but please update your config
.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The remote_logging option in [core]
has been moved to the remote_logging option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The task_log_reader option in [core]
 has been moved to the task_log_reader option in [logging] - the old setting has been used, but please update your config.
                  /home/airflow/airflow/lib/python3.6/site-packages/airflow/configuration.py:324 DeprecationWarning: The statsd_on option in [scheduler]
has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
                  Traceback (most recent call last):
                    File "/tmp/airflow/132495.py", line 4, in <module>
                      from airflow.contrib.auth.backends.password_auth import PasswordUser
                  ModuleNotFoundError: No module named 'airflow.contrib.auth'

Describe alternatives you've considered

Additional context

[BUG] The kerberos extra hangs pip install on Ubuntu 18.04

Your setup

Formula commit hash / release tag

Need to figure out is this upstream problem? For the moment do not use Kerberos extra.

Versions reports (master & minion)

Pillar / config used

Bug details

Describe the bug

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[FEATURE] pass args into sytemd

Is your feature request related to a problem?

Describe the solution you'd like

Example: airflow scheduler --num_runs

CELERYD_MAX_TASKS_PER_CHILD = n # new process/ latest code

Describe alternatives you've considered

Additional context

[BUG] webserver pid file blocked systemd start

Versions reports (master & minion)

Airflow 2.0rc1

Pillar / config used

See pillar.example - standard redis/postgres/airflow setup

Bug details

Describe the bug

After rebooting laptop vagrant had residual file blocking webserver from running.

Dec 10 19:53:28 ubuntu1804.localdomain airflow[4688]: Error: Already running on PID 2222 (or pid file '/home/airflow/airflow/airflow-webserver.pid

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

rm /home/airflow/airflow/airflow-webserver.pid

Additional context

[BUG] Install failed on Archlinux python3.9 Numpy issue

Versions reports (master & minion)

Pillar / config used

Bug details

Describe the bug

See numpy/numpy#17937

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[BUG] in python3.8 do not install typing via pip

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

Bug details

Describe the bug

Do not install typing when system is using python3.8+

See pypa/pip#8272 and ethereum/eth-abi#131

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[FEATURE] Kubernetes support

Is your feature request related to a problem?

Describe the solution you'd like

Salter and Saltstack-formulas already support Kubernetes and Docker. We should deliver an artifact YAML files for consmption by Kuberentes/Docker Swarm in this repo. Since community is moving from Travis CI to Gitlab maybe there is possibility to do CI/CD on Gitlab.

This is needed for SODA installer anyway (@noelmcloughlin)

[BUG] [archlinux] cannot install airflow on python 3.9

Versions reports (master & minion)

Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: 1.14.4
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.2
       libgit2: Not Installed
      M2Crypto: 0.36.0
          Mako: 1.1.3
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.0 (default, Oct  7 2020, 23:09:01)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 19.0.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: arch rolling n/a
        locale: utf-8
       machine: x86_64
       release: 5.9.11-arch2-1
        system: Linux
       version: Arch Linux rolling n/a

Pillar / config used

Bug details

Describe the bug

This is the error. It is apparently caused by https://github.com/Tinche/cattrs/issues/100

                  Traceback (most recent call last):
                    File "/home/airflow/airflow/bin/airflow", line 26, in <module>
                      from airflow.bin.cli import CLIFactory
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/bin/cli.py", line 94, in <module>
                      api_module = import_module(conf.get('cli', 'api_client'))  # type: Any
                    File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
                      return _bootstrap._gcd_import(name[level:], package, level)
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/api/client/local_client.py", line 24, in <module>
                      from airflow.api.common.experimental import delete_dag
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/api/common/experimental/delete_dag.py", line 26, in <module>
                      from airflow.models.serialized_dag import SerializedDagModel
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/models/serialized_dag.py", line 35, in <module>
                      from airflow.serialization.serialized_objects import SerializedDAG
                    File "/home/airflow/airflow/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 28, in <module>
                      import cattr
                    File "/home/airflow/airflow/lib/python3.9/site-packages/cattr/__init__.py", line 2, in <module>
                      from .converters import Converter, UnstructureStrategy
                    File "/home/airflow/airflow/lib/python3.9/site-packages/cattr/converters.py", line 15, in <module>
                      from ._compat import (
                    File "/home/airflow/airflow/lib/python3.9/site-packages/cattr/_compat.py", line 87, in <module>
                      from typing import _Union
                  ImportError: cannot import name '_Union' from 'typing' (/usr/lib/python3.9/typing.py)

Steps to reproduce the bug

Expected behaviour

It should install.

Attempts to fix the bug

[BUG] TypeError: entry_points() got an unexpected keyword argument 'group'

Your setup

Formula commit hash / release tag

Versions reports (master & minion)

Pillar / config used

Bug details

Describe the bug

The error TypeError: entry_points() got an unexpected keyword argument 'group' is reported despite package importlib-metadata==4.8.2 being installed in the virtualenv. This is happening across all OS.

This probably caused by OS package python3-importlib-metadata being loaded instead of pipenv library.

https://pkgs.org/search/?q=python3-importlib-metadata

The suggested solution is to remove python3-importlib-metadata package using the OS package manager and keep pip package.
See #85

Steps to reproduce the bug

Expected behaviour

Attempts to fix the bug

Additional context

[FEATURE] allow pip install airflow from local git

Is your feature request related to a problem?

If someone has clone of Apache airflow GitHub repository then they may want to install from that Repo.
pip install git+https://[email protected]/[email protected]#egg=airflow[gcp_api,mysql,crypto]==1.7.1.2+test

In this example, package is git+https://[email protected]/[email protected]#egg=airflow not apache-airflow

Describe the solution you'd like

Nice to support this. Maybe this works already?

[BUG] Apache Airflow is not at this location

Bug details

Describe the bug

Need to be careful with BASE_URL - The value of BASE_URL must have protocol http/https:

airflow-formula/airflow/files/default/airflowV2.cfg.jinja

Line 11 in 4e35a21

{%- set baseurl = basehost ~ ':' ~ baseport %}

Attempts to fix the bug

In airflow.cfg ensure base_url = http://<something> instead of base_url = <something>

Additional context

The automated release is failing 🚨

🚨 The automated release from the `master` branch failed. 🚨

I recommend you give this issue a high priority, so other packages depending on you could benefit from your bug fixes and new features.

You can find below the list of errors reported by semantic-release. Each one of them has to be resolved in order to automatically publish your package. I’m sure you can resolve this 💪.

Errors are usually caused by a misconfiguration or an authentication problem. With each error reported below you will find explanation and guidance to help you to resolve it.

Once all the errors are resolved, semantic-release will release your package the next time you push a commit to the master branch. You can also manually restart the failed CI job that runs semantic-release.

If you are not sure how to resolve this, here is some links that can help you:

If those don’t help, or if this issue is reporting something you think isn’t right, you can always ask the humans behind semantic-release.

Cannot push to the Git repository.

semantic-release cannot push the version tag to the branch master on the remote Git repository with URL https://[secure]@gitlab.com/saltstack-formulas/airflow-formula.git.

This can be caused by:

a misconfiguration of the repositoryUrl option
the repository being unavailable
or missing push permission for the user configured via the Git credentials on your CI environment

Good luck with your project ✨

Your semantic-release bot 📦🚀

[FEATURE] add support for airflow impersonation

This is needed here: https://airflow.apache.org/docs/apache-airflow/2.1.1/security/workload.html#impersonation

[FEATURE] Explore Windows Support

Is your feature request related to a problem?

Describe the solution you'd like

Windows Support for Airflow

Describe alternatives you've considered

Kubernetes/Docker Swarm maybe

[FEATURE] open firewall ports

Is your feature request related to a problem?

Describe the solution you'd like

When the firewall is running port must be allowed

firewall-cmd --zone=public --add-port=8080/tcp # default ui port
firewall-cmd --zone=public --add-port=18080/tcp # safer ui port (lots of things claim 8080 by default).