bahchis / airflow-cookbook Goto Github PK

Airflow workflow management platform chef cookbook.

License: Apache License 2.0

Ruby 75.03% HTML 24.97%

airflow-cookbook's Introduction

Airflow Chef Cookbook

Installs and configures Airflow workflow management platform. More information about Airflow can be found here: https://github.com/airbnb/airflow

Supported Platforms

Ubuntu (Tested on 14.04, 16.04). CentOS (Tested on 7.2).

Limitations

The Airflow all and oracle packages are not supported, this is due the Oracle package having dependencies which cannot be automatically installed. I will look how to solve this and add support for those packages at later stage.

Contributing

Please follow instructions in the contributing doc.

Usage

Use the relevant cookbooks to install and configure Airflow.
Use environment variable in /etc/default/airflow (for Ubuntu) or /etc/sysconfig/airflow (for CentOS) to configure Airflow during the startup process. (More info about Airflow environment variables at: Setting Configuration Options)
Make sure to run airflow initdb as part of your startup script.

Recipes

default - Executes other recipes.
directories - Creates required directories.
user - Creates OS user and group.
packages - Installs OS and pip packages.
config - Handles airflow.cfg
services - Creates services env file.
webserver - Configures service for webserver.
scheduler - Configures service for scheduler.
worker - Configures service for worker.
flower - Configures service for flower.
kerberos - Configures service for kerberos.
packages - Installs Airflow and supporting packages.

Attributes

User config

["airflow"]["airflow_package"] - Airflow package name, defaults to 'apache-airflow'. Use 'airflow' for installing version 1.8.0 or lower.
["airflow"]["version"] - The version of airflow to install, defaults to latest (nil).
["airflow"]["user"] - The user Airflow is executed with and owner of all related folders.
["airflow"]["group"] - Airflow user group.
["airflow"]["user_uid"] - Airflow user uid
["airflow"]["group_gid"] - Airflow group gid
["airflow"]["user_home_directory"] - Airflow user home directory.
["airflow"]["shell"] - Airflow user shell.

General config

["airflow"]["directories_mode"] - The permissions airflow and user directories are created.
["airflow"]["config_file_mode"] - The permissions airflow.cfg is created.
["airflow"]["bin_path"] - Path to the bin folder, default is based on platform.
["airflow"]["run_path"] - Pid files base directory
["airflow"]["is_upstart"] - Should upstart be used for services, determined automatiaclly.
["airflow"]["init_system"] - The init system to use when configuring services, only upstart or systemd are supported and defaults based on ["airflow"]["is_upstart"] value.
["airflow"]["env_path"] - The path to services env file, determined automatiaclly.

Python config

["airflow"]["python_runtime"] = Python runtime as used by poise-python cookbook.
["airflow"]["python_version"] = Python version to install as used by poise-python cookbook.
["airflow"]["pip_version"] = Pip version to install (true - installs latest) as used by poise-python cookbook.

Package config

default['airflow']['packages'] - The Python packages to install for Airflow.
default['airflow']['dependencies'] - The dependencies of the packages listed in default['airflow']['packages']. These are OS packages, not Python packages.

airflow.cfg

This cookbook enables to configure any airflow.cfg paramters dynamically by using attributes structure like (see the attributes file for airflow.cfg examples): ["airflow"]["config"]["CONFIG_SECTION"]["CONFIG_ENTRY"]

License

Apache 2.0 (http://www.apache.org/licenses/LICENSE-2.0)

Author

Sergey Bahchissaraitsev

airflow-cookbook's People

Contributors

Stargazers

Watchers

airflow-cookbook's Issues

Running on OpsWorks fails

It leaves this kind of trace

================================================================================
Recipe Compile Error in /var/lib/aws/opsworks/cache.stage2/cookbooks/airflow/resources/airflow.rb
================================================================================


ArgumentError
-------------
wrong number of arguments (1 for 0)


Cookbook Trace:
---------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/airflow/resources/airflow.rb:15:in `class_from_file'


Relevant File Content:
----------------------
/var/lib/aws/opsworks/cache.stage2/cookbooks/airflow/resources/airflow.rb:

8:  
9:  # Unless required by applicable law or agreed to in writing, software
10:  # distributed under the License is distributed on an "AS IS" BASIS,
11:  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12:  # See the License for the specific language governing permissions and
13:  # limitations under the License.
14:  
15>> resource_name :airflow
16:  
17:  actions :install
18:  default_action :install
19:  
20:  property :packages, String, default: "crypto,mysql"
21:  
22:  dependencies = {
23:    :all => [
24:        "libldap2-dev", "libsasl2-dev", "libssl-dev"

Airflow initdb command initiated in the wrong home directory

I am trying to use Airflow Cookbook on OpsWorks.
Looking at the folder var/chef/runs/xxxx I see:

/var/chef/runs/e8180f46-7954-47d4-8323-dfac8fcbe471

{
  "airflow": {
    "config": {
      "core": {
        "sql_alchemy_conn": "postgresql://DEVOPS:[email protected]:5432/airflow",
        "parallelism": 10,
        "executor": "LocalExecutor"
      }
    }
  },
  "run_list": [
    "recipe[airflow::webserver]",
    "recipe[airflow::scheduler]"
  ]
}

I see that the airflow.cfg is been generated on folder /usr/local/lib/airflow. So when I start Airflow it not uses this file, it uses the ~/airflow/airflow.cfg.

Does anyone can help me?

Is chef_version necessary in metadata.rb?

We are currently running an older version of chef

sudo env -i chef-client --version
Chef: 12.14.89

I just encountered the version bound specified in the metadata.rb file chef_version '~>12.19' which prevents me from using this cookbook because the converge fails.

Is the chef_version necessary?
915230a

Thank you for your time!

python_runtime error on python3, python_version == '3.5'

I'm trying to upgrade to Airflow 1.10.1 (with Python3) on CentOS 7 and getting an error related to the python_runtime version. I'm using the version numbers from your default and it's still failing.

I realize that this might not be addressed right away, if at all - it looks like this is because of a dependency in poise-python (which was archived without support for Python 3.5 or 3.6 for RHEL6).

# my Python config
default['airflow']['python_runtime'] = '3'
default['airflow']['python_version'] = '3.5'

error stack from chef-client:

  * python_runtime[3] action install
    * poise_languages_system[python] action install
      
      ================================================================================
      Error executing action `install` on resource 'poise_languages_system[python]'
      ================================================================================
      
      PoiseLanguages::Error
      ---------------------
      Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
      
      Cookbook Trace:
      ---------------
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
      /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
      /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
      /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
      
      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb
      
       32:         poise_languages_system system_package_name do
       33:           # Otherwise use the default install action.
       34:           action(:upgrade) if options['package_upgrade']
       35:           parent new_resource
       36:           # Don't pass true because we want the default computed behavior for that.
       37:           dev_package options['dev_package'] unless options['dev_package'] == true
       38:           dev_package_overrides dev_package_overrides
       39:           package_version options['package_version'] if options['package_version']
       40:           version options['version']
       41:           instance_exec(&block) if block
       42:         end
       43:       end
      
      Compiled Resource:
      ------------------
      # Declared in /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb:32:in `install_system_packages'
      
      poise_languages_system("python") do
        action [:install]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        declared_type :poise_languages_system
        cookbook_name "vox-airflow"
        parent # Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
      
      python_runtime("3") do
        provider PoisePython::PythonProviders::System
        action [:install]
        retries 0
        retry_delay 2
        default_guard_interpreter :default
        subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
        declared_type :python_runtime
        cookbook_name "vox-airflow"
        recipe_name "packages"
        version "3.5"
        pip_version "10.0.1"
        get_pip_url "https://bootstrap.pypa.io/get-pip.py"
        setuptools_version true
        virtualenv_version true
        wheel_version true
      end
      
        version "3.5"
        package_name "python"
        dev_package "python-devel"
      end
      
      System Info:
      ------------
      chef_version=12.22.3
      platform=centos
      platform_version=7.4.1708
      ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
      program_name=chef-client worker: ppid=30126;start=23:31:37;
      executable=/opt/chef/bin/chef-client
      
    
    ================================================================================
    Error executing action `install` on resource 'python_runtime[3]'
    ================================================================================
    
    PoiseLanguages::Error
    ---------------------
    poise_languages_system[python] (/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb line 32) had an error: PoiseLanguages::Error: Package python would install 2.7.5-86.el7, which does not match 3.5. Please set the package_name or package_version provider options.
    
    Cookbook Trace:
    ---------------
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:244:in `block (4 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:233:in `block (3 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `tap'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:232:in `block (2 levels) in patch_load_current_resource!'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:211:in `block (2 levels) in run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `each'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:200:in `block in run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:196:in `run_action_hack'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:102:in `block in action_install'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `instance_eval'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/subcontext_block.rb:54:in `subcontext_block'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:67:in `notifying_block'
    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb:100:in `action_install'
    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:45:in `action_install'
    
    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb
    
      1: python_runtime node['airflow']['python_runtime'] do
      2:   version node['airflow']['python_version']
      3:   provider :system
      4:   pip_version node['airflow']['pip_version']
      5: end
      6: 
    
    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/vox-airflow/recipes/packages.rb:1:in `from_file'
    
    python_runtime("3") do
      provider PoisePython::PythonProviders::System
      action [:install]
      retries 0
      retry_delay 2
      default_guard_interpreter :default
      subresources [python_package[apache-airflow], python_package[greenlet], python_package[eventlet], python_package[celery], python_package[cryptography], python_package[sphinx], python_package[sphinx-argparse], python_package[sphinx-rtd-theme], python_package[Sphinx-PyPI-upload], python_package[docker-py], python_package[druid], python_package[boto3], python_package[httplib2], python_package[google-auth-httplib2], python_package[google-api-python-client], python_package[oauth2client], python_package[PyOpenSSL], python_package[hive-thrift-py], python_package[pyhive], python_package[impyla], python_package[unicodecsv], python_package[jaydebeapi], python_package[pymssql], python_package[mysqlclient], python_package[librabbitmq], python_package[psycopg2], python_package[boto], python_package[filechunkio], python_package[pysmbclient], python_package[statsd], python_package[vertica-python], python_package[ldap3], python_package[pykerberos], python_package[thrift_sasl], python_package[snakebite[kerberos]], python_package[bcrypt], python_package[flask-bcrypt], python_package[Flask-OAuthlib], python_package[qds-sdk], python_package[cloudant], python_package[lxml], python_package[nose], python_package[nose-parameterized], python_package[mock], python_package[click], python_package[moto], python_package[freezegun]]
      declared_type :python_runtime
      cookbook_name "vox-airflow"
      recipe_name "packages"
      version "3.5"
      pip_version "10.0.1"
      get_pip_url "https://bootstrap.pypa.io/get-pip.py"
      setuptools_version true
      virtualenv_version true
      wheel_version true
    end
    
    System Info:
    ------------
    chef_version=12.22.3
    platform=centos
    platform_version=7.4.1708
    ruby=ruby 2.3.6p384 (2017-12-14 revision 61254) [x86_64-linux]
    program_name=chef-client worker: ppid=30126;start=23:31:37;
    executable=/opt/chef/bin/chef-client

Unable to install airflow_package via chef in kitchen environment

Ive been trying to bring up a clean kitchen instance for a couple days now, i was getting a setuptools issue until i was able to get past that with attribute assigning the pip_version to 18.0. But i can not seem to get past this error:

`* python_package[apache-airflow] action install[2018-10-18T15:19:20+00:00] INFO: Processing python_package[apache-airflow] action install (airflow::packages line 67)

     ================================================================================
     Error executing action `install` on resource 'python_package[apache-airflow]'
     ================================================================================

     Mixlib::ShellOut::ShellCommandFailed
     ------------------------------------
     Expected process to exit with [0], but received '1'
     ---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
     STDOUT: Collecting apache-airflow==1.10.0
       Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
         Complete output from command python setup.py egg_info:
         Traceback (most recent call last):
    File "<string>", line 20, in <module>
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
      do_setup()
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
      verify_gpl_dependency()
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
      raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
         RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE

         ----------------------------------------
     STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     You are using pip version 6.1.0, however version 18.1 is available.
     You should consider upgrading via the 'pip install --upgrade pip' command.
     The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
         Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
     ---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
     Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1

     Cookbook Trace:
     ---------------
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `tap'
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `language_command_shell_out!'
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:216:in `block in language_command_mixin'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:282:in `pip_command'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:296:in `pip_install'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:209:in `install_package'

     Resource Declaration:
     ---------------------
     # In /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb

      67: python_package node['airflow']['airflow_package'] do
      68:   version node['airflow']['version']
      69: end
      70:

     Compiled Resource:
     ------------------
     # Declared in /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb:67:in `from_file'

     python_package("apache-airflow") do
       package_name "apache-airflow"
       action [:install]
       retries 0
       retry_delay 2
       default_guard_interpreter :default
       declared_type :python_package
       cookbook_name "airflow"
       recipe_name "packages"
       parent_python python_runtime[2]
       timeout 900
     end

     Platform:
     ---------
     x86_64-linux

   [2018-10-18T15:19:25+00:00] INFO: Running queued delayed notifications before re-raising exception

   Running handlers:
   [2018-10-18T15:19:25+00:00] ERROR: Running exception handlers
   Running handlers complete
   [2018-10-18T15:19:25+00:00] ERROR: Exception handlers complete
   Chef Client failed. 38 resources updated in 02 minutes 31 seconds
   [2018-10-18T15:19:25+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
   [2018-10-18T15:19:25+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
   [2018-10-18T15:19:25+00:00] ERROR: python_package[apache-airflow] (airflow::packages line 67) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
   ---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
   STDOUT: Collecting apache-airflow==1.10.0
     Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
       Complete output from command python setup.py egg_info:
       Traceback (most recent call last):
         File "<string>", line 20, in <module>
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
    do_setup()
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
    verify_gpl_dependency()
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
    raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
       RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE

       ----------------------------------------
   STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   You are using pip version 6.1.0, however version 18.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
       Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
   ---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
   Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1
   [2018-10-18T15:19:25+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)`

How do I disable certain python packages from being installed.

Problem

I want to enable python_runtime 3 and some of the packages like librabbitmq are not compatible and throw an error when installing them.

WRT:
https://github.com/bahchis/airflow-cookbook/blob/master/attributes/packages.rb#L14-L66

I would like to disable the installation of certain python packages which are not python 3 compatible.

When I try setting the attributes in the role via:

  'airflow' => {
    'packages' => {
      celery: [{ name: 'celery', version: '>=3.1.15' }],
      crypto: [{ name: 'cryptography', version: '>=0.9.3' }],
      postgres: [{ name: 'psycopg2', version: '>=2.7.1' }],
      password: [{ name: 'bcrypt', version: '>=3.1.3' },
               { name: 'flask-bcrypt', version: '>=0.7.1' }]
    }
  }

chef actually merges in these changes with the default packages attributes.

This is due to the defined behavior in chef docs:

When an attribute value is a hash, that data is merged. When an attribute value is an array, if the attribute precedence levels are the same, then that data is merged. If the attribute value precedence levels in an array are different, then that data is replaced. For all other value types (such as strings, integers, etc.), that data is replaced.

from https://docs.chef.io/attributes.html

LoadError on kitchen converge. LoadError cannot load such file poise_archive/cheftie

I am getting the below issue when running

localhost@deppfx 00:11:45: ~/git/airflow-cookbook $ (master) kitchen converge airflow-scheduler-centos-72
-----> Starting Kitchen (v1.17.0)
-----> Converging <airflow-scheduler-centos-72>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 6.3.1...
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
       el 7 x86_64
       Getting information for chef stable 14.12.3 for el...
       downloading https://omnitruck.chef.io/stable/chef/metadata?v=14.12.3&p=el&pv=7&m=x86_64
         to file /tmp/install.sh.11431/metadata.txt
       trying wget...
       sha1	dea8e74571b15ebb65e03269f0b7c39b49819148
       sha256	a8baf9229d392d723ff42e6b529b7b18411f5909d13247dd39ec0a073b37bb8e
       url	https://packages.chef.io/files/stable/chef/14.12.3/el/7/chef-14.12.3-1.el7.x86_64.rpm
       version	14.12.3
       downloaded metadata file looks valid...
       downloading https://packages.chef.io/files/stable/chef/14.12.3/el/7/chef-14.12.3-1.el7.x86_64.rpm
         to file /tmp/install.sh.11431/chef-14.12.3-1.el7.x86_64.rpm
       trying wget...
       Comparing checksum with sha256sum...
       Installing chef 14.12.3
       installing with rpm...
       warning: /tmp/install.sh.11431/chef-14.12.3-1.el7.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
       Preparing...                          ################################# [100%]
       Updating / installing...
          1:chef-14.12.3-1.el7               ################################# [100%]
       Thank you for installing Chef!
       Transferring files to <airflow-scheduler-centos-72>
       Starting Chef Client, version 14.12.3
       Creating a new client identity for airflow-scheduler-centos-72 using the validator key.
       resolving cookbooks for run list: ["airflow::default", "airflow::scheduler"]
       Synchronizing Cookbooks:
         - apt (7.1.1)
         - airflow (1.2.4)
         - poise-python (1.7.0)
         - poise-languages (2.1.2)
         - poise-archive (1.5.0)
         - poise (2.8.2)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       
       ================================================================================
       Recipe Compile Error in /tmp/kitchen/cache/cookbooks/poise-archive/libraries/default.rb
       ================================================================================
       
       LoadError
       ---------
       cannot load such file -- poise_archive/cheftie
       
       Cookbook Trace:
       ---------------
         /tmp/kitchen/cache/cookbooks/poise-archive/libraries/default.rb:19:in `<top (required)>'
       
       Relevant File Content:
       ----------------------
       /tmp/kitchen/cache/cookbooks/poise-archive/libraries/default.rb:
       
        12:  # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        13:  # See the License for the specific language governing permissions and
        14:  # limitations under the License.
        15:  #
        16:  
        17:  raise 'Halite is not compatible with no_lazy_load false, please set no_lazy_load true in your Chef configuration file.' unless Chef::Config[:no_lazy_load]
        18:  $LOAD_PATH << File.expand_path('../../files/halite_gem', __FILE__)
        19>> require "poise_archive/cheftie"
        20:  
       
       System Info:
       ------------
       chef_version=14.12.3
       platform=centos
       platform_version=7.2.1511
       ruby=ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]
       program_name=/opt/chef/bin/chef-client
       executable=/opt/chef/bin/chef-client
       
       
       Running handlers:
       [2019-06-25T07:12:41+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2019-06-25T07:12:41+00:00] ERROR: Exception handlers complete
       Chef Client failed. 0 resources updated in 04 seconds
       [2019-06-25T07:12:41+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2019-06-25T07:12:41+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
       [2019-06-25T07:12:41+00:00] FATAL: LoadError: cannot load such file -- poise_archive/cheftie
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <airflow-scheduler-centos-72>.  Please see .kitchen/logs/airflow-scheduler-centos-72.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration