Giter VIP home page Giter VIP logo

upcloud-ansible's Introduction

upcloud-ansible

DEPRECATION NOTICE

This repo has been deprecated and won't receive more fixes, features or version updates. We suggest using UpCloud Terraform Provider for creating infrastructure and UpCloud Ansible Collection for provisioning and modification.

Dynamic inventory and modules for managing servers via UpCloud's API

The inventory script and modules contain documentation and examples as per Ansible's developer guidelines.

Dependencies and supported versions

  • upcloud-api>=2.0.0 must be installed, pip install upcloud-api or get the sources from Github
  • python 3.6 and higher versions are supported by upcloud-api
  • tested with ansible 2.10.9
  • It should work with whatever is the newest version of ansible, if not, please create an issue about it.

Note for OS X users:

  • install ansible with homebrew can make it hard to know what Python ansible is using, using pip install ansible is recommended

Inventory script

Installation

  • move to any location you wish, point to the script with ansible -i /path/to/script/upcloud.py
  • note that upcloud.ini and upcloud.py must be in the same folder; see .ini for settings
  • you may wish to use return_ip_addresses = True in .ini to ensure that SSH works (hostnames may not be in DNS)
  • information on configuring the inventory without specifying -i every time: http://stackoverflow.com/questions/21958727/where-to-store-ansible-host-file-on-osx
  • Define upcloud api user and password in the .ini file or in env variables.
  • Default timeout is defined either in the .ini file or as env variable. (default is 300s)

Usage

# match all servers
ansible all -m ping -i /path/to/upcloud.py

# match all servers from upcloud inventory script
ansible uc_all -m ping -i /path/to/upcloud.py

# inventory group servers by upcloud Tags
ansible <any-upcloud-tag> -m <module> -i <path-to-upcloud-inventory>

UpCloud modules

Installation

  • move the modules to a location of your choice
  • make sure to add the location of your choice into library path:
  • ...or provide module path when invoking ansible:
    • ansible-playbook -M /path/to/modules/dir playbook.yml

Usage

# you can specify inventory and Modules pathes via cli
ansible-playbook create-servers.yml -i /path/to/upcloud.py -M /path/to/upcloud/modules

See the source files for documentation and examples. You may also want to refer to UpCloud's API documentation

The following example shows off some of the features of upcloud, upcloud_tag and upcloud_firewall modules:

---
- hosts: localhost
  connection: local
  serial: 1
  gather_facts: no

  tasks:
    - name: Create upcloud server
      upcloud:
        state: present
        hostname: web1.example.com
        title: web1.example.com
        zone: uk-lon1
        plan: 1xCPU-1GB
        storage_devices:
          - { size: 30, os: 01000000-0000-4000-8000-000030200200 } # Note this is Ubuntu server 20.04 template UUID
          - { size: 100 }
        api_user: <YOUR UPCLOUD USERNAME>
        api_passwd: <YOUR UPCLOUD PASSWORD>
        ssh_keys:
          - ssh-rsa AAAAB3NzaC1yc2EAA[...]ptshi44x [email protected]
          - ssh-dss AAAAB3NzaC1kc3MAA[...]VHRzAA== [email protected]
      register: upcloud_server # upcloud_server.server will contain the API response body

    # upcloud_server.public_ip shortcut will contain a public IPv4 (preferred) or IPv6 address
    # this task is not needed if host_key_checking=False in ansible
    - name: remove new server from known_hosts in case of IP collision
      known_hosts:
        state: absent
        host: "{{ upcloud_server.public_ip }}"

    - name: Wait for SSH to come up
      wait_for: host={{ upcloud_server.public_ip }} port=22 delay=5 timeout=320 state=started

    - name: tag the created server
      upcloud_tag:
        state: present
        uuid: "{{ upcloud_server.server.uuid }}"
        tags: [webservers, london]

    - name: configure firewall
      upcloud_firewall:
        state: present
        uuid: "{{ upcloud_server.server.uuid }}"
        firewall_rules:
          - direction: in
            family: IPv4
            protocol: udp
            destination_port_start: 53
            destination_port_end: 53
            action: accept

          - direction: in
            family: IPv4
            protocol: tcp
            destination_port_start: 22
            destination_port_end: 22
            action: accept

          - direction: in
            family: IPv4
            protocol: tcp
            destination_port_start: 80
            destination_port_end: 80
            action: accept

          - direction: in
            family: IPv4
            protocol: tcp
            destination_port_start: 443
            destination_port_end: 443
            action: accept

          # default rule last:
          - direction: in
            action: drop

Testing

Tests located in project_root/test/ directory. Run with:

pytest test/

To test against all supported python versions, run (will also run flake8 checks):

tox

To check for possible vulnerabilities in python packages, run:

safety check

To check for flake8 errors, run:

flake8

For tests, flake8 and safety check to work - additional dependencies need to be installed:

pip install -r requirements-dev.txt

Bugs, Issues, Problems, Ideas

Please report issues and features requests through the issues page.

upcloud-ansible's People

Contributors

ajmyyra avatar alienhaxor avatar cmouse avatar dependabot[bot] avatar elnygren avatar faustasm avatar iler avatar mlackman avatar onnimonni avatar poposensei avatar scop avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

upcloud-ansible's Issues

Documentation seems to be erroneous

Default rule should be action: drop not "reject". There seems to be an option in API for turning firewall on, but documentation seems to lack that.

Provide ansible_distribution when building inventory

Hi,

We would need to know beforehand if a host if Windows, or Debain or CentOS, this way we can put them in appropriate groups.

For sure windows needs to be divided from the other hosts since other connection parameters are needed when using Ansible.

It would be great if more information of the server was specified.

Worst case just having the initial template name would be a great step forward ;)

Python3 support

the modules don't run properly with Python3, which causes a problem with OSes who dropped usage of Python2 or when Python3 is the default runtime. Furthermore Ansible is now running on Python3 and so is the upcloud-python-api.

2to3 handles the conversion without issues, except for SafeConfigParser() which now depecrated and requires simply ConfigParser()

Allow using `ip_address` as identifier for `upcloud_firewall` directive

Hey!

We would need to automate firewall changes to multiple Upcloud servers but our ansible inventory files only contain the server IP-addresses. Currently only hostname or uuid can be used as a identifier in the upcloud_firewall module.

We could save plenty of manual labour If we could use those IP-addresses as unique identifiers for the machines where we would want to alter the firewalls.

I will create a PR soon using the /ip_address/ endpoint found from your documentation: https://www.upcloud.com/static/downloads/upcloud-apidoc-1.2.1.pdf#page=69

This needs few small changes to the https://github.com/UpCloudLtd/upcloud-python-api code too.

Provide a separate directive for the default firewall rule

The current implementation is that the default firewall rule eg reject: all needs to be the last rule in the upcloud_firewall directive.

Because of that the last rule is completely different than the other rules and the ansible user needs to apply all sorts of hacks to really get this working.

for example if I run this:

- debug:
    var: upcloud_firewall_rules

- name: Setup firewalls for the backend groups
  upcloud_firewall:
    state: present
    ip_address: "{{ item }}"
    firewall_rules: "{{ upcloud_firewall_rules }}"
  with_flattened:
    - "{{ groups['db']|default([]) }}"
    - "{{ groups['web']|default([]) }}"

Works just fine and outputs this =>

 _________________________________
< TASK [upcloud-firewall : debug] >
 ---------------------------------

ok: [localhost] => {
    "upcloud_firewall_rules": [
        {
            "action": "accept",
            "comment": "Member of this cluster",
            "destination_port_end": "22",
            "destination_port_start": "22",
            "direction": "in",
            "family": "IPv4",
            "protocol": "tcp",
            "source_address_end": "94.237.29.230",
            "source_address_start": "94.237.29.230"
        },
        {
            "action": "reject",
            "direction": "in",
            "comment": "Default rule"
        }
    ]
}
______________________________________________________
/ TASK [upcloud-firewall : Setup firewalls for the backend groups ]
 ------------------------------------------------------

changed: [localhost] => (item=example-machine)

But If I later on change that machine 94.237.29.230 => 94.237.26.186 and apply these rules again the new machine gets just appended in the back of the list and the default rule isn't last one anymore. So then I'll get this:

image

Could you please provide better methods for updating the default rule?

default_timeout: A float is required

After the latest changes in master, we started encountering a type problem with the new "default_timeout" variable that was introduced. See the stack trace.

I'm sure a simple type conversion for "default_timeout" to float (or even int) before passing it to ServerManager or CloudManager does the trick.

I can try to provide more information if needed.

Here's the stack trace:

Traceback (most recent call last):
  File \"/tmp/ansible_aSi7uT/ansible_module_upcloud.py\", line 331, in main
    run(module, server_manager)
  File \"/tmp/ansible_aSi7uT/ansible_module_upcloud.py\", line 235, in run
    server = server_manager.find_server(uuid, hostname)
  File \"/tmp/ansible_aSi7uT/ansible_module_upcloud.py\", line 190, in find_server
    servers = self.manager.get_servers()
  File \"/usr/lib/python2.7/site-packages/upcloud_api/cloud_manager/server_mixin.py\", line 44, in get_servers
    servers = self.get_request(request)['servers']['server']
  File \"/usr/lib/python2.7/site-packages/upcloud_api/cloud_manager/base.py\", line 60, in get_request
    return self.request('GET', endpoint, timeout=timeout)
  File \"/usr/lib/python2.7/site-packages/upcloud_api/cloud_manager/base.py\", line 47, in request
    timeout=call_timeout)
  File \"/usr/lib/python2.7/site-packages/requests/api.py\", line 76, in get
    return request('get', url, params=params, **kwargs)
  File \"/usr/lib/python2.7/site-packages/requests/api.py\", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File \"/usr/lib/python2.7/site-packages/requests/sessions.py\", line 643, in send
    r = adapter.send(request, **kwargs)
  File \"/usr/lib/python2.7/site-packages/requests/adapters.py\", line 449, in send
    timeout=timeout
  File \"/usr/lib/python2.7/site-packages/urllib3/connectionpool.py\", line 677, in urlopen
    chunked=chunked,
  File \"/usr/lib/python2.7/site-packages/urllib3/connectionpool.py\", line 381, in _make_request
    self._validate_conn(conn)
  File \"/usr/lib/python2.7/site-packages/urllib3/connectionpool.py\", line 978, in _validate_conn
    conn.connect()
  File \"/usr/lib/python2.7/site-packages/urllib3/connection.py\", line 309, in connect
    conn = self._new_conn()
  File \"/usr/lib/python2.7/site-packages/urllib3/connection.py\", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File \"/usr/lib/python2.7/site-packages/urllib3/util/connection.py\", line 71, in create_connection
    sock.settimeout(timeout)
  File \"/usr/lib64/python2.7/socket.py\", line 224, in meth
    return getattr(self._sock,name)(*args)
TypeError: a float is required

Can't create tags with upcloud_tag using upcloud sub account

I can create servers with upcloud directive correctly but creating tags fails for me.

I tried to create new servers with tags using server uuids like this in my playbook:

tasks:
  - name: Create upcloud servers
      upcloud:
        state: "{{ item.state | default('present') }}"
        hostname: "{{ item.hostname }}"
        title: "{{ upcloud_project_name }} {{ item.title }}"
        zone: "{{ item.zone | default('fi-hel1') }}"
        timezone: "{{ item.timezone | default('Europe/Helsinki') }}"
        plan: "{{ item.plan }}"
        storage_devices: "{{ item.storage_devices }}"
        user: "{{ upcloud_user }}"
        ssh_keys: "{{infra_admin_ssh_keys}}"

        api_user: "{{ upcloud_api_user }}"
        api_passwd: "{{ upcloud_api_passwd }}"

      with_items: upcloud_server_spec_list
      register: upcloud_servers
      tags: ['upcloud']

    - debug: msg="{{upcloud_servers.results}}"

    - name: Tag the created servers
      upcloud_tag:
        state: present
        uuid: "{{ item.server.uuid }}"
        tags: ['helsinki']

        api_user: "{{ upcloud_api_user }}"
        api_passwd: "{{ upcloud_api_passwd }}"

      with_items: "{{ upcloud_servers.results }}"

and after that failed I also tried to put tags by using hostname:

    - name: Tag the created servers
      upcloud_tag:
        state: present
        hostname: "{{ item.hostname }}"
        tags: ['helsinki']

        api_user: "{{ upcloud_api_user }}"
        api_passwd: "{{ upcloud_api_passwd }}"

      with_items: "{{ upcloud_server_spec_list }}"

upcloud_tag module fails always with this error:

=> {
    "failed": true, "item": {
    "changed": false, "invocation": {
        "module_args": {
            "api_passwd": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "api_user": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "avoid_host": null,
            "boot_order": null, "core_number": null,
            "firewall": null, "hostname": "db1.client.geniem.io",
            "ip_addresses": null, "memory_amount": null, "nic_model": null,
            "password_delivery": null, "plan": "2xCPU-2GB",
            "ssh_keys": ["ssh-rsa AAAAB3Nz...", "ssh-rsa XYXYXY...."], 
            "state": "present",
            "storage_devices": [{"os": "Ubuntu 16.04", "size": 50}],
            "timezone": "Europe/Helsinki", "title": "client DB1",
            "user": "deploy", "uuid": null, "video_model": null,
            "vnc": null, "vnc_password": null, "zone": "fi-hel1"
        },
        "module_name": "upcloud"
    },
    "item": {
        "hostname": "server.client.geniem.io", "plan": "2xCPU-2GB", "state": "present",
        "storage_devices": [{"os": "Ubuntu 16.04", "size": 50}], "tags": ["db"],
        "title": "DB1", "zone": "fi-hel1"}, "public_ip": "80.69.172.184", "server": {
            "core_number": "2", "hostname": "db1.client.geniem.io", "license": 0,
            "memory_amount": "2048", "plan": "2xCPU-2GB", "state": "started", "tags": [],
            "title": "client DB1", "uuid": "000da9f5-8135-4f4a-ada2-445926fc3945", "zone": "fi-hel1"
        }
    },
    "msg": "{u'error': {u'error_message': u'The requested action is forbidden.', u'error_code': u'ACTION_FORBIDDEN'}
}
Traceback (most recent call last):
  File \"/tmp/ansible_aTownO/ansible_module_upcloud_tag.py\", line 230, in main
    run(module, tag_manager)
  File \"/tmp/ansible_aTownO/ansible_module_upcloud_tag.py\", line 164, in run
    tag_manager.create_missing_tags(tags)
  File \"/tmp/ansible_aTownO/ansible_module_upcloud_tag.py\", line 112, in create_missing_tags
    self.manager.create_tag(given_tag)
  File \"build/bdist.macosx-10.11-x86_64/egg/upcloud_api/cloud_manager/tag_mixin.py\", line 35, in create_tag
    res = self.request(\"POST\", \"/tag\", body)
  File \"build/bdist.macosx-10.11-x86_64/egg/upcloud_api/base.py\", line 47, in request
    return self.__error_middleware(res, res_json)
  File \"build/bdist.macosx-10.11-x86_64/egg/upcloud_api/base.py\", line 67, in __error_middleware
    raise Exception(res_json)
    Exception: {u'error': {u'error_message': u'The requested action is forbidden.', u'error_code': u'ACTION_FORBIDDEN'}}
"}

I tried to do this in admin panel and it works correctly with the same user credentials.

I also tried your api: https://api.upcloud.com straight from postman and got same results as in ansible:

screen shot 2016-06-16 at 11 47 01 pm

But still basic account information works correctly through the api:
https://api.upcloud.com/1.2/account returns:

{
  "account": {
    "username": "my-user-name-here"
  }
}

Better handling of IPv6 addresses

Currently the upcloud module that is used to create/destroy servers populates an attribute called public_ip that prefers IPv4 due to UpCloudLtd/upcloud-python-api#13. A better option would be to populate public_ipv4 and public_ipv6 attributes.

In addition, the same problem is inherent in the inventory script. Perhaps the .ini file could be used to control whether to use IPv4 or IPv6 (or which would be preferred as both are not guaranteed to exist).

Extreme slowness with big amount of servers

It seems that after increasing amount of servers, there happens an incremental delay on each operation that we execute.

Example with 500 dummy servers

  • Creating and starting a server with Ansible takes around 15 minutes
  • Starting a single server (shutdown -> started) with direct API command takes 8 minutes

This is causing an increasing problem due to changing of server plans and then starting the servers takes a huge amount of downtime.

I know this issue is directly withing the UpCloud API itself, but I could not find a way to contact developers directly.

Fails with ansible 2.11

Hi,

We tested it with 2.10.x without any issues, as soon as we upgraded to 2.11 the plugin seems broken. Error messages here:

 ansible-inventory -i upcloud.yml --graph --vars
[WARNING]:  * Failed to parse /runner/upcloud.yml with auto plugin: Invalid value "community.upcloud.upcloud" for configuration option "plugin_type: inventory plugin: ansible_collections.community.upcloud.plugins.inventory.upcloud setting:
plugin ", valid values are: ['upcloud']
[WARNING]:  * Failed to parse /runner/upcloud.yml with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]:  * Failed to parse /runner/upcloud.yml with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not allowed, this character is reserved to provide a port.
[WARNING]: Unable to parse /runner/upcloud.yml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.