Hi, I'm trying to install RKE but I'm receiving some errors.

Openstack version: "Train"

terraform.tvars:

edge_count    = 1
worker_count  = 1
master_count = 1

master_labels = { "node-role.kubernetes.io/master" = "true" }
edge_labels   = { "node-role.kubernetes.io/edge" = "true" }
public_net_name = "provider"
master_flavor_name = "a.large"
worker_flavor_name = "a.large"
edge_flavor_name = "a.large"

cluster_name = "rke"
ssh_keypair_name = "local"
nodes_net_cidr = "10.13.0.0/24"
dns_servers = ["8.8.8.8"]
dns_domain = "arkan.cloud."

use_ssh_agent = false

image_name = "ubuntu-18.04-minimal-cloudimg-amd64"
user_data_file = "rancher.yml"
acme_email = "[email protected]"

rancher.yml

#cloud-config

ssh_authorized_keys:
    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYQykR9v8sGtHV0fl1Otm8N3nGFUYg8iO5IhODQO1zHtIAB8/px0JSu8g1EifbXzvx3GUUYDW2lNBbUYPOP1Os27M6lYz68qYKxMjLXraEHt9jNe7aUVIyNu2iGc3ZAgSrkRabw7P05ijLdH3A6MscQnc4tqLE92a3z/QGcANvtymJvpkvhuE3iUz92NxyR9AHaj9ejGbzJ2vu9kISVx2cUEvymW6x8zGh/agljsIXcp/KYUVr/MvpfVrdk7tUrLg3vAN5+273pWRGNTrtEdwhxBvSAYyU/p/C66G7ZHAILKj45rm0kpNqCNeQh2UiwmDbFEqQmtSZsEHKwRNCbdAx arkan@DESKTOP-8FQ42NL

packages:
    - docker.io

# create the docker group
groups:
    - docker

# Add default auto created user to docker group
system_info:
    default_user:
        groups: [docker]

Debug with Errors

Some infrastructure dont play well with kubernetes integration. Make the openstack_cloud_provider conditionnal.

It would be great to support Octavia. We should then provision a LB for master HA and take advantage of Octavia Kubernetes cloud provider support.

When setting everything up, it fails

Docker logs says:

failed to run Kubelet: could not init cloud provider "openstack": Authentication failed

As per openshift/origin#19659 passwords with special characters must be provided with single quotes. Unfortunately I have a password with special characters and I can't do anything about it.

I looked into the code but found no place where it is defined.

I tested the script, but unfortunately the install fails, because no docker engine is installed. I tried with

wait_for_commands = [
"sudo apt update",
"sudo apt upgrade -y",
"sudo curl https://releases.rancher.com/install-docker/19.03.sh | sh"
]

but without any success, because the worker nodes have no direct internet connection. Only private IP. How do I do that?

I keep getting:

Error: Failed to instantiate provider "rke" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

What I've done:

• upgrade terrafom to 1.12
• Install terraform-provider-rke v1.0.0-rc4
• terraform init
• terraform plan -> this gives me the error above

$ tf providers
.
├── provider.openstack
└── module.rke
    ├── module.edge
    │   ├── provider.null
    │   └── provider.openstack (inherited)
    ├── module.keypair
    │   └── provider.openstack (inherited)
    ├── module.master
    │   ├── provider.null
    │   └── provider.openstack (inherited)
    ├── module.network
    │   └── provider.openstack (inherited)
    ├── module.rke
    │   ├── provider.local
    │   ├── provider.null
    │   ├── provider.openstack
    │   └── provider.rke
    ├── module.secgroup
    │   └── provider.openstack (inherited)
    └── module.worker
        ├── provider.null
        └── provider.openstack (inherited)

$ tf version
tf version
Terraform v0.12.24
+ provider.local v1.4.0
+ provider.null v2.1.2
+ provider.openstack v1.26.0
+ provider.rke v1.1
+ provider.rke v1.4

Hello,
first of all let me thank you very much for sharing this awesome module!

I have a small issue when using the latest release of the module (0.6.0) and Terraform 0.14.2 when setting enable_loadbalancer to true.

The simplest way to demonstrate it is to start from the first example of the README, after sourcing my openrc file, terraform init then terraform apply.
The config is:

variable "os_auth_url" {}
variable "os_password" {}

module "rke" {
  source             = "remche/rke/openstack"
  image_name         = "ubuntu-20.04-docker-x86_64" #An actual image in my Openstack project
  public_net_name    = "public"
  master_flavor_name = "m1.small"
  worker_flavor_name = "m1.small"
  os_auth_url        = var.os_auth_url
  os_password        = var.os_password

  enable_loadbalancer = true
}

Will result in the following output:

Error: Unsupported attribute

  on .terraform/modules/rke/output.tf line 33, in output "loadbalancer_floating_ip":
  33:   value       = var.enable_loadbalancer ? module.loadbalancer.floating_ip : ""
    |----------------
    | module.loadbalancer is tuple with 1 element

This value does not have any attributes.

I have done some experiment on my own and the culprit seems to be the fact that the module loadbalancer is conditionally instantiated (through count = var.enable_loadbalancer ? 1 : 0) and that confuses somehow Terraform that doesn't recognize the output within the module.

Unfortunately I wasn't able to devise a reasonable fix to the issue, otherwise I would have opened a PR myself.

Do you have any idea on how we can workaround this problem? (for the time being I'm using the module from a local clone where I've removed the output :D )

Thank you!

add custom taints to nodes group.

I get an entry in the logs about the "openstack" cloud provider being deprecated.

Since terraform 1.0.0-rc3, downscale does not work anymore :
rancher/terraform-provider-rke#178

Creating PVs fails, wehen installing in an openstack environment, where you have multiple AZs for compute (e.g. ams-a, ams-b, ams-c) but only one for storage (e.g. nova). Kubernetes tries to create them in the compute AZs, which openstack doesn't allow.

There is a solution in openshift/installer#2844 (comment)

Especially you need the "ignore_volume_az" option in cloud provider config.

When I set

  use_ssh_agent      = false
  ssh_keypair_name   = "thatcher" # existing key on openstack

The module fails with the error:

"rke" Failed initializing cluster err:Error while reading SSH key file: "file name too long", it also prints the full content of my RSA key. It is somehow trying to use the content of my SSH key as the filename.

setting the variable ssh_key_file doesn't seem to make a difference.

When I switched to using ssh-agent ssh-add, no arguments this error went away.

Since recent terraform_quality_gate change, code-quality action fails.
dallinwright/terraform_quality_gate#5

I was trying to make this Terraform module to work on, but I got a little confused about the use_ssh_agent. What exactly does it do?

The default is set to true, but that means you can't provide a private SSH key to access your newly provisioned nodes:

I was only getting timeouts with the default value:

Error: timeout - last error: Error connecting to bastion: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

When setting the use_ssh_agent on false, I could get connection.

we should be able to disable write of kubeconfig file on disk.

segmentio/terraform-docs

Make worker nodes an array to enable different worker type.
Wait for hashicorp/terraform#17519

This might be a bug in the RKE provider, however since I am not using that directly I figured it makes sense to report it here (first). Using the following config:

data "openstack_images_image_v2" "ubuntu" {
  name = "Ubuntu-18.04"
  most_recent = true
}

resource "openstack_compute_keypair_v2" "keypair" {
  name = "my-application-keypair-${var.environment}"
}

module "rke" {
  cluster_name       = "my-application-${var.environment}"
  source             = "remche/rke/openstack"
  version            = "0.5.4"
  image_name         = data.openstack_images_image_v2.ubuntu.name
  public_net_name    = "external"
  master_flavor_name = "m1.medium"
  worker_flavor_name = "m1.large"
  os_auth_url        = "https://myopenstackprovider.com:5000"
  os_password        = var.os_password
  edge_count         = 0
  worker_count       = 4
  master_count       = 1
  use_ssh_agent      = true
  ssh_keypair_name   = openstack_compute_keypair_v2.keypair.name
  master_labels      = { "node-role.kubernetes.io/master" = "true" }
  edge_labels        = { "node-role.kubernetes.io/edge" = "true" }
  user_data_file     = "cloud-init.yaml"
  system_user        = "ubuntu"
  nodes_config_drive = true
  deploy_traefik = true
  deploy_nginx = false
}

When I increase worker_count to 5 and do terraform apply -auto-approve, it spins up a new instance on my Openstack provider, however the instance does not register as a node with the RKE cluster that is already running on the existing instances. This used to be the case when I still used 0.4.2 of this provider, but is no longer the case with 0.5.4. I've tested on two separate existing clusters, both successfully create the new instance on Openstack but fail to recognize the new node. In both cases, the apply gets interrupted with:

time="2020-10-05T14:00:49+02:00" level=error msg="Failed to upgrade hosts: my-application-staging-worker-004 with error [Failed to verify healthcheck: Failed to check http://localhost:10248/healthz for service [kubelet] on host [192.168.42.42]: Get http://localhost:10248/healthz: Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: F1005 12:00:45.096391   25275 server.go:274] failed to run Kubelet: could not init cloud provider \"openstack\": Authentication failed]"                                                                                                                                                                 

Failed running cluster err:[workerPlane] Failed to upgrade Worker Plane: [Failed to verify healthcheck: Failed to check http://localhost:10248/healthz for service [kubelet] on host [192.168.42.42]: Get http://localhost:10248/healthz: Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: F1005 12:00:45.096391   25275 server.go:274] failed to run Kubelet: could not init cloud provider "openstack": Authentication failed]                                   
========================================                                                    

on .terraform/modules/rke/modules/rke/main.tf line 54, in resource "rke_cluster" "cluster":             
54: resource "rke_cluster" "cluster" {

However, in both cases just retrying terraform apply -auto-approve eventually results in Apply complete! Resources: 1 added, 0 changed, 0 destroyed..

Terraform v0.13.2
+ provider registry.terraform.io/hashicorp/local v1.4.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/rancher/rke v1.1.2
+ provider registry.terraform.io/terraform-provider-openstack/openstack v1.32.0
+ provider registry.terraform.io/terraform-providers/openstack v1.32.0

work in tf-13.0 branch :

• Providers changes
• Documentation
• hashicorp/terraform#25815

0.5.x will break Terraform 0.12.x compatibility, 0.4.x branch will remain compatible