terraform initialization stops after Install MicroOS and restart about terraform-hcloud-kube-hetzner HOT 17 CLOSED

omg - I forgot to check TF_LOG=trace probably there would be hint for that... and yes this was the issue of my ssh keys (I've used my existing one - which was not ed25519 🤦🏻‍♂️) - sorry for bothering you and thanks for support 🍻

It went more forward and exited during null_resource.kustomization - but I need to check it more deeply :)

from terraform-hcloud-kube-hetzner.

@drackows There is a timeout of 120 in the master.tf file, just search 120 and you will find it, double that, destroy and retry please.

from terraform-hcloud-kube-hetzner.

Thank you for quick answer, unfortunately it's the same

module.first_control_plane.hcloud_server.server: Still creating... [7m30s elapsed]
module.first_control_plane.hcloud_server.server: Still creating... [7m40s elapsed]
module.first_control_plane.hcloud_server.server: Still creating... [7m50s elapsed]
module.first_control_plane.hcloud_server.server: Still creating... [8m0s elapsed]
module.first_control_plane.hcloud_server.server (remote-exec): Connecting to remote host via SSH...
module.first_control_plane.hcloud_server.server (remote-exec):   Host: ....
module.first_control_plane.hcloud_server.server (remote-exec):   User: root
module.first_control_plane.hcloud_server.server (remote-exec):   Password: false
module.first_control_plane.hcloud_server.server (remote-exec):   Private key: true
module.first_control_plane.hcloud_server.server (remote-exec):   Certificate: false
module.first_control_plane.hcloud_server.server (remote-exec):   SSH Agent: true
module.first_control_plane.hcloud_server.server (remote-exec):   Checking Host Key: false
module.first_control_plane.hcloud_server.server (remote-exec):   Target Platform: unix
module.first_control_plane.hcloud_server.server: Still creating... [8m10s elapsed]
╷
│ Error: remote-exec provisioner error
│
│   with module.first_control_plane.hcloud_server.server,
│   on modules/host/main.tf line 60, in resource "hcloud_server" "server":
│   60:   provisioner "remote-exec" {
│
│ timeout - last error: dial tcp ............:22: i/o timeout
╵

🤔 what is interesting whole time of terraform execution doesn't change much

from terraform-hcloud-kube-hetzner.

cpx11 - all changed configuration is at the beginning of my message
I'm trying version fromn branch expose-load-balancer-ip 🤞

from terraform-hcloud-kube-hetzner.

I tried to detach server from created k3s firewall, or checked if port 22 is set as allowed and both looks good.
ssh was possible during rescue mode (what I tried earlier)
I'll check in console what's happening during installation and I'll put screenshots here - maybe You're right about hetzners problems 🤞🏻

from terraform-hcloud-kube-hetzner.

@drackows Sorry, can't answer your questions. But a few hints regarding debugging.

You could set TF_LOG=debug or even json, see https://www.terraform.io/internals/debugging . This produces a lot of log output, but can be useful if standard terraform output does not seem sufficient for debugging

But in this case, and as you mention that you can not reach the machine via SSH, it sounds as if networking is broken. You could try to set a user password via ignition_config in master.cf to set a user password and use that password to login via web console.
When you are logged in, please check if you are able to reach remote ips (i.e. ping 1.1.1.1) and if not, please post the output of ip a && ip r to check ip addresses and the routing table

from terraform-hcloud-kube-hetzner.

I've tried with this configuration and latest master version with the same result

location                  = "fsn1" # change to `ash` for us-east Ashburn, Virginia location
network_region            = "eu-central" # change to `us-east` if location is ash
agent_server_type         = "cpx21"
control_plane_server_type = "cpx11"
lb_server_type            = "lb11"

servers_num               = 3
agents_num                = 2

BUT I found clue (actually I found the reason why I cannot ssh to the machine and how to get round of it) 😃
I've checked iptables and there I found DROP rule with tallow list

tallow list

What can help me to ssh to the machine: iptables - drop rule 😅
I couldn't find anything in terraform-s related to iptables or ipset so it needs to be somewhere in image, right?
I've checked default machine in hetzner - it has clean iptables:

What is weird now I can ssh to the machine:

but terraform process does not continued (I've removed this rule when terraform was waiting for server to be available)

I'll give update here if I'll find something

from terraform-hcloud-kube-hetzner.

Ok, thanks, will investigate further. Keep you posted ASAP. Btw, what instance type did you use?

from terraform-hcloud-kube-hetzner.

@mnencia Any ideas?

from terraform-hcloud-kube-hetzner.

If I were you, make sure to start from a fresh Hetzner project. And make sure to not touch the firewall manually?! That could be the case, or Hetzner is having problems today, as we are also experiencing weird random behaviors tonight. But all port 22 are not accessible... That is weird.

Also, use the Hetzner console in the dashboard to have a look at the screen, see what it shows, if the node has rebooted correctly to MicroOS or not.

from terraform-hcloud-kube-hetzner.

@phaer thanks for hints - I'll try to check with that - good idea with setting up password in ignition_config 👍🏻
I've recorded script and console - but I can not see anything from that, it looks like you wrote: something wrong must be with network
https://www.youtube.com/watch?v=U2r5fbwyMxw at about 4:15 I've hit "Ctrl+Alt+Del" button because I wanted to show that restart server doesn't change anything, effect without this restart would be the same (video quality will be better soon after YT processing will be finished)

from terraform-hcloud-kube-hetzner.

Thanks for the video, it's great! 🙏

The network interfaces on Hetzner come up just fine! So that is not it.

from terraform-hcloud-kube-hetzner.

I believe there's something up with the firewall. Just delete the whole Hetzner project, create a new one, get a new token and start over, and this time never touch any Hetzner resource manually. Because if you touch it manually, it later screws up terraform.

Try this, and unless something on your network or machine is blocking SSH (in which case try via cloudflarewarp.com) it should work, there is no reason why not.

And you can also pull from master now, the recent PR has been merged. Good luck! :)

from terraform-hcloud-kube-hetzner.

@mysticaltech last thing I've done was new project with fresh token and without success :( I tried to attach different manually created firewall with "all open" (just after terraform failed). It looks like somehow this machine is hidden from outside because locally I'm able to ssh to the machine but after incorrect login I'm getting this state:

and now locally ssh localhost - it looks like way to find a solution 🤞🏻

I tried what @phaer suggested too (with set password and login in console) and here is the output:

With access to the machine I'll later try to check more things :)

from terraform-hcloud-kube-hetzner.

Yeah, please do. I have no idea why it would do that! Which network are you in? Try fsn1 just out of curiosity, and start fresh. Also you never need to modify the firewall manually otherwise it won't delete on destroy.

from terraform-hcloud-kube-hetzner.

Please pull from the latest and try again.

from terraform-hcloud-kube-hetzner.

Tallow is the microos protection software , it bans abusive IPs.

So only one explanation to you problem. You ssh config in the project is not correct so you get banned by the node after a few retries.

Do you have a passphrase in you ssh private key? Please re-read the section in the readme.

Also use advanced terraform logs like TF_LOG=trace

from terraform-hcloud-kube-hetzner.