Timeout on disk resize about terraform-provider-proxmox HOT 10 CLOSED

Thanks for the logs, and your conclusion is correct.

I think you're hitting an issue with a directory based storage similar to #1103. There is probably some FS locking or some other race condition in PVE that prevents resizing of a just attached disk.

Speaking about the fix, I double-checked the current API and I think we can get rid of running shell commands over SSH there. I'll take a look into that.

UPDATE: nope, PVE still does not allow to import e.g. a qcow2 image stored in the iso storage 😞 using import-disk attribute. So the SSH stuff remains, but the resize portion of it will be moved to API.

from terraform-provider-proxmox.

Thanks for the report @geekscrapy. That's interesting behaviour. Personally I don't use "directory" storage type for VMs, so it is not tested well.

The user name change is something strange tho. I just want to confirm your case: you simply ran terraform apply twice, without any changes in template or the provider config, and second time apply passed, but apparently was running under a different user? Is "super" user the one that used to connect to the PVE API? or is the one for SSH access?

from terraform-provider-proxmox.

Many thanks for taking a look. Tbh, I'm unsure about the boundary between the provider and proxmox function, so it might very well be a proxmox issue!

Yes, I'd do the apply them back to back (without modification to anything) and it would come inline with the defined resource as expected. The only modification in the second apply being the change in disk size. And to be clear, the apply is successful each time...!

super is my main admin user (the default admin is my breakglass). I have applied all permissions to the super user both in proxmox and the os level

When the error says timeout, what would you expect this to be? I timeout in the command, or maybe a timeout in issuing the command, or a timeout in the API call?

Tonight I'm going to look at debugging with Terraform, to see if it gives me any clues. I'm new to terraform and pretty new to proxmox (short of spinning up a few lxcs)

UPDATED FOR CLARIFICATION

from terraform-provider-proxmox.

I did an extremely quick bit of debugging and it appears that the first resize call is done over SSH (with the try_sudo prefix, which I guess is the reason for the root user), then the second is done via the API. I captured this via the terraform debug logs.

These lines are from the pve syslog output also indicating the same.

First apply

May 04 22:27:47 pve sudo[720523]:    super : PWD=/home/super ; USER=root ; COMMAND=/sbin/qm resize 102 scsi0 6G
May 04 22:27:47 pve sudo[720523]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
May 04 22:27:48 pve qm[720524]: <root@pam> starting task UPID:pve:000AFEB1:2DA328AC:66367E24:resize:102:root@pam:
May 04 22:27:48 pve qm[720561]: <root@pam> update VM 102: resize --disk scsi0 --size 6G
May 04 22:28:08 pve qm[720561]: command '/usr/bin/qemu-img resize -f qcow2 /mnt/pve/<redacted>/images/102/vm-102-disk-0.qcow2 6442450944' failed: got timeout
May 04 22:28:08 pve qm[720524]: <root@pam> end task UPID:pve:000AFEB1:2DA328AC:66367E24:resize:102:root@pam: command '/usr/bin/qemu-img resize -f qcow2 /mnt/pve/<redacted>/images/102/vm-102-disk-0.qcow2 6442450944' failed: got timeout
May 04 22:28:08 pve sudo[720523]: pam_unix(sudo:session): session closed for user root
May 04 22:28:08 pve sshd[720343]: pam_unix(sshd:session): session closed for user super

Second apply

May 04 22:29:51 pve pvedaemon[721987]: <super@pam!ansible> update VM 102: resize --disk scsi0 --size 6442450944
May 04 22:29:51 pve pvedaemon[75411]: <super@pam!ansible> end task UPID:pve:000B0443:2DA35878:66367E9F:resize:102:super@pam!ansible: OK

So I guess the question is, should the apply functionality be changed always use the API to resize the images?

from terraform-provider-proxmox.

Happy to test out the patch for you of course when its in the works 👍

from terraform-provider-proxmox.

Hi @geekscrapy, if you have a chance to test the code change from #1260 in your environment that would be great!

from terraform-provider-proxmox.

Ok, I'm going to merge the PR, and prob. cut out a release today.

from terraform-provider-proxmox.

Hi, sorry I didn't manage to test the PR

I did however download and test the new release, and good news!

1. The final VM is as expected with only having to do one apply 👍
2. It resizes the disk twice... 😄 The first time it fails, then 3 seconds later success. The user is as expected as well. So it seems to be a race condition as you say. Maybe add a delay of a few seconds before the resize as a stop gap?!

Either way, thanks for pushing it out so fast.

from terraform-provider-proxmox.

I don't want to optimize for a (relatively rare) failure case though, so I'd probably leave the current approach.

BTW, would you consider to set up a block storage for your VMs,like LVM-thin or ZFS? It would have better overall performance, without the FS overhead.

from terraform-provider-proxmox.

I don't want to optimize for a (relatively rare) failure case though, so I'd probably leave the current approach.

Understood, so the double resize is intentional then, I presume. Either way it works!

BTW, would you consider to set up a block storage for your VMs,like LVM-thin or ZFS? It would have better overall performance, without the FS overhead.

Yes I probably will, but it would require a bit of upheaval (getting everything setup in terraform is one step). But I might add another disk and do ZFS on that

from terraform-provider-proxmox.