aws_instance cloud-init mutipart encoding, etc. issues. about terraform-provider-template HOT 15 OPEN

This comment was originally opened by @jen20 as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

Hi @ohlol, thanks for opening the issue. It does appear there is a problem here. My feeling is that we should not automatically encode the user data as base 64 on aws_instance, and instead allow people to use the base64encode interpolation function if they need to. The issue here is that this will break backwards compatibility. Any thoughts @catsby?

from terraform-provider-template.

This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

There is also something else fishy in template_cloudinit_config.
Here's what happened: When updating an existing instance with regular user data to multipart user data cloud-init failed with the same error @ohlol mentions. But then changing around the gzip and base64_encode parameters and re-applying at some point it simply succeeded. He are the steps I took:

• existing instance with regular user data
• gzip = true, base64_encode=false cloud-init FAILS
• gzip = true, base64_encode=true cloud-init FAILS
• gzip = false, base64_encode=false terraform FAILS because of size limit
• gzip = true, base64_encode=false cloud-init SUCCEEDS

My code: https://gist.github.com/martonpe/659898d6e6d0fa4b7e7a

I have a feeling that terraform somehow doesn't always set the metadata for cloud-init correctly based on the current state of terraform.tfstate. Or maybe it has something to do with the resources having the same name?

I'll try to do some more testing and will post an update.

from terraform-provider-template.

This comment was originally opened by @boyand as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

@martonpe did you manage to get to the bottom of this. We are seeing very similar behaviour where gzipped userdata sporadically fails

from terraform-provider-template.

This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

@boyand not really. I ended up going with another solution to reduce the cloud-init config size.

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

I'm also experiencing some strange behaviour here. Under certain conditions, instances receive corrupt user data. I've been able to reproduce this under the following conditions:

1. Set gzip=true and base64_encode=false, and run terraform apply. The instance receives correct user-data and cloud-init succeeds.
2. Run terraform apply again. Terraform wrongly detects that the user-data needs to change, and destroys and reprovisions the instance. This time, the user-data is corrupt and cloud-init fails with a message like this:

2016-04-11 15:57:29,435 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'\\x1f\\\\ufffd\\x08\\x00\\x00\\tn\\\\ufffd\\x00\\\\ufff'...'

A comparison of the gzipped, rendered template in the statefile shows that between step 1 and 2, the encoding of the binary data within the JSON has changed. It seems as though some double-encoding has taken place somewhere, or perhaps the data has been wrongly decoded and re-encoded somewhere along the line, resulting in a corruption.

from terraform-provider-template.

This comment was originally opened by @boyand as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

This was exactly the behaviour I was seeing as well.

from terraform-provider-template.

This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

Yes, describes my experience very well too.

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

I believe I've isolated the source of this bug. As I suspected, it's due to bad decoding when unmarshalling the gzipped data from the JSON statefile.

On the first run, Terraform generates the gzipped data correctly and submits it to EC2, and also saves it to the statefile, using json.MarshalIndent. This escapes the binary data as unicode escape strings. On the subsequent runs, Terraform unmarshals the data in the statefile. The JSON library will only unmarshal a string, rather than a bytestream, and applies Unicode validation to it, this is documented here:

When unmarshaling quoted strings, invalid UTF-8 or invalid UTF-16 surrogate pairs are not treated as an error. Instead, they are replaced by the Unicode replacement character U+FFFD.

This process corrupts the bytestream, as we are dealing with a gzip file, not a Unicode string.

I've put together a gist demonstrating this: https://gist.github.com/dougneal/8fdc86516a76b69a51aa33fcadc4fdd1

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

Additionally, switching on base64_encode causes a different mode of failure. Although that neatly avoids the unmarshalling problem (and honestly I don't see a way round that; json isn't for storing raw binaries), the file lands on the EC2 instance in its base64 encoded state, meaning that it is double-base64 encoded on the way to EC2, as you can see here - the encoding is indiscriminate:

opts.UserData64 = aws.String(base64.StdEncoding.EncodeToString([]byte(d.Get("user_data").(string))))

IMHO a fix for this would be to:

• Disallow the combo of gzip=true and base64_encode=false as there is no way to safely record the state
• In resource_aws_instance, check whether the userdata is already base64 encoded, and pass it through as-is if so.

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

The changes in the linked PR are providing an adequate workaround for me at the moment.

from terraform-provider-template.

This comment was originally opened by @catsby as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

I just merged #6140 as a partial fix here, what issues remain here?

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

Thanks for the merge @catsby. The remaining issue is that the combination of base64_enable=false and gzip=true results in the binary gzip data being encoded into the JSON statefile in a way that it corrupts when read back out again.

from terraform-provider-template.

This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

Also not sure if this is exactly the same issue as @ohlol's bug report that I've kinda hijacked :)

from terraform-provider-template.

This comment was originally opened by @ohlol as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

@dougneal @catsby I'm no longer with my previous employer so unfortunately I have no way to verify that this fixes my problem :(

Feel free to close or address otherwise as you feel necessary!

from terraform-provider-template.

This comment was originally opened by @kwilczynski as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.

I have stumbled into the same issue.

from terraform-provider-template.