Comments (15)
This comment was originally opened by @jen20 as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
Hi @ohlol, thanks for opening the issue. It does appear there is a problem here. My feeling is that we should not automatically encode the user data as base 64 on aws_instance
, and instead allow people to use the base64encode
interpolation function if they need to. The issue here is that this will break backwards compatibility. Any thoughts @catsby?
from terraform-provider-template.
This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
There is also something else fishy in template_cloudinit_config
.
Here's what happened: When updating an existing instance with regular user data to multipart user data cloud-init failed with the same error @ohlol mentions. But then changing around the gzip
and base64_encode
parameters and re-applying at some point it simply succeeded. He are the steps I took:
- existing instance with regular user data
gzip = true
,base64_encode=false
cloud-init FAILSgzip = true
,base64_encode=true
cloud-init FAILSgzip = false
,base64_encode=false
terraform FAILS because of size limitgzip = true
,base64_encode=false
cloud-init SUCCEEDS
My code: https://gist.github.com/martonpe/659898d6e6d0fa4b7e7a
I have a feeling that terraform somehow doesn't always set the metadata for cloud-init correctly based on the current state of terraform.tfstate. Or maybe it has something to do with the resources having the same name?
I'll try to do some more testing and will post an update.
from terraform-provider-template.
This comment was originally opened by @boyand as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
@martonpe did you manage to get to the bottom of this. We are seeing very similar behaviour where gzipped userdata sporadically fails
from terraform-provider-template.
This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
@boyand not really. I ended up going with another solution to reduce the cloud-init config size.
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
I'm also experiencing some strange behaviour here. Under certain conditions, instances receive corrupt user data. I've been able to reproduce this under the following conditions:
- Set
gzip=true
andbase64_encode=false
, and runterraform apply
. The instance receives correct user-data and cloud-init succeeds. - Run
terraform apply
again. Terraform wrongly detects that the user-data needs to change, and destroys and reprovisions the instance. This time, the user-data is corrupt and cloud-init fails with a message like this:
2016-04-11 15:57:29,435 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'\\x1f\\\\ufffd\\x08\\x00\\x00\\tn\\\\ufffd\\x00\\\\ufff'...'
A comparison of the gzipped, rendered template in the statefile shows that between step 1 and 2, the encoding of the binary data within the JSON has changed. It seems as though some double-encoding has taken place somewhere, or perhaps the data has been wrongly decoded and re-encoded somewhere along the line, resulting in a corruption.
from terraform-provider-template.
This comment was originally opened by @boyand as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
This was exactly the behaviour I was seeing as well.
from terraform-provider-template.
This comment was originally opened by @martonpe as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
Yes, describes my experience very well too.
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
I believe I've isolated the source of this bug. As I suspected, it's due to bad decoding when unmarshalling the gzipped data from the JSON statefile.
On the first run, Terraform generates the gzipped data correctly and submits it to EC2, and also saves it to the statefile, using json.MarshalIndent
. This escapes the binary data as unicode escape strings. On the subsequent runs, Terraform unmarshals the data in the statefile. The JSON library will only unmarshal a string, rather than a bytestream, and applies Unicode validation to it, this is documented here:
When unmarshaling quoted strings, invalid UTF-8 or invalid UTF-16 surrogate pairs are not treated as an error. Instead, they are replaced by the Unicode replacement character U+FFFD.
This process corrupts the bytestream, as we are dealing with a gzip file, not a Unicode string.
I've put together a gist demonstrating this: https://gist.github.com/dougneal/8fdc86516a76b69a51aa33fcadc4fdd1
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
Additionally, switching on base64_encode
causes a different mode of failure. Although that neatly avoids the unmarshalling problem (and honestly I don't see a way round that; json isn't for storing raw binaries), the file lands on the EC2 instance in its base64 encoded state, meaning that it is double-base64 encoded on the way to EC2, as you can see here - the encoding is indiscriminate:
opts.UserData64 = aws.String(base64.StdEncoding.EncodeToString([]byte(d.Get("user_data").(string))))
IMHO a fix for this would be to:
- Disallow the combo of
gzip=true
andbase64_encode=false
as there is no way to safely record the state - In
resource_aws_instance
, check whether the userdata is already base64 encoded, and pass it through as-is if so.
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
The changes in the linked PR are providing an adequate workaround for me at the moment.
from terraform-provider-template.
This comment was originally opened by @catsby as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
I just merged #6140 as a partial fix here, what issues remain here?
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
Thanks for the merge @catsby. The remaining issue is that the combination of base64_enable=false
and gzip=true
results in the binary gzip data being encoded into the JSON statefile in a way that it corrupts when read back out again.
from terraform-provider-template.
This comment was originally opened by @dougneal as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
Also not sure if this is exactly the same issue as @ohlol's bug report that I've kinda hijacked :)
from terraform-provider-template.
This comment was originally opened by @ohlol as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
@dougneal @catsby I'm no longer with my previous employer so unfortunately I have no way to verify that this fixes my problem :(
Feel free to close or address otherwise as you feel necessary!
from terraform-provider-template.
This comment was originally opened by @kwilczynski as hashicorp/terraform#4794 (comment). It was migrated here as part of the provider split. The original comment is below.
I have stumbled into the same issue.
from terraform-provider-template.
Related Issues (20)
- Template 2.0.0 tries to interpulate percent (%) signs followed by curly-braces {} HOT 3
- Breaking change in substitution in v2.0.0 HOT 1
- Extremely unprofessional backward incompatible release HOT 1
- The new 2.0.0 doesn't parse %${var} core HOT 6
- Make Terraform validate template_file on plan phase
- Feature request: Render variables in filenames in template_dir
- Template provider - Better error message at template parsing HOT 2
- Terraform template creating the wrong type of numbers HOT 5
- template_dir doesn't work on Terraform v0.12.4 HOT 2
- template_dir doesn't skip hidden files HOT 1
- [regression] provider-template fails rendering when template file has % in the content starting from v2.0.0 HOT 3
- go module uses invalid version HOT 4
- Passing Lists as Template Variables HOT 2
- template_dir can't handle subdirectories correctly on windows: open <filename>: The system cannot find the path specified.
- depends_on should wait for template_dir generation HOT 3
- Provider cannot be migrated to terraform-plugin-sdk as it uses `lang` pkg directly
- Terragrunt panic due to api body size. HOT 2
- Unsupported functions in template file HOT 1
- Changes from nearly a year ago not published
- Provider archived
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from terraform-provider-template.