hashicorp / terraform-provider-consul Goto Github PK

This issue was originally opened by @vasilij-icabbi as hashicorp/terraform#20966. It was migrated here as a result of the provider split. The original body of the issue is below.

Hello,
I am using Consul provider with Terraform and I want it to be secured with Consul ACL token.

Question is: What are minimum required ACL rules for policy for Consul Provider to proper work?

Thank you

This issue was originally opened by @VAdamec as hashicorp/terraform#15609. It was migrated here as a result of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.9.11

Terraform Configuration Files

resource "consul_key_prefix" "bastion-host" {
  path_prefix                = "demo/env/"
  subkeys                    = {
    "BASTION"                = "${baremetal_core_instance.proxy.0.public_ip}"
  }
}

resource "consul_key_prefix" "proxy-host" {
  path_prefix                = "demo/env/"
  subkeys                    = {
    "PROXY"                  = "${format("http://${baremetal_core_instance.proxy.0.private_ip}:3128/")}"
  }
}

Debug Output

https://gist.github.com/VAdamec/e3e3c2fe4a597707ac03c60e4fcebebb

Expected Behavior

add values to Consul KV and keep them till any change occurse

Actual Behavior

add and remove value with every run (first run add them, next remove them, next add them, ...)

Steps to Reproduce

Just run apply, no changes in code or vars

1. terraform apply

Important Factoids

Using BMCS provider https://github.com/oracle/terraform-provider-baremetal

Preamble: Consul allows you to set an unsigned integer along with a KV that can be used however the provider/consumer wish. Often, this data contains a bitmask to let consumers know metadata about the key being read (is this value encrypted?, is it json encoded?, is the data binary?). Terraform current does not support management of the Flags attribute.

Terraform Version

0.11.7

Affected Resource(s)

• consul_keys
• consul_key_prefix

Terraform Configuration Files

proposal for interface changes...

consul_keys

because the consul_keys resource uses a sub-object, adding a flags attribute is a non-breaking change (with a caveat). See "expected behavior" below.

resource "consul_keys" "app" {
  datacenter = "nyc1"
  token      = "abcd"

  # Set the CNAME of our load balancer as a key
  key {
    path  = "service/app/elb_address"
    value = "${aws_elb.app.dns_name}"
    flags = 64
  }
}

consul_key_prefix

Consul key prefix uses a map to describe the subkeys, as such, a larger contract change would be necessary. There are a couple approaches to this, but the most backwards compatible way I can think of is...

resource "consul_key_prefix" "myapp_config" {
  datacenter = "nyc1"
  token      = "abcd"

  # Prefix to add to prepend to all of the subkey names below.
  path_prefix = "myapp/config/"

  subkeys = {
    "elb_cname"         = "${aws_elb.app.dns_name}"
    "s3_bucket_name"    = "${aws_s3_bucket.app.bucket}"
    "database/hostname" = "${aws_db_instance.app.address}"
    "database/port"     = "${aws_db_instance.app.port}"
    "database/username" = "${aws_db_instance.app.username}"
    "database/name"     = "${aws_db_instance.app.name}"
  }

  subkey {
    path="database/password"
    value="${aws_db_instance.app.password}"
    flags=64
  }
}

it could either be ok to provide both subkeys and [many] subkey elements, or only one or the other could be allowed on any resource.

Debug Output

N/A

Panic Output

N/A

Expected Behavior

Terraform should be able to manage the Flags attributes of a KV, and should ensure that the Flags of a managed KV are always correct. (See Actual Behavior)

Actual Behavior

Currently, Terraform ignores flags altogether, so it is possible to use a different tool to write values under a consul_key_prefix resource and Terraform will not "put it back". I'd argue this is a bug, but I imagine people are taking advantage of this (like I am) to overlay flags from another tool, while setting the "content" of the value from Terraform.

If terraform begins to manage flags as well, I also believe that terraform should assume that a key is tainted if the modify index does not match the modify index at the time terraform last set it (but this should probably fixed in a separate PR).

Steps to Reproduce

N/A

Important Factoids

N/A

References

https://www.consul.io/api/kv.html#flags
https://www.consul.io/api/kv.html#flags-1
https://www.consul.io/api/kv.html#modifyindex

This issue was originally opened by @blalor as hashicorp/terraform#12163. It was migrated here as part of the provider split. The original body of the issue is below.

From looking at the documentation, it's unclear how a user should choose between the above three resources. Furthermore, both consul_service and consul_agent_service appear to do effectively the same thing, although the docs for consul_service indicate that registering a service with the catalog is something intended for the future. I believe the design of consul_service is confusing: if you configure the provider to use a non-local agent (such as consul.service.consul:8500 or some other mechanism that could non-deterministically choose a different agent on every Terraform plan or apply), subsequent Terraform invocations can fail if they're unable to find a service that was registered with a different agent previously.

It would be great to have somewhere in terraform a resource that would generate a gossip key compatible to the format generated by consul keygen. This would help setting up clusters with consul using terraform.

This issue was originally opened by @juliangamble as hashicorp/terraform#15716. It was migrated here as a result of the provider split. The original body of the issue is below.

On the page here:
https://www.terraform.io/docs/providers/consul/index.html

It's not clear how the consul provider relates to the consul backend - or what the difference is. Could you please include a note about the difference?

The current data source for consul keys doesn't support using an entire folder, you are required to specify every key. For my use case, Im versioning config, but want to grab all the key/vales under a folder, for which the keys may have changed.

This issue was originally opened by @wv-cfromm as hashicorp/terraform#15356. It was migrated here as part of the provider split. The original body of the issue is below.

My project is a self-serve build/compile/deploy on immutable servers. In numerous places, I have a need for complex structures as variables, such as a structure to hold ip/port manifest for binding and security group creation. Maps/Lists (or nesting both) works fine as cli vars or tfvars, but Consul KV retrieved with data.consul_keys is all String. This is rough because I need Consul to carry my vars from other apps into TF.

Am I missing something? Is there some way to interpret KV as other than String?

If this needs to be a feature request, I believe there is a number of approaches. Interpreting Consul KV as other than string is one, such as storing HCL in Consul . It might be better to enhance the various interpolation functions to support accepting a single string input. For example, map function only works when supplied key/value, key/value, etc. However, the map function could accept a single string that includes a map structure. Same goes for List, etc. Or separate function that cast a string into map, list, etc assuming standard map/list/etc syntax. Or what about accepting HCL/JSON as input?

When a consul_intention resource is deleted manually, subsequent plans panic when reading this resource.

This is very similar to #69

As far as I can tell after a quick glance at other resources, they seem not to have this bug although a more in-depth check should be done.

Steps to Reproduce

➜  terraform-provider-consul git:(catalog-fix) cat example.tf
resource "consul_intention" "database" {
  source_name      = "api"
  destination_name = "db"
  action           = "allow"
}
➜  terraform-provider-consul git:(catalog-fix) terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + consul_intention.database
      id:               <computed>
      action:           "allow"
      destination_name: "db"
      source_name:      "api"


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

consul_intention.database: Creating...
  action:           "" => "allow"
  destination_name: "" => "db"
  source_name:      "" => "api"
consul_intention.database: Creation complete after 0s (ID: 2056a436-8578-9dc4-3a9d-eaa257bfd628)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
➜  terraform-provider-consul git:(catalog-fix) consul intention delete api db
Intention deleted.
➜  terraform-provider-consul git:(catalog-fix) terraform apply
consul_intention.database: Refreshing state... (ID: 2056a436-8578-9dc4-3a9d-eaa257bfd628)

Error: Error refreshing state: 1 error(s) occurred:

* consul_intention.database: 1 error(s) occurred:

* consul_intention.database: consul_intention.database: unexpected EOF


panic: runtime error: invalid memory address or nil pointer dereference
2018-12-27T16:02:34.913+0100 [DEBUG] plugin.terraform-provider-consul: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1822e02]
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: goroutine 8 [running]:
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: github.com/terraform-providers/terraform-provider-consul/consul.resourceConsulIntentionRead(0xc000201960, 0x1a04cc0, 0xc0001ec3c0, 0xc000201960, 0x0)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /Users/remi/go/src/github.com/terraform-providers/terraform-provider-consul/consul/resource_consul_intention.go:175 +0x242
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc0002677a0, 0xc0003bef50, 0x1a04cc0, 0xc0001ec3c0, 0xc0003ab510, 0x10bf501, 0x189b1e0)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /Users/remi/go/src/github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:352 +0x160
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc000267810, 0xc0003bef00, 0xc0003bef50, 0xc0003cc400, 0x18, 0x2539000)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /Users/remi/go/src/github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x92
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc0001a4140, 0xc00052ecb0, 0xc00052ef00, 0x0, 0x0)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /Users/remi/go/src/github.com/terraform-providers/terraform-provider-consul/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: reflect.Value.call(0xc0000a35c0, 0xc0003c00e0, 0x13, 0x1a312c0, 0x4, 0xc000079f18, 0x3, 0x3, 0xc0000c1700, 0x0, ...)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /usr/local/Cellar/go/1.11.4/libexec/src/reflect/value.go:447 +0x454
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: reflect.Value.Call(0xc0000a35c0, 0xc0003c00e0, 0x13, 0xc0002df718, 0x3, 0x3, 0x0, 0x0, 0x0)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /usr/local/Cellar/go/1.11.4/libexec/src/reflect/value.go:308 +0xa4
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: net/rpc.(*service).call(0xc0003e6a40, 0xc0000d6730, 0xc0000be700, 0xc0000be870, 0xc000198f00, 0xc000161e00, 0x189b1a0, 0xc00052ecb0, 0x16, 0x189b1e0, ...)
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /usr/local/Cellar/go/1.11.4/libexec/src/net/rpc/server.go:384 +0x14e
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul: created by net/rpc.(*Server).ServeCodec
2018-12-27T16:02:34.914+0100 [DEBUG] plugin.terraform-provider-consul:  /usr/local/Cellar/go/1.11.4/libexec/src/net/rpc/server.go:481 +0x47e
2018-12-27T16:02:34.916+0100 [DEBUG] plugin: plugin process exited: path=/Users/remi/.terraform.d/plugins/terraform-provider-consul
2018/12/27 16:02:34 [ERROR] root: eval: *terraform.EvalRefresh, err: consul_intention.database: unexpected EOF
2018/12/27 16:02:34 [ERROR] root: eval: *terraform.EvalSequence, err: consul_intention.database: unexpected EOF
2018/12/27 16:02:34 [TRACE] [walkRefresh] Exiting eval tree: consul_intention.database
2018/12/27 16:02:34 [TRACE] dag/walk: upstream errored, not walking "provider.consul (close)"
2018/12/27 16:02:34 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-12-27T16:02:34.916+0100 [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

As part of the preparation for Terraform v0.12, we would like to migrate all providers to use Go Modules. We plan to continue checking dependencies into vendor/ to remain compatible with existing tooling/CI for a period of time, however go modules will be used for management. Go Modules is the official solution for the go programming language, we understand some providers might not want this change yet, however we encourage providers to begin looking towards the switch as this is how we will be managing all Go projects in the future. Would maintainers please react with 👍 for support, or 👎 if you wish to have this provider omitted from the first wave of pull requests. If your provider is in support, we would ask that you avoid merging any pull requests that mutate the dependencies while the Go Modules PR is open (in fact a total codefreeze would be even more helpful), otherwise we will need to close that PR and re-run go mod init. Once merged, dependencies can be added or updated as follows:

$ GO111MODULE=on go get github.com/some/module@master
$ GO111MODULE=on go mod tidy
$ GO111MODULE=on go mod vendor

GO111MODULE=on might be unnecessary depending on your environment, this example will fetch a module @ master and record it in your project's go.mod and go.sum files. It's a good idea to tidy up afterward and then copy the dependencies into vendor/. To remove dependencies from your project, simply remove all usage from your codebase and run:

$ GO111MODULE=on go mody tidy
$ GO111MODULE=on go mod vendor

Thank you sincerely for all your time, contributions, and cooperation!

Terraform Version

Refer to #56

Affected Resource(s)

Refer to #56

Terraform Configuration Files

Refer to #56

Expected Behavior

Refer to #56

Actual Behavior

Refer to #56

Steps to Reproduce

Refer to #56

Important Factoids

Refer to #56

References

Refer to #56

This issue was originally opened by @juliangamble as hashicorp/terraform#15719. It was migrated here as a result of the provider split. The original body of the issue is below.

In the example here:
https://github.com/terraform-providers/terraform-provider-consul/blob/master/examples/kv/main.tf

There isn't an example of reading values from consul, only writing to it.

As this uses the terraform 6 syntax mostly - it's confusing for people trying to figure out how to use the consul key reading features of data.

Could you please expand this example to show reading values from consul as well?

Affected Resource(s)

• consul_service

I would like to request for the ability to define health checks for services registered with the consul_service resource.

As far as I understand, the resource is meant for use with external services and it would be great if we can also define health checks for the external service. Then, we can use something else like Consul ESM to do the actual health checking.

Terraform Version

Terraform v0.11.1

• provider.consul v1.0.0

Affected Resource

• data consul_catalog_service

Terraform Configuration Files

provider "consul" {
  address    = "pilot.example.net"
  datacenter = "dc1"
}

data "consul_catalog_service" "read-consul-primary-service" {
  tag = "primary",
  name = "service"
}

resource "null_resource" "primary-configuration" {
  connection {
    user = "admin"
    password = "admin"
    host = "${lookup(data.consul_catalog_service.read-consul-primary-service.service[0], "node_address")}"
  }

  provisioner "remote-exec" {
    inline = [
      "...",
    ]
  }
}

Debug Output

$ /c/terraform/terraform.exe plan

Error: data.consul_catalog_service.read-consul-primary-ns: "tag": this field cannot be set

Panic Output

N/A

Expected Behavior

I am not sure. This is either a documentation bug or an actual product bug. I tend to think that the documentation is right and that we should be able to use the tag argument in this data source.

Actual Behavior

consul_catalog_service data source argument reference mentions that one of the supported argument is "tag - (Optional) A single tag that can be used to filter the list of nodes to return based on a single matching tag.." This seems to indicate that the tag argument could be set.

In data_source_consul_catalog_service.go, this argument seems to be defined as Computed:

catalogServiceTag: &schema.Schema{
  // Used in the query, must be stored and force a refresh if the value
  // changes.
  Computed: true,
  Type:     schema.TypeString,
  ForceNew: true,
},

Because it is not Required or Optional, validation in schema.validate(string, *Schema, *terraform.ResourceConfig) fails:

if !schema.Required && !schema.Optional {
  // This is a computed-only field
  return nil, []error{fmt.Errorf(
    "%q: this field cannot be set", k)}
}

I did modify data_source_consul_catalog_service.go locally to make the tag argument Optional. It worked, but I am not sure enough about the consequences to send in a Pull Request.

catalogServiceTag: &schema.Schema{
  // Used in the query, must be stored and force a refresh if the value
  // changes.
  Optional: true,
  Type:     schema.TypeString,
},

Steps to Reproduce

1. Create a consul provider
2. Add a consul_catalog_service with a tag argument
3. terraform plan or terraform apply

Important Factoids

None.

References

None.

Terraform Version

Terraform v0.11.10
+ provider.consul v2.2.0
+ provider.random v2.0.0

Affected Resource(s)

• consul_keys

Terraform Configuration Files

provider "consul" {
  version = "~> 2.2.0"
}

provider "random" {
  version = "~> 2.0.0"
}

resource "random_id" "prometheus_infra_passwd" {
  byte_length = 16
}

resource "consul_keys" "all_environments" {
  key {
    path  = "sqsc/services/prometheus-infra/htpasswd"
    value = "${bcrypt(random_id.prometheus_infra_passwd.b64_url)}"
  }
}

output "prometheus_infra_passwd" {
  value     = "${random_id.prometheus_infra_passwd.b64_url}"
  sensitive = false // So it shows in logs
}

Output

$ terraform-0.11.10 init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "random" (2.0.0)...
- Downloading plugin for provider "consul" (2.2.0)...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ TF_LOG=TRACE TF_LOG_PATH=terraform.log terraform-0.11.10 apply -auto-approve
random_id.prometheus_infra_passwd: Refreshing state... (ID: Ns9F6Hb2nw22Uzs_PoJ6nQ)
consul_keys.all_environments: Refreshing state... (ID: consul)

Error: Error applying plan:

1 error(s) occurred:

* consul_keys.all_environments: consul_keys.all_environments: diffs didn't match during apply. This is a bug with Terraform and should be reported as a GitHub Issue.

Please include the following information in your report:

    Terraform Version: 0.11.10
    Resource ID: consul_keys.all_environments
    Mismatch reason: attribute mismatch: key.1982130857.default
    Diff One (usually from plan): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"key.505010295.delete":*terraform.ResourceAttrDiff{Old:"false", New:"false", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1982130857.value":*terraform.ResourceAttrDiff{Old:"", New:"$2a$10$e5kSPh0ibbwm7Hu4Kk8RpeQNYjtQctDrPKRr.te5tCNlVrq/PlbAS", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1982130857.name":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1982130857.default":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.default":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1982130857.delete":*terraform.ResourceAttrDiff{Old:"", New:"false", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1982130857.path":*terraform.ResourceAttrDiff{Old:"", New:"sqsc/services/prometheus-infra/htpasswd", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.path":*terraform.ResourceAttrDiff{Old:"sqsc/services/prometheus-infra/htpasswd", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.name":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}
    Diff Two (usually from apply): *terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{"key.1516627792.name":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1516627792.path":*terraform.ResourceAttrDiff{Old:"", New:"sqsc/services/prometheus-infra/htpasswd", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1516627792.delete":*terraform.ResourceAttrDiff{Old:"", New:"false", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.delete":*terraform.ResourceAttrDiff{Old:"false", New:"false", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.name":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.path":*terraform.ResourceAttrDiff{Old:"sqsc/services/prometheus-infra/htpasswd", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.505010295.default":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:true, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1516627792.default":*terraform.ResourceAttrDiff{Old:"", New:"", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, "key.1516627792.value":*terraform.ResourceAttrDiff{Old:"", New:"$2a$10$jaiPt998TLR6.oTeSoW04u/4JuMbK6VM2AcZV12jShc8w7htefTqa", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, Meta:map[string]interface {}(nil)}

Also include as much context as you can about your config, state, and the steps you performed to trigger this error.


Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Debug Output

https://gist.github.com/mildred/4e4f078147963e527405ac3c68809f3c

Steps to Reproduce

1. Write the terraform configuraton above to a example.tf file
2. Run consul agent -dev somewhere
3. Run terraform init followed by terraform apply
4. Notice the bug. If not, try to apply again, with terraform 0.11.8 the bug appeared on the second apply.

Important Factoids

Using bcrypt to generate a value for a consul key.

This issue was originally opened by @juliangamble as hashicorp/terraform#15717. It was migrated here as a result of the provider split. The original body of the issue is below.

Could you please expand the example here:
https://www.terraform.io/docs/providers/consul/r/keys.html

To show a consul provider in addition to the consul resource?

this is a feature request to support the ca_path configuration option from consul

currently the only supported provider feature is ca_file, which requires a path to a single file

because a valid configuration for consul is to have a directory containing ca files used, I believe that the consul terraform provider should support it, as well

Terraform Version

0.11.5

Affected Resource(s)

Please list the resources as a list, for example:
"consul_key_prefix"

Terraform Configuration Files

This is part of the file... but as you see, i'm using a variable for the acl.

resource "consul_key_prefix" "myapp_config" {
  token      = "${var.acltoken}"

  # Prefix to add to prepend to all of the subkey names below.
  path_prefix = "${var.kvprefix}"
#Using remote data to get information
  subkeys = "${merge(local.common_poop, var.staticmap)}"
}

Debug Output

This is from terraform

2018/04/18 17:54:30 [DEBUG] [aws-sdk-go]
2018/04/18 17:54:31 [INFO] Resource 2018-04-19 00:54:31.347287328 +0000 UTC has dynamic attributes
2018/04/18 17:54:31 [DEBUG] Attaching resource state to "consul_key_prefix.myapp_config": &terraform.ResourceState{Type:"consul_key_prefix", Dependencies:[]string{"local.common_poop"}, Primary:(*terraform.InstanceState)(0xc420733c70), Deposed:[]*terraform.InstanceState{}, Provider:"provider.consul", mu:sync.Mutex{state:0, sema:0x0}}
2018/04/18 17:54:31 [DEBUG] ReferenceTransformer: "consul_key_prefix.myapp_config" references: []
consul_key_prefix.myapp_config: Refreshing state... (ID: devops/poopenvironment/)
2018-04-18T17:54:31.362-0700 [DEBUG] plugin.terraform-provider-consul_v1.0.0_x4: 2018/04/18 17:54:31 [DEBUG] Listing keys under 'devops/poopenvironment/' in us-east-1-staging
2018/04/18 17:54:31 [ERROR] root: eval: *terraform.EvalRefresh, err: consul_key_prefix.myapp_config: Failed to list Consul keys under prefix 'devops/poopenvironment/': Unexpected response code: 403
2018/04/18 17:54:31 [ERROR] root: eval: *terraform.EvalSequence, err: consul_key_prefix.myapp_config: Failed to list Consul keys under prefix 'devops/poopenvironment/': Unexpected response code: 403
2018/04/18 17:54:31 [TRACE] [walkRefresh] Exiting eval tree: consul_key_prefix.myapp_config

This is from consul

2018/04/19 00:54:31 [ERR] http: Request GET /v1/kv/devops/poopenvironment/?dc=us-east-1-staging&recurse=, error: rpc error making call: ACL not found from=10.7.0.106:57145

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

After I change the variable back to the correct ACL, it should allow the K/V to be setup again.

Actual Behavior

It keeps 403-ing, complaining of a bad ACL....

Steps to Reproduce

1. `terraform apply with a the right ACL is successful in putting in the K/V'
2. A change of the ACL variable to a bad ACL, one without the rights. A terraform plan is OK, but a terraform apply complains to a 403... ONLY IF THERE IS A KEY CHANGE, as it should. If the change is only the ACL itself, the terraform apply is successful.
3. Change acltoken back to the correct ACL.
4. Subsequent plans an applys, even with the right ACL, keeps complaining of the 403.
5. At this point, I can't plan, apply, refresh, or destroy. I had go and delete the state file and the k/v and start over.
6. A subsequent apply works just fine since it had no reference of what's in the past.

Important Factoids

Looking further into this, it would appear that, when I do a plan and an apply, either way, the bad ACL gets put in the state file. However, when I put the good ACL back to variable and do a plan, it apparently still tries to use bad ACL to access the K/V, even though I have a good ACL in my variable now, and thus it errors out before it even tries the corrected ACL.

I was able to verify this by versioning the S3 bucket where the state file is and rolling it back one version, back to where the good ACL is in the state file. Then a TF plan and apply have no problems.

It would make sense to have some sort of test to see whether the ACL works before putting it in the state file, so that it doesn't get in this "stuck" state?

Terraform Version

$ terraform -v
Terraform v0.11.3

• provider.aws v1.9.0
• provider.template v1.0.0

Affected Resource(s)

hashicorp/consul/aws, at least I've not tried any more, it may affect other modules.

Expected Behavior

Terraform should plan

Actual Behavior

Terraform returns an error:

	status code: 403, request id: 9faded3b-1684-11e8-bc4a-b18ad23d67a1

Steps to Reproduce

Given a simple terraform config:

$ ls -la
total 8
drwxr-xr-x   3 will  staff   96 20 Feb 21:09 .
drwxr-xr-x  29 will  staff  928 20 Feb 21:09 ..
-rw-r--r--   1 will  staff  534 20 Feb 21:09 example.tf

$ cat example.tf 
provider "aws" {
  access_key = "<snip>"
  secret_key = "<snip, but it contained only alphanumerics>"
  region     = "us-east-2"
}

resource "aws_instance" "example" {
  ami           = "ami-167f5773"
  instance_type = "t2.micro"
}

$ terraform init

Initializing provider plugins...
<snip>
commands will detect it and remind you to do so if necessary.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_instance.example
      id:                           <computed>
      ami:                          "ami-167f5773"
      associate_public_ip_address:  <computed>
      availability_zone:            <computed>
      ebs_block_device.#:           <computed>
      ephemeral_block_device.#:     <computed>
      instance_state:               <computed>
      instance_type:                "t2.micro"
      ipv6_address_count:           <computed>
      ipv6_addresses.#:             <computed>
      key_name:                     <computed>
      network_interface.#:          <computed>
      network_interface_id:         <computed>
      placement_group:              <computed>
      primary_network_interface_id: <computed>
      private_dns:                  <computed>
      private_ip:                   <computed>
      public_dns:                   <computed>
      public_ip:                    <computed>
      root_block_device.#:          <computed>
      security_groups.#:            <computed>
      source_dest_check:            "true"
      subnet_id:                    <computed>
      tenancy:                      <computed>
      volume_tags.%:                <computed>
      vpc_security_group_ids.#:     <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Adding in hashicorp/consul/aws causes an error when trying to plan:

$ cat example.tf 
provider "aws" {
  access_key = "<snip, as before>"
  secret_key = "<snip, as before>"
  region     = "us-east-2"
}

resource "aws_instance" "example" {
  ami           = "ami-167f5773"
  instance_type = "t2.micro"
}

# The only new section
module "consul" {
  source = "hashicorp/consul/aws"
  aws_region  = "us-east-2"
  num_servers = "3"
}

$ terraform init
Initializing modules...
- module.consul
  Found version 0.1.1 of hashicorp/consul/aws on registry.terraform.io
  Getting source "hashicorp/consul/aws"
- module.consul.consul_servers
  Getting source "./modules/consul-cluster"
- module.consul.consul_clients
  Getting source "./modules/consul-cluster"
- module.consul.consul_servers.security_group_rules
  Getting source "../consul-security-group-rules"
- module.consul.consul_servers.iam_policies
  Getting source "../consul-iam-policies"
- module.consul.consul_clients.security_group_rules
  Getting source "../consul-security-group-rules"
- module.consul.consul_clients.iam_policies
  Getting source "../consul-iam-policies"

<snip>
commands will detect it and remind you to do so if necessary.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.template_file.user_data_server: Refreshing state...
data.template_file.user_data_client: Refreshing state...

Error: Error refreshing state: 1 error(s) occurred:

* module.consul.provider.aws: InvalidClientTokenId: The security token included in the request is invalid.
	status code: 403, request id: e5679375-1682-11e8-9062-1f8341bee52e

If I subsequently remove hashicorp/consul/aws and add in terraform-aws-modules/vpc/aws the plan works:

$ cat example.tf 
provider "aws" {
  access_key = "<snip, as before>"
  secret_key = "<snip, as before>"
  region     = "us-east-2"
}

resource "aws_instance" "example" {
  ami           = "ami-167f5773"
  instance_type = "t2.micro"
}

# Swap hashicorp/consul/aws for terraform-aws-modules/vpc/aws
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "my-vpc"
  cidr = "10.0.0.0/16"

  azs             = ["eu-west-1a", "eu-west-1b", "eu-west-1c"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24", "10.0.103.0/24"]

  enable_nat_gateway = true
  enable_vpn_gateway = true

  tags = {
    Terraform = "true"
    Environment = "dev"
  }
}

$ terraform init
Initializing modules...
- module.vpc

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 1.9"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_instance.example
      id:                               <computed>
      ami:                              "ami-167f5773"
      associate_public_ip_address:      <computed>
      availability_zone:                <computed>
      ebs_block_device.#:               <computed>
      ephemeral_block_device.#:         <computed>
      instance_state:                   <computed>

There is a closed issue regarding this ( #35 ), but I have a question about the choice of the implementation.

Why is a "delete" argument necessary? If I'm running a destroy, I expect all of the resources to be destroyed. I don't understand why you explicitly have to flag them as "delete."

The plan and destroy output even indicate that the resource is going to be and was destroyed respectively.

This issue was originally opened by @juliangamble as hashicorp/terraform#15718. It was migrated here as a result of the provider split. The original body of the issue is below.

In the example here:
https://www.terraform.io/docs/providers/consul/d/keys.html

The provider is not shown in the example - and this is confusing in how it relates (or more specifically doesn't relate) to the consul backend. (In terraform 6 - the terraform backend did both - and so the upgrade is confusing).

Could you please expand the example to show the consul provider?

In order to be transparent with the roadmap and broadcast some substantial changes coming to the provider, I'm opening this issue in the hopes that interested users will see it. Any changes will be versioned and included in changelogs, with old versions continuing to work (as long as the upstream Consul API continues to support them). You can learn more about pinning a provider version here.

In time I will create milestones within GitHub with some stand-alone issues for each of the below changes. Timeline across this will likely be flexible -- any help is welcome, just feel free to make an comment here.

These changes are primarily to simplify the number of resources, ensure we are using the correct APIs/design provided by Consul for something like Terraform, and remove resources that can no longer be supported by the current version of the Consul API.

Existing Resources

New Resources

These are resources that we intend to add in the future.

* This assumes that the Consul API supports some notion of a "token accessor", otherwise I don't think it would be wise to support it given the security implications.

This issue was originally opened by @cbarbour as hashicorp/terraform#8760. It was migrated here as part of the provider split. The original body of the issue is below.

Description

It would be nice if the consul provider could accept certificates as data rather than as a file reference. This would simplify the process of generating self-signed certificates using the tls provider.

Currently. the only solution to both generate and consume the certificates from terraform is to write the certificates to disk using a local-exec provisioner, and then reference them from the provider.

Having a local-file provider would also simplify this process.

This approach also introduces a number of dependency ordering problems; the provider now depends on the TLS resources, but doesn't directly reference them, preventing Terraform from establishing dependency relationships between the provider and resources.

Code showing my current solution is provided below.

Terraform Version

0.7.3

Affected Provider

• consul

Terraform Configuration Files

resource "tls_private_key" "consul_ca" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "consul_ca" {
  is_ca_certificate = true
  key_algorithm = "RSA"
  private_key_pem = "${tls_private_key.consul_ca.private_key_pem}"
  validity_period_hours = 8760

  allowed_uses = [
    "cert_signing",
    "client_auth",
    "server_auth",
  ]

  subject {
    common_name = "${var.region}.compute.internal"
    organization = "${coalesce(var.cluster_name, var.subnet_id)}"
  }

  provisioner "local-exec" {
    command = "python -c 'print \"\"\"${self.cert_pem}\"\"\"' > ${path.root}/consul_ca_cert.pem"
  }

}

resource "tls_private_key" "consul_node" {
  algorithm = "RSA"

  provisioner "local-exec" {
    command = "python -c 'print \"\"\"${self.private_key_pem}\"\"\"' > ${path.root}/consul_private_key.pem"
  }
}

data "tls_cert_request" "consul_node" {
  key_algorithm = "RSA"
  private_key_pem = "${tls_private_key.consul_node.private_key_pem}"

  subject {
    common_name = "${var.region}.compute.internal"
    organization = "${coalesce(var.cluster_name, var.subnet_id)}"
  }
}

resource "tls_locally_signed_cert" "consul_node" {
  cert_request_pem = "${data.tls_cert_request.consul_node.cert_request_pem}"

  ca_cert_pem = "${tls_self_signed_cert.consul_ca.cert_pem}"
  ca_key_algorithm = "RSA"
  ca_private_key_pem = "${tls_private_key.consul_ca.private_key_pem}"
  validity_period_hours = 8760

  allowed_uses = [
    "client_auth",
    "server_auth",
  ]

  provisioner "local-exec" {
    command = "python -c 'print \"\"\"${self.cert_pem}\"\"\"' > ${path.root}/consul_cert.pem"
  }
}

data "template_file" "consul_tls" {
  template = "consul_ca_cert.pem"
  depends_on = [
    "tls_self_signed_cert.consul_ca",
    "tls_private_key.consul_node",
    "tls_locally_signed_cert.consul_node",
  ]
  vars {
    ca_file = "${path.root}/consul_ca_cert.pem"
    cert_file = "${path.root}/consul_cert.pem"
    key_file = "${path.root}/consul_private_key.pem"
  }
}

provider "consul" {
  address = "${aws_instance.master.public_ip}:8080"
  datacenter = "${coalesce(var.cluster_name, var.subnet_id)}"
  scheme = "https"

  # Use of templates is a nasty hack to force dependency ordering
  ca_file = "${data.template_file.consul_tls.vars.ca_file}"
  cert_file = "${data.template_file.consul_tls.vars.cert_file}"
  key_file = "${data.template_file.consul_tls.vars.key_file}"
}

References

• GH3379

This issue was originally opened by @Alars-ALIT as hashicorp/terraform#9070. It was migrated here as part of the provider split. The original body of the issue is below.

Terraform Version

0.7.4

Affected Resource(s)

• consul_service
• consul_catalog_entry

Expected Behavior

When the address or port fields of a service is updated and applied, the update should be reflected in consul.

Actual Behavior

When the address or port fields of a service is updated and applied, the service is removed in consul

Steps to Reproduce

1. Apply a consul_service/consul_catalog_entry resource
2. Update address or port
3. Apply the update

Terraform Version

Terraform v0.11.10

• provider.consul v2.2.0

Affected Resource(s)

consul_key_prefix

Enhancement

I use terraform to manage configuration stored in consul using the consul_key_prefix resource. Terraform configuration is stored in a git repo and promotion of config changes are managed using pull requests. The git repo is meant to the the source of truth.

With the current setup of the consul provider, if the terraform state is ever lost or corrupted, it requires a complete key purge of the consul_key_prefix resources in order for terraform to control this prefix again. If keys exist, the following error is given:

consul_key_prefix.example: X keys already exist under config/example/; delete them before managing this prefix with Terraform

I understand this is done to prevent accidentally destroying existing consul KV's. I also understand TF state is important and shouldn't be lost. However with a consul_key_prefix resource it is very disruptive to have to purge all existing keys in a live environment because the TF state was lost / corrupt. It would be very useful to have the ability to force the resource creation even with existing keys. Alternatively a way to "bootstrap" tfstate with an existing consul_keys_prefix would solve this as well.

This issue was originally opened by @failshell as hashicorp/terraform#2792. It was migrated here as part of the provider split. The original body of the issue is below.

As the current Consul implementation is done, if using tokens, it's required to define them in each consul_keys resources. It makes a lot of sense as it provides flexibility. But it's currently not possible to define a default token to be used in the provider definition though.

In my current use case, we only have one token. It would be great if we could define it in the provider and don't have to define it in every consul_keys we have.

Currently, data.consul_service has almost no use for me, as I discovered it returns from the catalog and does not exercise the health check results.

Ideally, I'd like to use data.consul_service to return the healthy addresses for a service. I'm not sure what use cases people have where they want to get unhealthy results, but I don't have one.

I need to configure terraform's vault provider with the address of a healthy vault instance. With the current data.consul_service behavior, I end up getting back node addresses of unhealthy or left nodes.

Hello, details follows. Summary is that when using consul_keys resource, the token which is provided
via -var to terraform plan, it shows the token in plain text.

Terraform Version

$ terraform -v
Terraform v0.11.7
+ provider.consul v1.0.0

Affected Resource(s)

• consul_keys

Terraform Configuration Files

variable "app1_version_token" {}           
                                           
resource "consul_keys" "app1_version" {    
  datacenter = "dc1"                       
  token = "${var.app1_version_token}"      
   key {                                   
    path  = "app1/version"                 
    value = "0.1"                          
  }                                        
}                                          

Debug Output

Not relevant

Panic Output

NA

Expected Behavior

Consul token marked "computed" or not shown at all.

Actual Behavior

consul_keys.app1_version: Creating...
  datacenter:             "" => "dc1"
  key.#:                  "" => "1"
  key.1209483366.default: "" => ""
  key.1209483366.delete:  "" => "false"
  key.1209483366.name:    "" => ""
  key.1209483366.path:    "" => "app1/version"
  key.1209483366.value:   "" => "0.1"
  token:                  "" => "cf0ed539-cb44-3a8b-17d0-f4dc03f47438"
  var.%:                  "" => "<computed>"

Steps to Reproduce

Run the following script:

#!/bin/bash
terraform_token="mysecret"
terraform apply -target=consul_keys.app1_version -var "app1_version_token=$terraform_token"

Important Factoids

References

Hi there,
I have some keys in Consul with dots in their names. When I try to fetch them - I've got very strange results.

Example key are someserver.servername.pattern=<#hash>.domain.ltd and someserver.dispatcher.queue.size=65535

I am fetching data this way:
output "consul-variables" { value = "${data.consul_key_prefix.web.subkeys}" }

Terraform Version

terraform -v
Terraform v0.11.8

• provider.consul v2.1.0

Expected Behavior

Outputs:

consul-variables = [
,someserver.servername.pattern=<#hash>.domain.ltd
,someserver.dispatcher.queue.size=65535
]

Actual Behavior

Outputs:

consul-variables = {
someserver= map[servername:map[pattern:<#hash>.domain.ltd] dispatcher:map[queue:map[size:65535]]]
}

Steps to Reproduce

terraform apply

In addtion to #95 - it would be good if we can update ACL Tokens and Policies that already exisit.

As an example - the Anonymous token - providing it with a new updated ACL Policy that allows it read on all nodes:

node_prefix "" {
   policy = "example
}

Currently can create a new token and policy but can not update an existing token's policy etc.

The Consul UI now respects a special metadata field on services that will show a Terraform icon next to service names if the correct metadata exists on the service.

We should add this to all service registrations performed by the Terraform provider (the consul_service resource).

There is more detail here: hashicorp/consul#4640
And an example of it in the k8s catalog sync tool: https://github.com/hashicorp/consul-k8s/blob/ee548ca16e5e3cf94ca8613e4eeba0159ecdc427/catalog/from-k8s/resource.go#L235-L238

Hello,

We have an issue where consul keys had been set via terraform (using the config pasted below) and making it to state file just fine but when they had been deleted in the console, terraform wasn't detecting that those keys were missing and needed to get recreated on subsequent plans. It's almost as if it's running with -refresh=false and the consul_keys resource isn't reaching out to consul to perform a refresh at all.

To be clear, we are not passing in the -refresh=false flag here when running terraform plan. It just seems like it is being passed with this resource.

Terraform Version

0.11.2

Affected Resource(s)

Please list the resources as a list, for example:

• consul_keys

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

resource "consul_keys" "dns" {
  key {
    path   = "global/env_vars/EXAMPLE_API_URI"
    value  = "http://example.com"
    delete = true
  }

  key {
    path   = "global/env_vars/EXAMPLE_API_PRIV_URI"
    value  = "http://example.priv"
    delete = true
  }
}

Expected Behavior

Missing consul keys that had been previously set by terraform, and are in state file, should be detected and recreated on subsequent terraform plan/apply.

Actual Behavior

Missing consul keys aren't being detected and a zero diff is output for a terraform plan.

Steps to Reproduce

1. terraform apply the config above and get consul keys created via terraform.
2. delete those keys from console manually (outside of terraform's control)
3. run terraform plan and see no detected changes (where it should be trying to add those keys again.

Hello,

Are there any plans to release Consul provider which will be working with Terraform 0.12, and if so is there any timeline?

Currently when I try to use newest Terraform 0.12 with Consul, there is an error:

./terraform plan

Error: Failed to instantiate provider "consul" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

Terraform Version

./terraform version
Terraform v0.12.0-alpha4 (2c36829d3265661d8edbd5014de8090ea7e2a076)
+ provider.aws v1.40.0-6-gb23683732-dev
+ provider.consul v2.2.0

consul --version
Consul v1.4.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Affected Resource(s)

Please list the resources as a list, for example:

• multiple resources

Terraform Configuration Files

data "consul_key_prefix" "common" {
  datacenter = "dc1"
  # Prefix to add to prepend to all of the subkey names below.
  path_prefix = "common/service1"
}

module "alb" {
...
source = "./modules/alb"
CommonAlbRules   = "${data.consul_key_prefix.common.subkeys}"
...

resource "aws_lb_listener_rule" "CommonAlb" {
  count        = "${length("${var.CommonAlbRules}")}"
  ...
}

Expected Behavior

Terraform plan/apply should succeed.

Actual Behavior

Error: Failed to instantiate provider "consul" to obtain schema: Incompatible API version with plugin. Plugin version: 4, Client versions: [5]

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Download Terraform 0.12 and Consul
2. Populate Consul with simple KV
   consul kv put common/service1 service1,8080,http,8080
3. Configure Consul as a simple Data source
4. terraform apply

References

Possibly this one:
hashicorp/terraform#19221

Terraform Version

0.9.8

Affected Resource(s)

• consul_key data source

Terraform Configuration Files

data "consul_keys" "vpc" {
  key {
    name = "vpc_id"
    path = "tf/outputs/vpc_id"
  }
}

Actual Behavior

If tf/outputs/vpc_id doesn't exist in Consul, the default behavior is to have an empty string within the data source at the expected data.consul_keys.vpc.var.vpc_id location. This type of behavior may be desired in some circumstances, but in others it'd be desired to have the key in Consul be a hard dependency (which may be populated through some mechanism other than Terraform).

It would be nice if there were a required option or something else that will cause the Terraform run to fail if the key is not found within Consul. It would also expose Consul or other upstream misconfigurations rather than silently returning empty strings.

When a Consul service is removed out-of-band from the Terraform Consul provider, for instance manually, the provider can no longer refresh its state.

Terraform-Provider-Consul Version

The provider name in .terraform/plugins/darwin_amd64/ is terraform-provider-consul_v0.1.0_x4

Affected Resource

• consul_service (I have not verified other resources from the Consul provider)

Expected Behavior

I should be able to run plan or apply

Actual Behavior

Running plan or apply failed with Failed to get service 'reqs_redis' from Consul agent

Steps to Reproduce

1. Create infrastructure using terraform apply which includes service registration with Consul.
2. Manually delete the service or re-create the single -dev instance of your Consul cluster.
3. Run terraform plan or terraform apply
4. See

Error refreshing state: 1 error(s) occurred:

* consul_service.redis: 1 error(s) occurred:

* consul_service.redis: consul_service.redis: Failed to get service 'reqs_redis' from Consul agent

This issue was originally opened by @wv-cfromm as hashicorp/terraform#13481. It was migrated here as part of the provider split. The original body of the issue is below.

Seems quite bizarre, but I get the same response out of both consul_service and consul_agent_service fairly often, but random.

�[0m�[1mmodule.events_qa.asg.asg_lb.consul_service.app: Creating...�[0m
  address:    "" => "demo-test1.wvhtest.com"
  name:       "" => "demo"
  port:       "" => "443"
  service_id: "" => "<computed>"�[0m
�[31mError applying plan:

1 error(s) occurred:

* module.events_qa.module.asg.module.asg_lb.consul_service.app: 1 error(s) occurred:

* consul_service.app: Failed to register service 'demo' with Consul agent: Put http://127.0.0.1:8500/v1/agent/service/register?dc=test1: dial tcp 127.0.0.1:8500: getsockopt: connection refused

Config is nothing special, except that I'm feeding consul address from a seperate consul datasource. But where would it get 127.0.0.1 from? That's definitely not in my datasource. When it doesn't do this, it appears to work, but it does this so often I cant tell.

provider "consul" {
    address = "elb-docker-consul-util.wvendpoints.com:8500"
        datacenter = "USE1"
}
provider "consul" {
    alias = "localvpc"
    address = "${data.consul_keys.vpc.var.dns_lb_ip}:8500"
    datacenter = "${var.environment}"
}
resource "consul_service" "app" {
  provider = "consul.localvpc"
  address = "${aws_route53_record.alb.fqdn}"
  name    = "${var.project}"
  port    = "${element(split("|", element(split(":", data.consul_keys.alb.var.app_lb_map), 1)),0)}"
}

This issue was originally opened by @blalor as hashicorp/terraform#13944. It was migrated here as part of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.9.3

Affected Resource(s)

• consul_catalog_entry

Terraform Configuration Files

provider "consul" {}

resource "consul_catalog_entry" "redis" {
    node = "redis"
    address = "127.0.0.2"
    
    service = {
        name = "my-redis"
        port = "6379"
        tags = ["master"]
    }
}

Debug Output

Log for 2nd terraform apply: apply.log.txt

Console output:

$ TF_LOG=TRACE TF_LOG_PATH=apply.log.txt terraform apply
consul_catalog_entry.redis: Refreshing state... (ID: redis-127.0.0.2-[my-redis])

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Expected Behavior

Terraform should have noticed that the catalog entry was missing and recreated.

Actual Behavior

Nuttin', honey.

Steps to Reproduce

0. consul agent -dev in terminal 0
1. terraform apply in terminal 1
2. confirm service registered: curl -sfS localhost:8500/v1/catalog/service/my-redis
3. stop consul agent in terminal 0
4. consul agent -dev in terminal 0 (restarting it)
5. confirm service missing: curl -sfS localhost:8500/v1/catalog/service/my-redis
6. terraform apply in terminal 1
7. confirm service still missing: curl -sfS localhost:8500/v1/catalog/service/my-redis

This issue was originally opened by @hexedpackets as hashicorp/terraform#9310. It was migrated here as part of the provider split. The original body of the issue is below.

Terraform Version

0.7.5

Affected Resource(s)

• consul provider
• consul_keys data source

Terraform Configuration Files

provider "consul" {
  address = "consul.service.consul:8500"
  datacenter = "${var.datacenter}"
}

data "consul_keys" "foobar" {
  key {
    name = "foobar"
    path = "services/foo/bar"
  }
}

outut "foobar" {
  value = "${data.consul_keys.foobar.var.foobar}"
}

Expected Behavior

The datacenter specified in the provider is used to retrieve the key.

Actual Behavior

The agent's default datacenter is used instead of the one specified in the provider, resulting in an empty or incorrect value.

Adding the datacenter argument to the consul_keys stanza as a workaround does cause the correct datacenter to be used.

Terraform Version

0.9.11.

Affected Resource(s)

• consul_key_prefix

Terraform Configuration Files

resource "consul_key_prefix" "micro_service" {

  path_prefix = "config/micro-service/"

  subkeys = {
    "database/url"      = "${var.database_host}:3306/${mysql_database.app.name}"
    "database/password" = "${var.root_database_password}"
    // ...
  }
}

Expected Behavior

When deleting a consul cluster and starting a new one from scratch, I expect Terraform to be able to re-provision all our microservice's (Spring Boot & Cloud) configurations based on the consul_key_prefix resource.
This way I could keep all my configuration of the microservices versioned in Git, along with the other deployment aspects (S3 configuration, Route53, Load balancing, ECS, ...).

Actual Behavior

Error refreshing state: 1 error(s) occurred:
* consul_key_prefix.micro_service: consul_key_prefix.micro_service: Failed to list Consul keys under prefix 'config/micro-service/': Unexpected response code: 500

It seems that terraform compares the state against the information (not) found in Consul, and in the case that it did not find the expected data there it does not try to re-insert them into the cluster, but rather errors.

Steps to Reproduce

1. Standard terraform apply to a fresh (empty) Consul cluster (we use 3 nodes in HA config).
2. Kill and purge consul cluster so that data is lost.
3. Re-run the previous terrraform apply, expecting that terraform restores the state in consul.
4. See terraform error.

I think this resource should rather do an "upsert" (Insert or Update) like opration instead of failing when it does not find anything.

This issue was originally opened by @gozer as hashicorp/terraform#8841. It was migrated here as part of the provider split. The original body of the issue is below.

Right now, working around it by setting :

CONSUL_HTTP_SSL_VERIFY=1

in my environment, but would be so much cleaner if I could use the provider to do it explicitely:

provider "consul" {
  address    = "some.host.name:443"
  scheme     = "https"
  ssl_verify = "false"
}

This issue was originally opened by @amanjeev as hashicorp/terraform#14451. It was migrated here as part of the provider split. The original body of the issue is below.

The documentation at https://www.terraform.io/docs/providers/consul/r/key_prefix.html says that datacenter is option and that it by default uses from the provider setup. But that was not true in our case.

Terraform Version

v0.9.3

Affected Resource(s)

Please list the resources as a list, for example:

• consul_key_prefix

Terraform Configuration Files

Provider:

variable "consul_dc" {}
variable "consul_acl_token" {}

provider "consul" {
  datacenter = "${var.consul_dc}"
  token = "${var.consul_acl_token}"
}
module "dtabs"  { source = "modules/dtabs"  dc = "${var.consul_dc}" }

Resource (without datacenter) original:

resource "consul_key_prefix" "keys" {
  path_prefix = "${var.prefix}${var.path}"
  subkeys = "${data.external.dtabs_keys.result}"
}

Resource (with datacenter) modified/fixed:

resource "consul_key_prefix" "keys" {
  datacenter = "${var.dc}" 
  path_prefix = "${var.prefix}${var.path}"
  subkeys = "${data.external.dtabs_keys.result}"
}

Expected Behavior

Since datacenter is already passed in via provider "consul", it should have chosen the correct datacenter.

Actual Behavior

It chose the wrong datacenter.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Use "Resource (without datacenter) original" resource above
2. terraform apply

This issue was originally opened by @samber as hashicorp/terraform#10292. It was migrated here as part of the provider split. The original body of the issue is below.

Hi guys,

It would be very helpful to be able to set multiple consul-kv variables from a .env file, like following:

1- Traditional way

env.tf

resource "consul_keys" "env" {
    datacenter = "abcd"
    token = "abcd"

    key {
        name = "MYSQL_USERNAME"
        path = "eu-central-1/my-powerful-api/env"
        value = "root"
    }
}

2- Proposal A

env.tf

resource "consul_keys" "env" {
    datacenter = "abcd"
    token = "abcd"

    dot_env = "${file("${path.module}/.env")}"
}

.env

eu-central-1/my-powerful-api/env/MYSQL_USERNAME=root

3- Proposal B

env.tf

resource "consul_keys" "env" {
    datacenter = "abcd"
    token = "abcd"

    dot_env {
         file = "${file("${path.module}/.env")}"
         path = "eu-central-1/my-powerful-api/env"
    }
}

.env

MYSQL_USERNAME=root

WDYT ?

This issue was originally opened by @sheldonh as hashicorp/terraform#2331. It was migrated here as part of the provider split. The original body of the issue is below.

I've love support for Consul ACL resources.

I've taken a quick look, and there's a bit of a decision to be made. Either I have to

• use Consul's /v1/acl/list endpoint to scroll through all tokens, looking for the one that matches the policy I have in tfstate, or
• save tokens to tfstate so I can use /v1/acl/info/<id>.

I'm keen to hear what people think of this trade-off. My instinct is that relying on /v1/acl/list will turn into a mess. In fact, it might not even be possible when there are multiple tokens with the same policy.

So we might be stuck with the less secure idea of saving tokens to tfstate?

Hi,
I need the NodeMeta feature (https://www.consul.io/api/catalog.html#nodemeta).
Will it be implemented soon?
I've done a pull request (#38) for this. What do you think about it?
Thanks for your help

Terraform Version

0.11.3

Terraform Version

$ terraform -v
Terraform v0.11.2
+ provider.consul v1.0.0

Affected Resource(s)

Please list the resources as a list, for example:

• consul_keys

Terraform Configuration Files

provider "consul" {
  address = "localhost:8500"
  datacenter = "dc1"
}
resource "consul_keys" "test" {
  key {
    path = "foo"
    value = "bar"
  }
}

During the second run it is:

provider "consul" {
  address = "localhost:8500"
  datacenter = "dc1"
}

Debug Output

https://gist.github.com/sakaru/e68c9918674bba98428f79ae56ae272d

Panic Output

n/a

Expected Behavior

The consul_keys.test resource should be destroyed from the consul service.

Actual Behavior

The resource is not destroyed. Despite what the output says.

Steps to Reproduce

1. terraform apply
2. Remove consul_keys.test resource from the main.tf file
3. Re-run terraform apply (and approve)

Important Factoids

$ consul version
Consul v1.0.2

References

None

Terraform Version

$ terraform version
Terraform v0.11.7
+ provider.consul v2.1.0

Affected Resource(s)

• consul_node

Terraform Configuration Files

• provider.tf

provider "consul" {
  datacenter = "dc1"
  scheme = "https"
  address    = "server.example.com"
}

• app.tf:

resource "consul_node" "app-nodes" {
  name = "some-name"
  address = "100.127.20.51"
}

resource "consul_service" "app-consul-service" {
  name    = "some-service-name"
  node    = "some-name"
  port    = 80
  tags    = ["app"]
  datacenter = "dc1"
  depends_on = ["consul_node.app-nodes"]
}

Expected Behavior

The node resource should have been created.

Actual Behavior

 consul_node.app-nodes: Failed to get datacenter from Consul agent: Unexpected response code: 403 (Permission denied)

Steps to Reproduce

1. terraform apply

Important Factoids

The Permission Denied message is correct because the token doesn't have the rights to /agent/self.

However, we wanted to simulate the creation of External Services in consul - which worked via curl - via Terraform. We were able to achieve it after explicitly giving the attribute inside the consul_node resource block:

resource "consul_node" "app-nodes" {
  name = "some-name"
  address = "100.127.20.51" 
+  datacenter = "dc1"
}

However this is not described in the documentation - we had to pull it out from the code. It would be good if there would be also the description of the behavior in the documentation.

Since datacenter is already defined in the provider block, this should have not happened - I'll open another issue because of this.

References

Documentation page in question: https://www.terraform.io/docs/providers/consul/r/node.html

#60 introduces support for ACL with consul_acl_policy and consul_acl_token. Both require the ACL to have been previously bootstrapped manually to work.

Bootstrapping ACLs requires to keep track of the master token given by https://www.consul.io/api/acl/acl.html#bootstrap-acls so it would be written as clear text in the terraform state.
The change in ACLs in Consul 1.4 was made so that the secret IDs would not need to be saved in unsecure situation and accessor IDs could be used instead.

Once ACLs are bootstrapped, the operator also needs to update the provider configuration and to set token or set the CONSUL_HTTP_TOKEN environment variable.

It would be nice to have a secure way to automate this.

Related: hashicorp/terraform#9556

This issue was originally opened by @pll as hashicorp/terraform#8690. It was migrated here as part of the provider split. The original body of the issue is below.

This is essentially a feature request for the consul provider and/or the consul_keys resource to provide a means of insecurely connecting to an SSL protected consul server analogous to curl's -k|--insecure or wget's --no-check-certificate options.

Terraform Version

$ terraform -v
Terraform v0.6.16

Version is probably irrelevant, given the current documentation does not indicate this feature even exists for consul or consul_keys.

Affected Resource(s)

• cosul provider
• consul_keys resource

And probably other consul related resources such as consul_key_prefix.

Actual Behavior

Error applying plan:
1 error(s) occurred:

* consul_keys.jobservice: Failed to write Consul key 'service/data/elbEndpoint': Put 
https://consul.myco.com/v1/kv/jobservice/data/elbEndpoint?dc=aws: x509: certificate is valid for
consul-service, not consul-myco.com

Steps to Reproduce

1. Set up a consul service with a self-signed cert where the CN in no way remotely resembles to actual ELB name.
2. Try to register a key in that consul environment.

I'm trying to use data.consul_key_prefix to batch read keys from consul, but terraform fails with "data variables must be four parts: data.TYPE.NAME.ATTR" even though I'm following this document.

Terraform Version

Terraform v0.11.8

• provider.consul v2.1.0

Affected Resource(s)

• consul_key_prefix

Terraform Configuration Files

provider "consul" {
  address = "consul.local:443"
  scheme  = "https"
}

data "consul_key_prefix" "aws-environment" {
  token = "yeah-right"
  datacenter = "local"

  path_prefix = "terraform/config/aws-env/${terraform.workspace}/"
}

# Module Outputs
output "vpc_id" {
  value = "${data.consul_key_prefix.aws-environment["vpc_id"]}"
}

Debug Output

2018/09/28 15:56:39 [INFO] Terraform version: 0.11.8 7a5c1d221ac209bbac66947c369815cd9ca70ed5
2018/09/28 15:56:39 [INFO] Go runtime version: go1.10.1
2018/09/28 15:56:39 [INFO] CLI args: []string{"/usr/local/bin/terraform", "plan"}
2018/09/28 15:56:39 [DEBUG] Attempting to open CLI config file: /home/tiago_wegner/.terraformrc
2018/09/28 15:56:39 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2018/09/28 15:56:39 [INFO] CLI command args: []string{"plan"}
2018/09/28 15:56:39 [DEBUG] plugin: waiting for all plugin processes to complete...

Error: Error loading /home/tiago_wegner/repo/tf-modules/aws-environment/main.tf: Error reading config for output account_id: data.consul_key_prefix.aws-environment: data variables must be four parts: data.TYPE.NAME.ATTR in:

${data.consul_key_prefix.aws-environment["vpc_id"]}

Panic Output

Expected Behavior

Keys should be correctly read from consul kv store.

Actual Behavior

Error: Error loading /home/tiago_wegner/repo/tf-modules/aws-environment/main.tf: Error reading config for output account_id: data.consul_key_prefix.aws-environment: data variables must be four parts: data.TYPE.NAME.ATTR in:

${data.consul_key_prefix.aws-environment["vpc_id"]}

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan

Important Factoids

Nothing in particular

References

None

This issue was originally opened by @jniesen as hashicorp/terraform#7438. It was migrated here as part of the provider split. The original body of the issue is below.

Terraform Version

~ terraform -v
Terraform v0.7.0-rc2 (46a0709bba004d8b6e0eedad411270b3ae135a9e)

Affected Resource(s)

• consul_keys

Terraform Configuration Files

provider "consul" {
  address    = "${var.consul_addr}"
  datacenter = "${var.consul_dc}"
  scheme     = "${var.consul_scheme}"
}

resource "consul_keys" "ro" {
  key {
    name = "ami"
    path = "${var.ami_path}/${var.ami_version}"
  }
}

resource "aws_launch_configuration" "a" {
  name_prefix   = "artifactory-"
  instance_type = "t2.large"
  image_id      = "${consul_keys.ro.var.ami}"
  key_name      = "${var.ssh_key_name}"
  security_groups = [
    "${aws_security_group.a.id}"
  ]

  lifecycle {
    create_before_destroy = true
  }
}

Debug Output

https://gist.github.com/jniesen/7997ba2f475a0d8637a1a58f16cbe2da

Expected Behavior

Successful plan.

Actual Behavior

~ terraform plan .
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

data.terraform_remote_state.manheim2_vpc: Refreshing state...

Error running plan: 1 error(s) occurred:

* Resource 'consul_keys.ro' does not have attribute 'var.ami' for variable 'consul_keys.ro.var.ami'

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan