As-is, this project doesn't show up on a GitHub search for "terraform" + "state"

We have a common AWS account to store statefiles and manage locking for our terraform installations. It would be fantastic if you could allow the use of a specific "profile" in ~/.aws/credentials.

See: https://www.terraform.io/docs/state/remote/s3.html#profile

Configuration might look like this:

lock = {
  backend = "dynamodb"
  config {
    state_file_id = "my-app"
    aws_region = "us-east-1"
    table_name = "terragrunt_locks"
    max_lock_retries = 360
    profile = "terraform_state_prod"
  }
}

The credentials file might look like this:

[default]
aws_access_key_id = AKADADF
aws_secret_access_key = Wpaduaded32466

[terraform_state_prod]
aws_access_key_id = AKEASDFRE
aws_secret_access_key = zasdf3QWEEdae                                                                 

It would also be nice to specify the credentials file location, the default obviously being "~/.aws/credentials"

When you run terragrunt apply-all or terragrunt destroy-all, they can make changes across a ton of different modules, in parallel. This is great, but as mentioned in #71, the amount of log output is overwhelming, and it's easy to miss something important. One small step in helping with this is to store the log output for each module somewhere. Perhaps within the module or a tmp directory will do.

terragrunt version: v0.5.1

This is a possible duplicate of #41, but I get errors that look like this on a fairly regular basis. Any chance someone can confirm this as a duplicate, or even better provider a fix?

https://gist.github.com/jdubeau123/6f5286fd43019bf66b71ca62e5a0cafe

Currently, terragrunt configures your localdev's terraform remote state with this code.

The problem is that this doesn't guarantee that the contents of the .terragrunt file match the current remote state configuration. This particularly becomes an issue when I've made a mistake in the .terragrunt file, correct it, and re-run terragrunt expecting terragrunt to pick up the change.

We should consider creating a hash of the .terragrunt file and writing it to the .terraform folder so that terragrunt can check whether any value in the .terragrunt file has changed, and if so, re-execute terraform remote config.

It would be really nice if you could use terragrunt with a environment argument, and have it map the argument value to a state file configured.

Something like terragrunt -environment=staging plan and then be able to do something like

remoteState = {
  backend = "s3"
  backendConfigs = {
    encrypt = "true"
    bucket = "my-bucket"
    key = "${environment}.tfstate"
    region = "us-east-1"
  }
}

I have terragrunt config files per account-region. The root config file should also have the matching region info and a different bucket name. Could you please support a different root config file instead of operating under the assumption that the root file will be named .terragrunt?

Thanks for the app, it's been super useful!

Would like the ability to lock/unlock a configuration from further changes.

Terragrunt does not pick up on changes to modules that it has already downloaded. I know this is noted in the comments where that code is located... just wanted to log an issue to note that it's already known and maybe someone will pick it up and work on a fix.

Great tool here! Helps solves my team's issue with state conflicts while we figure out our long term solution.

I was wondering if it makes sense to optionally lock at the "terraform plan" stage instead of apply? From what I understood in terragrunt, the locking is only at the "apply" stage (which is great).

The terraform workflow I've been following is

1. terraform plan and output a plan file
2. Get a peer review on the plan and approve it if needed
3. terraform apply against the plan file

So in general, we always ran "terraform apply" against a plan file.

At a previous organisation, we had a bespoke solution where we locked the state file when we begin "plan" stage. That meant that

• no one can change (apply) the infrastructure until that plan has been applied or the user released the lock for that plan.
• other users can still run plan but when they apply their changes on a locked state, they will be prompetd to wait for that other plan to be run and re-run their plan against the new changes
• the user's plan is going to be applied against a state that did not change during the peer review period. There is no chance between the "plan" and "apply" stage that the infrastructure got changed by someone else.

Any thoughts?

I get this error once in a while:

fatal error: systemstack called from unexpected goroutine
...
### same line repeated 500+ times ###
...
fatal error: systemstack called from unexpected goroutine
./terraform.sh: line 210: 23108 Segmentation fault: 11  terragrunt get $LAYER_WORK_DIR

Rerun of the same command helps.

Here is nice stack trace to review - https://gist.github.com/antonbabenko/87bfdd7c070146d49d1021be424458d4

Per #26, I have a lot of commonalities between .terragrunt files in different directories. I'd really like to be able to do something like:

# devops/infrastructure/.terragrunt
dynamoDbLock = { stateFileId = "my_project_terraform_db_lock" }

remoteState = {
  backend = "s3"
  backendConfigs = {
    encrypt = "true"
    bucket = "my-bucket"
    region = "us-east-1"
  }
}

Then, in each directory, I could do something like:

# devops/infrastructure/application-qa/.terragrunt
import("../.terragrunt")
remoteState = {
  backendConfigs = {
    key = "terragrunt/application-qa.tfstate"
  }
}

The idea being that if I do an include, anything I do from that point onwards would override what was imported.

Given the following folder structure:

infrastructure-live
- stage
  - data-stores
    - postgres
    - redis
  - ecs-cluster
  - kms-master-key
  - vpc

I ran terragrunt spin-up and expected Terragrunt to apply all the above folders as long as each had a .terragrunt file. But the stage/data-stores is ignored, even though both stage/data-stores/postgres and stage/data-stores/redis are confirmed to have a .terragrunt file .

Here's a zip file of the full infrastructure-live directory in a private Google Drive folder available to Gruntworkers only so someone else can reproduce.

Here's the output. Note the inclusion of ignored modules.

josh@Joshs-MacBook-Pro /r/g/b/i/stage> terragrunt spin-up
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/vpc depends on module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc as well! (y/n) y
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster depends on module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host as well! (y/n) y
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster depends on module /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics as well! (y/n) y
[terragrunt] 2016/12/19 15:07:23 Stack at /repos/gruntwork-clients/acme/infrastructure-live/stage:
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/stage/vpc, /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host, /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/kms-master-key (dependencies: [])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/vpc (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc (dependencies: [])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc, /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics (dependencies: [])
[terragrunt]  Are you sure you want to run 'terragrunt apply' in each folder of the stack described above? (y/n)

Not sure when this started happening and why it's apparently not happening in CI, but Terragrunt tests consistently fail when I run them locally. It's late here, so I'm just recording this but haven't dug into why yet. Here's the test output:

[terragrunt] 2016/12/05 01:26:46 Reading Terragrunt config file at /var/folders/g0/0gkdx4sj4qv5gzxfcznscdpw0000gn/T/.terragrunt053106030
[terragrunt] 2016/12/05 01:26:46 Initializing remote state for the s3 backend
Remote state S3 bucket terragrunt-test-bucket-oapzqi does not exist or you don't have permissions to access it. Would you Terragrunt to create it? (y/n)
[terragrunt] 2016/12/05 01:26:46 The non-interactive flag is set to true, so assuming 'yes' for all prompts
[terragrunt] 2016/12/05 01:26:46 Creating S3 bucket terragrunt-test-bucket-oapzqi
[terragrunt] 2016/12/05 01:26:47 S3 bucket terragrunt-test-bucket-oapzqi created.
[terragrunt] 2016/12/05 01:26:47 Enabling versioning on S3 bucket terragrunt-test-bucket-oapzqi
[terragrunt] 2016/12/05 01:26:48 Configuring remote state for the s3 backend
[terragrunt] 2016/12/05 01:26:48 Running command: terraform remote config -backend s3 -backend-config=encrypt=true -backend-config=bucket=terragrunt-test-bucket-oapzqi -backend-config=key=terraform.tfstate -backend-config=region=us-west-2
Initialized blank state with remote state enabled!
Remote state configured and pulled.
[terragrunt] 2016/12/05 01:26:49 Attempting to acquire lock for state file terragrunt-test-fixture in DynamoDB
[terragrunt] 2016/12/05 01:26:49 Lock table terragrunt_locks does not exist in DynamoDB. Will need to create it just this first time.
[terragrunt] 2016/12/05 01:26:49 Creating table terragrunt_locks in DynamoDB
[terragrunt] 2016/12/05 01:26:50 Table terragrunt_locks is not yet in active state. Will check again after 10s.
[terragrunt] 2016/12/05 01:27:00 Success! Table terragrunt_locks is now in active state.
[terragrunt] 2016/12/05 01:27:00 Attempting to create lock item for state file terragrunt-test-fixture in DynamoDB table terragrunt_locks
[terragrunt] 2016/12/05 01:27:00 Lock acquired!
[terragrunt] 2016/12/05 01:27:00 Running command: terraform apply fixture/
data.template_file.test: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

rendered_template =
[terragrunt] 2016/12/05 01:27:01 Attempting to release lock for state file terragrunt-test-fixture in DynamoDB
[terragrunt] 2016/12/05 01:27:01 Lock released!
[terragrunt] 2016/12/05 01:27:03 Reading Terragrunt config file at fixture-lock/.terragrunt
Are you sure you want to acquire a long-term lock? (y/n)
[terragrunt] 2016/12/05 01:27:03 The non-interactive flag is set to true, so assuming 'yes' for all prompts
[terragrunt] 2016/12/05 01:27:03 Acquiring long-term lock. To release the lock, use the release-lock command.
[terragrunt] 2016/12/05 01:27:03 Attempting to acquire lock for state file terragrunt-test-fixture-lock in DynamoDB
[terragrunt] 2016/12/05 01:27:03 Attempting to create lock item for state file terragrunt-test-fixture-lock in DynamoDB table terragrunt_locks
[terragrunt] 2016/12/05 01:27:03 Lock acquired!
[terragrunt] 2016/12/05 01:27:03 Reading Terragrunt config file at fixture-lock/.terragrunt
[terragrunt] 2016/12/05 01:27:03 Attempting to acquire lock for state file terragrunt-test-fixture-lock in DynamoDB
[terragrunt] 2016/12/05 01:27:03 Attempting to create lock item for state file terragrunt-test-fixture-lock in DynamoDB table terragrunt_locks
[terragrunt] 2016/12/05 01:27:03 Someone already has a lock on state file terragrunt-test-fixture-lock! arn:aws:iam::087285199408:user/[email protected]@10.0.1.9 acquired the lock on 2016-12-05 08:27:03.216425023 +0000 UTC.
Unable to acquire lock for item terragrunt-test-fixture-lock after 1 retries.
exit status 1
FAIL	github.com/gruntwork-io/terragrunt/test	17.631s

Hi,

How does terragrunt interact with different AWS account credentials? Is there an option to set the profile in the config or will it respect the AWS_DEFAULT_PROFILE variable to get the correct API keys for use.

Thanks

Mark

Similar to #85

Terragrunt Version: v0.7.3
OS: Ubuntu 14.04 LTS
Terraform Version: 0.8.2

Repro:

• set up a custom AWS credentials file with multiple account access keys ad secrets
• set the path to file into the global environment AWS_SHARED_CREDENTIALS_FILE
• run terraform plan
• run terragrunt plan

Result

• Terraform runs as expected
• Terragrunt gives a credential error. Example output

terragrunt plan -var-file=/my_path/env.tfvars -out=/my_path/env.tfplan --terragrunt-config /my_path/env.terragrunt
[terragrunt] 2017/01/03 22:43:02 Reading Terragrunt config file at /my_path/env.terragrunt
[terragrunt] 2017/01/03 22:43:02 Initializing remote state for the s3 backend
[terragrunt] 2017/01/03 22:43:07 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated. 
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

The official AWS GO support the global AWS_SHARED_CREDENTIALS_FILE - https://github.com/aws/aws-sdk-go/blob/b2dc98bb584e48b0f5f39c93110633173c5da43c/aws/credentials/shared_credentials_provider.go#L30

So in theory terragrunt should just be able to read it but I am not sure what could be messing with it. Any ideas?

Similar to how it verifies existence of dynamodb lock table and create it if it does not exist.

I think it should create versioned S3 bucket (default, acl=private) when remote_state section is defined.

If you run terragrunt release-lock when a corresponding lock that doesn't exist, you get the same output as if the lock did exist.

I am working on a special environment, and I would like the ability to choose a different .terragrunt file than the one on the current directory, or, even better, the ability to specify the options on the command line or environment variables. Currently, I'm automatically generating the .terragrunt file.

I would like to use terraform directly if possible. With this layer on top it is possible for someone to just invoke terraform directly instead of using terragrunt.

Is it possible to make terragrunt a terraform plugin or possibly a data source such that I can put 1 line into a terraform file that will ensure that terragrunt is installed for terraform plan to be able to be run successfully?

Terragrunt has a relatively lightweight footprint with respect to terraform, but we should automate our validation that terragrunt works with any given version of terraform.

terraform plan -detailed-exitcode may return 0, 1, 2. Official documentation

terragrunt plan -detailed-exitcode currently return 0 or 1.

We have a bunch of different Terraform state files, each handling .tf files in different directories. There's a .terragrunt file in each directory that looks something like:

dynamoDbLock = { stateFileId = "my_project_terraform_db_lock" }

remoteState = {
  backend = "s3"
  backendConfigs = {
    encrypt = "true"
    bucket = "my-bucket"
    key = "terragrunt/<dirname>.tfstate"
    region = "us-east-1"
  }
}

I'd really like to be able to use a single .terragrunt file for all the directories and be able do something like:

# Skip a bunch of common code shown above
remoteState = {
  backendConfigs = {
    key = "terragrunt/${dirname()}.tfstate"
  }
}

That way, I can symlink to a common .terragrunt in the parent directory and be assured I'm not screwing anything up when I copy the .terragrunt file into a new directory.

Hi guys,

It's me again. Seems that #36 was fixed, because I was able to use 0.1.1 for several days without any issues, but today during "terragrunt plan" it crashed:

https://gist.github.com/antonbabenko/9eae6e5ded37633307d78894e1b6c833

Rerun fixes the problem and I continue.

Any ideas ?

I've just setup Terragrunt, and after running a couple of times went to check DynamoDB to see what's happening there. Every time I've run it, the output has included something like the following:

Attempting to acquire lock for state file terraform-example in DynamoDB
Attempting to create lock item for state file terraform-example in DynamoDB table terragrunt_locks
Lock acquired!

Yet, I can't locate any such table in AWS. I've checked all regions. Originally, I wasn't specifying a region, which I've updated now. But still, no actual table.

So my question is... is the lock really happening? Or is Terragrunt failing silently in some way and giving me a false sense of security?

(I'm not running Terragrunt on an EC2 instance here, I'm running it on my local machine. Which I assume is normal, but something I read elsewhere made me wonder about this.)

Terragrunt writes private IP address into DynamoDB lock table, while CloudTrail shows public IP address in "Source IP address".

I think it would be great to store public IP also in lock table. (I don't have a clear use case for this now, but it may potentially simplify debug when we can see same kind of IP in both places. Especially when terragrunt show-lock is ready).

Feature request ? :)

Trying to set up terragrunt for the first time using this command:
terragrunt plan --var-file=..\terraform.tfvars -out .\plan.out -var "username=$env:username"

and I get:
Error parsing Terragrunt config file .terragrunt: state_file_id cannot be empty

.terragrunt file contents:

# Configure Terragrunt to use DynamoDB for locking
lock = {
	backend = "dynamodb"
	config {
		stateFileId = "my-apps"
		awsRegion = "us-west-2"
		tableName = "terragrunt_locks"
		maxLockRetries = 360
	}
}

# Configure Terragrunt to automatically store tfstate files in an S3 bucket
remoteState = {
  backend = "s3"
  backendConfigs = {
    encrypt = "true"
    bucket = "tfremotestate"
    key = "terraform.tfstate"
    region = "us-west-2"
  }
}

There is an S3 bucket created, called tfremotestate, and I have full Admin rights.

When you run terragrunt apply-all, Terragrunt will first ask you if you want to apply each external dependency. Whether you say yes or no, it includes the external dependency in the list above "Are you sure you want to run 'terragrunt apply' in each folder of the stack described above?"

The expected behavior is external modules that are not to be applied should be omitted from the list.

josh@Joshs-MacBook-Pro /r/g/b/i/stage> terragrunt apply-all
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/vpc depends on module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc as well! (y/n) y
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster depends on module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host as well! (y/n) y
[terragrunt]  Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster depends on module /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics as well! (y/n) y
[terragrunt] 2016/12/19 15:07:23 Stack at /repos/gruntwork-clients/acme/infrastructure-live/stage:
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/stage/vpc, /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host, /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/kms-master-key (dependencies: [])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/stage/vpc (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc (dependencies: [])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host (dependencies: [/repos/gruntwork-clients/acme/infrastructure-live/mgmt/vpc, /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics])
  => Module /repos/gruntwork-clients/acme/infrastructure-live/global/sns-topics (dependencies: [])
[terragrunt]  Are you sure you want to run 'terragrunt apply' in each folder of the stack described above? (y/n)

Nice guide here: https://medium.com/@mthenw/distributed-locks-with-consul-and-golang-c4eccc217dd5#.tr3569pv8

When you run apply-all on multiple modules, it's not obvious whether any modules had an error because Terragrunt itself won't exit with an error. Instead, you have to just scroll up to look for red text. It'd be better if Terragrunt reported its errors in a summary at the end like this:

[terragrunt] 2016/12/19 14:38:49 Terragrunt spin-up completed:

   5 Modules applied successfully [green text]
   1 Module failed to apply [red text]

Failed Modules
- ecs-cluster: Error downloading modules: module ecs_cluster: not found, may need to be downloaded using 'terraform get'

Hi,

I'm in the process of adding support for locking via Azure Storage Blobs and have a suggestion to make it simpler to implement a new lock provider. By using a more generic config format similar to Terraform's remote state configuration with a backend name and config map.

Example:

dynamoDbLock = {
  stateFileId = "my-app"
}

# Would become

lock = {
  backend = "dynamodb"
  config {
    state_file_id = "my-app"
  }
}

This would allow lock-providing packages to expose a factory function like in Terraform, which would assume the role of FillDefaults and Validate.

I've also noticed that keys in the .terragrunt file use camelCase rather than snake_case like Terraform, is there a reason for this?

I'll continue to work on my Azure lock in the current code style based off the DynamoDB implementation.

Hi all,
This is an enhancement I suppose. :)
Today a new version of terragrunt is out and I was interested in reading about the changes but there is no CHANGELOG.md to check. It would help us users understand if there is any new functionality/breaking changes/bug fix that may impact us without checking the blog.

Thanks you,
F.

Just a cosmetics thing, if you run terragrunt graph | dot -Tpng > mygraph.png
you will get an error like so Error: <stdin>: syntax error in line 1 near '[' because terragrunt adds the two lines [terragrunt] 2016/08/30 11:41:34 Remote state is already configured for backend s3 [terragrunt] 2016/08/30 11:41:34 Running command: terraform graph

It's easy to just run terraform graph instead, but I wanted to give a heads up

After you run the tests, a bunch of DynamoDB tables are left behind. These should be cleaned up automatically.

Terraform has a command that syncs the remote state storage with the state stored locally:

terraform remote push

This obviously updates the remote state, so Terragrunt should obtain a lock before doing it.

Hi,

Great work on the latest release! :) Really looking forward to trying out the aquire-lock function.

But this release does not seem to have any binaries to download from?
https://github.com/gruntwork-io/terragrunt/releases/tag/v0.3.0

Following up from #48 (comment), the lock section of the .terragrunt config file allows the user to specify a region. Failing to specify a region uses a hard-coded default of us-east-1. The Go SDK supports the AWS standard AWS_REGION environment variable. Not honoring this setting violates the POLA. 😉

Suggested order of priority:

1. region value from .terragrunt
2. AWS_REGION environment variable
3. Fall back us-east-1 region

Thanks for the great tool!

I've created the dynamodb table with Terraform and I've naively set the partition key as "state_file_id".
Terragrunt would not work because the expected partition key is StateFileId.

Let's update the README.md to make that explicit

I am trying to use the nested terragrunt file on Windows. The directory structure is:

.terragrunt
env
  dev
    web
      .terragrunt

The root terragrunt file content.

lock = {
  backend = "dynamodb"
  config {
    state_file_id = "${path_relative_to_include()}"
    aws_region = "eu-west-1"
    table_name = "terragrunt_locks"
  }
}

remote_state = {
  backend = "s3"
  config {
    encrypt = "true"
    bucket = "tfstate"
    key = "${path_relative_to_include()}/terraform.tfstate"
    region = "eu-west-1"
  }
}

The content of terragrunt file in dev/web.

include = {
  path = "${find_in_parent_folders()}"
}

Running terragrunt plan in dev/web, I got the following error:

[terragrunt] 2016/12/27 15:56:03 Reading Terragrunt config file at C:\Users\userA\dev\test-aws\envs\dev\web\.terragrunt
[terragrunt] 2016/12/27 15:56:03 Error parsing Terragrunt config file C:\Users\userA\dev\test-aws\envs\dev\web\.terragrunt: At 2:20: illegal char escape

When I replace ${find_in_parent_folders()} with hardcoded path: ../../../.terragrunt, I got error further along.

[terragrunt] 2016/12/27 15:52:07 Reading Terragrunt config file at C:\Users\userA\dev\test-aws\envs\dev\web\.terragrunt
[terragrunt] 2016/12/27 15:52:07 Error parsing Terragrunt config file C:\Users\userA\dev\test-aws\envs\dev\web\.terragrunt: Error parsing Terragrunt config file C:\Users
\userA\dev\test-aws\.terragrunt: At 15:17: illegal char escape

Could this be another Windows issue?

When following Terraform state management best practices, you often end up with the all the components of a single environment broken into many folders. For example, here is a typical production environment:

prod
  └ vpc
  └ services
      └ frontend-app
      └ backend-app
          └ vars.tf
          └ outputs.tf
          └ main.tf
          └ .terragrunt
  └ data-storage
      └ mysql
      └ redis

The reason we break out these components into separate folders rather than defining them all in one set of templates provides is to ensure isolation. That way, when you're making changes to one component (e.g. deploying the frontend-app), you aren't risking breaking all the others (e.g. breaking the VPC, data storage, etc).

The question is, how do you "spin up" an entire environment without having to manually go into each folder and run terragrunt apply?

One option is to add a command like terragrunt spin-up which could find all the Terraform modules in the folder structure (based on the presence of a .tf files), parse the code in those modules to figure out the dependencies between them (that is, those dependencies explicitly specified via the usage of the terraform_remote_state resource), and automatically call terragrunt apply on that dependency tree in the right order. There could be an analogous terragrunt spin-down command too. If all the modules contain .terragrunt files (or there is a root .terragrunt file if we implement #26 (comment)), then Terragrunt would know how they manage remote state, and could therefore match the remote state configuration against the usage of the terraform_remote_state resource within each module.

This would work as long as all dependencies between modules are specified via the terraform_remote_state resource. If there are an implicit dependencies—for example, if the iam module creates an IAM role and the kms module has the name of that IAM role copy/pasted and just assumes it'll exist—then you may get errors. The good news is that fixing those errors would be very simply: just add a terraform_remote_state resource to the kms module that points to the remote state of the iam module. In other words, adding a terraform_remote_state resource would be like adding a depends_on parameter.

There is also the tricky question of what to do if Terragrunt hits an error part of the way through spinning up an environment. Probably the right thing to do is to exit, just as Terraform does. If you fix the issue and re-run terragrunt spin-up, you'd probably lose some time running terraform apply on modules that have already been applied, but since there won't be any actual changes to apply, that shouldn't take too long.

It would very helpful if we can create a terragrunt lock without performing any other operation.

The use case is, during the time between when User 1 does a terragrunt plan and does the subsequent terraform apply, someone else (User 2) could come in and do a plan + apply. This could create a situation where the outcome of User 1's apply results in something unexpected. If User 1 could create the lock before doing anything, that would ensure that only one user was doing anything terraform-related.

Hi,

The brew package is still at v0.3.

Any chance to get it updated to the latest?

According to the documentation:

The terraform refresh command is used to reconcile the state Terraform knows about (via its state file) with the real-world infrastructure. This can be used to detect any drift from the last-known state, and to update the state file.

This does not modify infrastructure, but does modify the state file. If the state is changed, this may cause changes to occur during the next plan or apply.

If someone is running an apply with terragrunt, another user could run refresh and potentially write a broken state in s3.

On SIGINT (Interrupt from keyboard), terragrunt release the lock, but not on SIGTERM (Termination signal). This should probably be fixed. See locks/lock.go.

diff --git a/locks/lock.go b/locks/lock.go
index 58ace31..13af32b 100644
--- a/locks/lock.go
+++ b/locks/lock.go
@@ -5,6 +5,7 @@ import (
        "github.com/gruntwork-io/terragrunt/errors"
        "os"
        "os/signal"
+       "syscall"
 )
 
 // Every type of lock must implement this interface
@@ -48,6 +49,7 @@ func WithLock(lock Lock, action func() error) (finalErr error) {
        // the blocking call to action() to return normally.
        signalChannel := make(chan os.Signal, 1)
        signal.Notify(signalChannel, os.Interrupt)
+       signal.Notify(signalChannel, syscall.SIGTERM)
        go func() { util.Logger.Printf("Caught signal '%s'. Terraform should be shutting down gracefully now.", <- signalChannel) }()
 
        return action()

When you run terragrunt where a .terragrunt file specifies an S3 Bucket that exists but for which versioning has not been enabled, it panics when instead it should issue a warning. Here are the steps to reproduce.

1. Create an S3 Bucket but do NOT enable versioning.
2. Specify that S3 Bucket Name in a .terragrunt file.
3. Run terragrunt plan or terragrunt apply.
4. Terragrunt will panic.

I received this output:

> terragrunt plan
[terragrunt] 2016/12/05 01:30:59 Reading Terragrunt config file at .terragrunt
[terragrunt] 2016/12/05 01:30:59 Initializing remote state for the s3 backend
[terragrunt] 2016/12/05 01:31:00 runtime.errorString runtime error: invalid memory address or nil pointer dereference
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/errors/errors.go:70 (0x69e31)
/usr/local/go/src/runtime/asm_amd64.s:472 (0x59d8e)
	call32: MOVQ	DX, 0(SP);			\
/usr/local/go/src/runtime/panic.go:443 (0x2ba89)
	gopanic: freedefer(d)
/usr/local/go/src/runtime/panic.go:62 (0x2a145)
	panicmem: panic(memoryError)
/usr/local/go/src/runtime/sigpanic_unix.go:24 (0x40a39)
	sigpanic: panicmem()
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/remote/remote_state_s3.go:113 (0xca055)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/remote/remote_state_s3.go:49 (0xc96b1)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/remote/remote_state.go:45 (0xc83c8)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/remote/remote_state.go:60 (0xc85a7)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:199 (0x68601)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:102 (0x67475)
/usr/local/go/src/runtime/asm_amd64.s:472 (0x59d8e)
	call32: MOVQ	DX, 0(SP);			\
/usr/local/go/src/reflect/value.go:435 (0x12bc5d)
	Value.call:
/usr/local/go/src/reflect/value.go:303 (0x12a921)
	Value.Call: }
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:480 (0xd13be)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:241 (0xceac5)
/home/ubuntu/.go_workspace/src/github.com/gruntwork-io/terragrunt/main.go:19 (0x20ce)
/usr/local/go/src/runtime/proc.go:188 (0x2d5a0)
	main: // Make racy client program work: if panicking on
/usr/local/go/src/runtime/asm_amd64.s:1998 (0x5c441)
	goexit: endofpage:

The fix is straightforward. I'll submit a PR for it now, but would prefer to fix #64 so that we can write a failing test for it.

Currently, the bucket is hard-coded to an S3 bucket in the Gruntwork AWS account. This means no contributors can run the test without making (non-obvious) changes locally.

To fix this issue, the test should be updated to:

• Create a randomly named S3 Bucket
• Generate a new .terragrunt file with the new S3 Bucket Name
• Run the test
• Delete all files in the S3 Bucket
• Delete the S3 Bucket

After using terragrunt apply-all for a bit, I'm starting to find this formatting frustrating:

Module /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster depends on module /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host, which is an external dependency outside of the current working directory. Should Terragrunt skip over this external dependency? Warning, if you say 'no', Terragrunt will make changes in /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host as well! (y/n)

It's just a lot to read, especially when you're working fast. What do you think of a format like this?

Should Terragrunt apply the following external dependency? (y/n)

   Module:              /repos/gruntwork-clients/acme/infrastructure-live/stage/ecs-cluster
   External Dependency: /repos/gruntwork-clients/acme/infrastructure-live/mgmt/bastion-host

I'm going to leave this as an open issue to report and collect feedback on terragrunt spin-up and terragrunt tear-down.

Overall, it's a joy to use! But was it Thoreau who warned that with each new technology we both gain and lose? With spin-up the gains are the ability to launch a lot of infra fast while preserving terraform state isolation, and by extension the ability to launch a whole environment fast. These are wonderful wins. The loss is that I'm less aware of the details of each individual terragrunt apply.

For example, I had an issue where a git tag of one of my Terraform modules (module-security) was downloaded, but it was out of date. I'd needed to re-issue a git tag to fix a minor issue, but terraform won't automatically check for new versions of a git tag of an already downloaded module (which is reasonable behavior). If I hadn't manually inspected the results, I wouldn't have seen that in fact terraform was using an outdated module version.

It made me wonder if the spin-up paradigm demands a new level of guardrails. For example, should terragrunt do a terraform get -update on every module to be safe? Perhaps that's an option since there will be a performance hit with that.

Not sure, so just recording initial thoughts.

As per issue #21

@brikis98 looks like I'm still having this issue.

[karl.barbour@xxx tf_poc]$ terragrunt apply
[terragrunt] 2016/12/21 15:04:13 Reading Terragrunt config file at .terragrunt
[terragrunt] 2016/12/21 15:04:13 Initializing remote state for the s3 backend
[terragrunt] 2016/12/21 15:04:13 Error finding AWS credentials (did you set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables?): NoCredentialProviders: no valid providers in chain. Deprecated. 
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
[karl.barbour@xxx tf_poc]$ echo $AWS_DEFAULT_PROFILE 
poc-account
[karl.barbour@xxx tf_poc]$ echo $AWS_PROFILE 
poc-account

As you can see, the env vars are working.

Versions:
terragrunt 0.7.1
aws-cli 1.10.47
Darwin 15.5.0

Any ideas?

With these ENV vars set, I can use the CLI fine (i.e. aws ec2 describe-instances), so it doesn't appear to be a credential issue.

Terragrunt provides locking for state files, but you can still have "concurrency" issues with the following two scenarios:

1. You forget to run git pull before running terragrunt apply, so you end up applying out-of-date templates.
2. You forget to run git commit and git push after running terragrunt apply, so even if your other teammates remember to run git pull, they will end up applying out-of-date templates.

Perhaps we should add a new setting in .terragrunt:

enforce_version_control = {
  backend = "git"
  config = {
    branch = "master"
    remote = "origin"
  }
}

If you have that config in your file, then every time you run terragrunt apply, Terragrunt will:

1. Obtain a lock (if one is specified).
2. Check if you are on master and if not, exit with an error.
3. Check if you have any uncommitted or unpushed files in master, and if you do, exit with an error.
4. Run git pull origin master.
5. Then run your actual Terraform command.

Note that you'd probably only want this config in "shared" environments such as production. For test environments, you will likely make many changes with Terraform that are not committed to version control, as a way to test those changes before committing.