Terragrunt Provider Cache not working properly when using it with Atlantis about terragrunt HOT 17 CLOSED

Hi @amontalban,

Thanks for looking this issue, I have built a test case for easier testing.
You can find it here: https://github.com/amontalban/terragrunt-issue-3076

Thank you, this should really help, I will try it out.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this rejected the given authentication credentials error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command. The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.

That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.

Hi @levkohimins,

Thank you for your explanation! I have now tested this in Atlantis and the rejected the given authentication credentials. error disappeared after using a fixed TERRAGRUNT_PROVIDER_CACHE_TOKEN value for all processes.

However I am now getting this error in some of the plans (Here are the stats 42 failing, 1 pending, and 652 successful checks):

╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider
│ hashicorp/null: could not query provider registry for
│ registry.terraform.io/hashicorp/null: the request failed after 2 attempts,
│ please try again later: Get
│ "http://127.0.0.1:36847/v1/providers/registry.terraform.io/hashicorp/null/versions":
│ dial tcp 127.0.0.1:36847: connect: connection refused

Here is the full log just in case: https://gist.github.com/amontalban/a5dbe6f5831d9ea79fd2923793a53323

Just in case I'm going to test with v0.57.6 and report back because I have been testing with v0.57.2.

Thanks once again for the help!

from terragrunt.

Hi @amontalban,

Thanks for looking this issue, I have built a test case for easier testing.

You can find it here: https://github.com/amontalban/terragrunt-issue-3076

Thank you, this should really help, I will try it out.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

Each Terragrunt process runs its own cache server (starts listening on a free port) and generates its own token, they shouldn't conflict. I suppose that this rejected the given authentication credentials error occurs when Terraform tries to connect to a server other than the one whose process run the terraform command.
The logic for determining a free port is implemented in Golang code, so the likelihood of a bug is unlikely.

That's all thoughts for now, I'll try your test if I encounter this error I will be able to say more.

from terragrunt.

Hi @amontalban,
Thanks for the test case. I figured out why it doesn't work, --terragrunt-provider-cache conflicts with --terragrunt-download-dir. I have started working on fixing this issue and will update you once it is done.

from terragrunt.

Hey @levkohimins, sorry I missed your previous message. I will try to test it out tomorrow and report back.

Thank you for the quick fix!

from terragrunt.

@sshepel, I could not reproduce the issue.

from terragrunt.

I think the init logs are missleading (so it's working). Because with terragrunt cache enabled I can see this:

Initializing provider plugins...
- Finding latest version of hashicorp/template...
....
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (unauthenticated)

Where as with TF_PLUGIN_CACHE_DIR var set I saw something like: used from cache for the same provider plugin. But I have checked the .terraform folder under .terragrunt-cache for specific module after the init step with cache and I can see that the binaries for provider plugins are linked to the cached folder as mentioned in Provider Caching docs.

We use provider_installation instead of plugin_cache_dir so that Terraform has read-only access to the cache directory. Since the logs you mentioned are Terrafrom's log we cannot change them in any way.

Also there is a message about incomplete lock file information for providers (which seems to be fine, but very annoying).

We are trying to solve it in #3089

from terragrunt.

@levkohimins: I am still getting

│ Error: Failed to resolve provider packages
│ 
│ Could not resolve provider hashicorp/time: could not query provider
│ registry for registry.opentofu.org/hashicorp/time: the request failed after
│ 2 attempts, please try again later: Get
│ "http://127.0.0.1:46489/v1/providers//registry.opentofu.org/hashicorp/time/versions":
│ dial tcp 127.0.0.1:46489: connect: connection refused

with v0.58.3

from terragrunt.

Hi @amontalban, Unfortunately, this is not enough information to understand the cause of the issue.

│ Could not retrieve the list of available versions for provider
│ hashicorp/aws: host registry.terraform.io rejected the given authentication
│ credentials

This error is in no way related to the number of concurrent processes.
Please review how caching works https://terragrunt.gruntwork.io/docs/features/provider-cache/
Most likely, the values, TF_TOKEN_*, are somehow overwritten, which is why you cannot connect to the cache server and get rejected the given authentication credentials.

from terragrunt.

I would suggest also checking the credentials file.

from terragrunt.

Hi @levkohimins,

Thanks for looking this issue, I have built a test case for easier testing.

You can find it here: https://github.com/amontalban/terragrunt-issue-3076

If you set your machine with those requirements (Mainly GNU Parallel, Docker and the specified Terraform/Terragrunt versions) then if you run make parallel you will see the error I specified in the report, on the other side if you run make sequentially it will work properly because it is executed one at a time.

Let me know if that helps or you need anything else.

Thanks once again!

from terragrunt.

@levkohimins after building this test case and running some tests, I think you were on the right track.

The problem is that when I run multiple parallel plans the generated token which is an UUID will be different for each of the processes and therefore causing this authentication issue.

If I set the TERRAGRUNT_PROVIDER_CACHE_TOKEN env var or --terragrunt-provider-cache-token to be the same for all processes then this works in my test case.

I'm going to test it out in Atlantis to confirm and get back to you to confirm that solves my problem.

from terragrunt.

Hi @amontalban @nsb413, the issue should be resolved in v0.57.9 release, could you please confirm this?

from terragrunt.

Still an issue for me.

TG: 0.57.9
TF: 1.5.6

TERRAGRUNT_PROVIDER_CACHE_DIR="somepath"
TERRAGRUNT_PROVIDER_CACHE="true"
TF_PLUGIN_CACHE_DIR is not set
TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE should be set automatically based on the caching docs.

Might be some different problem, because cache is there after the first run, but during the second run on terrgarunt run-all init -upgrade I can see that provides been installed again even though that caching server is running from what I can see in logs.

from terragrunt.

@sshepel, I could not reproduce the issue.

I think the init logs are missleading (so it's working). Because with terragrunt cache enabled I can see this:

Initializing provider plugins...
- Finding latest version of hashicorp/template...
....
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (unauthenticated)

Where as with TF_PLUGIN_CACHE_DIR var set I saw something like: used from cache for the same provider plugin.
But I have checked the .terraform folder under .terragrunt-cache for specific module after the init step with cache and I can see that the binaries for provider plugins are linked to the cached folder as mentioned in Provider Caching docs.

Also there is a message about incomplete lock file information for providers (which seems to be fine, but very annoying).

from terragrunt.

Hi @amontalban, could you please confirm that the issue is resolved?

from terragrunt.

Hi @norman-zon, your issue is different, not connected to this one, this is already resolved. Please created a new one with the full log --terragrunt-log-level debug.

from terragrunt.

I'm closing the issue since it seems to be resolved. Feel free to reopen if this is not the case.

from terragrunt.