`run-all init` fails with parallelism about terragrunt HOT 16 OPEN

@GMartinez-Sisti @HamoucheTF1

I had the same issue with Atlantis. I saw the error Required plugins are not installed during drift detection runs for 2-3 modules out of more than 300 overall modules. I discovered that errors may arise when Atlantis runs plans for two dependent modules in parallel. The error Required plugins are not installed occurs when Terragrunt tries to parse dependencies. It seems there is a race condition where one Terragrunt process generates the .terraform.lock.hcl file while another process tries to initialize the module to get outputs.

I resolved it by adding execution order groups for modules in the Atlantis config. Now, plans for dependent modules do not run in parallel, and the issue has been resolved. In the last ~30 consecutive drift detection runs, this error has not occurred. Previously, each run contained 2-3 failed runs with this error. Hope it helps 🙃

@levkohimins Thanks a lot for the great feature for providers caching! It sped up our CI by more than 3 times 🔥

from terragrunt.

Hi @ajax-ryzhyi-r ,

Thank you very much for your response and suggestion. By implementing execution order groups, I was able to resolve the errors with parallel plans. I haven't encountered any errors related to required plugins not being installed since making this change.

However, I am now facing a different issue. Specifically, I'm getting the following error related to the AWS TFLint plugin:

Failed to initialize plugins; fork/exec /path/to/.tflint.d/plugins/github.com/terraform-linters/tflint-ruleset-aws/0.30.0/tflint-ruleset-aws: text file busy Have you by any chance come across this error? Any insights you could share would be greatly appreciated.

Thank you in advance for your help!

We have the same issue with tflint plugins, but we use a cache for these plugins (enabled via the TFLINT_PLUGIN_DIR environment variable), so we only face this issue when we update the plugin versions. We haven't found a solution for tflint concurrency yet. We just fill the cache with the new plugin versions before updating the versions in the tflint configs. Actually, it's not related to terragrunt - it's a tflint concurrency issue 🥲

from terragrunt.

I already tried to reproduce it without success as it only occurs on projects with multiple different modules / configuration and doesn't occurs 100% of the time (seems to be around 70%).
I'll try to find a way to reproduce it though.

from terragrunt.

Thank you @ajax-ryzhyi-r!

Terragrunt's provider cache is safe when used in parallel with the same provider cache directory, but this is not true if multiple Terragrunt processes are running with the same working directory.

from terragrunt.

Hi @gnuletik, most likely you are using the Provider Plugin Cache. Make sure to read Provider Caching.

from terragrunt.

Hi @levkohimins, thanks for the feedback !

I was using the Terraform's provider plugin cache but I disabled it since I migrated to Terragrunt's Provider Caching Server.

I was able to reproduce this issue with:

• Terragrunt's provider caching
• without any caching mechanism enabled

Also, previous releases of Terragrunt didn't have this issue because the parallelism was disabled during init (because of the Terraform's provider plugin cache issue).

from terragrunt.

Hi @gnuletik,
It would be most effective if you make a minimal set of configuration files that will reproduce this issue and upload them to a public repository, because I cannot reproduce the issue from the data, you provided in the description. Thank you!

from terragrunt.

To clarify, before the implementation of Terragrunt Provider Caching, with Provider Plugin Cache enabled and running run-all init command, Terragrunt automatically set TERRAGRUNT_PARALLELISM=1, but with the advent of Terragrunt Provider Caching, this behavior was removed.

Therefore, I asked you to check whether this feautre is enabled in the Terraform CLI config or in the environment variable

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"

export TF_PLUGIN_CACHE_DIR="$HOME/.terraform.d/plugin-cache"

from terragrunt.

@levkohimins Yes I can confirm that this configuration is not enabled.
Also, this behavior has been reproduced both locally and on our CI.

from terragrunt.

Hi, I'm getting the same behavior while using Provider Plugin Cache with Atlantis.

Atlantis has ATLANTIS_PARALLEL_POOL_SIZE=4. I'm not using run-all though, just plain run. Without Provider Plugin Cache this issue doesn't happen, but each plan will pull the providers on it's own. I'm seeing that 25% of the plans will fail with the same Error: Required plugins are not installed error.

UPD: interesting that some of the errors happen while running terragrunt show. While using atlantis, we need to run terragrunt plan [...] -out=file.json and then atlantis needs the terraform compatible terraform show from that file to report the status.

from terragrunt.

Hi @GMartinez-Sisti, I would be happy to help you solve this issue, but I need a sample from you.
You could create it with docker-composer and put it in some repository, for example https://github.com/amontalban/terragrunt-issue-3076, once I can reproduce the issue locally I'll start working on a solution.

from terragrunt.

Hi @GMartinez-Sisti, I would be happy to help you solve this issue, but I need a sample from you. You could create it with docker-composer and put it in some repository, for example https://github.com/amontalban/terragrunt-issue-3076, once I can reproduce the issue locally I'll start working on a solution.

This is a great starting point, I might be able to make it happen just by forking this repo and enabling the Provider Plugin Cache. Will let you know!

from terragrunt.

Hi,

I wanted to ask if there have been any updates regarding this issue. I'm experiencing the same problem when running terragrunt plan in parallel with the Terragrunt provider cache enabled.

Any information or progress on this would be greatly appreciated.

Thank you!

from terragrunt.

To provide more context, I am working on implementing Terragrunt drift detection. As part of this process, I run multiple terragrunt plan commands in parallel, which seems to trigger the error described in this issue.

from terragrunt.

Hi @ajax-ryzhyi-r ,

Thank you very much for your response and suggestion. By implementing execution order groups, I was able to resolve the errors with parallel plans. I haven't encountered any errors related to required plugins not being installed since making this change.

However, I am now facing a different issue. Specifically, I'm getting the following error related to the AWS TFLint plugin:

Failed to initialize plugins; fork/exec /path/to/.tflint.d/plugins/github.com/terraform-linters/tflint-ruleset-aws/0.30.0/tflint-ruleset-aws: text file busy
Have you by any chance come across this error? Any insights you could share would be greatly appreciated.

Thank you in advance for your help!

from terragrunt.

Hi @ajax-ryzhyi-r ,
Thank you very much for your response and suggestion. By implementing execution order groups, I was able to resolve the errors with parallel plans. I haven't encountered any errors related to required plugins not being installed since making this change.
However, I am now facing a different issue. Specifically, I'm getting the following error related to the AWS TFLint plugin:
Failed to initialize plugins; fork/exec /path/to/.tflint.d/plugins/github.com/terraform-linters/tflint-ruleset-aws/0.30.0/tflint-ruleset-aws: text file busy Have you by any chance come across this error? Any insights you could share would be greatly appreciated.
Thank you in advance for your help!

We have the same issue with tflint plugins, but we use a cache for these plugins (enabled via the TFLINT_PLUGIN_DIR environment variable), so we only face this issue when we update the plugin versions. We haven't found a solution for tflint concurrency yet. We just fill the cache with the new plugin versions before updating the versions in the tflint configs. Actually, it's not related to terragrunt - it's a tflint concurrency issue 🥲

Thank you @ajax-ryzhyi-r

from terragrunt.