Injection secrets as ENV vars? about vault-k8s HOT 45 OPEN

Vault Agent currently doesn't support something this, so it's unlikely this kind of feature would be available for some time. We can definitely see the value in supporting environment variables and hope to support it some day, if we can figure out a way that:

* Doesn't expose secrets in pod metadata

* the injector doesn't fetch secrets for the pod - having a service that impersonates an application is a security concern

* Doesn't mutate the original pod containers beyond mounting a volume.

One idea is to generate an envrc file, put it in the secret mount and have the user modify their container or entrypoint to source the file at startup.

Open to ideas here!

Hi @jasonodonnell.
I think as many other people it will be good if we can inject as env vars.
I will try to implement that in vault-k8s, I will try to update you guys.

from vault-k8s.

FYI for those commenting +1, you can just subscribe to the thread for updates without commenting. If you want to add your clout then give a thumbs-up to the OP. It's rather annoying to keep getting notifications about this thread thinking it's something useful that turns out to just be another "+1" without anything else to contribute. Sorry in advance to those that got this notification, I'm aware I'm not really adding anything of value here other than protecting my own sanity.

from vault-k8s.

+1

from vault-k8s.

@povils yeah exactly! I didn't want to mention banzai but this is exactly what I was had in mind:

env:
- name: AWS_SECRET_ACCESS_KEY
  value: "vault:secret/data/accounts/aws#AWS_SECRET_ACCESS_KEY"

from vault-k8s.

Yeah it would be wonderful to have an option of how we want to inject these variables. Specially when you are running apps where you have no control of them and they are using envs. This is how Banzai bank-vaults solves env injection: https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/

Of course secret rotation with envs would be tricky

from vault-k8s.

+1

from vault-k8s.

+1 for this feature! Also in combination with helm charts it would be nice to prepare an environment with secrets in env for a helm install.

from vault-k8s.

Hi all, just wanted to give an update on this. We've recently released the vault-csi-provider project, which supports syncing Vault secrets to K8s secrets. This means that project supports environment variables. This would be an alternative to the Vault Agent Injector and we would love some input on this new project.

As for environment variable support in the agent injector, we've explored this quite a bit on our end, but no solutions have been found that would not include injecting a binary (such as consul-env) and overriding an entry point.

One possible solution is a sync init container that takes the value from the filesystem and populates it into K8s. While this does work, we're hesitant to publish secrets to K8s with the Vault Agent Injector. CSI supports that out of the box so we would likely recommend that technology.

We're still looking for different ways to support envs in the injector based on some criteria listed above: #14 (comment).

Happy to explore any other ideas you might have!

from vault-k8s.

Using this approach:

{{- with secret "internal/data/development/api" -}}
{{- range $key, $value := .Data.data }}
export {{ $key }}={{ $value }}
{{- end }}
{{- end -}}

command: ["/bin/sh", "-c"]
args:
[ "for f in /vault/secrets/*.env; do source $f; done && npm start" ]

from vault-k8s.

Vault Agent currently doesn't support something this, so it's unlikely this kind of feature would be available for some time. We can definitely see the value in supporting environment variables and hope to support it some day, if we can figure out a way that:

• Doesn't expose secrets in pod metadata
• the injector doesn't fetch secrets for the pod - having a service that impersonates an application is a security concern
• Doesn't mutate the original pod containers beyond mounting a volume.

One idea is to generate an envrc file, put it in the secret mount and have the user modify their container or entrypoint to source the file at startup.

Open to ideas here!

from vault-k8s.

We have worked around this issue by having the template write to a .env file. Then in our entrypoint we just source that file (if exists) before running the main command.

This allows the vars to not be in the kubernetes metadata and still allows for customization for other kinds of outputs.

from vault-k8s.

Hi @tehmoon, there is this solution (External Secrets CRD) to updates vault secrets with k8s Secrets https://github.com/godaddy/kubernetes-external-secrets.

I would like to know how to configure the vault to rotate secrets and restart the pod automatically.

I solved the question above changing the kv engine to version 2.

from vault-k8s.

Hi folks, another update on this. We recently released a beta of vault-secrets-operator that will sync Vault secrets to Kubernetes secrets, which can then be used as environment variables natively: [blog] [docs]

from vault-k8s.

hello!

Any updates on this feature? to vault in kubernetes

from vault-k8s.

You can achieve the same by using ENV/BASH_ENV shell environment variable without hacking the entrypoint.
https://stackoverflow.com/a/78056940/10215243

from vault-k8s.

Hi,

I'm currently using this azure-key-vault-to-kubernetes solution but I want to move over to Hashi's Vault. In terms of implementation here, perhaps it would be pertinent to have a look at how that solution works first? I'm not saying you won't find issues or things you don't like with that solution, just that it might be a "leg up" in thinking about it.

An example feature that I like about that solution is the ability to take JSON or YAML data in secrets (as long as the depth is 1) and return a Kubernetes secret with multiple key/value pairs.

I am really keen on a 12factor solution using environment variables for injecting these secrets.

Also they use a mutating admission controller to automatically override the entrypoint to pull in the secret and set up the env var prior to running the original entrypoint/CMD. This is fantastic, but in some ways a curse - as it has to inspect those images in order to do that and on larger images it can take a while to pull the image to inspect it.

Anyway - just thought I'd point this solution out in case it helps with patterns, etc.

from vault-k8s.

https://www.vaultproject.io/docs/platform/k8s/injector/examples#environment-variable-example

This seems to suggest vault-agent can do it? But I can't get it to work.

Edit: My bad I didn't realised that it isn't the injection that creates the environment variable, the script in the arg creates them... Tunnel visioned

from vault-k8s.

I'd like to contribute. So any heads up? @g-sponda @alvin-huang

from vault-k8s.

https://www.vaultproject.io/docs/platform/k8s/injector/examples#environment-variable-example

This seems to suggest vault-agent can do it? But I can't get it to work.

Edit: My bad I didn't realised that it isn't the injection that creates the environment variable, the script in the arg creates them... Tunnel visioned

For me looks very ugly, need to redefine command/args parameters with shellizm, but what happened if these parameters are not supported in operator's CDR resources?

from vault-k8s.

I dream of having the same opportunity as 1password

https://developer.1password.com/docs/connect/k8s-injector

from vault-k8s.

You should check out Infisical which provides this functionality.

from vault-k8s.

Hi.

Any updates on this feature?

from vault-k8s.

Definitely a must to avoid the headache of refactoring the apps.
Let us know how it go @g-sponda ! You rock
Cheers

from vault-k8s.

Also very interested in this too as it eliminates the need for sealed secrets and / or helm + vault-secrets-operator

from vault-k8s.

@JnMik I am reading the code of banzaicloud and the vault-k8s to try to implement vaultenv for vault-k8s, but It's been more complex than I thought.

from vault-k8s.

Hi there! Any updates ? Really need this feature!

from vault-k8s.

@mitchellmaler yes I know this way, but store values in memory more security solution. It would be great if some one can implement that in vault-k8s

from vault-k8s.

+10086 same problem.

from vault-k8s.

+1

from vault-k8s.

+1

from vault-k8s.

First, just want to say thank you for the hard work, vault is an amazing product.
Along with others, I've also been myself trying to think of a workflow and don't want to have to rewrite a bunch of entrypoint, arg, command for x containers. I love the idea of the template and injection, but it is not as practical if the app itself is not aware of vault.

I know what people are going to say, but I feel like there could be a way to make it work with native kubernetes secrets. I am still new to this world and security focus, but I think that kubernetes secrets are here to stay, there are some tooling/deployments/templates that require and make use of a secret.

One thing that I'm thinking about is having that sidecar vault injector that synchronize a vault template to a kubernetes secret. There are lifecycles that will restart the pod in case a secret is change (I think).
No matter what, I think that kubernetes secrets need to be taken into the security scope, the good thing about vault is that it's taking care of the rotation, and rotation is more important than storage if you had to choose one or the other. One could think of a daily secret rotation that restarts pods once a day (or even shorter).

Let me know what you all think.

from vault-k8s.

So the "config" script in the arg - that's fine if you ask me. It's the same as the solution I posted really, except more explicit - you could argue that's better. The solution I posted hides the implementation away a little and uses a mutating admission controller to effectively alter the YAML to look something like this Vault implementation.

I'd have to check a few things out (it is a consideration for us whether the contents of the secret land in k8s at all - e.g. as a native k8s secret, or a volume for example or if they are only exposed inside this container), but if we can get this to work it may just suffice. Thanks for giving me something to ponder though.

from vault-k8s.

https://stackoverflow.com/a/66057099/11952470

from vault-k8s.

Hello. I hope everybody is doing fine. Is this task planned or is it in any road map please ?

from vault-k8s.

I'd like to contribute. So any heads up? @g-sponda @alvin-huang

@AfrouzMashayekhi Unfortunately, I didn't make any progress on that issue.
It still open to someone work on it.

from vault-k8s.

Hi all, just wanted to give an update on this. We've recently released the vault-csi-provider project, which supports syncing Vault secrets to K8s secrets. This means that project supports environment variables. This would be an alternative to the Vault Agent Injector and we would love some input on this new project.

We use it already in production. Works very good for us!

from vault-k8s.

hi @jasonodonnell @eyenx

I can't make this work. For some reason, I get a 500 invalid iss claim back from vault when csi-provider tries to log in.

Opened an issue here: hashicorp/vault-csi-provider#87

Could you guys please take a look on the config?

from vault-k8s.

hi @jasonodonnell @eyenx

I can't make this work. For some reason, I get a 500 invalid iss claim back from vault when csi-provider tries to log in.

Opened an issue here: hashicorp/vault-csi-provider#87

Could you guys please take a look on the config?

Hi, you need to set the issuer correctly when creating the vault-auth binding (auth/kubernetes) as done here:

https://github.com/hashicorp/vault-csi-provider/blob/8f370e05cdc23007e93143f36a905aed8ca16f58/test/bats/provider.bats#L37

from vault-k8s.

hi @jasonodonnell @eyenx
I can't make this work. For some reason, I get a 500 invalid iss claim back from vault when csi-provider tries to log in.

Opened an issue here: hashicorp/vault-csi-provider#87
Could you guys please take a look on the config?

Hi, you need to set the issuer correctly when creating the vault-auth binding (auth/kubernetes) as done here:

https://github.com/hashicorp/vault-csi-provider/blob/8f370e05cdc23007e93143f36a905aed8ca16f58/test/bats/provider.bats#L37

You will most likely need to specify the issuer parameter as the CSI driver seems to fetch the issuer from a different AIP endpoint.

from vault-k8s.

In case it helps, for AKS (azure) clusters the issuer turned out to be the external API for the cluster.
So we were able to run
export TF_VAR_csi_issuer=$(yq e -j ~/.kube/config | jq '.clusters[] | select(.name | contains("NameOFCluster")) | .cluster.server' | sed 's#https://##' | sed 's#:443##' | sed 's#"#"#g')
which evaluated to ahash.hcp.uksouth.azmk8s.io

from vault-k8s.

Using this approach:

image: jweissig/app:0.0.1
command: ["/bin/sh", "-c"]
args: [". /vault/secrets/config && env >> /app/secret.txt && sleep 600"]

I was able to set up the environment variable. How can I use it in .NET application? As soon as I substitute jweissig/app:0.0.1 with my own image, image deployed successfully, but kills the assigned port's connection. Port forwarding doesn't work. What is wrong here?

from vault-k8s.

Using this approach:

{{- with secret "internal/data/development/api" -}} {{- range $key, $value := .Data.data }} export {{ $key }}={{ $value }} {{- end }} {{- end -}}

command: ["/bin/sh", "-c"] args: [ "for f in /vault/secrets/*.env; do source $f; done && npm start" ]

Or you can use this other approach:

command: ["/bin/sh", "-c"] args: [ "source <(cat /vault/secrets/*.env) && npm start" ]

from vault-k8s.

Using this approach:

image: jweissig/app:0.0.1 command: ["/bin/sh", "-c"] args: [". /vault/secrets/config && env >> /app/secret.txt && sleep 600"]

I was able to set up the environment variable. How can I use it in .NET application? As soon as I substitute jweissig/app:0.0.1 with my own image, image deployed successfully, but kills the assigned port's connection. Port forwarding doesn't work. What is wrong here?

Hello @sline-git
I couldn't have worked Vault agent injection for env var method with dotnet applications.
Were you able to fix the problem? If you were able to run, what is your ENTRYPOINT and args combination?

Thanks & Regards

from vault-k8s.

Hi @tvoran This is quite nice!

Are there any plans to allow directly injecting secrets into application's environment variables without involving k8s secrets? Currently we use another project for this purpose.

• https://github.com/banzaicloud/bank-vaults
• https://banzaicloud.com/blog/inject-secrets-into-pods-vault-revisited/

from vault-k8s.

Hi All,
After creating the secret in file and sourcing . after completion of all steps in entrypoint i want to delete the secrets file .. i updated rm -f /sw/secrets/vaultkeys in entry point script but it fails with
operation not permitted unable to remove the file.. any suggestions pls ?

rm: cannot remove '/sw/secrets/vaultkeys': operation not permitted

from vault-k8s.