Giter VIP home page Giter VIP logo

external-secrets / external-secrets Goto Github PK

View Code? Open in Web Editor NEW
4.0K 45.0 720.0 113.21 MB

External Secrets Operator reads information from a third-party service like AWS Secrets Manager and automatically injects the values as Kubernetes Secrets.

Home Page: https://external-secrets.io/main

License: Apache License 2.0

Dockerfile 0.13% Makefile 0.67% Go 97.24% Smarty 0.36% Shell 0.46% HCL 1.01% HTML 0.01% Python 0.12%
external-secrets kubernetes kubernetes-secrets secrets-manager hacktoberfest

external-secrets's Introduction

external-secrets

External Secrets

ci CII Best Practices OpenSSF Scorecard Go Report Card FOSSA Status Artifact Hub operatorhub.io

External Secrets Operator is a Kubernetes operator that integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, IBM Cloud Secrets Manager, Akeyless, CyberArk Conjur, Pulumi ESC and many more. The operator reads information from external APIs and automatically injects the values into a Kubernetes Secret.

Multiple people and organizations are joining efforts to create a single External Secrets solution based on existing projects. If you are curious about the origins of this project, check out this issue and this PR.

Documentation

External Secrets Operator guides and reference documentation is available at external-secrets.io. Also see our stability and support policy.

Contributing

We welcome and encourage contributions to this project! Please read the Developer and Contribution process guides. Also make sure to check the Code of Conduct and adhere to its guidelines.

Sponsoring

Please consider sponsoring this project, there are many ways you can help us with: engineering time, providing infrastructure, donating money, etc. We are open to cooperations, feel free to approach as and we discuss how this could look like. We can keep your contribution anonymized if that's required (depending on the type of contribution), and anonymous donations are possible inside Opencollective.

Bi-weekly Development Meeting

We host our development meeting every odd wednesday on Jitsi. We run the meeting with alternating times 8:00 PM Berlin Time and 1:00 PM Berlin Time, we'll announce the time in our Kubernetes Slack channel. Meeting notes are recorded on hackmd.

Anyone is welcome to join. Feel free to ask questions, request feedback, raise awareness for an issue, or just say hi. ;)

Security

Please report vulnerabilities by email to [email protected]. Also see our SECURITY.md file for details.

software bill of materials

We attach SBOM and provenance file to our GitHub release. Also, they are attached to container images.

Adopters

Please create a PR and add your company or project to our ADOPTERS.md file if you are using our project!

Roadmap

You can find the roadmap in our documentation: https://external-secrets.io/main/contributing/roadmap/

Kicked off by

Sponsored by

License

FOSSA Status

external-secrets's People

Contributors

adustyoldmuffin avatar albertollamaso avatar andreyzamyslov avatar arthurpaivat avatar atzedevries avatar dependabot[bot] avatar domizei385 avatar eladgabay avatar elsachelala avatar eso-service-account-app[bot] avatar gusfcarvalho avatar haf-tech avatar hydeenoble avatar jabray5 avatar kiantigger avatar kinyat avatar knelasevero avatar mcavoyk avatar moolen avatar omateusp avatar paul-the-alien[bot] avatar renanaakeyless avatar ricardotorresdacosta avatar rodrmartinez avatar sebagomez avatar shanti-g avatar shuheiktgw avatar skarlso avatar vladarts avatar william-young-97 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

external-secrets's Issues

Consider using Paul, Derek or Prow

Describe the solution you'd like
I think we could have some kind of automation bot.

What is the added value?
Help with some managerial tasks. Specially useful if we want some approvers and maintainers not necessarily added to the org.

Give us examples of the outcome

Wanna vouch for Paul here. We use it in some internal projects, and it was developed by a fellow CSer. My favorite feature being the ability to label a PR with 'merge', when you kinda know that the CI will pass with the last commit that you sent, and you don't want to sit in front of the laptop waiting for it to click merge. Paul will merge PRs labelled with 'merge' in the next hour if CI passed. It is also very simple to configure (and contribute to, if we need something specific). I think Prow is too complicated for such a small project like ESO. Derek is also an option.

Paul: https://github.com/apps/paul-the-alien
Derek: https://github.com/alexellis/derek
Prow: https://github.com/kubernetes/test-infra/tree/master/prow

Metrics first iteration

As a Cluster Operator i need metrics/alerts to ensure ESO runs smoothly, secrets are beings synced and SecretStores are properly configured

AC:

Vault Provider does not support cluster-scoped secret store

Describe the solution you'd like
The vault provider implementation should support providing data to external secrets from the cluster secret store resource. It seems that the initial implementation stubbed this out and did not implement the cluster-scoped lookup.

What is the added value?
Cluster secret stores will be able to provide secrets in an identical manner as namespaced secret stores.

Use TokenRequest API to fetch service account tokens

Describe the solution you'd like
Use the TokenRequest API to fetch tokens used for authentication on demand.

What is the added value?
The TokenRequest API makes it possible to fetch tokens with a configurable TTL and audience so that they can be used for a specific purpose (such as authenticating with Vault with JWT auth or AWS via IRSA) and are only good for a short period of time before they expire. Single purpose, short lived tokens are better for security.

The service account token that can currently be used may never expire and is intended for use with the Kubernetes API itself. Using it with other services increases the chance that it may leak and be used against the cluster.

Give us examples of the outcome

This would extend the vault provider's kubernetes auth property in SecretStore and ClusterSecretStore and add a new property named tokenRequestAPI. This object would include the TokenRequestSpec object content. It would also need to include a serviceAccountName property. ClusterSecretStore would also need to include a namespace property.

The same new property would be added directly to the the aws provider's auth object. This would enable the use of IRSA when authenticating with AWS since IRSA depends on using dedicated tokens.

Observations (Constraints, Context, etc):

The following YAML can be sent to the k8s API to receive a token:

apiVersion: authentication.k8s.io/v1
kind: TokenRequest
spec:
  audiences:
    - bar
    - baz
  expirationSeconds: 630

This will request a token with a TTL of 10 minutes and 30 seconds with the audience list set to bar and baz. A token created for the demo service account in the default namespace can be retrieved by sending that content to the API server at api/v1/namespaces/default/serviceaccounts/foo/token. This can be demonstrated with curl as follows assuming you have already run kubectl proxy and that the YAML has bee saved to token-request.yaml:

curl -X POST localhost:8001/api/v1/namespaces/default/serviceaccounts/foo/token -H 'Content-Type: application/yaml; charset=utf-8' --data-binary @token-request.yaml

The only real limitation I have found so far is that the minimum TTL for tokens is 10 minutes. The API enforces this, and I haven't found away to change that yet.

This requires the TokenRequest feature gate to be enabled (on by default since Kubernetes 1.12, and available since 1.10). The following options for the kube-apiserver also need to be specified:

  • --service-account-signing-key-file
  • --service-account-key-file
  • --service-account-issuer
  • --service-account-api-audiences

These are apparently enabled by default in EKS. ClusterAPI also enables them by default.

template functions

Once #8 is in we need some basic templating functions, e.g.:

  • extract certificates/keys from pkcs12 (#22)
  • basic transformations (e.g. via sprig)

Start with tweaks approach for test cases structs before we have more tests

Describe the solution you'd like
We could have a base test case struct with most of the default happy path fields set at the top, and use tweak anonymous functions to customize the test case for each specific desired scenario.

What is the added value?
Sometimes we have to include a lot of fields set, for validation or because it is a required field, and when reading tests you have to really look closer to find what is changing between each case. With tweaks changing exactly what is needed for each scenario it is easier to quickly see what the test is about, and what is relevant to the test.

Give us examples of the outcome

My wife was grabbing some clean up tasks from k/k and she is doing exactly that for some tests. Saw Moritz starting documentation early in the project and imagined this being another thing worth doing before we have big test suites.

Here is the reference issue on the k/k: kubernetes/kubernetes#99005

Her PR with tweaks and stuff: kubernetes/kubernetes#99165

Obs:

I think this is a umbrela issue and we can let people grab specific tests, or specific test files:

  • TestGetSecret in pkg/provider/aws/secretsmanager/secretsmanager_test.go (@gabibeyer)
  • TestGetSecretMap in pkg/provider/aws/secretsmanager/secretsmanager_test.go

People that want to do this can comment in this issue, and start working on it. I will add other ones to this list that have multiple case scenarios (anyone can just suggest any as well, and I update the issue here).

Local Development Guidelines Update

Hi there,

This is not an issue, more like an idea to contribute to the Development Guidelines. I'm using these commands to get the operator up and running in a Kind cluster locally (if you want to test some K8S integration that needs more than make run).

# get cluster up and running
kind create cluster
# does everything else
export TAG=v2
export IMAGE=eso-local

GOOS=linux make build
docker build . -t $IMAGE:$TAG
kind load docker-image $IMAGE:$TAG
make helm.generate
helm upgrade --install external-secrets ./deploy/charts/external-secrets/ --set image.repository=$IMAGE --set image.tag=$TAG

If need a new version, I just update TAG and run the last script all over again.

What do you think about adding this info to the contribution guidelines?

Implement the Providers interface

Expected Behavior

Have an interface to setup all the Providers (AWS SM, GCP SM, Vault etc).

We should have a common interface with the methods used by all Providers, something like:

type Client interface {
	New(...) (Client, error)
	GetSecret(...) ([]byte, error)
	GetSecretMap(...) (map[string][]byte, error)
}

We should also have some form of registration of the implemented backends during the controller startup.

Actual Behavior

Not implemented yet.

Separate the SecretStore Provider into two interfaces

Currently the Provider interface has functionality for New(...), and GetSecret(...). This ends up being a bit confused as the controller gets a Provider from the internal schema, but then calls New() on the Provider to get.. another Provider

Provider's New(...) then has to be careful not to mutate itself, but to return a New instantiation of its own type, if a Provider failed to do so, the mutation may have undefined side-effects on future ExternalSecrets using the same provider during the runtime of the controller.

Proposal

type Provider interface {
 	// New returns a SecretsClient for the given store definition
	New(...) (SecretsClient, error)
}

type SecretsClient interface {
	// GetSecret returns a single secret from the provider
	GetSecret(...) ([]byte, error)

	// GetSecretMap returns multiple k/v pairs from the provider
	GetSecretMap(...) (map[string][]byte, error)
}

Besides the increased clarity for the controller using this interface, this also simplifies the use-case for the same cloud provider having multiple SecretsClient discussed in #27 (comment). AWS SecretsManager and AWS Parameter Store could share the same Provider interface, and based on the Service type field - return different SecretsClient.

External Secret default fields do not work as described in documentation

Describe the solution you'd like
Default fields should not be required to be explicitly supplied by users creating resources.

Observations (Constraints, Context, etc):

For example:

$ kubectl explain secretstore.spec.provider.vault

FIELDS:
...
   path <string> -required-
     Path is the mount path of the Vault KV backend endpoint, e.g: "secret". The
     v2 KV secret engine version specific "/data" path suffix for fetching
     secrets from Vault is optional and will be appended if not present in
     specified path.
...
   version	<string> -required-
     Version is the Vault KV secret engine version. This can be either "v1" or
     "v2". Version defaults to "v2".

But when creating a secret store resource, without specifying the vault version or secret path mount :

apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: example-secretstore
spec:
  provider:
    vault:
      server: "https://vault.example.com"
#      version: v2
#      path: secret
      auth:
        kubernetes:
          mountPath: "kubernetes/example-cluster"
          serviceAccountRef:
            name: "default"
          role: "default"

kubectl prevents creating this resource:

$ kubectl apply -f /tmp/ss.yaml

error: error validating "/tmp/ss.yaml": error validating data: [ValidationError(SecretStore.spec.provider.vault): missing required field "path" in io.external-secrets.v1alpha1.SecretStore.spec.provider.vault, ValidationError(SecretStore.spec.provider.vault): missing required field "version" in io.external-secrets.v1alpha1.SecretStore.spec.provider.vault]; if you choose to ignore these errors, turn validation off with --validate=false

1password.com support would be usable by a lot

Describe the solution you'd like
external-secrets should allow for mapping to passwords from ones store on 1password.com

What is the added value?
Supporting getting passwords from an ordinary password manager used by many, such as 1password.com - would mean you could simply map externalsecrets to the place you already keep your password - instead of having to move all passwords, or duplicating them.

I know a LOT of people who use 1password.com for its simplicity.. I personally don't - but know both companies and friends who do, and who need a better solution than the current sealed-secrets - which leads to secrets duplication (as they still store them in either in their passwordstore.org personal git repo, or 1password.com account).

Fix helm chart install

The helm chart install as documented here is failing with this error
Installing the chart as documented here fails with the below error

`% helm repo add external-secrets https://external-secrets.github.io/external-secrets

Error: looks like "https://external-secrets.github.io/external-secrets" is not a valid chart repository or cannot be reached: failed to fetch https://external-secrets.github.io/external-secrets/index.yaml : 404 Not Found`

Tried installing the chart directly from the chart source and ran into the below with default values

Failed to pull image "ghcr.io/external-secrets/external-secrets:0.1.0": rpc error: code = Unknown desc = Error reading manifest 0.1.0 in ghcr.io/external-secrets/external-secrets: name unknown

passwordstore.org (ie. gpg files from git url) integration?

Describe the solution you'd like
It would be fantastic if I could just point external-secrets to my pass (see https://passwordstore.org) git repo - wherever I host it.
pass uses GPG keys to encrypt (and decrypt keys) - and allows for multiple recipients (using gpg).

What is the added value?
This would allow me to store secrets in a git repo using tools that I already know (and that are well developed), and I could then simply add a kubernetes cluster GPG key to "the files/secrets" I want it to be able to decrypt.. and they would be decryptable by that recipient.

This allows me to SHARE a secret amongst multiple recipients (clusters and actual people for backup/recovery reasons) - and since its GPG keys - our personal keys can be stored on a yubikey (so they can't be stolen) - AND if one wanted to - one could also use a yubikey on a server - for storing the gpg key for a k8s deployment - making it impossible to steal the decryption key.

CLA signing from PR

Describe the solution you'd like
Have a way to easily sign a CLA from a PR.

What is the added value?
Avoiding legal problems regarding code ownership when PRs are up but not merged. Also like this we follow CNCF standard.

Implement the `ClusterSecretStore`

Expected Behavior

We need a cluster version of the SecretStore CRD. With the implementation, we might to add its spec on the crd-spec repositoty.

Actual Behavior

Not implemented.

Remove kubebuilder kustomize manifests

With a helm chart available, the kustomize manifest is just unnecessary config in the repo.

My suggestion would to follow similar route as cert-manager's deployment manifests if we want to publish manifest artifacts with Github releases. This uses a default helm values file and helm template to generate the yaml to kubectl apply.

documentation first iteration

We need a static site for our documentation. For the first iteration it should be enough to have a basic scaffolding without (much) content.

AC:

  • basic static site generator pipeline with github actions (e.g. hugo)
  • basic scaffolding for:
    • User Guide: Getting Started / Installation / use-cases / Configuration
    • Design: Security
    • Contribution Guidelines / development setup

Provider: AWS Secrets Manager

Implement AWS Secrets Manager Provider

AC:

  • SecretStore should expose AWS Secrets Manager as a backend
  • Authentication mechanisms implemented: pull credentials from secret, use ambient credentials
  • implement assumeRole mechanics

External Secrets templates use []byte instead of string.

Describe the solution you'd like
External Secret templates require the user specify the template contents in []byte format, which means all templates have to be base64'd by hand by the user.

Give us examples of the outcome

External Secrets should support alternate methods of configuring templates (e.g stringData) or use strings directly instead of []byte.

Add helm chart

Describe the solution you'd like
Add Helm chart to install external-secrets controller.

Fix flaking tests

Describe the solution you'd like
We have some tests failing because of timeout or some other flaky reasons, we should have a look an fix.

Vault LDAP auth method

First of all - thanks to you for the great product!

Describe the solution you'd like
Possibility to get a Vault token using LDAP method.
https://www.vaultproject.io/docs/auth/ldap

What is the added value?
In case of short-living token usage an additional controller required to rotate the token.

Give us examples of the outcome

Add ldap section to vault auth section.

What do you think about such feature? I am ready to implement it by myself and send a Pull Request.

Thanks!

Add support for Vault k/v store

Describe the solution you'd like
SecretStore resources should support Vault as a backend. Authentication should support at least AppRole and Kubernetes Auth.

Security of SecretStore provider credential secrets resources

In our last community meeting we talked briefly about the security concerns on the SecretStore resources, since it has access keys to different Providers.

We talked about keeping SecretStores in the same namespace as the controller by default, or we can create a different namespace to manage it.

What do you think? What are the best practices around managing custom resources with sensitive information?

Cheers ๐Ÿ‘‹

External Secret status is incorrect when previous errors occurred reconciling the resource

Describe the solution you'd like
When an external secret is created and has an error within the resource (e.g no secret store exists), then that error case is resolved (by creating a secret store), and additional errors occur, the status of the external secret never updates to newer failed status conditions.

Observations (Constraints, Context, etc):

My initial speculation is that one of the conditions in SetExternalSecretCondition is incorrect, which prevents a status update:

unc SetExternalSecretCondition(status *esv1alpha1.ExternalSecretStatus, condition esv1alpha1.ExternalSecretStatusCondition) {
	currentCond := GetExternalSecretCondition(*status, condition.Type)
	if currentCond != nil && currentCond.Status == condition.Status && currentCond.Reason == condition.Reason {
		return
	}
	// Do not update lastTransitionTime if the status of the condition doesn't change.
	if currentCond != nil && currentCond.Status == condition.Status {
		condition.LastTransitionTime = currentCond.LastTransitionTime
	}
	status.Conditions = append(filterOutCondition(status.Conditions, condition.Type), condition)
}

Unclear which of these and what the correct fix would be though.

e2e tests

We will integrate a lot of external APIs and we need to test them end-to-end. Ideally we test the "real" upstream APIs rather than fakes, however this will cost a few cents.

AC:

  • find sponsors for AWS, Azure & GCP
  • consider running HasiCorp's Vault Cluster using an IaaS Provider in kind

With AWS Secrets Manager backend, make one AWS API call per secret even when not using dataFrom

(I had initially posted this the Kubernetes External Secrets project as external-secrets/kubernetes-external-secrets#704 , but it was pointed out to me that it might more sense to talk about this here since this is where the project's moving.)

I'm using secretsManager as my backend for some k8s secrets; my ExternalSecret object looks a bit like this.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: secretsManager
  data:
    - key: /path/to/secret
      name: username
      property: username
    - key: /path/to/secret
      name: password
      property: password`

Notice that I have one Secrets Manager secret, called /path/to/secret, that I'm getting two values from (username and password) and surfacing as two different pieces of data in my secret.

This works, but I can see through CloudTrail that the External Secrets pod makes two API requests to Secrets Manager (one per item in the data list). I'm wondering if there's a way to get I can make it get both values in just one API call, since I imagine the response it's getting from the first call also has the values needed for the second. I'd like to minimize the number of API calls I make to AWS to avoid getting throttled.

I understand that dataFrom would solve my problem, but I'm hoping to avoid using that if I can; we'd like to be explicit about what secrets we're including in our applications, and dataFrom doesn't make it clear when looking at the YAML which values I'm bringing into my application. I'd be just as happy with an option to explicitly call out which keys I'm getting from dataFrom.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: secretsManager
  dataFromSubset:
    secret: /path/to/secret
    keys:
      - username
      - password

Where "dataFromSubset" acted just like "dataFrom", but only surfaced the keys specifically called out here.

Implement a basic CI pipeline

Expected Behavior

We should have a pipeline to test the code and deploy the Docker image, respectively:

  • Run make test on every pull request/commit
  • Run make docker-build and make docker-push when tagging (using semantic versioning)
  • Implement code coverage (codecov.io)

Still need to decide:

  • Platform? I'd use GitHub actions, simple and already here
  • For the Docker image, maybe GitHub Container Registry (since DockerHub now has the pull limits)

Actual Behavior

There's no pipeline :(

Move CRD Specification into this repo

The CRD specification is currently in another repo https://github.com/external-secrets/crd-spec, this can make it difficult for members to find the specification or make changes.

Originally the CRD repo existed because there was no project for this operator but we wanted a place for discussion and design, but we should be able to move all that discussion to this project now.

Installation instructions from Getting Started not working

Describe the solution you'd like
I'd like to have working installation instructions from the Getting Started guide.

What is the added value?
Make a better initial developer experience.

The current command under Install the controller actually fails:

kubectl kustomize "github.com/external-secrets/external-secrets/config/default" \
| kubectl apply -f -
Failed to pull image "controller:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/controller:latest": failed to resolve reference "docker.io/library/controller:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

This is because the image name inside the config/manager/manager.yaml is controller:latest.

From the top of my head, we can fix this in some ways:

  • Change the config/manager/kustomization.yaml file to replace the image name with the latest version
  • Change directly the config/manager/manager.yaml to run the public latest image
  • Change the installation instructions to set an image version, like the etcd Operatior

What do you think? I can work on any of the changes โœจ

feat: Add mechanism to disable `externalSecret` metadata propagation

Describe the solution you'd like

Enable a mechanism to disable externalSecret label and annotation propagation (total or selective).

What is the added value?

In some scenarios, labels are used to determine ownership of resources, especially when they are managed from outside the cluster. For example, some GitOps operators, like ArgoCD, use labels to identify what resources are part of a particular ArgoCD application. Propagating the labels to the resources created by the externalSecret instance will transfer the ownership of the Secrets created to ArgoCD.

Give us examples of the outcome

When the externalSecret, created by the GitOps operator, propagates the labels, it adds metadata to the Secret resource that will end up with ArgoCD taking the "ownership" of the resource. In a GitOps scenario, that leads to the deletion of the resources, as is not present in the repository. That triggers the recreation of the secret by the external-secrets operator, and the loop goes on. Using proper ownerReferences is not viable as in multi-cluster scenarios, the GitOps operator manage the resources from an external cluster. This is one of the scenarios that we found, put probably there are other scenarios where this propagation may lead to an undesired behaviour.

Observations (Constraints, Context, etc):

Is not really tied to a particular version.

Helm chart does not include rbac for leader election

Describe the solution you'd like
The helm chart should also configure the RBAC necessary when leader election is enabled.

Currently, leader election is broken in the helm chart because the RBAC is missing access to coordination.k8s.io/leases

Vault Secret Store should allow specifying the service account name, instead of the SA token name.

Describe the solution you'd like
The vault secret store provider should allow specifying the kubernetes service account name, as an alternative to specifying the secret created for the service account. The service account secret token is an autogenerated name and discoverable via the service account name.it

What is the added value?
Secret stores are easier to configure and do not contain autogenerated secret references.

Give us examples of the outcome

apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: secretstore
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      auth:
        kubernetes:
          serviceAccount:
            name: "external-secrets-sa"
...

Cluster scoped ExternalSecret for creating secrets in multiple namespaces

Describe the solution you'd like

Imagine a cluster with 50+ or even more namespaces where we have to create a secret(eg. image pull secret ) using ExternalSecret. We will have to create same ExternalSecret in each namespaces. Currently we have 50+ external secrets for the same secret.

Adding a cluster scoped externalsecret with a field spec.namespaces which will accept a list of namespaces will solve this type of scenarios. One ExternalSecret create secret in all the listed namespaces.

What is the added value?
This will eliminate the need of creating ExternalSecrets in all namespaces for same secret.

Give us examples of the outcome

Example of ClusterExternalSecret

apiVersion: kubernetes-client.io/v1
kind: ClusterExternalSecret
metadata:
  name: docker-registry-internal
spec:
  backendType: secretsManager
  data:
  - key: arn:aws:secretsmanager:eu-central-1:75XXXXXXXX61:secret:secrets/nonprod-eu/docker-secret-internal
    name: .dockerconfigjson
  template:
    type: kubernetes.io/dockerconfigjson
  namespaces:
  - sandbox-app1
  - sandbox-app2
  - sandbox-app3
  - integration
  - demo

External Secrets w/templated keys are overwritten by remote keys of the same name

Describe the solution you'd like
External Secret resources that reference remote keys (e.g data or dataFrom) and then also define explicit template key-value pairs in the target secret are not templated. The remote keys replace the templated key contents. External Secrets can not contain templated keys that collide with remote keys, because the remote keys always win the merge.

What is the added value?
External Secrets can support materializing secrets with keys that contain templates with keys that exactly match the names of remote keys.

Give us examples of the outcome

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: foo-es
spec:
  data:
  - remoteRef:
      key: "projects/foo-project/conf"
      property: foo
    secretKey: foo
  refreshInterval: 4h
  secretStoreRef:
    kind: ClusterSecretStore
    name: foo-store
  target:
    name: foo-secret
    template:
      type: "Opaque"
      data:
        foo: aGVsbG8gd29ybGQhCg== ## 'hello world!'

I would expect a secret foo-secret resource with a single key, foo would be created, with the base64 decoded value "hello world!. Instead foo-secret exists, with the single key, foo, containing the contents of the secret from the cluster secret store. I would expect the template contents to overwrite the remote value in the final resource output.

Observations (Constraints, Context, etc):

This is trivial to work around when using data as you can rename keys, but dataFrom is less flexible, as it imports the contents of all the key-values under the path. Users that update their vault (or whatever backend) with new secret keys that collide with external secret templates will discover they have clobbered their templated secrets.

External Secrets can have surprising template behavior

Describe the solution you'd like
External Secrets templates do not handle edge cases well. For example, if the template {{ . | toString }} is supplied for any key, the resulting secret resource has that key, with the value of the template, unrendered.

Other invalid templates, for example {{ .nonexistent_key | toString }}, produce a different result. The secret is created, and the key for the template has a value of Error: provided key not found in secret after base64 decoding.

What is the added value?
Template errors should be visible on the status of the external secret, and the logs of the controller. I saw no error with this particular edge case in any of the controller logs or status fields.

Give us examples of the outcome

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: foo-es
spec:
  data:
  - remoteRef:
      key: "projects/foo-project/conf"
      property: foo
    secretKey: foo
  refreshInterval: 4h
  secretStoreRef:
    kind: ClusterSecretStore
    name: foo-store
  target:
    name: foo-secret
    template:
      type: "Opaque"
      data:
        bar: e3sgLiB8IHRvU3RyaW5nIH19Cg== ## '{{ . | toString }}'

Produced the following secret:

apiVersion: v1
data:
  foo: Zm9vCg==
  bar: e3sgLiB8IHRvU3RyaW5nIH19Cg==
kind: Secret
metadata:
  creationTimestamp: "2021-04-28T17:17:30Z"
  name: foo-secret
  ownerReferences:
  - apiVersion: external-secrets.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ExternalSecret
    name: test
    uid: 44246fa1-e28b-4ecc-a76e-b39212d387ec
  resourceVersion: "131974233"
  uid: 3ffa418e-3d3b-4bfc-a184-32c097207210
type: Opaque

Observations (Constraints, Context, etc):

No errors in the controller were logged. The sync status on the external secret resource was "SyncSuccess".

I'm aware that the template {{ .foo | toString }} would correctly render the secret contents, but I was surprised by the behavior with this example. I would have expected garbage secret data, or an error, or anything else except exposing the template contents unrendered.

Implement main controller reconciliation

Describe the solution you'd like
ExternalSecret controller should connect to the referenced SecretStore and fetch the secret material referenced by the ExternalSecrets resource. The controller should update the status of the ExternalSecret resource with the outcome of this reconciliation.

Trying to apply the chart to GKE/Autipilot results in error

This is what happens:

$ helm repo add external-secrets https://charts.external-secrets.io
"external-secrets" has been added to your repositories
$ helm install helm-chart-0.1.1 external-secrets/external-secrets
Error: chart requires kubeVersion: >= 1.11.0 which is incompatible with Kubernetes v1.18.16-gke.2100

Trying this in vanilla kubernetes works with no problems

Create Helm chart repository for this project.

There is no chart repository available for this project or organization. We should be able to use Github pages for this and then possibly setup a CNAME for charts.external-secrets.io.

Once setup the main branch can push canary charts to this repository and tags can push charts for our releases.

ParameterStore: intelligent updates

Parameter Store has API limits for different tiers.
We need a scheduling mechanism to distribute the API calls more evenly so we won't run into Limits if a user has a lot parameters to fetch.
Things that came to my mind:

  • some sort of jitter for retries in the Reconcile func on the controller level (i'd assume that this is implemented in controller-runtime)
  • some mechanic that throttles/defers API calls so we won't run into limits on the provider level

see: #27 (comment)
also: external-secrets/kubernetes-external-secrets#655

Provider: AWS Parameter Store

Implement AWS Parameter Store Provider

AC:

  • SecretStore should expose AWS Parameter Store as a backend
  • Authentication mechanisms implemented: pull credentials from secret, use ambient credentials

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.