cruise-automation / daytona Goto Github PK

I would have expected this to work, but it returns an error:

╰─ go install github.com/cruise-automation/[email protected]
go: downloading github.com/cruise-automation/daytona v1.2.2
go install: github.com/cruise-automation/[email protected]: module github.com/cruise-automation/[email protected] found, but does not contain package github.com/cruise-automation/daytona

Generally daytona seems to swallow a bunch of different errors when fetching secrets fails.
Here is an example I ran into recently, but there are other instances I ran into this:

use these environment variables:

        - name: VAULT_SECRET_DATABASE_URL
          value: my/valid/vault/path/DATABASE_URL
        - name: DAYTONA_SECRET_DESTINATION_DATABASE_URL
          value: /secrets/folder_that_does_not_yet_exist/DATABASE_URL

In this example I specified a destination folder that does not yet exist.
What I would expect is that daytona either:

• creates the target directory
• errors out with a message that the target directory does not exist and errors with a non-zero exit-code.

What daytona really prints is this:

DAYTONA - 2020/08/14 08:16:45 Starting secret fetch
DAYTONA - 2020/08/14 08:16:45 Certificate or private key output path is empty, will not attempt to get certificate

and exits with a 0 exit code.

So TLDR it simply skips that secret above without any indication/error which makes it very difficult to catch and debug these configuration errors.

I'm not sure what the last message "Certificate or private key output path is empty" means but I seem to get this error messages also in cases where the secret fetch actually works.

Is there any reason to have -entrypoint without -secret-env? Can -secret-env be removed and rolled into -entrypoint as the default behavior?

If you specify multiple secrets using multiple VAULT_SECRET_ definitions and use VAULT_SECRET_PATH without any DAYTONA_SECRET_DESTINATION_, the secrets all write to the same file, but sequentially, overwriting the file every time so that only the last written secret remains.

Also, if you specify VAULT_SECRET_PATH in combination with a DAYTONA_SECRET_DESTINATION_, it writes to both places, exhibiting the same overwrite behavior with multiple secrets. This means you can't write some secrets to a shared file and other secrets to explicit files at the same time.

Daytona logs to stdout with no option for leveled logging, GCP picks up these logs as errors, and we get paged for "service throwing errors!" when in fact it was just normal Daytona logs.

Can we configure Daytona to log in JSON format with a severity so platforms like GCP don't count it as error? It could be as easy as:

LOG_FORMAT=json (could be any format that the logger supports)
LOG_LEVEL_KEY=severity (can default to `level`, but GCP expects `severity`)

I would suggest ZeroLog but anything that can log in JSON format is good.

I would like to be able to define a secret or directory of secrets and have them written to disk in a hierarchy that matches the secret path hierarchy.

The obvious exception is for secrets that have both data and act as a directory, but i feel like those should be treated as exceptional and ignored unless specified explicitly.

Example:
VAULT_SECRET_PATH could point to a directory like secrets/application/projectx/envy/componentz and then DAYTONA_SECRET_PATH could point to a filesystem path like /home/vault/componentz. Then every secret in the secret path could be copied to a file in the filesystem path, recrusively. Secrets that are both directory and contain data could just be treated as a directory when written to the filesystem, requiring explicit secret/destination extraction, if desired.

This would make it just two env vars to extract a whole hierarchy of secrets, instead of requiring them all to be extracted with explicit secret and destination vars.

When following the docs to write a single secret it is written to use

VAULT_SECRET_THING=secret/whatever/thing
DAYTONA_SECRET_DESTINATION_THING=/tmp/top-secret
VAULT_VALUE_KEY_THING=api_key

If one would only like to get env variables and not write a secret to a file, one would expect to not set DAYTONA_SECRET_DESTINATION_THING, and set SECRET_ENV='true'.

However, that doesn't work because the condition that wraps the case to write a single secret expects a destination path:

I believe I found a bug in how daytona treats the names of environment variables when fetching individual secrets.

Consider I start daytona with those environment variables:

VAULT_SECRET_1: secret/application/myapp/service_account.json
DAYTONA_SECRET_DESTINATION_1: /secrets/service_account.json

What I would expect to happen is that daytona reads the key at secret/application/myapp/service_account.json and writes it's value to /secrets/service_account.json.

What actually happens is, well, basically nothing.
Daytona seems to silently skip reading / writing the key and the logs simply contain this:

DAYTONA - 2020/01/27 01:22:02 Starting secret fetch
DAYTONA - 2020/01/27 01:22:02 Certificate or private key output path is empty, will not attempt to get certificate

Through some extensive try-and-error I found that apparently daytona expects the suffix of the environment variable to match the last path of the vault secret path, in this case service_account.json.
In other words:

VAULT_SECRET_service_account.json: secret/application/myapp/service_account.json
DAYTONA_SECRET_DESTINATION_service_account.json: /secrets/service_account.json

works, but

VAULT_SECRET_1: secret/application/myapp/service_account.json
DAYTONA_SECRET_DESTINATION_1: /secrets/service_account.json

does not.

This is a bit surprising since the docs here https://github.com/cruise-automation/daytona#secret-fetching clearly mention

Any unique value can be appended to VAULT_SECRET_ in order to provide the ability to supply multiple secret paths.

Furthermore requiring the suffix to match last path of the secret basically makes it impossible to fetch secrets from different vault paths if their key name (file name) by chance happens to be the same, which will inevitably lead to a name conflict in their environment variables.

Also daytona should generally provide some error message if environment variables starting with VAULT_SECRET_ somehow could not be processed correctly.

Thanks.

After using daytona in an initContainer with the secrets stored to a file, it isn't easy to expose those secrets as environment variables in subsequent containers. I propose to export secrets in a ".env" file, so containers can source that file before starting. Something like:

# in vault:
secret/application/foo/MY_VAR value=fizzbuzz
secret/application/foo/SOME_URL value=http://example.com

# kubernetes
initContainers:
  - name: daytona
    env:
    - name: SECRET_ENV_PATH
      value: /home/vault/secrets.env
    - name: VAULT_SECRETS_APP
      value: secret/application/foo
containers:
  - name: my-app
    command: ["/bin/bash", "-c"]
    args: |
    - source /home/vault/secrets.env
      ./my-app

# /home/vault/secrets.env
export MY_VAR=fizzbuzz
export SOME_URL=http://example.com

https://github.com/pmezard/go-difflib (a partial port of Python difflib) has been updated to contain "IS NO LONGER MAINTAINED." It's docs are at: https://pkg.go.dev/github.com/pmezard/go-difflib/difflib?tab=doc

The help says

  -k8s-auth-mount string
        The vault mount where k8s auth takes place (env: K8S_AUTH_MOUNT, note: will infer via k8s metadata api if left unset) (default "kubernetes")

I'm setting the env variable via

env:
        - name: K8S_AUTH
          value: 'true'
        - name: K8S_AUTH_MOUNT
          value: 'kubernetes'
        - name: VAULT_ADDR
          value: 'http://vault.vault-system.svc.cluster.local:8200'

on a kubernetes init container running daytona.

But the logs on that init container show me that it's trying to infer the auth endpoint from GCPs metadata API:

{"level":"info","applicationName":"daytona","time":"2022-12-20T16:24:39Z","message":"Attempting kubernetes auth.."}
{"level":"error","applicationName":"daytona","error":"Error making API request.\n\nURL: PUT http://vault.vault-system.svc.cluster.local:8200/v1/auth/kubernetes-gcp-<cluster-name>/login\nCode: 403. Errors:\n\n* permission denied","time":"2022-12-20T16:24:39Z","message":"failed to retrieve vault token"}

I would expect that the auth endpoint is http://vault.vault-system.svc.cluster.local:8200/v1/auth/kubernetes

Tested on latest version v1.2.4

It would be great if a platform was able to inject an always up to date version of Daytona into arbitrary K8s workloads.

A common way to do this in the K8s world is to create a mutating admission webhook that can rewrite a pod to do the following:

1. Inject an initContainer and a shared memory volume between the initContainer and the other containers.
2. Make the binary in the init container copy itself to the shared volume.
3. Rewrite the command on the application containers to execute the binary that was copied to the shared volume.

If Daytona supported a flag like --copy-to that would allow Daytona to copy itself to a destination then it would be possible to do the above while still using a feature like --secret-env/--entrypoint.

On a kubernetes workload that is supposed to write a secrets file as well as a single secret to file the following env vars are configured:

          - name: DAYTONA_SECRET_DESTINATION_PATH
            value: '/home/vault/secrets'
          - name: DAYTONA_SECRET_DESTINATION_A_KEY
            value: '/home/vault/a-key'

However, that only writes the /home/vault/secrets file, not the /home/vault/a-key file, which is unexpected. It should write both files.

If the DAYTONA_SECRET_DESTINATION_PATH var is removed like so:

          # - name: DAYTONA_SECRET_DESTINATION_PATH
          #   value: '/home/vault/secrets'
          - name: DAYTONA_SECRET_DESTINATION_A_KEY
            value: '/home/vault/a-key'

Then it writes the /home/vault/a-key file, which is expected.

The reason this is flagged as bug is because it still works with the deprecated SECRET_PATH env variable:

          - name: SECRET_PATH
            value: '/home/vault/secrets'
          - name: DAYTONA_SECRET_DESTINATION_A_KEY
            value: '/home/vault/a-key'

Writes both files, /home/vault/secrets, and /home/vault/a-key, which is expected. So once the SECRET_PATH variable is finally removed this fix would not work anymore.

Maybe the solution to this problem is actually to not deprecate the SECRET_PATH env variable. Because its replacement DAYTONA_SECRET_DESTINATION_PATH has the flaw that it's unclear what it's supposed to do for secrets actually called path. Is it then supposed to write all secrets in that directory into one file, or is it only supposed to write that one secret called path?

I would like to be able to login via Daytona from a VM behind a VM ScaleSet in Azure. Currently Daytona tries to retrieve VM metadata from 'Microsoft.Compute/virtualMachines/{instanceName}', but for a VMSS instance it should instead be using 'Microsoft.Compute/virtualMachineScaleSets/{vmScaleSetName}/virtualMachines/{instanceId}'

In kubernetes < 1.21 this worked:

But with kubernetes 1.21 jwt tokens now look like this:

{
  "aud": [
    "https://container.googleapis.com/v1/projects/cruise-paas-dev-0e95/locations/us-west1/clusters/paas-dev-us-west1"
  ],
  "exp": 1683235981,
  "iat": 1651699981,
  "iss": "https://container.googleapis.com/v1/projects/cruise-paas-dev-0e95/locations/us-west1/clusters/paas-dev-us-west1",
  "kubernetes.io": {
    "namespace": "juno",
    "pod": {
      "name": "juno-provisioner-vault2-7fd97dfcdc-chlct",
      "uid": "95bb6f8a-d018-485f-9414-2cb518c4cf52"
    },
    "serviceaccount": {
      "name": "juno-provisioner-vault2",
      "uid": "afcb128e-9d97-4e89-b20a-18d07732b014"
    },
    "warnafter": 1651703588
  },
  "nbf": 1651699981,
  "sub": "system:serviceaccount:juno:juno-provisioner-vault2"
}

So the new path for service account name is now "kubernetes.io".serviceaccount.name (vs previously kubernetes.io/serviceaccount/service-account.name).

Daytona should get updated to infer role names before and after kubernetes version 1.21.

I'm still getting this error when I try to fetch the latest master commit of daytona.

> go get github.com/cruise-automation/daytona@master
...
> go mod tidy
go: finding github.com/cruise-automation/daytona/cmd latest
go: github.com/cruise-automation/[email protected]: parsing go.mod: unexpected module path "github.robot.car/cruise/daytona"

For reasons I don't understand, even though I'm fetching a post v1.0.0 commit, go is still inspecting previous versions. When it tries to inspect the v1.0.0 commit, it fails because the module name was wrong at that point.

I don't know if this will fix it, but can you please try cutting a v1.0.1 release?

Daytona fails to ignore subpaths when it's reading through the secrets in a specified path using VAULT_SECRETS_FOO=secrets/foo.

Assume a path secrets/foo that has a subpath bar/:

% vault list secrets/foo
Keys
----
bar/
foo1
foo2

% vault list secrets/foo/bar
Keys
----
bar1
bar2

This will cause Daytona to fail with:

DAYTONA - 2020/03/24 22:50:11 Starting v1.1.0...
DAYTONA - 2020/03/24 22:50:11 Attempting to automatically infer some k8s configuration data
DAYTONA - 2020/03/24 22:50:11 Checking for an existing, valid vault token
DAYTONA - 2020/03/24 22:50:11 No token found in VAULT_TOKEN env, checking path
DAYTONA - 2020/03/24 22:50:11 Found an existing token at /home/vault/.vault-token
DAYTONA - 2020/03/24 22:50:11 Starting secret fetch
DAYTONA - 2020/03/24 22:50:11 Starting iteration on secrets/foo
DAYTONA - 2020/03/24 22:50:11 Vault listed a secret 'bar', but got not-found trying to read it at 'secrets/foo/bar'; very strange

I would like to suggest fixing this by ignoring/printing a warning if Daytona encounters a subpath instead of a key/value pair in the path it's iterating on.

For example, application is a directory that has no secrets under it, only the my-application directory (which holds a secret called my-secret)

Structure and error looks like:

secret/
└── application
    └── my-application
        └── my-secret

DAYTONA - 2019/05/28 19:07:14 starting secret fetch
DAYTONA - 2019/05/28 19:07:14 starting iteration on secret/application/
DAYTONA - 2019/05/28 19:07:35 vault listed a secret 'my-application/', but got not-found trying to read it at 'secret/application/my-application'; very strange

Probably don't actually want to write secrets in this structure, but it leaves daytona in a weird state. We should look for this type of error and try to do something intelligent to recover- skip it and log the issue?

I can use vault to read a gcp token roleset with vault read -field=token gcp/token/foo-token-dev. When I try to read that roleset with Daytona by setting VAULT_SECRET_FOO=gcp/token/foo-token-dev I get an error:

DAYTONA - 2019/07/01 22:02:19 secret 'foo-token-dev_token_ttl' has non-string value: "3599"

Perhaps Daytona could support flags like VAULT_SECRET_FOO=-field=token gcp/token/foo-token-dev?

DAYTONA - 2019/08/27 19:46:11 Will exec:  []
panic: runtime error: index out of range

goroutine 1 [running]:
main.main()
	/go/src/github.robot.car/cruise/gulp/vendor/github.com/cruise-automation/daytona/cmd/daytona/main.go:181 +0xc03

Here's the panicking code in main.go.

if config.Entrypoint {
    args := flag.Args()
    log.Println("Will exec: ", args)
    binary, err := exec.LookPath(args[0]) // <<<< panic is here. args list is empty
    if err != nil {
        log.Fatalf("Error finding '%s' to exec: %s\n", args[0], err)
    }
    err = syscall.Exec(binary, args, os.Environ())
    if err != nil {
        log.Fatalf("Error from exec: %s\n", err)
    }
}

I reproduced the structure from your example, but all trying ended with errors.

1. I have secrets on the following path, that can be read and listed.
   any/path/daytona_test/{api-key|database}
   I exported the variable VAULT_SECRETS_APPLICATION=any/path/daytona_test and try to fetching that secrets, but i got 405 error.

URL: GET https://awesome.vault/v1/secret/data/any/path/daytona_test?list=true
Code: 405. Errors:

* 1 error occurred:
        * unsupported operation

2. Trying to fetch secret by direct path. Use the variable: VAULT_SECRET_APPLICATION=any/path/daytona_test/api-key. Got non-string value error.

DAYTONA - 2019/08/15 12:18:05 Starting secret fetch
DAYTONA - 2019/08/15 12:18:06 secret 'api-key_data' has non-string value: map[string]interface {}{"value":"1234"}

Try again:

DAYTONA - 2019/08/15 12:21:17 Starting secret fetch
DAYTONA - 2019/08/15 12:21:17 secret 'api-key_metadata' has non-string value: map[string]interface {}{"destroyed":false, "version":"1", "created_time":"2019-08-15T05:02:57.750127434Z", "deletion_time":""}

If i send in the API by Postman i get next:

{
    "request_id": "a273f8e1-342f-7ee1-550f-de0734456154",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data": {
        "data": {
            "value": "1234"
        },
        "metadata": {
            "created_time": "2019-08-15T05:02:57.750127434Z",
            "deletion_time": "",
            "destroyed": false,
            "version": 1
        }
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Is this an unexpected response for Daytona? What am I doing wrong?
Vault version: 1.1.1

Currently Daytona supports secrets fetching with the use of List and Read capabilities. These translate well into other endpoints (such as the GCP Secrets backend) that use the same capabilities, but does not do similarly for endpoints that require additional capabilities.

Today, if we were interested in injecting certificates and keys into an application with the use of the Vault PKI backend we would first need to create the certificate and then store them in the Secrets backend to be fetched & injected.

This Issue is to request functionality for Daytona to provision certificates by interacting directly with the /sign or /issue endpoints of the Vault PKI backend and performing the same secret fetching & injection thereafter. This removes the need to store certificates in the Secrets backend, further aligning with the pattern of short lived certificates and service lifetimes.

We receive panic sometimes when daytona tries to refresh token:

Found an existing token at token path, setting as client token
Starting secret fetch
Starting the token renewer service
token ttl of 0 is below threshold of 7200 , renewing to 43200

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xb1d31e]
goroutine 1 [running]:
github.com/cruise-automation/daytona/pkg/auth.RenewService(_, {{0x0, 0x0}, {0xc00003a00b, 0x18}, {0xcd09d2, 0x33}, 0x1, {0xc00003804f, 0x29}, ...})
	/go/src/github.com/cruise-automation/daytona/pkg/auth/auth.go:189 +0x3de
main.main()
	/go/src/github.com/cruise-automation/daytona/cmd/daytona/main.go:257 +0x825

I would like to be able to :

1. extract the whole data payload as raw json
2. extract the whole secret payload as raw json with metadata, like TTL
3. I would like to be able to specify which field to extract as a raw string
4. I would like to be able to specify multiple fields to extract as keys in a json payload

One way to do this might be to support jsonpath, the way kubectl does.

An example might be DAYTONA_SECRET_JSONPATH_SUFFIX=.data or DAYTONA_SECRET_JSONPATH_SUFFIX=.data.field. If it's a map or list, print json. If it's a string or number, print raw.

If a non string-value is returned in a secret payload, addSecrets will panic. The type should be inspected and handled before it is cast to a string.

Currently when using daytona as entrypoint via --secret-env it injects secrets as environment variables with the same name as the secret in vault.
This forces the user to name vault secrets exactly as the application expects them, which is impractical for applications or services which expect very generic environment variable names like CLIENT_ID CLIENT_SECRET and whose source is not in control of the user.
Writing secrets with the same generic name to vault makes it hard for developers to understand what a particular secret actually refers and where it is coming from (i.e. by which IDP a secret is issues).

For example let's say you have an app which expects an OIDC_SECRET.
In your particular setup this secret may be issued by Auth0, and as such I'd prefer to write this secret as AUTH0_OIDC_SECRET to vault which makes it much clearer where this came from.

So what I'm proposing is to decouple the vault secret name from the environment name and introduce something like DAYTONA_SECRET_ENV_<suffix> to set the environment variable name.

For example:

VAULT_SECRET_AUTH0_OIDC_SECRET=/vault/path/to/secret/AUTH0_OIDC_SECRET
DAYTONA_SECRET_ENV_AUTH0_OIDC_SECRET=OIDC_SECRET