Giter VIP home page Giter VIP logo

Comments (11)

jessebot avatar jessebot commented on May 30, 2024 2

I think those commands are incomplete for postgresql specifically. If I run all of the commands in this directory as the postgresql user in my initDBScript for zitadel, I still get errors in the setup job:

time="2023-11-14T10:38:52Z" level=info msg="setup started" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:63"

time="2023-11-14T10:38:52Z" level=warning msg="postgres is currently in beta" caller="/home/runner/work/zitadel/zitadel/internal/database/postgres/config.go:65"

time="2023-11-14T10:38:52Z" level=info msg="verify migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:39" name=01_tables

time="2023-11-14T10:38:52Z" level=info msg="query failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/repository/sql/query.go:98" error="ERROR: column \"creation_date\" does not exist (SQLSTATE 42703)"

time="2023-11-14T10:38:52Z" level=fatal msg="unable to migrate step 1" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:117" error="ID=SQL-KyeAx Message=unable to filter events Parent=(ERROR: column \"creation_date\" does not exist (SQLSTATE 42703))"
zitadel-argocd-application-set.yaml for reference of all values passed in
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: zitadel-web-app-set
  namespace: argocd
  annotations:
    pref.argocd.argoproj.io/default-view: "network"
    pref.argocd.argoproj.io/default-pod-sort: "topLevelResource"
spec:
  goTemplate: true
  # generator allows us to source specific values from an external k8s secret
  generators:
    - plugin:
        configMapRef:
          name: secret-var-plugin-generator
        input:
          parameters:
            secret_vars:
              - zitadel_hostname
              - global_cluster_issuer
  template:
    metadata:
      name: zitadel-web-app
      annotations:
        argocd.argoproj.io/sync-wave: "4"
    spec:
      project: zitadel
      destination:
        server: https://kubernetes.default.svc
        namespace: zitadel
      syncPolicy:
        syncOptions:
          - ApplyOutOfSyncOnly=true
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://zitadel.github.io/zitadel-charts
        chart: zitadel
        targetRevision: 7.1.0
        helm:
          releaseName: zitadel
          # https://github.com/zitadel/zitadel-charts/blob/main/charts/zitadel/values.yaml
          values: |
            replicaCount: 1

            # Overrides the image tag to the latest version
            # as kept up to date by renovateBot
            image:
              tag: "v2.35.0"

            zitadel:
              # See all defaults here:
              # https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml
              configmapConfig:
                DefaultInstance:
                  LoginPolicy:
                    # disable registration AKA signups
                    AllowRegister: false
                Database:
                  Postgres:
                    Host: zitadel-postgres-rw.zitadel.svc
                    Port: 5432
                    Database: zitadel
                    User:
                      Username: zitadel
                      SSL:
                        Mode: verify-full
                    #Admin:
                    #  Username: postgres
                    #  SSL:
                    #    Mode: verify-full

                ExternalDomain: {{ .zitadel_hostname }}

                TLS:
                  # off until https://github.com/zitadel/zitadel-charts/pull/141
                  # or a similar easy fix would be merged
                  Enabled: false

                # specifies if ZITADEL is exposed externally through TLS this
                # must be set to true even if TLS is not enabled on ZITADEL itself
                # but TLS traffic is terminated on a reverse proxy
                # !!! Changing this after initial setup breaks your system !!!
                ExternalSecure: true
                ExternalPort: 443

                Machine:
                  Identification:
                    Hostname:
                      Enabled: true
                    Webhook:
                      Enabled: false

                # setup ZITADEL with a service account
                FirstInstance:
                  Org:
                    Machine:
                      Machine:
                        # Creates a service account with the name zitadel-admin-sa,
                        # which results in a secret 'zitadel-admin-sa' with a key 'zitadel-admin-sa.json'
                        Username: zitadel-admin-sa
                        Name: Admin
                      MachineKey:
                        Type: 1

              # Reference the name of the secret that contains the masterkey.
              # The key should be named "masterkey".
              masterkeySecretName: "zitadel-core-key"

              # The Secret containing the CA certificate at key ca.crt needed for establishing secure database connections
              dbSslCaCrtSecret: "zitadel-postgres-server-ca-key-pair"

              # The db admins secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
              # dbSslAdminCrtSecret: "zitadel-postgres-postgres-cert"

              # The db users secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
              dbSslUserCrtSecret: "zitadel-postgres-zitadel-cert"

            initJob:
              # Once ZITADEL is installed, the initJob can be disabled.
              enabled: false

            ingress:
              enabled: true
              className: "nginx"
              annotations:
                kubernetes.io/tls-acme: "true"
                cert-manager.io/cluster-issuer: {{ .global_cluster_issuer }}
              hosts:
                - host: {{ .zitadel_hostname }}
                  paths:
                    - path: /
                      pathType: Prefix
              tls:
                - secretName: zitadel-tls
                  hosts:
                    - {{ .zitadel_hostname }}

            metrics:
              enabled: false
              serviceMonitor:
                enabled: false

            readinessProbe:
              enabled: true
              initialDelaySeconds: 20
              periodSeconds: 15
              failureThreshold: 6

            livenessProbe:
              enabled: true
              initialDelaySeconds: 20
              periodSeconds: 15
              failureThreshold: 6

This is after also making sure to pass in additional commands related to schema ownership. Here's the full set of commands I run using the CloudNativePG operator for creating postgresql cluster:

postgresql-cluster-crd-argocd-applicationset.yaml
---
# webapp is deployed 2nd because we need secrets and persistent volumes up 1st
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: zitadel-postgres-app-set
  namespace: argocd
spec:
  goTemplate: true
  # generator allows us to source specific values from an external k8s secret
  generators:
    - plugin:
        configMapRef:
          name: secret-var-plugin-generator
        input:
          parameters:
            secret_vars:
              - zitadel_s3_endpoint
              - zitadel_s3_bucket
  template:
    metadata:
      name: zitadel-postgres-cluster
      namespace: zitadel
      annotations:
        argocd.argoproj.io/sync-wave: "3"
    spec:
      project: zitadel
      destination:
        server: "https://kubernetes.default.svc"
        namespace: zitadel
      syncPolicy:
        syncOptions:
          - ApplyOutOfSyncOnly=true
        automated:
          prune: true
          selfHeal: true
      source:
        repoURL: https://small-hack.github.io/cloudnative-pg-cluster-chart
        chart: cnpg-cluster
        targetRevision: 0.3.9
        helm:
          releaseName: zitadel-postgres-cluster
          values: |
            name: zitadel-postgres
            instances: 1

            bootstrap:
              initdb:
                database: zitadel
                owner: zitadel
                postInitApplicationSQLRefs:
                  secretRefs:
                    - name: zitadel-postgres-init-script
                      key: init.sql

            enableSuperuserAccess: true

            backup:
              # barman is a utility for backing up postgres to s3
              barmanObjectStore:
                destinationPath: "s3://{{ .zitadel_s3_bucket }}"
                endpointURL: "https://{{ .zitadel_s3_endpoint }}"
                s3Credentials:
                  accessKeyId:
                    name: zitadel-db-credentials
                    key : "ACCESS_KEY"
                  secretAccessKey:
                    name: zitadel-db-credentials
                    key : "SECRET_KEY"
              retentionPolicy: "30d"

            certificates:
              server:
                enabled: true
                generate: true
              client:
                enabled: true
                generate: true
              user:
                enabled: true
                username:
                  - zitadel
                  - postgres

            scheduledBackup:
              name: zitadel-pg-backup
              spec:
                schedule: "0 0 0 * * *"
                backupOwnerReference: self
                cluster:
                  name: pg-backup

            monitoring:
              enablePodMonitor: false

            postgresql:
              pg_hba:
                - hostnossl all all 0.0.0.0/0 reject
                - hostssl all all 0.0.0.0/0 cert clientcert=verify-full

It automatically creates a user named zitadel and a database named zitadel that the user zitadel owns and default has all permissions on. It uses this secret for init sql statements that it runs as the postgres super user (had to change events2 to events everywhere due to the fact that events2 is only created if events doesn't exist even though the SQL commands in that referenced directory do not actually create a schema called events and I'm unsure where that happens):

init.sql secret
apiVersion: v1
kind: Secret
metadata:
  name: zitadel-postgres-init-script
type: Opaque
stringData:
  init.sql: |
    BEGIN;
    CREATE SCHEMA IF NOT EXISTS eventstore;
    CREATE SCHEMA IF NOT EXISTS projections;
    CREATE SCHEMA IF NOT EXISTS system;
    CREATE TABLE IF NOT EXISTS system.encryption_keys (id TEXT NOT NULL, key TEXT NOT NULL, PRIMARY KEY (id));
    CREATE TABLE IF NOT EXISTS eventstore.events (
        instance_id TEXT NOT NULL
        , aggregate_type TEXT NOT NULL
        , aggregate_id TEXT NOT NULL
        , event_type TEXT NOT NULL
        , "sequence" BIGINT NOT NULL
        , revision SMALLINT NOT NULL
        , created_at TIMESTAMPTZ NOT NULL
        , payload JSONB
        , creator TEXT NOT NULL
        , "owner" TEXT NOT NULL
        , "position" DECIMAL NOT NULL
        , in_tx_order INTEGER NOT NULL
        , PRIMARY KEY (instance_id, aggregate_type, aggregate_id, "sequence"));
    CREATE INDEX IF NOT EXISTS es_active_instances ON eventstore.events (created_at DESC, instance_id);
    CREATE INDEX IF NOT EXISTS es_wm ON eventstore.events (aggregate_id, instance_id, aggregate_type, event_type);
    CREATE INDEX IF NOT EXISTS es_projection ON eventstore.events (
        instance_id
        , aggregate_type
        , event_type
        , "position");
    CREATE SEQUENCE IF NOT EXISTS eventstore.system_seq;
    CREATE TABLE IF NOT EXISTS eventstore.unique_constraints (
        instance_id TEXT
        , unique_type TEXT
        , unique_field TEXT
        , PRIMARY KEY (instance_id, unique_type, unique_field));
    GRANT ALL ON SCHEMA system TO zitadel;
    GRANT ALL ON ALL TABLES IN SCHEMA system TO zitadel;
    GRANT ALL ON SCHEMA eventstore TO zitadel;
    GRANT ALL ON ALL TABLES IN SCHEMA eventstore TO zitadel;
    GRANT ALL ON SCHEMA projections TO zitadel;
    GRANT ALL ON ALL TABLES IN SCHEMA projections TO zitadel;
    COMMIT;

I can also run all of those SQL commands directly as the postgres user against the postgres cluster and they all work, but still don't result in a functional zitadel install via the helm chart. I think it's because the init job and setup job are not properly encapsulated.

I unfortunately am out of time for this project, so I will have to give zitadel admin access anyway, which may be an issue during an audit, as this zitadel helm chart is in Argo CD as an ApplicationSet and so we'd have to write some additional logic outside of the IaC to enable the init job declaratively, and then create a new git commit to disable the init job and remove the additional superuser cert secrets from both the zitadel ApplicationSet and the Cluster CRD that needed to be generated and then disable the admin user's external access to the cluster. This makes thing's a little harder and more manual.

I will try to come back and look into this more later and post any solutions I come to. Maybe it makes sense to just have cert manager rotate the admin cert after this, that way it can be a Kubernetes Job still managed in the same declaritve repo and directly.

from zitadel-charts.

eliobischof avatar eliobischof commented on May 30, 2024 1

The scripts that ZITADEL uses are here and I see there is also a description about what they do.

from zitadel-charts.

jessebot avatar jessebot commented on May 30, 2024 1

Excellent, Thanks for your sleuthing! I will try to set this up and report back later today.

from zitadel-charts.

jessebot avatar jessebot commented on May 30, 2024

There is also no way to open an issue that isn't a doc issue, but helm charts will continue to need improvements beyond docs as Kubernetes grows. I recommend having additional issue templates for reporting bugs other than direct security vulnerabilities.

from zitadel-charts.

eliobischof avatar eliobischof commented on May 30, 2024

Hi @jessebot
The admin user is only needed for the init phase
You can manually set up your database and skip the init phase by running zitadel setup or zitadel start-from-setup.
D
If this doesn't help, please reopen the issue.

from zitadel-charts.

jessebot avatar jessebot commented on May 30, 2024

If this doesn't help, please reopen the issue.

You have not added permissions for users to reopen issues, so I cannot do that. You should allow users more time than 1 minute to respond before closing an issue. If your metrics are calculated based on time to close an issue, your product manager should adjust them.

The admin user is only needed for the init phase
You can manually set up your database and skip the init phase by running zitadel setup or zitadel start-from-setup.

how do you do that via the helm chart though? It is not explained here

from zitadel-charts.

jessebot avatar jessebot commented on May 30, 2024
  1. can I just not include the zitadel.configmapConfig.Database.Postgres.Admin object and dbSslAdminCertSecret?

  2. If I do that, which SQL commands need need to be run for zitadel? In the init docs you linked it just says:

  • If not already done, it grants the necessary permissions ZITADEL needs to the non privileged user.
  • If they don’t exist already, it creates all schemas and some basic tables.

Here is my current values.yaml in a collapsible:

click me for values.yaml
replicaCount: 1

# Overrides the image tag to the latest version
# as kept up to date by renovateBot
image:
  tag: "v2.35.0"

zitadel:
  # See all defaults here:
  # https://github.com/zitadel/zitadel/blob/main/cmd/defaults.yaml
  configmapConfig:
    DefaultInstance:
      LoginPolicy:
        # disable registration AKA signups
        AllowRegister: false
    Database:
      Postgres:
        Host: zitadel-postgres-rw.zitadel.svc
        Port: 5432
        Database: zitadel
        User:
          Username: zitadel
          SSL:
            Mode: verify-full
        Admin:
          SSL:
            Mode: verify-full

    ExternalDomain: myzitadel.example.com

    TLS:
      # off until https://github.com/zitadel/zitadel-charts/pull/141
      # or a similar easy fix would be merged
      Enabled: false

    # specifies if ZITADEL is exposed externally through TLS this
    # must be set to true even if TLS is not enabled on ZITADEL itself
    # but TLS traffic is terminated on a reverse proxy
    # !!! Changing this after initial setup breaks your system !!!
    ExternalSecure: true
    ExternalPort: 443

    Machine:
      Identification:
        Hostname:
          Enabled: true
        Webhook:
          Enabled: false

    # setup ZITADEL with a service account
    FirstInstance:
      Org:
        Machine:
          Machine:
            # Creates a service account with the name zitadel-admin-sa,
            # which results in a secret 'zitadel-admin-sa' with a key 'zitadel-admin-sa.json'
            Username: zitadel-admin-sa
            Name: Admin
          MachineKey:
            Type: 1

  # Reference the name of the secret that contains the masterkey.
  # The key should be named "masterkey".
  masterkeySecretName: "zitadel-core-key"

  # The Secret containing the CA certificate at key ca.crt needed for establishing secure database connections
  dbSslCaCrtSecret: "zitadel-postgres-server-cert"

  # The db admins secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
  dbSslAdminCrtSecret: "zitadel-postgres-server-cert"

  # The db users secret containing the client certificate and key at tls.crt and tls.key needed for establishing secure database connections
  dbSslUserCrtSecret: "zitadel-postgres-zitadel-cert"

ingress:
  enabled: true
  className: "nginx"
  annotations:
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-prod
  hosts:
    - host: myzitadel.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: zitadel-tls
      hosts:
        - myzitadel.example.com

metrics:
  enabled: false
  serviceMonitor:
    enabled: false

readinessProbe:
  enabled: true
  initialDelaySeconds: 20
  periodSeconds: 15
  failureThreshold: 6

livenessProbe:
  enabled: true
  initialDelaySeconds: 20
  periodSeconds: 15
  failureThreshold: 6

from zitadel-charts.

eliobischof avatar eliobischof commented on May 30, 2024

You have not added permissions for users to reopen issues

Ok, unfortunately, I couldn't find an option to allow everybody to reopen issues πŸ™ I will not close them immediately anymore.

can I just not include the zitadel.configmapConfig.Database.Postgres.Admin object and dbSslAdminCertSecret?

Yes, if you don't intend to run zitadel init.

which SQL commands need need to be run for zitadel

We are not maintaining and publishing such a list. I recommend you let ZITADEL initialize the DB and then remove the admin credentials. Else, you can also initialize a local database and record the SQL statements.

from zitadel-charts.

jessebot avatar jessebot commented on May 30, 2024

Ok, unfortunately, I couldn't find an option to allow everybody to reopen issues πŸ™ I will not close them immediately anymore.

thanks for looking!

Yes, if you don't intend to run zitadel init.

I will try that.

We are not maintaining and publishing such a list. I recommend you let ZITADEL initialize the DB and then remove the admin credentials. Else, you can also initialize a local database and record the SQL statements.

wait, then how can I setup the zitadel database and user ahead of time if you don't tell me what permissions it needs? I don't want to let zitadel have admin access to my production cluster. This is a bit confusing. I just need to know what grants you need and what schemas you may need access to. If that is closed source info, this project is not fully open source.

from zitadel-charts.

lukasredev avatar lukasredev commented on May 30, 2024

@jessebot I would be interested in a solution for this as well. Right now, my workaround would be to have a separate Postgres server for Zitadel ...

from zitadel-charts.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.