Giter VIP home page Giter VIP logo

Comments (13)

Fillamug avatar Fillamug commented on June 2, 2024 3

Hello!

@hymgg @maxcaresywwforever I did the following, hope it helps.

First off I modified the webhook.yaml file a bit like so:

Click to expand!
apiVersion: v1
kind: ServiceAccount
metadata:
  name: danm-webhook
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: caas:danm-webhook
rules:
- apiGroups:
  - danm.k8s.io
  resources:
  - tenantconfigs
  verbs: [ "*" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: caas:danm-webhook
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: caas:danm-webhook
subjects:
- kind: ServiceAccount
  name: danm-webhook
  namespace: kube-system
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: danm-webhook-config
  namespace: kube-system
webhooks:
  - name: danm-netvalidation.nokia.k8s.io
    clientConfig:
      service:
        name: danm-webhook-svc
        namespace: kube-system
        path: "/netvalidation"
      # Configure your pre-generated certificate matching the details of your environment
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: ["CREATE","UPDATE"]
        apiGroups: ["danm.k8s.io"]
        apiVersions: ["v1"]
        resources: ["danmnets","clusternetworks","tenantnetworks"]
    failurePolicy: Fail
  - name: danm-configvalidation.nokia.k8s.io
    clientConfig:
      service:
        name: danm-webhook-svc
        namespace: kube-system
        path: "/confvalidation"
      # Configure your pre-generated certificate matching the details of your environment
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: ["CREATE","UPDATE"]
        apiGroups: ["danm.k8s.io"]
        apiVersions: ["v1"]
        resources: ["tenantconfigs"]
    failurePolicy: Fail
  - name: danm-netdeletion.nokia.k8s.io
    clientConfig:
      service:
        name: danm-webhook-svc
        namespace: kube-system
        path: "/netdeletion"
      # Configure your pre-generated certificate matching the details of your environment
      caBundle: ${CA_BUNDLE}
    rules:
      - operations: ["DELETE"]
        apiGroups: ["danm.k8s.io"]
        apiVersions: ["v1"]
        resources: ["tenantnetworks"]
    failurePolicy: Fail
---
apiVersion: v1
kind: Service
metadata:
  name: danm-webhook-svc
  namespace: kube-system
  labels:
    danm: webhook
spec:
  ports:
  - name: webhook
    port: 443
    targetPort: 8443
  selector:
    danm: webhook
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: danm-webhook-deployment
  namespace: kube-system
  labels:
    danm: webhook
spec:
  selector:
    matchLabels:
     danm: webhook
  template:
    metadata:
      annotations:
        # Adapt to your own network environment!
        danm.k8s.io/interfaces: |
          [
            {
              "network":"flannel"
            }
          ]
      name: danm-webhook
      labels:
        danm: webhook
    spec:
      serviceAccountName: danm-webhook
      containers:
        - name: danm-webhook
          image: danm_webhook
          command: [ "/usr/local/bin/webhook", "-tls-cert-bundle=/etc/webhook/certs/cert.pem", "-tls-private-key-file=/etc/webhook/certs/key.pem", "bind-port=8443" ]
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
     # Configure the directory holding the Webhook's server certificates
      volumes:
        - name: webhook-certs
          secret:
            secretName: danm-webhook-certs

Then I generated the cert and key files and put them into a kubernetes secret with the following webhook-create-signed-cert.sh script:

Click to expand!
#!/bin/bash

set -e

usage() {
    cat <<EOF
Generate certificate suitable for use with an sidecar-injector webhook service.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with sidecar-injector webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.
       --service          Service name of webhook.
       --namespace        Namespace where webhook service and secret reside.
       --secret           Secret name for CA certificate and server certificate/key pair.
EOF
    exit 1
}

while [[ $# -gt 0 ]]; do
    case ${1} in
        --service)
            service="$2"
            shift
            ;;
        --secret)
            secret="$2"
            shift
            ;;
        --namespace)
            namespace="$2"
            shift
            ;;
        *)
            usage
            ;;
    esac
    shift
done

[ -z ${service} ] && service=danm-webhook-svc
[ -z ${secret} ] && secret=danm-webhook-certs
[ -z ${namespace} ] && namespace=kube-system

if [ ! -x "$(command -v openssl)" ]; then
    echo "openssl not found"
    exit 1
fi

csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "

cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf

# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true

# create  server cert/key CSR and  send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

# verify CSR has been created
while true; do
    kubectl get csr ${csrName}
    if [ "$?" -eq 0 ]; then
        break
    fi
done

# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=${tmpdir}/server-key.pem \
        --from-file=cert.pem=${tmpdir}/server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

After this, I filled the CA bundles in the webhook.yaml file by giving this command:

cat ./webhook.yaml | ./webhook-patch-ca-bundle.sh > ./webhook-ca-bundle.yaml

Using the following webhook-patch-ca-bundle.sh script:

Click to expand!
#!/bin/bash

ROOT=$(cd $(dirname $0)/../../; pwd)

set -o errexit
set -o nounset
set -o pipefail

export CA_BUNDLE=$(kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"')

if command -v envsubst >/dev/null 2>&1; then
    envsubst
else
    sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
fi

Then I simply used the kubectl create -f webhook-ca-bundle.yaml command to set it up, and it works fine for me. (The only thing you need to modify still is the image you use.)

from danm.

Levovar avatar Levovar commented on June 2, 2024

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#webhookclientconfig-v1beta1-admissionregistration-k8s-io

basically the CA used in your cluster to set-up TLS with the webhook
if you did not secure webhook, or its instance certificate was signed by a root trusted CA I guess you can leave it empty

from danm.

hymgg avatar hymgg commented on June 2, 2024

Ok, thank you, haven't used webhook before.

Removed caBundle entries, then it complained about these files don't exist, of course:
"-tls-cert-bundle=/etc/webhook/certs/danm_webhook.crt", "-tls-private-key-file=/etc/webhook/certs/danm_webhook.key"

Tried remove them, so it's just, command: [ "/usr/local/bin/webhook", "bind-port=8443" ]
Failed too.
$ kubectl logs danm-webhook-deployment-84cd64568d-5ksqp -n kube-system
2019/08/08 21:20:21 ERROR: TLS configuration could not be initialized, because:open : no such file or directory

Looked at
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

So if webhook require client auth, then apiserver itself needs to be reconfigured???
May I use the webhook without any auth?

Thanks. -Jessica

from danm.

Levovar avatar Levovar commented on June 2, 2024

the webserver is the server in this transaction, the API server is the client.
you don't need to reconfigure the API server as per se because of the TLS, because that's what the WebhookConfiguration API is for, and it is a dynamic API.
but yes, the admission controller feature needs to be enabled in your cluster to be able to use webhooks (secured or not)

and no, currently there is no way to start-up an insecure webhook. I consider TLS a basic feature in my projects

from danm.

hymgg avatar hymgg commented on June 2, 2024

So don't need to do what's described here?
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers

Going by these information, looks like the 2 controllers are enabled by default?
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options

--enable-admission-plugins stringSlice

  | admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, PersistentVolumeClaimResize, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, DefaultStorageClass, DefaultTolerationSeconds, DenyEscalatingExec, DenyExecOnPrivileged, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodPreset, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

So I just need to generate the cert and key files, right?

from danm.

maxcaresywwforever avatar maxcaresywwforever commented on June 2, 2024

Would it be possible to provide one script/guide to generate these: danm_webhook.crt, danm_webhook.key and CA_BUNDLE?

Just like what is did in CPU-Pooler

from danm.

Levovar avatar Levovar commented on June 2, 2024

possible, yes
planned, no
feel free to contribute if you feel this is needed

but I don't consider it strictly part of the project, as it is basically plain Linux openssl stuff

@hymgg : yes, I guess these controllers are added by default. so yes, you only need to set-up TLS

from danm.

hymgg avatar hymgg commented on June 2, 2024

@Fillamug Thank you so much!!! worked for me. gonna move on to next steps...

-Jessica

from danm.

maxcaresywwforever avatar maxcaresywwforever commented on June 2, 2024

@Fillamug that also works for me. Could this be merged to project so that other users could make use of this to easily bring webhook service up and running.

After webhook up and running, i setup one danmnet and create one pod a top. There is one error saying webhook does not have permission to list danmeps resources. After i change the webhook cluster role to enable this, error disappears. @Levovar does webhook service requires premission to danmeps resources, could you confirm?

from danm.

Levovar avatar Levovar commented on June 2, 2024

we appreciate if this would be turned to a PR, looks really useful :)

regarding DanmEp: yes, you are right! the recent validation enhancements I made to modify and delete operations indeed require access to DanmEp API
thanks for the pointer, going to correct!

will close the issue, as it's original scope is now solved I think

from danm.

maxcaresywwforever avatar maxcaresywwforever commented on June 2, 2024

@Fillamug do you have plan to contribute this to danm project?

If not, would it ok for you that I turn this into a PR? Since i have a few scripts to integrate danm into Kuberneters via a automatic way, where the changes here would definitely help a lot.

from danm.

Fillamug avatar Fillamug commented on June 2, 2024

@maxcaresywwforever Hello!

Yes, I will make a PR about it in a couple days time.

from danm.

maxcaresywwforever avatar maxcaresywwforever commented on June 2, 2024

@maxcaresywwforever Hello!

Yes, I will make a PR about it in a couple days

great, thanks.

from danm.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.