Comments (13)
Hello!
@hymgg @maxcaresywwforever I did the following, hope it helps.
First off I modified the webhook.yaml file a bit like so:
Click to expand!
apiVersion: v1
kind: ServiceAccount
metadata:
name: danm-webhook
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: caas:danm-webhook
rules:
- apiGroups:
- danm.k8s.io
resources:
- tenantconfigs
verbs: [ "*" ]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: caas:danm-webhook
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: caas:danm-webhook
subjects:
- kind: ServiceAccount
name: danm-webhook
namespace: kube-system
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
name: danm-webhook-config
namespace: kube-system
webhooks:
- name: danm-netvalidation.nokia.k8s.io
clientConfig:
service:
name: danm-webhook-svc
namespace: kube-system
path: "/netvalidation"
# Configure your pre-generated certificate matching the details of your environment
caBundle: ${CA_BUNDLE}
rules:
- operations: ["CREATE","UPDATE"]
apiGroups: ["danm.k8s.io"]
apiVersions: ["v1"]
resources: ["danmnets","clusternetworks","tenantnetworks"]
failurePolicy: Fail
- name: danm-configvalidation.nokia.k8s.io
clientConfig:
service:
name: danm-webhook-svc
namespace: kube-system
path: "/confvalidation"
# Configure your pre-generated certificate matching the details of your environment
caBundle: ${CA_BUNDLE}
rules:
- operations: ["CREATE","UPDATE"]
apiGroups: ["danm.k8s.io"]
apiVersions: ["v1"]
resources: ["tenantconfigs"]
failurePolicy: Fail
- name: danm-netdeletion.nokia.k8s.io
clientConfig:
service:
name: danm-webhook-svc
namespace: kube-system
path: "/netdeletion"
# Configure your pre-generated certificate matching the details of your environment
caBundle: ${CA_BUNDLE}
rules:
- operations: ["DELETE"]
apiGroups: ["danm.k8s.io"]
apiVersions: ["v1"]
resources: ["tenantnetworks"]
failurePolicy: Fail
---
apiVersion: v1
kind: Service
metadata:
name: danm-webhook-svc
namespace: kube-system
labels:
danm: webhook
spec:
ports:
- name: webhook
port: 443
targetPort: 8443
selector:
danm: webhook
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: danm-webhook-deployment
namespace: kube-system
labels:
danm: webhook
spec:
selector:
matchLabels:
danm: webhook
template:
metadata:
annotations:
# Adapt to your own network environment!
danm.k8s.io/interfaces: |
[
{
"network":"flannel"
}
]
name: danm-webhook
labels:
danm: webhook
spec:
serviceAccountName: danm-webhook
containers:
- name: danm-webhook
image: danm_webhook
command: [ "/usr/local/bin/webhook", "-tls-cert-bundle=/etc/webhook/certs/cert.pem", "-tls-private-key-file=/etc/webhook/certs/key.pem", "bind-port=8443" ]
imagePullPolicy: IfNotPresent
volumeMounts:
- name: webhook-certs
mountPath: /etc/webhook/certs
readOnly: true
# Configure the directory holding the Webhook's server certificates
volumes:
- name: webhook-certs
secret:
secretName: danm-webhook-certs
Then I generated the cert and key files and put them into a kubernetes secret with the following webhook-create-signed-cert.sh script:
Click to expand!
#!/bin/bash
set -e
usage() {
cat <<EOF
Generate certificate suitable for use with an sidecar-injector webhook service.
This script uses k8s' CertificateSigningRequest API to a generate a
certificate signed by k8s CA suitable for use with sidecar-injector webhook
services. This requires permissions to create and approve CSR. See
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
detailed explantion and additional instructions.
The server key/cert k8s CA cert are stored in a k8s secret.
usage: ${0} [OPTIONS]
The following flags are required.
--service Service name of webhook.
--namespace Namespace where webhook service and secret reside.
--secret Secret name for CA certificate and server certificate/key pair.
EOF
exit 1
}
while [[ $# -gt 0 ]]; do
case ${1} in
--service)
service="$2"
shift
;;
--secret)
secret="$2"
shift
;;
--namespace)
namespace="$2"
shift
;;
*)
usage
;;
esac
shift
done
[ -z ${service} ] && service=danm-webhook-svc
[ -z ${secret} ] && secret=danm-webhook-certs
[ -z ${namespace} ] && namespace=kube-system
if [ ! -x "$(command -v openssl)" ]; then
echo "openssl not found"
exit 1
fi
csrName=${service}.${namespace}
tmpdir=$(mktemp -d)
echo "creating certs in tmpdir ${tmpdir} "
cat <<EOF >> ${tmpdir}/csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF
openssl genrsa -out ${tmpdir}/server-key.pem 2048
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
# clean-up any previously created CSR for our service. Ignore errors if not present.
kubectl delete csr ${csrName} 2>/dev/null || true
# create server cert/key CSR and send to k8s API
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: ${csrName}
spec:
groups:
- system:authenticated
request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- server auth
EOF
# verify CSR has been created
while true; do
kubectl get csr ${csrName}
if [ "$?" -eq 0 ]; then
break
fi
done
# approve and fetch the signed certificate
kubectl certificate approve ${csrName}
# verify certificate has been signed
for x in $(seq 10); do
serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
if [[ ${serverCert} != '' ]]; then
break
fi
sleep 1
done
if [[ ${serverCert} == '' ]]; then
echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
exit 1
fi
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
--from-file=key.pem=${tmpdir}/server-key.pem \
--from-file=cert.pem=${tmpdir}/server-cert.pem \
--dry-run -o yaml |
kubectl -n ${namespace} apply -f -
After this, I filled the CA bundles in the webhook.yaml file by giving this command:
cat ./webhook.yaml | ./webhook-patch-ca-bundle.sh > ./webhook-ca-bundle.yaml
Using the following webhook-patch-ca-bundle.sh script:
Click to expand!
#!/bin/bash
ROOT=$(cd $(dirname $0)/../../; pwd)
set -o errexit
set -o nounset
set -o pipefail
export CA_BUNDLE=$(kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"')
if command -v envsubst >/dev/null 2>&1; then
envsubst
else
sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
fi
Then I simply used the kubectl create -f webhook-ca-bundle.yaml
command to set it up, and it works fine for me. (The only thing you need to modify still is the image you use.)
from danm.
basically the CA used in your cluster to set-up TLS with the webhook
if you did not secure webhook, or its instance certificate was signed by a root trusted CA I guess you can leave it empty
from danm.
Ok, thank you, haven't used webhook before.
Removed caBundle entries, then it complained about these files don't exist, of course:
"-tls-cert-bundle=/etc/webhook/certs/danm_webhook.crt", "-tls-private-key-file=/etc/webhook/certs/danm_webhook.key"
Tried remove them, so it's just, command: [ "/usr/local/bin/webhook", "bind-port=8443" ]
Failed too.
$ kubectl logs danm-webhook-deployment-84cd64568d-5ksqp -n kube-system
2019/08/08 21:20:21 ERROR: TLS configuration could not be initialized, because:open : no such file or directory
Looked at
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
So if webhook require client auth, then apiserver itself needs to be reconfigured???
May I use the webhook without any auth?
Thanks. -Jessica
from danm.
the webserver is the server in this transaction, the API server is the client.
you don't need to reconfigure the API server as per se because of the TLS, because that's what the WebhookConfiguration API is for, and it is a dynamic API.
but yes, the admission controller feature needs to be enabled in your cluster to be able to use webhooks (secured or not)
and no, currently there is no way to start-up an insecure webhook. I consider TLS a basic feature in my projects
from danm.
So don't need to do what's described here?
https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers
Going by these information, looks like the 2 controllers are enabled by default?
https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#is-there-a-recommended-set-of-admission-controllers-to-use
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#options
--enable-admission-plugins stringSlice
| admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, PersistentVolumeClaimResize, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, DefaultStorageClass, DefaultTolerationSeconds, DenyEscalatingExec, DenyExecOnPrivileged, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodPreset, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
So I just need to generate the cert and key files, right?
from danm.
Would it be possible to provide one script/guide to generate these: danm_webhook.crt
, danm_webhook.key
and CA_BUNDLE
?
Just like what is did in CPU-Pooler
from danm.
possible, yes
planned, no
feel free to contribute if you feel this is needed
but I don't consider it strictly part of the project, as it is basically plain Linux openssl stuff
@hymgg : yes, I guess these controllers are added by default. so yes, you only need to set-up TLS
from danm.
@Fillamug Thank you so much!!! worked for me. gonna move on to next steps...
-Jessica
from danm.
@Fillamug that also works for me. Could this be merged to project so that other users could make use of this to easily bring webhook
service up and running.
After webhook
up and running, i setup one danmnet
and create one pod a top. There is one error saying webhook
does not have permission to list danmeps
resources. After i change the webhook
cluster role to enable this, error disappears. @Levovar does webhook
service requires premission to danmeps
resources, could you confirm?
from danm.
we appreciate if this would be turned to a PR, looks really useful :)
regarding DanmEp: yes, you are right! the recent validation enhancements I made to modify and delete operations indeed require access to DanmEp API
thanks for the pointer, going to correct!
will close the issue, as it's original scope is now solved I think
from danm.
@Fillamug do you have plan to contribute this to danm
project?
If not, would it ok for you that I turn this into a PR? Since i have a few scripts to integrate danm
into Kuberneters via a automatic way, where the changes here would definitely help a lot.
from danm.
@maxcaresywwforever Hello!
Yes, I will make a PR about it in a couple days time.
from danm.
@maxcaresywwforever Hello!
Yes, I will make a PR about it in a couple days
great, thanks.
from danm.
Related Issues (20)
- Build fail HOT 1
- failed to get Pod info from K8s API server due to:Unauthorized HOT 1
- cannot create pod due to `Error delegating ADD to CNI plugin:flannel because:OS exec call failed:missing network name` HOT 2
- Unable to deploy the pod with SRIOV-VF's HOT 4
- add ipv6 address to network interface fail HOT 3
- Support building danm with default CRI HOT 2
- IP Address allocation fails HOT 15
- Tenant network always loss ipvlan link HOT 4
- How to check what is the VF getting assigned to a POD/container while created? HOT 1
- SRIOV VF not released back to resource pool HOT 3
- Error delegating ADD to CNI plugin:calico because:OS exec call failed:invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable HOT 2
- Support for NodePort service on secondary POD interfaces HOT 3
- How to use host-device plugin? HOT 1
- "Static IP allocation failed", requested IP address already in use HOT 2
- CNI delegation failed due to error:Error delegating ADD to CNI plugin:sriov because:OS exec call faild:netplugin failed with no error message HOT 2
- danm/calico: pod to pod communication does not traverse nodes HOT 6
- Not able to deploy Danm 4.3.0 in kubernetes 1.21.8 using installer job. HOT 1
- [v4.3.0] - invalid version: module contains a go.mod file, so module path must match major version HOT 1
- Ipv6 global mngtmpaddr dynamic address observed in interface HOT 2
- Is this project dead? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from danm.