joe-elliott / cert-exporter Goto Github PK

View Code? Open in Web Editor NEW

314.0 314.0 87.0 14.61 MB

A Prometheus exporter that publishes cert expirations on disk and in Kubernetes secrets

License: Apache License 2.0

Go 72.22% Shell 23.04% Dockerfile 0.38% Makefile 0.56% Mustache 3.80%

cert-exporter's People

Contributors

Stargazers

Watchers

Forkers

cmcgalliard treydock michalfikejs nomeelnoj lukascoppens llambeau aravee tafsiralam zoopster abhishekjiitr rgl rakeshnetsil dan-vaughan berkiadam jitendrachopra insannik be-mobilenv jonasdkhansen dadez izolight lfranchini31 sweetops djagannath-asapp philyuchkoff taha1012 tehlers320 abcsa5s hakhundov ladmitry apsega messiahua luckmy46 benzch adhara-tech sass1997 shashankm pawandwivedi dmathern86 augustine-censys-io phuong39 faust64 diebietse oliverbaehler loaiq1 devopstoday11 ducnm0711 vlint zargha velkjaer chrisbsmith asatt krasttt dongchengjueshen110 thameezb praymann infa-kparida gitgrave lianghao208 danielrozenblum enriquejosesanjuanelorobles-tomtom yanggongwang infa-ralluri ivan-lozano dpivazyan fadecore hvauto 0x51 bryce-soghigian agoethals tiwalter angrysushi simonwilli teletrackingtechnologies blackhawk312 croesnick mamccorm jiri-lazensky wolfi-chainguard-demo ssiwek-dxc kigeorge chainguard-wolfi-bites-back carstenson martin-weiss pajikos mistral-valaise

cert-exporter's Issues

Error : requesting secrets secrets is forbidden:

i am running into following issue. i installed cert-exporter at default namespace level. what could be reason?

E0914 19:58:32.999795       1 periodicSecretChecker.go:83] Error requesting secrets secrets is forbidden: User "system:serviceaccount:default:cert-exporter" cannot list resource "secrets" in API group "" at the cluster scope
I0914 19:58:32.999811       1 periodicSecretChecker.go:59] Begin periodic check

deployed at default namespace

jichopra-macOS:prometheus-monitoring-Fork jichopra$ kubectl apply -f cert.yaml 
deployment.apps/cert-exporter created
clusterrole.rbac.authorization.k8s.io/secret-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/cert-exporter configured
serviceaccount/cert-exporter unchanged```


Pod is running but error:
```text 
jichopra-macOS:prometheus-monitoring-Fork jichopra$ kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
cert-exporter-66744f7f96-x5j29   1/1     Running   0          88s```

here is configuration:
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cert-exporter
  name: cert-exporter
spec:
  selector:
    matchLabels:
      name: cert-exporter
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        name: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
        - image: joeelliott/cert-exporter:v2.3.2
          name: cert-exporter
          command: ["./app"]
          args:
            - --secrets-annotation-selector=cert-manager.io/certificate-name
            - --secrets-include-glob=*.crt
            - --logtostderr
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-exporter
subjects:
  - kind: ServiceAccount
    name: cert-exporter
    namespace: prometheusdevirl1
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-exporter```

correct values for grafana

i'm succesfully started cert-exporter service, exporter scraping from pem files

but which unit need to select in grafana for show correct values? because its shows 49 years lol.

I am testing in a kubeadm k8s cluster with version 1.17.9. I have deployed the helm chart (https://github.com/joe-elliott/cert-exporter/tree/master/helm/cert-exporter) by editing the args for the container in the values.yaml as

- --include-cert-glob=/var/lib/k8s/kubernetes/pki/*.cert
       - --include-kubeconfig-glob=/var/lib/k8s/kubernetes/*.conf
       - --secrets-annotation-selector=cert-manager.io/certificate-name
       - --secrets-include-glob=*.crt
       - --logtostderr

I have prometheus operator up and running, so enabled the servicemonitor as well.
Once its deployed in the prometheus I can only see the metrics of
cert_exporter_error_total nothing else.

when I hit the endpoint with /metrics I am getting like this only

HELP cert_exporter_error_total Cert Exporter Errors
TYPE cert_exporter_error_total counter
cert_exporter_error_total 0
HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.1325e-05

I am also not able to see any metrics even though in the deployment pod logs I can see the app is reviewing every secret I have in different namespaces.

So the pods logs are like

I0405 07:44:39.716109       1 main.go:69] Starting cert-exporter (version unknown; commit unknown; date unknown)
I0405 07:44:39.716197       1 periodicCertChecker.go:38] Begin periodic check
W0405 07:44:39.716212       1 client_config.go:551] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0405 07:44:39.716251       1 periodicCertChecker.go:38] Begin periodic check
I0405 07:44:39.718227       1 periodicSecretChecker.go:61] Begin periodic check
I0405 07:44:40.963007       1 periodicSecretChecker.go:109] Reviewing secret

Can you please assist me if something is wrong here, or did I missed any step, because I not able find rest of the 4 exported metrics as mentioned in the doc.

Support for pfx format

Is it possible to add PFX/ PKCS#12 format support? I guess it can be done using this library https://pkg.go.dev/golang.org/x/crypto/pkcs12 ?

Cert-exporter couldn't be able to extract PEM format certificates

Cert-exporter working as expected with the '.crt' format certificates but not with the '.pem format certificates. Would it be possible to make it work with the PEM format certificates? If this is possible, we can also extract OpenSSL certs metrics from the cluster to track.

Single metric available - cert_exporter_error_total

The only metric available in the metrics endpoint of cert-exporter is - cert_exporter_error_total

Running version v2.3.0 and the endpoint is showing as up in Prometheus.

Also confirmed this is the only available metric by manually curling the metrics endpoint.

See full kubernetes manifest file below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cert-exporter
  namespace: kube-monitoring
  labels:
    app: cert-exporter
spec:
  selector:
    matchLabels:
      name: cert-exporter
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        name: cert-exporter
        app: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
      - image: joeelliott/cert-exporter:v2.3.0
        name: cert-exporter
        command: ["./app"]
        args:
        - --secrets-annotation-selector='cert-manager.io/certificate-name'
        - --secrets-include-glob='*.crt'
        - --alsologtostderr
        resources:
            limits:
              cpu: 50m
              memory: 64Mi
            requests:
              cpu: 25m
              memory: 32Mi
---
apiVersion: v1
kind: Service
metadata:
  name: cert-exporter-metrics
  namespace: kube-monitoring
  labels:
    app: cert-exporter-metrics
spec:
  selector:
    app: cert-exporter
  ports:
    - name: metrics
      port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-operator-cert-exporter
  namespace: kube-monitoring
  labels:
    app: prometheus-operator-operator
    prometheus: prometheus-operator-prometheus
    release: prometheus-operator
spec:
  endpoints:
  - port: metrics
    path: /metrics
    interval: 20s
  jobLabel: cert-exporter-metrics
  namespaceSelector:
    matchNames:
    - kube-monitoring
  selector:
    matchLabels:
      app: cert-exporter-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cert-exporter
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-exporter
subjects:
- kind: ServiceAccount
  name: cert-exporter
  namespace: kube-monitoring
roleRef:
  kind: ClusterRole
  name: cert-exporter
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-exporter
  namespace: kube-monitoring

How to configure cert-exporter on existing prometheus NS?

Hello @joe-elliott ,

I recently configured cert-exporter on sandbox environment. I used helm for the installation and installed it in cert-exporter namespace.

I can see Pod, Service, and ReplicaSet were created on cert-exporter namespace.

 kubectl -n cert-exporter get all
NAME                                              READY   STATUS    RESTARTS   AGE
pod/cert-exporter-cert-manager-67c8ffb99d-djf62   1/1     Running   0          3d21h

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/cert-exporter   ClusterIP   XX.XX.XX.XX   <none>        8080/TCP   3d21h

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-exporter-cert-manager   1/1     1            1           3d21h

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-exporter-cert-manager-67c8ffb99d   1         1         1       3d21h

I issued kubectl port-forward --namespace cert-exporter service/cert-exporter 8080:8080 and was able to see:

cert_exporter_error_total, and
cert_exporter_secret_expires_in_seconds

from http://localhost:8080/metrics

Since, I have Prometheus configured in prometheus namespace on sandbox environment, how can I scrap the above metrics from prometheus instead of cert-exporter namespace?

Should I just configure cert-exporter helm chart to be installed on prometheus instead of cert-exporter namespace?

I appreciate your input/help on this.

Thank you,
Laurentius

Can cert-exporter get router and registry cert secret from k8s v1.6(ocp 3.6)?

Hello,I try to deploy cert-exporter v1.3.0 to get certificate info from router and registry cert secret in ocp 3.6. In logs or metrics ,I cannot get info about router and registry cert secret in ocp v3.6. I wonder know that cert-export support to get cert secret from ocp v3.6 or kubernetes v1.6.

secrets-annotation-selector does't support multiple values

Would it be possible to select two annotations values with --secrets-annotation-selector? For example --secrets-annotation-selector=test --secrets-annotation-selector=test2

#feature request

Incorrect cert CN

Certificate CN shows incorrectly when exporter parse crt files.

Here is example:
cert_exporter_cert_expires_in_seconds{cn="DigiCert SHA2 Secure Server CA",filename="mycert.crt",instance="localhost:9088",issuer="DigiCert Global Root CA",job="cert-exporter"}

Figure out attaching binaries to a release

Currently when pushing a version tag release artifacts are being built correclty:

https://github.com/joe-elliott/cert-exporter/actions/runs/220712419

Figure out the best method of attaching these to a release.

@rgl
If I attempt to create a release through the github UI there is no option to attach the artifacts of the build to the release. It seems I'd have to manually download and attach to the release. Do you know of any automation options here? Am I missing something obvious?

Option to add annotations to Service (at least) and maybe pods in Helm Chart

Not everyone uses the Prometheus Operator.

Currently its impossible to scrape Cert Exporter using just the standard prometheus.io/scrape annotation.

Would you be open to a PR to add this? (just an option to add annotations to the service, not adding any 'preset' annotations).

how to debug exporter?

There is no error exporter logs but still i don't see all metrics of certificate exporter. i am not sure what i am missing so want to turn on debugger. is there a way?

jichopra-macOS:prometheus-monitoring-Fork jichopra$ kubectl logs cert-exporter-5bc895c76f-5rlw9 
I0914 22:47:43.893219       1 main.go:60] Starting cert-exporter (version unknown; commit unknown; date unknown)
I0914 22:47:43.894582       1 periodicSecretChecker.go:59] Begin periodic check
I0914 22:47:43.924293       1 periodicSecretChecker.go:89] Reviewing secret cert-exporter-token-fb6v6 in default
I0914 22:47:43.924311       1 periodicSecretChecker.go:89] Reviewing secret default-token-k2ggh in default
I0914 22:47:43.924315       1 periodicSecretChecker.go:89] Reviewing secret default-token-f87sh in kube-node-lease
I0914 22:47:43.924319       1 periodicSecretChecker.go:89] Reviewing secret default-token-mcd7z in kube-public
I0914 22:47:43.924322       1 periodicSecretChecker.go:89] Reviewing secret attachdetach-controller-token-f56sj in kube-

metrics endpoint:

# HELP cert_exporter_error_total Cert Exporter Errors
# TYPE cert_exporter_error_total counter
cert_exporter_error_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.0675e-05
go_gc_duration_seconds{quantile="0.25"} 1.346e-05
go_gc_duration_seconds{quantile="0.5"} 2.5262e-05
go_gc_duration_seconds{quantile="0.75"} 2.5526e-05
go_gc_duration_seconds{quantile="1"} 3.4061e-05
go_gc_duration_seconds_sum 0.000148167
go_gc_duration_seconds_count 7

arm builds to run the exporter in AWS graviton

@joe-elliott: Our infrastructure is moving to get the benefit of ARM based AWS Graviton.We have been using this exporter in all our environments but it does not work due to different architecture.

Is it possible to build this exporter for arm please?

armv7 and arm64

value address before relabel does not contain the port 8080

The image cert-exporter:v2.4.0 is instantiated in a container running on the master node of an upstream K8s cluster to monitor the certificates of the cluster.
There is a prometheus framework installed and operational, keeping an eye on amongst others services and infrastructure.

In the manifest of the DaemonSet, we have annotations for prometheus where we specify to scrape, the path and the port. The latter is set to 8080.

In our global configuration of prometheus, we configured a job for the pods in the cluster. For those that are visible in prometheus, the pod for the cert-exporter is not showing port (8080) in the label address. It only contains the IP address, where other pods are showing the combination of IP address and port. This results in a status Down for that endpoint.

-polling-period duration not working

Hello, thanks for the cert-exporter, it works very well - however I could not change the polling interval.

How to scrape secrets from multiple namespaces

Dear @joe-elliott ,

I am looking to scrape secrets from multiple namespaces like below, but I am not able to get the expected result.

- args:
            - '--secrets-namespace=<firstnamespace> <second namespace>'
            - '--secrets-include-glob=*' 
            - '--secret-include-types=Opaque'
            - '--logtostderr

Could you please help me to understand how can I achieve the same?

cert-manager.io/v1alpha2 not supported

I have tested Cert-exporter successfully with the node daemonset but not with the cert-manager.
We have been using cert-manager successfully in our cluster to issue and renewal TLS certificates.

I have deployed cert-exporter deployment as here https://github.com/joe-elliott/cert-exporter/blob/master/docs/cert-manager.yaml with the following args:

args:

--secrets-label-selector=cert-manager.io/certificate-name
--alsologtostderr

Note
We are using cert-manager.io/v1alpha2 apiVersion for the certificates. So, I have modified the arg to - --secrets-label-selector=cert-manager.io/certificate-name

Logs show as below:

I0309 14:02:37.901374 1 main.go:47] Application Starting
I0309 14:02:37.911237 1 periodicSecretChecker.go:53] Begin periodic check

I don't see any data using the metric cert_exporter_secret_expires_in_seconds in the Prometheus. I think this exporter is not capable of export certs using cert-manager.io/v1alpha2 apiVersion

Please kindly provide me with suggestions to make it work.

Support for the metrics with NotBefore timestamp

Could it be possible to add support to extra metrics which would say since when certficiate can be used
It should be fairly simple .NotBefore from https://pkg.go.dev/crypto/x509#pkg-variables

I've never touched go - and it would be super cool to have such metrics.

It would literally allow to compare NotBefore from k8s secret with the time when pod was started. And automatically restart pods when they were spawned before this cert (so cert was renewed)

Export deployment fails in AKS

I'm trying to deploy this to an AKS cluster and it gives me errors as below. I already have Prometheus and Grafana installed on the cluster.

Error 1

error: unable to recognize "cert-exporter-prom.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

Error 2

Unable to mount volumes for pod "cert-exporter-687jg_monitoring(6ed154a9-f624-460c-9501-e3735a92c707)": timeout expired waiting for volumes to attach or mount for pod "monitoring"/"cert-exporter-687jg". list of unmounted volumes=[kube-proxy kube-root]. list of unattached volumes=[kubelet kube-proxy kube-root default-token-x9gwr]

MountVolume.SetUp failed for volume "kube-proxy" : hostPath type check failed: /var/lib/kube-proxy is not a directory

MountVolume.SetUp failed for volume "kube-root" : hostPath type check failed: /srv/kubernetes is not a directory

Any insight into how to resolve this

Lifecycle

Hi Joe,

Would it be possible that you can do some lifecycle on the GO side? Upgrade to the latest state etc.
If you agree I will create a PR to upgrade the Dockerfile to the newest status/versions.

Cert-exporter send metrics for CA instead of service.

Hi,
I configured cert-exporter to scrape tls.crt. The problem I am facing is cert_exporter_secret_expires_in_seconds metric in Grafana is is always 3.5~ years to all secrets. After some investigation I found that tls.crt contains both service pem and CA pem.
My guess is since CA pem is the last certificate, cert-exporter take the metric from it instead of the service pem.
Any idea how to fix this?

Not able to get details of JKS keystore as a secret

We are using cert exporter to monitor all out kubernetes tls secrets and it works perfectly fine.We have requirement to monitor secret created from jks file which is throwing error like

Error exporting secret Failed to parse as a pem

I feel only := pem.Decode(certBytes) is not able to handle secrets created from JKS.
The secret content will be like

kafka.client.keystore.jks: MIIeCQIBAzCCH................... kafka.client.truststore.jks: /u3+7QAAAAIAAAC..................

Breaks in Kubernetes V0.24.8

⨯ release failed after 0.94s error=failed to build for linux_amd64: # github.com/joe-elliott/cert-exporter/src/checkers
../../../go/pkg/mod/github.com/joe-elliott/[email protected]/src/checkers/periodicSecretChecker.go:63:57: not enough arguments in call to client.CoreV1().Secrets(p.namespace).List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
../../../go/pkg/mod/github.com/joe-elliott/[email protected]/src/checkers/periodicSecretChecker.go:74:56: not enough arguments in call to client.CoreV1().Secrets(p.namespace).List
have ("k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)
want (context.Context, "k8s.io/apimachinery/pkg/apis/meta/v1".ListOptions)

User cannot list resource secrets in API group "" at the cluster scope

Hi.
I've applied this manifest from the docs:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: cert-exporter
  name: cert-exporter
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: cert-exporter
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        name: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
      - image: joeelliott/cert-exporter:v2.0.0
        name: cert-exporter
        command: ["./app"]
        args:
        - --secrets-annotation-selector='cert-manager.io/certificate-name'
        - --secrets-include-glob='*.crt'
        - --alsologtostderr
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-exporter
subjects:
- kind: ServiceAccount
  name: cert-exporter
  namespace: default
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-exporter
  namespace: kube-system

There are some issues in pod's log:

$ kc logs cert-exporter-65956cb7c6-sv844
I0403 07:19:43.229744       1 main.go:53] Application Starting
I0403 07:19:43.232963       1 periodicSecretChecker.go:60] Begin periodic check
E0403 07:19:43.248757       1 periodicSecretChecker.go:86] Error requesting secrets secrets is forbidden: User "system:serviceaccount:kube-system:cert-exporter" cannot list resource "secrets" in API group "" at the cluster scope
...

Have you ever struggled with that? Thanks.

Kubernetes: 1.13.12

Does cert-exporter work with kubeadm clusters?

Hello,

Does the cert-exporter work with kubeadm clusters? The config for kube-proxy is in a configmap which is mounted in kube-proxy pod, so it is unclear how to check the certs from kube-proxy.

Kind regards,
WS

Renewed certificates persist in metrics

For example if file on node /var/lib/kubelet/pki/kubelet.crt is renewed (deleted and recreated), cert-exporter maintains reference to old certificate in metrics:

Is it expected behavior?

Ceph rbd is not unmapped automatically. Need to add an option to change volume mount propagation (mountPropagation).

Hi there.

I've started to use cert-exporter and faced a problem:
when I delete a pod, that use ceph rbd, that rbd is not unmapped/unmount automatically from k8s node and the pod cannot be scheduled on another node.
Kubelet logs:

Aug 11 09:17:42 kube-01 d8-kubelet-forker[14693]: E0811 09:17:42.907060   14694 nestedpendingoperations.go:301] 
Operation for "{volumeName:kubernetes.io/rbd/kube:kubernetes-dynamic-pvc-018409f2-e715-4452-b483-c011772acec9 
podName: nodeName:}" failed. No retries permitted until 2021-08-11 09:17:43.407003409 +0300 MSK 
m=+1698521.295086326 (durationBeforeRetry 500ms). Error: "UnmountDevice failed for volume \"pvc-323d06cc-947b-
4c64-aaa0-e2f8963d27e5\" (UniqueName: \"kubernetes.io/rbd/kube:kubernetes-dynamic-pvc-018409f2-e715-4452-b483-
c011772acec9\") on node \"kube-01\" : rbd: failed to unmap device /dev/rbd10, error exit status 16, rbd output: [114 98 100 58 
32 115 121 115 102 115 32 119 114 105 116 101 32 102 97 105 108 101 100 10 114 98 100 58 32 117 110 109 97 112 32 102 97 105 
108 101 100 58 32 40 49 54 41 32 68 101 118 105 99 101 32 111 114 32 114 101 115 111 117 114 99 101 32 98 117 115 121 10]"
...
Aug 11 09:17:43 kube-01 d8-kubelet-forker[14693]: E0811 09:17:43.429490   14694 nestedpendingoperations.go:301] 
Operation for "{volumeName:kubernetes.io/rbd/kube:kubernetes-dynamic-pvc-018409f2-e715-4452-b483-c011772acec9 
podName: nodeName:}" failed. No retries permitted until 2021-08-11 09:17:44.429433563 +0300 MSK 
m=+1698522.317516538 (durationBeforeRetry 1s). Error: "UnmountDevice failed for volume \"pvc-323d06cc-947b-4c64-
aaa0-e2f8963d27e5\" (UniqueName: \"kubernetes.io/rbd/kube:kubernetes-dynamic-pvc-018409f2-e715-4452-b483-
c011772acec9\") on node \"kube-01\" : Unmount failed: exit status 32\nUnmounting arguments: 
/var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-018409f2-e715-4452-b483-
c011772acec9\nOutput: umount: /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-
018409f2-e715-4452-b483-c011772acec9: not mounted\n\n"

After some research I found the root of this issue (thanks to this guy - https://cloud.tencent.com/developer/article/1469532). This is a cert-exporter pod running on the same k8s node.
cert-exporter pod mounts /var/lib/kubelet and pods, that use ceph rbd, mounts over it to /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/blablabla

Related issue kubernetes/kubernetes#54214.
Related PR in prometheus-node-exporter https://github.com/helm/charts/pull/11194/files

The solution is to add an option to configure mountPropagation in daemonsets like that:

volumeMounts:
        - mountPath: /var/lib/kubelet
          mountPropagation: HostToContainer
          name: kubelet
          readOnly: true

Custom secrets?

Running into a weird issue where this exporter is not pulling in our own secrets. We manage client certs outside of cert manager for connecting to different external resources and we keep them in secrets in k8s.

I have added - --secrets-label-selector=monitor-cert

And the secrets have:

metadata:
  labels:
    monitor-cert: "true"

Sorry if this is a dumb question, but am I missing something?

Release Windows binaries?

Can you please start releasing binaries?

https://github.com/ribbybibby/ssl_exporter is using goreleaser to manage the releases, would that be welcomed here? if so, I can supply a PR (like I did at ribbybibby/ssl_exporter#43 for adding the windows binaries to ssl_exporter).

In concrete, I'm looking for Windows.

I was able to do a go build and use the resulting binary to successfully look into certificates that are inside a given directory:

.\cert-exporter.exe `
  -logtostderr `
  -prometheus-listen-address :9000 `
  -include-cert-glob c:\certs\*-crt.pem

How to get the cert details of ETCD node

Hi , Hope you are doing great

I have implemented cert-exporter by following the below document
https://github.com/joe-elliott/cert-exporter/blob/master/docs/kops-masters.yaml

Problem I am facing : I am using Legacy Kubernetes which is setup on ec2 instance not EKS , I have deployed above daemonset which is running on master and my master certs are located at /etc/kubernetes/pki changing the argument in the DS, it started capturing the master certs

Now the issue is how to monitor ETCD cert where the certificates will be located in my etcd node at /etc/kubernetes/pki/etcd , Etcd certs will be only in etcd node for legacy kubernetes we can not find it in master node

certs on file in different places

How do I handle certs on file in different places, I only found that I can specify one folder location.
Do I have to work with symlinks?

Supporting CRL

Hi,
Does cert-exporter supports CRL (Certificate Revocation List) ?

I tried it using 2 ways :

Add CRL in .pem format in my CA Secret
I created my Secret with

kubectl create secret generic ca --from-file=ca.crt=ca.crt --from-file=ca.crl=crl.pem --namespace ingress-nginx

As a result in cert-exporter Pod, I got the following logs :

I0325 10:58:50.549648       1 periodicSecretChecker.go:109] Reviewing secret ca in ingress-nginx
I0325 10:58:50.549673       1 periodicSecretChecker.go:126] Annotations matched. Parsing Secret.
I0325 10:58:50.549680       1 periodicSecretChecker.go:158] Publishing ca/ingress-nginx metrics ca.crl
E0325 10:58:50.549719       1 periodicSecretChecker.go:161] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:23 length:13 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} validity @100

💡 The ca.crt of my CA Secret is successfully recognized by cert-exporter.

Add CRL in .pem format as an independant Secret
I created my Secret with

kubectl create secret generic crl --from-file=crl.pem -n ingress-nginx

As a result in cert-exporter Pod, I got the following logs :

I0325 10:58:50.549886       1 periodicSecretChecker.go:109] Reviewing secret crl in ingress-nginx
I0325 10:58:50.549899       1 periodicSecretChecker.go:126] Annotations matched. Parsing Secret.
I0325 10:58:50.549939       1 periodicSecretChecker.go:158] Publishing crl/ingress-nginx metrics crl.pem
E0325 10:58:50.549972       1 periodicSecretChecker.go:161] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:23 length:13 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} validity @100

My cert-exporter deployment is

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cert-exporter
  labels:
    app: cert-exporter
    name: cert-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: cert-exporter
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        app: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
      - image: joeelliott/cert-exporter:v2.5.0
        name: cert-exporter
        command: ["./app"]
        args:
        - --secrets-include-glob=*.crt
        - --secrets-include-glob=*.crl
        - --secrets-include-glob=*.pem
        - --logtostderr
        ports:
          - name: metrics
            containerPort: 8080
            protocol: TCP

A CRL format looks like

-----BEGIN X509 CRL-----
...
-----END X509 CRL-----

Thanks for your help!

Getting "Error exporting secret asn1: structure error"

I'm using label sector method for discovering two of the secrets, but only one is getting discovered and in the cert-exporter pod log I am seeing the below error. What can be the issue?

I0115 10:34:26.082742       1 periodicSecretChecker.go:89] Reviewing secret tls-secret in dev
I0115 10:34:26.082751       1 periodicSecretChecker.go:106] Annotations matched. Parsing Secret.
I0115 10:34:26.082756       1 periodicSecretChecker.go:138] Publishing tls-secret/dev metrics tls.crt
I0115 10:34:26.083061       1 periodicSecretChecker.go:138] Publishing tls-secret/dev metrics tls.key
E0115 10:34:26.083084       1 periodicSecretChecker.go:141] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2

Deleted cert still shows up?

Hi,

I am a new user of your product, so sorry if I missed something in the docs...

The dashboard alerted me of an expired cert in a secret, which was for an Ingress we don't use anymore. I deleted the secret, and waited for an hour, but it still showed up in the dashboard. When I restarted the pod, it disappeared. Our current --pollling-period is set to 15m...

Missing release artifacts?

Hi,

It seems like version 2.7.0 is last one having binaries released.
Is that a deliberate move or some kind of bug?

connection refused :8080/metrics

we facing an issue with the metrics endpoint - any ideas ?

/app $ wget localhost:8080/metrics
Connecting to localhost:8080 ([::1]:8080)
wget: can't open 'metrics': Permission denied

Formulate looks wrong

HI,

The formulae to calculate cert expiration count seems wrong, it should looks like this instead,

Right now if the first expression for example the below one returns empty , then even if second or 3rd expression returns value greater than 0 , it's not getting taken into account as undefined + anything + anything is undefined

count(15724800 < cert_exporter_cert_expires_in_seconds and cert_exporter_cert_expires_in_seconds <= 31536000)

 (count(15724800 < cert_exporter_cert_expires_in_seconds and cert_exporter_cert_expires_in_seconds <= 31536000) OR on() vector(0) ) 
+(count(15724800 < cert_exporter_kubeconfig_expires_in_seconds and cert_exporter_kubeconfig_expires_in_seconds <= 31536000) OR on() vector(0) )
+(count(15724800 < cert_exporter_secret_expires_in_seconds and cert_exporter_secret_expires_in_seconds <= 31536000) OR on() vector(0))

Export secrets based on secrets-label-selector OR secrets-annotation-selector

The selectors for secrets, secrets-label-selector and secrets-annotation-selector, are currently ANDed. So, for example, if I wanted to export cert-manager secrets (with annotations) as well as some custom secrets, then I'd have to commit to using annotations only. Wondering if this is intentional as it seems natural to expect that one can mix and match label and annotation selectors.

Helm install doesn't work

I am trying to install cert-exporter by helm.

helm version: v3.3.4

Sure that I added the repo. helm search repo cert-exporter shows the right exporter. After helm repo update and seeing "Successfully got an update from the "cert-exporter" chart repository", helm install fails...

With debug flag, I was able to see that

Error: failed to fetch https://github.com/joe-elliott/cert-exporter/releases/download/cert-exporter-0.1.1/cert-exporter-0.1.1.tgz : 404 Not Found

I think the url should be https://github.com/joe-elliott/cert-exporter/archive/cert-exporter-0.1.1.tar.gz . It's weird...

Dockerhub images

On Dockerhub, the last release available is some v2.7.0 (https://hub.docker.com/r/joeelliott/cert-exporter/tags)
Is it normal the last releases were not published?

Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false})

We are facing the issue with monitoring custom certificate from secret for my ingress
Certificate and metrics are visible, but there is always 1 error on dashboard:

can it be fixed?

Logs:

I0410 12:07:41.260784       1 main.go:53] Application Starting
I0410 12:07:41.262521       1 periodicSecretChecker.go:60] Begin periodic check
I0410 12:07:41.456012       1 periodicSecretChecker.go:92] Reviewing secret app-ssl-certificate in et-prod
I0410 12:07:41.456033       1 periodicSecretChecker.go:111] Annotations matched.  Parsing Secret.
E0410 12:07:41.456257       1 periodicSecretChecker.go:146] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2
I0410 13:07:41.262607       1 periodicSecretChecker.go:60] Begin periodic check
I0410 13:07:41.325997       1 periodicSecretChecker.go:92] Reviewing secret app-ssl-certificate in et-prod
I0410 13:07:41.326020       1 periodicSecretChecker.go:111] Annotations matched.  Parsing Secret.
E0410 13:07:41.326208       1 periodicSecretChecker.go:146] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2

the secret contains both cert and key

apiVersion: v1
data:
  tls.crt: YYYYYYYYYYYYYYYYYYYYYYY
  tls.key: XXXXXXXXXXXXXXXXXXXXXXX
kind: Secret
metadata:
  labels:
    monitor-cert: "true"
  name: app-ssl-certificate
  namespace: default
type: kubernetes.io/tls

docker-compose version

hi is there any docker-compose version of this exporter?

Grafana miss-interpreting data

Hi,
I am using https://github.com/joe-elliott/cert-exporter/blob/master/docs/daemonset-prom-operator.yaml but the result looks like:

Is there some place where I can start looking?

$ kubectl -n monitoring logs -f cert-exporter-sh458
I0701 16:01:41.483538       1 main.go:54] Application Starting
I0701 16:01:41.483957       1 periodicCertChecker.go:38] Begin periodic check
I0701 16:01:41.484302       1 periodicCertChecker.go:38] Begin periodic check
I0701 16:01:41.484403       1 periodicCertChecker.go:44] Publishing dev-kubernetes-master node metrics /etc/kubernetes/pki/etcd/ca.crt
I0701 16:01:41.484760       1 periodicCertChecker.go:44] Publishing dev-kubernetes-master node metrics /etc/kubernetes/pki/etcd/healthcheck-client.crt
I0701 16:01:41.484906       1 periodicCertChecker.go:44] Publishing dev-kubernetes-master node metrics /etc/kubernetes/pki/etcd/peer.crt
I0701 16:01:41.485052       1 periodicCertChecker.go:44] Publishing dev-kubernetes-master node metrics /etc/kubernetes/pki/etcd/server.crt

This is the only log I get from the exporter

metrics endpoint of exporter:

/app # curl localhost:8080/metrics
# HELP cert_exporter_cert_expires_in_seconds Number of seconds til the cert expires.
# TYPE cert_exporter_cert_expires_in_seconds gauge
cert_exporter_cert_expires_in_seconds{cn="dev-kubernetes-master",filename="/etc/kubernetes/pki/etcd/peer.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.9752335513827194e+07
cert_exporter_cert_expires_in_seconds{cn="dev-kubernetes-master",filename="/etc/kubernetes/pki/etcd/server.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.9752333513707895e+07
cert_exporter_cert_expires_in_seconds{cn="etcd-ca",filename="/etc/kubernetes/pki/etcd/ca.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 2.809848265141798e+08
cert_exporter_cert_expires_in_seconds{cn="kube-etcd-healthcheck-client",filename="/etc/kubernetes/pki/etcd/healthcheck-client.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.9752336513950195e+07
# HELP cert_exporter_cert_not_after_timestamp Timestamp of when the certificate expires.
# TYPE cert_exporter_cert_not_after_timestamp gauge
cert_exporter_cert_not_after_timestamp{cn="dev-kubernetes-master",filename="/etc/kubernetes/pki/etcd/peer.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.613375237e+09
cert_exporter_cert_not_after_timestamp{cn="dev-kubernetes-master",filename="/etc/kubernetes/pki/etcd/server.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.613375235e+09
cert_exporter_cert_not_after_timestamp{cn="etcd-ca",filename="/etc/kubernetes/pki/etcd/ca.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.874607728e+09
cert_exporter_cert_not_after_timestamp{cn="kube-etcd-healthcheck-client",filename="/etc/kubernetes/pki/etcd/healthcheck-client.crt",issuer="etcd-ca",nodename="dev-kubernetes-master"} 1.613375238e+09
# HELP cert_exporter_error_total Cert Exporter Errors
# TYPE cert_exporter_error_total counter
cert_exporter_error_total 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.09e-05
go_gc_duration_seconds{quantile="0.25"} 2.2801e-05
go_gc_duration_seconds{quantile="0.5"} 4.83e-05
go_gc_duration_seconds{quantile="0.75"} 8.67e-05
go_gc_duration_seconds{quantile="1"} 0.000935604
go_gc_duration_seconds_sum 0.005797025
go_gc_duration_seconds_count 83
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 13
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.17"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.870464e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 2.33454432e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.455693e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 481429
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 3.5187927214700534e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.377728e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.870464e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.193152e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.685824e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 8691
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.1505536e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6617344e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5936250631249027e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 490120
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 37584
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 754347
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.1762168e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.58
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 10
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.3037568e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.59361929992e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.32849664e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 578
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

here prometheus data:

It looks like data flow is ok and prometheus receives valid data. Must be grafana board then - any idea?
I will check if this is a better option: https://github.com/joe-elliott/cert-exporter/blob/master/docs/sample-dashboard.json

addLabels for auditing

is there any objection to adding this feature back?

values.yaml used to have

  deployment:
    replicaCount: 1
    additionalLabels: {}

Support configmap and mutatingwebhookconfiguration/validatingwebhookconfiguration when it comes to admission webhook

Our k8s clusters use admission webhooks in production environment, some of the webhooks mount their certificate files from configmap, and the ca certificate will be in Mutatingwebhookconfiguration and Validatingwebhookconfiguration object. I wonder if cert-exporter will take this senario into account. If so, I can submit PRs to add some supports.

Docker run issue

I pulled the image (docker pull joeelliott/cert-exporter).. but when I try to run following docker run command it give following error. Would you please provide the docker run command :

docker run -d -p 9120:9117 joeelliott/cert-exporter --logtostderr=true --include-glob /etc/ssl/certs/*.pem

18b06542338fe91a9a9732c75405cc7ff81e6d0ec5425404fb9ba318d8f22028
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "--logtostderr=true": executable file not found in $PATH: unknown.

Docker Image libraries need to be updated

Our security tool has discovered few vulnerabilities related to the docker image.
Can we upgrade below mentioned packages to the latest version and create a new release.
Busybox package -->1.31.1-r20 fixed in version-->1.31.1-r21
apk-tools current version --> 2.10.6-r0 fixed in version--> 2.10.7-r0

get secrets only from namespace where cert-exporter is running

hello,

we have the situation, that we using a shared cluster and only have rights in our namespace - so we would check our certificates in secrets for ("kubernetes.io/tls") ...
so we created RBAC Rules only for our local namespace but the cert-exporter will get all secrets from cluster-level - is there a solution only get secrets from specific namespace ?

thanks

Support Multiple -secrets-data-glob

Support for multiple -secrets-label-selectors and -secrets-annotation-selectors exist, but not -secrets-data-glob.

How should this work?

Option 1: Create a way to match up label/annotation selectors with data globs
- Would be flexible but cause configuration to be overly complex and difficult to understand
Option 2: Accept multiple secrets data globs. Add support for -exclude-data-glob and apply the globs to all secrets discovered by the selectors equally.
- Not as flexible but more straightforward configuration.

I am partial to option 2. If someone's config needs to be that complicated it would be easy to run multiple cert-exporter's with different combinations of parameters.

@ananth07reddy, @haarchri, you have shown interest in these features in the past. Feel free to voice an opinion!