Giter VIP home page Giter VIP logo

Comments (7)

joe-elliott avatar joe-elliott commented on September 27, 2024

This is the line that is logging that the secrets are being reviewed:

https://github.com/joe-elliott/cert-exporter/blob/master/src/checkers/periodicSecretChecker.go#L89

This line is never logged:

https://github.com/joe-elliott/cert-exporter/blob/master/src/checkers/periodicSecretChecker.go#L106

So that suggests that you have annotation selectors that are not being matched. I would review your annotation selectors and the annotations on your secrets.

from cert-exporter.

jitendrachopra avatar jitendrachopra commented on September 27, 2024

Thanks for response. I have annotation defined. what i am missing?
note : i have deployed below config on default namespace. prometheus is scrapping at namespace level.
do i need to make any adjustment so prometheus can listen at default namespace level?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cert-exporter
  name: cert-exporter
spec:
  selector:
    matchLabels:
      name: cert-exporter
  template:
    metadata:
      **annotations**:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        name: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
        - image: joeelliott/cert-exporter:v2.3.2
          name: cert-exporter
          command: ["./app"]
          args:
            - --secrets-annotation-selector=cert-manager.io/certificate-name
            - --secrets-include-glob=*
            - --logtostderr
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-exporter
subjects:
  - kind: ServiceAccount
    name: cert-exporter
    namespace: default
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-exporter

from cert-exporter.

joe-elliott avatar joe-elliott commented on September 27, 2024

What annotations are on the cert that you want to cert-exporter to parse?

from cert-exporter.

jitendrachopra avatar jitendrachopra commented on September 27, 2024

thanks Joe. it was working after removing following annotations. my secrects didn't have any annotations.
kubectl describe secret secretname

- --secrets-annotation-selector=cert-manager.io/certificate-name

i used prometheus-operator and i can see metrics by local port forwarding.

prometheus doesn't show metrics. where to look issue?
i think service monitor is not able to look at cert-exporter job
note:
i have one prometheus operator but will it creates new one with prometheus-operator-cert-exporter. i don't see prometheus-operator-cert-exporter while kubectl get pods -n namespace is it okay?

- job_name: prometheusdevor2/prometheus-operator-cert-exporter/0
  honor_timestamps: true
  scrape_interval: 20s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - prometheusdevor2
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: cert-exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_cert_exporter_metrics]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace



- job_name: prometheusdevor2/cert-exporter/0
  honor_timestamps: true
  scrape_interval: 20s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - prometheusdevor2
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: cert-exporter-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_cert_exporter_metrics]
    separator: ;
    regex: (.+)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: metrics
    action: replace

pods

jichopra-macOS:prometheus-monitoring-master jichopra$ kubectl get pods -n prometheusdevor2
NAME                                               READY   STATUS    RESTARTS   AGE
alertmanager-prometheusdevor2-0                    2/2     Running   0          41h
alertmanager-prometheusdevor2-1                    2/2     Running   0          41h
alertmanager-prometheusdevor2-2                    2/2     Running   0          41h
cert-exporter-6dbd758d68-rhppn                     1/1     Running   0          4m40s
cloudwatch-exporter-68bf57b7b7-jrbnw               1/1     Running   0          41h
grafana-7c79f5697f-6v8ps                           1/1     Running   0          41h
kube-state-metrics-79db5c76d9-gg64j                3/3     Running   0          41h
node-exporter-4qwmz                                2/2     Running   0          41h
node-exporter-7mjhf                                2/2     Running   0          41h
node-exporter-9krzp                                2/2     Running   0          41h
node-exporter-mkwgw                                2/2     Running   0          3d7h
prometheus-operator-cdb65c6c5-g5cvt                1/1     Running   0          41h
prometheus-prometheusdevor2-0                      3/3     Running   0          41h
prometheus-prometheusdevor2-1                      3/3     Running   0          41h
sso-5f6454cb9c-574d7                               1/1     Running   0          41h
yet-another-cloudwatch-exporter-578c4f9dc9-xbq4l   1/1     Running   0          41h

Screen Shot 2020-09-25 at 10 38 13 PM

I0926 05:37:06.881611       1 periodicSecretChecker.go:138] Publishing sso-ingress-sso/prometheusdevor2 metrics tls.key
E0926 05:37:06.881643       1 periodicSecretChecker.go:141] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2
I0926 05:37:06.881652       1 periodicSecretChecker.go:89] Reviewing secret sso-sso in prometheusdevor2
I0926 05:37:06.881657       1 periodicSecretChecker.go:106] Annotations matched. Parsing Secret.
I0926 05:37:06.881662       1 periodicSecretChecker.go:138] Publishing sso-sso/prometheusdevor2 metrics metadata.xml
E0926 05:37:06.881667       1 periodicSecretChecker.go:141] Error exporting secret Failed to parse as a pem
I0926 05:37:06.881671       1 periodicSecretChecker.go:138] Publishing sso-sso/prometheusdevor2 metrics certificate.cert
I0926 05:37:06.881759       1 periodicSecretChecker.go:138] Publishing sso-sso/prometheusdevor2 metrics certificate.key
E0926 05:37:06.881782       1 periodicSecretChecker.go:141] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2
I0926 05:37:06.881789       1 periodicSecretChecker.go:138] Publishing sso-sso/prometheusdevor2 metrics config.json
E0926 05:37:06.881796       1 periodicSecretChecker.go:141] Error exporting secret Failed to parse as a pem
I0926 05:37:06.881800       1 periodicSecretChecker.go:89] Reviewing secret tls-config in prometheusdevor2
I0926 05:37:06.881804       1 periodicSecretChecker.go:106] Annotations matched. Parsing Secret.
I0926 05:37:06.881808       1 periodicSecretChecker.go:138] Publishing tls-config/prometheusdevor2 metrics cert.pem
I0926 05:37:06.881870       1 periodicSecretChecker.go:138] Publishing tls-config/prometheusdevor2 metrics key.pem
E0926 05:37:06.881893       1 periodicSecretChecker.go:141] Error exporting secret asn1: structure error: tags don't match (16 vs {class:0 tag:2 length:1 isCompound:false}) {optional:false explicit:false application:false private:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2

Result with port forwarding
kubectl port-forward cert-exporter-6dbd758d68-rhppn 8080:8080 -n prometheusdevor2

cert_exporter_secret_not_after_timestamp{cn="kubernetes",issuer="kubernetes",key_name="ca.crt",secret_name="ttl-controller-token-jdmwg",secret_namespace="kube-system"} 1.904809495e+09
cert_exporter_secret_not_after_timestamp{cn="monitoring-or2.dev.adobelogin.com",issuer="DigiCert SHA2 Secure Server CA",key_name="certificate.cert",secret_name="sso-sso",secret_namespace="prometheusdevor2"} 1.641384e+09
cert_exporter_secret_not_after_timestamp{cn="monitoring-or2.dev.adobelogin.com",issuer="DigiCert SHA2 Secure Server CA",key_name="tls.crt",secret_name="sso-ingress-sso",secret_namespace="prometheusdevor2"} 1.641384e+09
cert_exporter_secret_not_after_timestamp{cn="monitoring-va6.dev.adobelogin.com",issuer="intermediate.gmon.cloudops.adobe.com",key_name="cert.pem",secret_name="tls-config",secret_namespace="prometheusdevor2"} 1.60753098e+09


apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: cert-exporter
  name: cert-exporter
spec:
  selector:
    matchLabels:
      name: cert-exporter
  template:
    metadata:
      annotations:
        prometheus.io/port: "8080"
        prometheus.io/scrape: "true"
      labels:
        name: cert-exporter
    spec:
      serviceAccountName: cert-exporter
      containers:
        - image: joeelliott/cert-exporter:v2.3.2
          name: cert-exporter
          command: ["./app"]
          args:
            - --secrets-include-glob=*
            - --logtostderr
---
apiVersion: v1
kind: Service
metadata:
  name: cert-exporter-metrics
  namespace: prometheusdevor2
  labels:
    app: cert-exporter-metrics
spec:
  selector:
    app: cert-exporter
  ports:
    - name: metrics
      port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-operator-cert-exporter
  namespace: prometheusdevor2
  labels:
    app: prometheus-operator-operator
    prometheus: prometheus-operator-prometheus
    release: prometheus-operator
spec:
  endpoints:
    - port: metrics
      path: /metrics
      interval: 20s
  jobLabel: cert-exporter-metrics
  namespaceSelector:
    matchNames:
      - prometheusdevor2
  selector:
    matchLabels:
      app: cert-exporter-metrics
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secret-reader
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-exporter
subjects:
  - kind: ServiceAccount
    name: cert-exporter
    namespace: prometheusdevor2
roleRef:
  kind: ClusterRole
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-exporter

kubectl describe po -n prometheusdevor2

Name:         cert-exporter-6dbd758d68-42pwt
Namespace:    prometheusdevor2
Priority:     0
Node:         ip-10-71-116-210.us-west-2.compute.internal/10.71.116.210
Start Time:   Sat, 26 Sep 2020 06:57:46 -0700
Labels:       name=cert-exporter
              pod-template-hash=6dbd758d68
Annotations:  kubernetes.io/psp: eks.privileged
              prometheus.io/port: 8080
              prometheus.io/scrape: true
Status:       Running
IP:           10.71.116.219
IPs:
  IP:           10.71.116.219
Controlled By:  ReplicaSet/cert-exporter-6dbd758d68
Containers:
  cert-exporter:
    Container ID:  docker://4f249b6c59e94c6b923233abb7dab6bb0aac6bb91eb13dc1e51a98fb7bf9ba41
    Image:         joeelliott/cert-exporter:v2.3.2
    Image ID:      docker-pullable://joeelliott/cert-exporter@sha256:865801a802b3753d97f91a0235c65e70df37b9c70edcfc12e3b22184301966f0
    Port:          <none>
    Host Port:     <none>
    Command:
      ./app
    Args:
      --secrets-include-glob=*
      --include-cert-glob=*.pem
      --logtostderr
    State:          Running
      Started:      Sat, 26 Sep 2020 06:57:47 -0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cert-exporter-token-gqwfz (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  cert-exporter-token-gqwfz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cert-exporter-token-gqwfz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From                                                  Message
  ----    ------     ----  ----                                                  -------
  Normal  Scheduled  24m   default-scheduler                                     Successfully assigned prometheusdevor2/cert-exporter-6dbd758d68-42pwt to ip-10-71-16-210.us-west-2.compute.internal
  Normal  Pulled     24m   kubelet, ip-10-71-16-20.us-west-2.compute.internal  Container image "joeelliott/cert-exporter:v2.3.2" already present on machine
  Normal  Created    24m   kubelet, ip-10-71-11-20.us-west-2.compute.internal  Created container cert-exporter
  Normal  Started    24m   kubelet, ip-10-71-16-21.us-west-2.compute.internal  Started container cert-exporter

from cert-exporter.

joe-elliott avatar joe-elliott commented on September 27, 2024

I've used the prometheus operator some, but honestly don't know a ton about it. If you need help using it I'd refer you to their documentation/help channels.

from cert-exporter.

jitendrachopra avatar jitendrachopra commented on September 27, 2024

i have kubernetes version(v1.16.13-eks-2ba888) that's why above prometheus-operator didn't work. need to adjust config. now it is working fine.

config

      serviceAccountName: cert-exporter
      containers:
        - image: joeelliott/cert-exporter:v2.3.2
          name: cert-exporter
          command: ["./app"]
          args:
            - --secrets-include-glob=*
            - --include-kubeconfig-glob=*
            - --include-cert-glob=*
            - --logtostderr

But i am not able to see below metrics.

  1. cert_exporter_kubeconfig_expires_in_seconds
  2. cert_exporter_cert_expires_in_seconds

Screen Shot 2020-09-27 at 8 37 04 PM

cert exporter logs:
I0928 03:21:39.057695 1 periodicSecretChecker.go:138] Publishing node-exporter-token-86977/prometheusdevor2 metrics token
E0928 03:21:39.057700 1 periodicSecretChecker.go:141] Error exporting secret Failed to parse as a pem

from cert-exporter.

joe-elliott avatar joe-elliott commented on September 27, 2024

Glad you were able to get your issue fixed!

from cert-exporter.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.