Description

This project was originally meant to only support core/v1 Events; however, as the new Event API is planned to go GA in Kubernetes v1.19, we might want to consider supporting it.

Impactful changes

EventRecorder

The core API EventRecorder was designed to update the apiserver every time an Event was generated. This was really useful to this project as it means that every time an Event was generated we could be noticed right away by watching the resource on the apiserver.

However, in order to reduce the number of apiserver calls made by the EventRecorder, this behavior was changed in the new implementation. In the new implementation there are 3 mechanisms in place updating the apiserver:

• A singleton Event is created
• An EventSeries is broken (no new isomorphic Event in the last 6 minutes)
• A 30minutes heartbeat send updates about the existing EventSeries

These changes reduce the precision we will be able to have on Events as we will only be able to receive updates from them every 30 minutes. This might require redefining the goals of this project.

Event structure

The InvolvedObject field was renamed Regarding to make the semantic more obvious. Since we were using the InvolvedObject naming in our labels we might also want to consider renaming:

involved_object_kind => regarding_kind
involved_object_namespace => regarding_namespace

--
Event API proposal
Event API GA

Description

When an Event is created, we increment the kube_events_total counter by 1. However, in some cases, a single Event creation can result in multiple isomorphic Events being created.

When an Event is updated, this is handled correctly because we add the difference between the counts of the old Event and the new one to the counter.

Event creation

"k8s.io/client-go/tools/record/" EventRecorder

There are 2 cases where an Event is created with this recorder:

• The Event count is equal to 1
• The recorder tried to patch an Event that does not exist anymore on the apiserver

With this recorder, each Event generation results in one request to the apiserver. Thus, the assumption of one Create event resulting in one Event creation is correct.

"k8s.io/client-go/tools/events/" EventRecorder

There are 2 cases where an Event is created with this recorder:

• The Event series is nil
• The recorder tried to patch an Event that does not exist anymore on the apiserver

Both behaviour are really similar however, in this particular case, in order to reduce the number of call made to the apiserver, it was chosen to only update the Events on the apiserver every 30 minutes via a heartbeat (goroutine). Meaning that all updates during these 30 minutes gets merged into a single request. With a PATCH request, we won't have any problem to know how many Events were generated during this period since we will be able to get both the previous count and the new one. However, if the PATCH request fail, the Event will be recreated and we won't be able to know the previous count.

Potential solutions

• When an Event is created add the Event count to the counter.

We can't do that because, in the case of a deleted Event, the next update will result in this Event recreation with a count equal to all the older Events + the new ones.

Example: Delete Ev{Count=100} (retention) => Update Ev{Count=101} => Fail (does not exist) => Create Ev{Count=101}

• When an Event is created for the first time if we increment the counter by one and store this Event + its count in an internal cache. We will then keep the data up to date in the internal cache and use it as the old Event count when updating the counter.

We will have inconsistencies when multiple Events are created with the same apiserver request and seen for the first time by the exporter.

• Have a sync period before starting collecting metrics in order to save existing Event counts in an internal cache.

This sync period needs to be longer than the period between 2 heartbeats (+30min). However, we won't have any inconsistencies with this method.

--
Current event handler in charge of updating the kube_events_total counter.

Currently, the polling mechanism used in the e2e tests to wait for the exporter to be running is based on the ready replicas count of the exporter deployment. This introduces flakes when creating Events right after creating the exporter as the informer might not be running yet.

In order to fix that, we should poll the /healthz endpoint instead.

As suggested in the following comment #3 (comment) the exporter should expose metrics about successful/unsuccessful metrics collection.

Instead of relying on passing apiserver address as cli option in

This could be autodiscovered for example by parsing KUBERNETES_* environment variables. Or better with go-lang k8s client ;)

Note: kubeconfig should have precedence over autodiscovery.

Deployment object references latest image even when labels are pointing to 0.1.0

Source: https://github.com/rhobs/kube-events-exporter/blob/master/manifests/kube-events-exporter-deployment.yaml#L24

Follow-up issue to the following comment: #5 (comment)

The collector should handle the following Counter metric:

kube_events_total{type=””, involved_object_namespace=””, involved_object_kind=””, reason=””}

At first, this ListWatch will be unfiltered and will watch Kubernetes Events across all namespaces.

The exporter should expose the following metrics about itself:

• kube_events_exporter_version
• kube_events_exporter_requests_total
• kube_events_exporter_requests_in_flight
• kube_events_exporter_request_duration_seconds_sum
• kube_events_exporter_request_duration_seconds_count
• kube_events_exporter_request_duration_seconds_bucket