Summary
- Register a collector per instance/disk.
- Goroutine that listens for new instances/disks to register.
- Methods to Unregister and Reregister instances/disks as labels change?
Exploration
Due to the dynamic nature of this exporter we may want to investigate using an Unchecked exporter. One which returns no description of metrics. We really need a better understanding of the downsides of using an Unchecked exporter OR if we can find a better method to manage collectors that allows the collectors to remain checked.
The github.com/prometheus/client_golang/prometheus documentation indicates that our use case is expected and describes the pitfalls of checked vs. unchecked.
There is a more involved use case, too: If you already have metrics available, created outside of the Prometheus context, you don't need the interface of the various Metric types. You essentially want to mirror the existing numbers into Prometheus Metrics during collection.
Creation of the Metric instance happens in the Collect method. The Describe method has to return separate Desc instances, representative of the “throw-away” metrics to be created later. NewDesc comes in handy to create those Desc instances. Alternatively, you could return no Desc at all, which will mark the Collector “unchecked”. No checks are performed at registration time, but metric consistency will still be ensured at scrape time, i.e. any inconsistencies will lead to scrape errors. Thus, with unchecked Collectors, the responsibility to not collect metrics that lead to inconsistencies in the total scrape result lies with the implementer of the Collector. While this is not a desirable state, it is sometimes necessary. The typical use case is a situation where the exact metrics to be returned by a Collector cannot be predicted at registration time, but the implementer has sufficient knowledge of the whole system to guarantee metric consistency.
The big question now is what do they mean by "metric consistency"? I think I have seen an example of this inconsistency with the stackdriver exporter. It reported duplicate versions of a metric which caused a panic and crash.
I think though that the stackdriver exporter sets a good example for how we can keep our exporters checked. Currently we only register 2 collectors at startup. One for disks, and one for processes.
The stack driver exporter creates Collectors per project you ask it to scrape metrics from: https://github.com/prometheus-community/stackdriver_exporter/blob/16401d6cce781e5d99615e9518f220dbf56d6f0b/stackdriver_exporter.go#L124-L131
So what we would want is to create a collector per instance/disk, register with the metrics found when the collector starts, and success!
If an instance restarts, many of its labels and metrics will change. So we should unregister the collector and register a new one if the labels change.