Configure monitoring about kadalu HOT 11 CLOSED

To start with we can start with the following metrics. Metric exporter will run in all Pods as container or as shared process within the container in case accessing required resource is challenge.

• Storage utilization(Size and inodes) csi-provisioner will have mounts of all Kadalu Storage Volumes(Hosting Volumes), so it is the best candidate to export all the utilization metrics(kadalu_storage_count, kadalu_storage_health, kadalu_storage_capacity_bytes, kadalu_storage_capacity_used_bytes, kadalu_storage_capacity_free_bytes, kadalu_storage_inodes_count, kadalu_storage_inodes_used_count, kadalu_storage_inodes_free_count etc)
• PV utilization Simple Quota Crawler is anyways doing the crawl and collecting the stat of each subvols/Pvs, This can export the metrics related to the PVs(kadalu_pv_count, kadalu_pv_capacity_bytes, kadalu_pv_capacity_used_bytes, kadalu_pv_capacity_free_bytes, kadalu_pv_inodes_count, kadalu_pv_inodes_used_count, kadalu_pv_inodes_free_count etc).
• Memory, CPU, Uptime and other metrics Export these metrics for all the processes in Kadalu managed Containers.
• Heal related metrics CSI Provisioner or Operator can run glfsheal command and export the heal metrics.

Two options to export these metrics to Prometheus:

• Expose single /metrics endpoint from Operator. When Prometheus requests the metrics then internally collect from other pods using kubectl exec.
• Expose metrics from respective containers itself. Prometheus Server(Prometheus Operator) can collect metrics from these individual exporters by using Pod/Container label.

Bonus: kubectl kadalu storage-health CLI can be created by consuming these metrics. So that Admins can get the realtime status without using Prometheus or Grafana.

Let us know if any other metrics required other than the ones listed above?

from kadalu.

Thank you for your contributions.
We are marking this issue as stale because it has not had recent activity. It will be closed in 2 weeks if no further activity occurs.

from kadalu.

Still needed.

from kadalu.

Thank you for your contributions.
Noticed that this issue is idle since 180 days! There is a possibility that this issue is already fixed in later releases. Please upgrade and check! If I don't hear any update in this issue in next 2 weeks, will be closing the issue.
That doesn't mean one can't re-open the issue! Just comment on the issue, and click 'Reopen', if you still have the issue.

from kadalu.

still needed.

from kadalu.

like the storage-health command idea, not always convenient to bring up grafana for viewing health status

from kadalu.

Expose single /metrics endpoint from Operator. When Prometheus requests the metrics then internally collect from other pods using kubectl exec.

Little change to this approach. Each containers exposes ReST APIs and Operator will collect from all these containers and exposes a single API.

Operator(GET /metrics) -> Collect metrics from all the containers in Kadalu namespace by calling respective ReST APIs(internal APIs).

Advantages:

• Easy to integrate with internal or externally hosted Prometheus server.
• Easily avoid collecting duplicate metrics. Operator can intelligently pick any one storage pod to get the metrics.

Additionally expose GET /metrics.json for all non-Prometheus purposes.

This approach is very similar to Gluster Metrics Exporter project.

from kadalu.

@aravindavk can this issue be closed?

from kadalu.

ping @aravindavk ^^, can this be closed?

from kadalu.

Thank you for your contributions.
Noticed that this issue is idle since 180 days! There is a possibility that this issue is already fixed in later releases. Please upgrade and check! If I don't hear any update in this issue in next 2 weeks, will be closing the issue.
That doesn't mean one can't re-open the issue! Just comment on the issue, and click 'Reopen', if you still have the issue.

from kadalu.

from kadalu.