how to remove stopped task from metrics about dm HOT 7 CLOSED

@mxlxm DM's metrics are stored in Prometheus, you can try https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series.

from dm.

@mxlxm DM's metrics are stored in Prometheus, you can try https://prometheus.io/docs/prometheus/latest/querying/api/#delete-series.

no, I mean some stopped task's metric info still present on dm-worker's metric page

from dm.

stopped task's metrics already stored in Prometheus, then Grafana still presents them.
so, you should delete them in Prometheus.

from dm.

no, there's nothing about prometheus.
I created some test task, and then stopped them, query-status shows empty, but on dm-worker's x.x.x.x:8262/metrics page, those test task's metric still present. is there any possible way to remove them?

from dm.

@mxlxm I understand your question now.
This is because when stopping the task, metrics with task-name label were not removed. Maybe we should call some functions like DeleteLabelValues for metrics.

Now, I think you should restart the DM-worker to remove these metrics, and we can remove them automatically when stopping the task later. Or, you can open a PR to do that if you want. 😁

from dm.

😅 haven't read the code, I'll try to do this this weekend.

from dm.

fixed in #575.

from dm.