Metrics data loss in K8S controller about agones HOT 9 OPEN

This sounds like works as intended.

1. If a fleet is deleted and we restart the controller, we can't create the old metrics - it's all in memory.
2. If a fleet is deleted we specifically remove it from all metrics reporting to ensure a memory leak / metric explosion doesn't happen (we have to do a full reset to do it).

See #2478 for context.

from agones.

This sounds like works as intended.

1. If a fleet is deleted and we restart the controller, we can't create the old metrics - it's all in memory.
2. If a fleet is deleted we specifically remove it from all metrics reporting to ensure a memory leak / metric explosion doesn't happen (we have to do a full reset to do it).

See #2478 for context.

In my specific scenario, the controller metrics has already malfunctioned before the restart.

1. Start the K8S controller - Agones Controller.
2. Create "shipping-mode1-map1-3568" fleet
3. Check the metrics for the "shipping-mode1-map1-3568" battle server.
4. Observe that the "agones_gameservers_total" metric is being collected, but "agones_gameservers_count" metric is not.

from agones.

That's a good point - will have to attempt to replicate 🤔

from agones.

That's a good point - will have to attempt to replicate 🤔

In our use case, Agones is configured with 10 fleets, and each fleet has a fleet autoscaler enabled. Additionally, 10 separate gameservers have been configured, which are not managed by the fleets.

Hope this can help you successfully reproduce the issue. Thank you for your hard work.

from agones.

That's a good point - will have to attempt to replicate 🤔

Hello，markmandel,

I hope this message finds you well. I wanted to follow up on the issue. Furthermore, I understand that replicating the issue can sometimes be challenging, and I'm wondering if there's any additional information or assistance I can provide to facilitate the process.

If any specific scenarios, logs, or system configurations would be helpful, please let me know. I’m also willing to assist with testing or any other tasks that might help you address the issue more efficiently.

Looking forward to your guidance on how I can best support your efforts. Thank you for your time and attention to this matter.

Best regards

from agones.

Sorry this isn't currently at the top of my priority queue, so haven't had a chance to look at it. Would definitely be happy to provide pointers if you wanted to dig into it?

from agones.

from agones.

If you would like to go digging (and i encourage it!), all these metrics are managed here: https://github.com/googleforgames/agones/tree/main/pkg/metrics

Feel free to drop questions here, or in #development channel on our Slack!

from agones.

We have replicated this issue locally and the agones_gameservers_total is missing after restarting the agones-controller.

Before restart:

After:

from agones.