Support CloudWatch high-resolution metrics about servo HOT 5 CLOSED

We haven't received much demand internally at this point so it hasn't been a priority for us. That said it is probably a simple change, pull requests are welcome.

As an FYI, here are some of the concerns we would have for actually using it at Netflix:

1. It consolidates fairly quickly (3h), this means it could be much harder to debug why scaling activities (our main reason for using CloudWatch) took effect.
2. Cost on instance of collecting and processing the data. A number of our clusters have a lot of metrics per instance and the cost of polling much more frequently would need to be tested. In particular with gauges because the users can put in user defined functions which we have seen in many cases can be more expensive than desired. Not everyone pays attention to the documentation saying the gauge functions should be cheap and avoid any inline IO or network calls.

That said, no objection to supporting it.

from servo.

It consolidates fairly quickly (3h)

That's true. But if you are stuck with CloudWatch, 3h of high-res is better than no high-res.

Cost on instance of collecting and processing the data.

Very valid. What's your typical polling frequency at Netflix? I think we can safely go to something like 5, 10 or 15s but would be interested by any feedback.

That said it is probably a simple change, pull requests are welcome.

Should be a one-liner with the right sdk. Might be doable with the current one, but I will have to check that AwsWebServiceRequest#putCustomQueryParameter does the trick. Any opinion on upgrading the sdk?

from servo.

That's true. But if you are stuck with CloudWatch, 3h of high-res is better than no high-res.

Depends on what you use it for. To just look at I would agree. For the most part we only use CloudWatch to drive auto-scaling, and if there are problems we like to be able to dig in and understand why a certain action was triggered or it behaved a certain way. With only 3h it is likely the relevant data will be lost before the deeper investigation takes place.

What's your typical polling frequency at Netflix? I think we can safely go to something like 5, 10 or 15s but would be interested by any feedback.

The main polling interval is 60s. For some apps we run a finer grained polling interval that defaults to 10s, but some teams vary it. I know at least one team that increased it to 30s because of overhead, but that team produces a lot of metrics. So I would say it depends on your work load.

Should be a one-liner with the right sdk.

Is there any downside to enabling the higher resolution on the call if it isn't actually used? If so, then we would probably want a config option so a user could choose whether or not to use it.

Any opinion on upgrading the sdk?

No big concerns. A lot of our tools are using 1.11.227. So at least up to that version we haven't seen any major issues.

from servo.

Is there any downside to enabling the higher resolution on the call if it isn't actually used?

None that I'm aware of. I will ask for confirmation to the CloudWatch team next week.

from servo.

I just got feedback from AWS support. They initially discouraged me to turn on high-res by default but eventually agreed their is no fundamental issue to do so. I am unsure whether I can share the exchange publicly, but will happily forward it to you.

That being said, I don't know when I will have to time to perform this experiment and will likely do it in micrometer rather than Servo. I will keep you posted on the outcomes.

from servo.