Giter VIP home page Giter VIP logo

codahale-aggregated-metrics-cloudwatch-reporter's Introduction

CircleCI build master branch Maven Central

CodaHale Aggregated Metrics CloudWatch Reporter

This is a CloudWatch Reporter for the stable version of Dropwizard Metrics (formerly CodaHale & Yammer Metrics). The reporter is an implementation of ScheduledReporter from Dropwizard Metrics v4.1.8

Table of Contents

Prerequisites

  • Java 1.8

Testing

  • ./gradlew test runs unit tests

Code Coverage

A code coverage report will be generated under build/reports/jacoco/html/. You need to open index.html in a browser.

  • ./gradlew clean jacocoTestReport

AWS Java SDK v1 and v2

The master branch is now on AWS Java SDK v2.x.x. If you want to submit pull requests for v1.x.x, please use com_amazonaws_aws_cloudwatch_1.x.x branch.

Summary

  • This CloudWatchReporter reports the metric data to CloudWatch asynchronously using the CloudWatchAsyncClient (AWS) interface
  • Each reportable value in CodeHale Metric is reported as a separate MetricDatum (AWS)
  • When reporting Meter, Counter, Histogram and Timer count metrics (getCount()) as MetricDatum (AWS), only the count difference since the last report is reported. This way the counters do not require a reset within the application using this reporter.
  • If configured, each Snapshot translated into StatisticSet (AWS) in the most direct way possible.
  • If configured, JVM statistic is reported

Reportable Metrics

Currently the only metric values that are reportable through configuration are:

Please note:

  • Histogram values (the percentiles, min, max, sum, arithmetic mean & std-dev from Snapshot) are reported raw.
  • Timer values (the percentiles, min, max, sum, arithmetic mean & std-dev from Snapshot) are reported after conversion by a duration factor was applied. The duration factor is calculated by converting 1 unit of duration unit type to nanoseconds (see ScheduledReporter)
  • Meter values (the 1min rate, 5min rate, 15min rate & mean rate) are reported after conversion by a rate factor was applied. The rate factor is calculated by converting 1 unit of rate unit type to seconds (see ScheduledReporter). The Unit of the sent MetricDatum will default to that rate unit if not changed with builder.withMeterUnitSentToCW().

Metric Dimensions

A dimension is a name/value pair that helps you to uniquely identify a metric. Every metric has specific characteristics that describe it, and you can think of dimensions as categories for those characteristics.

Global Dimensions

The reporter can be configured with a set of global dimensions. These will be added to all reported metrics.

Per Metric Dimensions

The reporter will look for dimensions encoded in the metric name when the metric is reported. To add dimensions to a metric, encode them inside square brackets at the end of the metric name;

"metricName[dimension1:value1,dimension2:value2]"

Use DimensionedName to more easily create metric names with dimensions;

final DimensionedName dimensionedName = DimensionedName.withName("test")
        .withDimension("key1", "val1")
        .withDimension("key2", "val2")
        .withDimension("key3", "val3")
        .build();

metricRegistry.counter(dimensionedName.encode()).inc();

It's also possible to derive a new DimensionedName from an existing one;

final DimensionedName dimensionedName = DimensionedName
        .withName("test")
        .withDimension("key1", "val1")
        .withDimension("key2", "val2")
        .withDimension("key3", "val3")
        .build();

final DimensionedName derivedDimensionedName = dimensionedName
        .withDimension("key3", "replaced_value")
        .withDimension("key4", "val4")
        .build();

metricRegistry.counter(dimensionedName.encode()).inc();
metricRegistry.counter(derivedDimensionedName.encode()).inc();

Since DimensionedName is immutable, the string representation returned by encode() is cached. Multiple calls to encode() will return the same String.

Defaults

The Reporter uses the following defaults which can be configured:

  • Rate metrics are in TimeUnit.SECONDS
  • Duration metrics are in TimeUnit.MILLISECONDS
  • MetricFilter.ALL will be used for the Filter
  • Clock.defaultClock() will be used for the Clock (Unconfigurable)
  • Metrics are reported using standard resolution (can be changed to high resolution)
  • Empty global Dimension (AWS) list
  • The reporter adds a Type Dimension (AWS) to each reported metric, e.g:
Type Metric Name
1-min-mean-rate [per-second] com.example.component.SomeComponent.timer
snapshot-mean [in-milliseconds] com.example.component.SomeComponent.timer
snapshot-mean com.example.component.SomeComponent.histogram
95% com.example.component.SomeComponent.timer
99.5% com.example.component.SomeComponent.timer
99.5% com.example.component.SomeComponent.histogram
count com.example.component.SomeComponent.counter

The only metrics that are reportable by default are:

All other metrics have to be configured for reporting by invoking their respective withXXXX() methods on the CloudWatchReporter.Builder instance

Adding cloudwatch reporter to your project

The library artifact ID is dropwizard-metrics-cloudwatch if you want to search for it on Maven Central

The following is an example how to include the library in your project using Gradle:

repositories {
    mavenLocal()
    maven { url "https://repo.maven.apache.org/maven2" }
    jcenter { url "https://jcenter.bintray.com" }
}

dependencies { 
    implementation("io.github.azagniotov:dropwizard-metrics-cloudwatch:2.0.8")
}

Usage

The reporter provides a fine-grained configuration options through its builder to configure what metrics should be reported to CloudWatch. Since AWS costs money, you probably do not want to report all the values from Metric classes or Snapshot, but only what's really useful to you.

    final CloudWatchAsyncClient amazonCloudWatchAsync =
            CloudWatchAsyncClient
                    .builder()
                    .region(Region.US_WEST_2)
                    .build();

    final CloudWatchReporter cloudWatchReporter =
            CloudWatchReporter.forRegistry(metricRegistry, amazonCloudWatchAsync, Main.class.getName())
                    .convertRatesTo(TimeUnit.SECONDS)
                    .convertDurationsTo(TimeUnit.MILLISECONDS)
                    .filter(MetricFilter.ALL)
                    .withPercentiles(Percentile.P75, Percentile.P99)
                    .withOneMinuteMeanRate()
                    .withFiveMinuteMeanRate()
                    .withFifteenMinuteMeanRate()
                    .withMeanRate()
                    .withArithmeticMean()
                    .withStdDev()
                    .withStatisticSet()
                    .withZeroValuesSubmission()
                    .withReportRawCountValue()
                    .withHighResolution()
                    .withMeterUnitSentToCW(StandardUnit.BYTES)
                    .withJvmMetrics()
                    .withGlobalDimensions("Region=us-west-2", "Instance=stage")
                    .withDryRun()
                    .build();

    cloudWatchReporter.start(10, TimeUnit.SECONDS);

Dry run

The reporter can be configured to run in DRY RUN mode by invoking .withDryRun() on the Builder. In that case, the reporter will log.DEBUG the created instance of PutMetricDataRequest (AWS), instead of doing a real POST to CloudWatch.

Credits

Changelog

2.0.8

  • Changes to build.gradle to ensure that upon publishing an artifact, the dependencies section in a generated POM has a compile scope

2.0.7

  • Some clean up

2.0.6

2.0.5

2.0.4

2.0.3

2.0.2

2.0.1

2.0.0

1.0.13

1.0.12

1.0.11

1.0.10

1.0.9

1.0.8

1.0.7

1.0.6

  • Issue #8: Make it configurable to send zero values.

1.0.5

  • Upgraded Metrics to v3.2.3 due to #1115
  • Upgraded AWS Java SDK to v1.11.179

1.0.4

  • Issue #4: Not reporting metric zero values.
  • PR #6: Reporting Histogram snapshot raw values as StatisticSet, without applying a conversion by duration factor (https://github.com/williedoran)
  • Checking isDebugEnabled when logging debug information

1.0.3

1.0.2

  • PR #2: Updated AWS SDK to com.amazonaws:aws-java-sdk-cloudwatch:1.11.86 (https://github.com/MeiSign)
  • Reporting Histogram snapshot Arithemtic Mean & StdDev raw values, without applying a conversion by duration factor

1.0.1

  • Revisited Javadoc
  • Added dependency on metrics-jvm module in order to be able to export JVM metrics
  • Code clean-up

1.0.0

  • Initial release

License

MIT

codahale-aggregated-metrics-cloudwatch-reporter's People

Contributors

anilkumarmyla avatar azagniotov avatar efenderbosch avatar fyi-coursera avatar jebl01 avatar madanepuri avatar marcelboettcher4007 avatar meisign avatar mmccuiston avatar skuehn avatar tjarkog avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

codahale-aggregated-metrics-cloudwatch-reporter's Issues

Add an working example runtime

Hi, first of all thanks for the work, I've been searching for a codahale metrics cloudwatch reporter and found yours with the best documentation amongst others.

Even though, I wish to ask if there is any working example of how to use it? I must be missing something obvious, but I cant figure out how I do configure connection with AWS Cloudwatch just like i used to do with graphite reporter.

I've used the Readme example configuration... If there is no example, I would be happy to open a PR as soon as i understand how to do so.

Also, I cant use the option withJvmMetrics as it throws ClassNotFoundException: com.codahale.metrics.jvm.ClassLoadingGaugeSet

Add the ability to define Dimensions by metric

In one MetricRegistry there might be multiple metrics, with different data. It would be nice to parse their name and make dimensions from them:

String nameAndDimensions = "Name:dim1=val1,dim2=val2";

metricRegistry.meter.update(nameAndDimensions).mark(value);

Use case: I am measuring response times of my REST API endpoints, and want to have metrics for each one of them. I could make something like:

...

String url = request.url()
String response = response.status().toString();
metricRegistry.meter("Endpoint:url="+url+",status="+status).mark(responseTime);

This way I'd get metrics with dimensions on status and url.

JVMMetrics: A metric named jvm.buffers.direct.count already exists

Hi,

We were adding CloudWatchReporter 2.0.5 to our Dropwizard application.

However, if we call the .withJvmMetrics() method, the application does not start and gives the error

java.lang.IllegalArgumentException: A metric named jvm.buffers.direct.count already exists
at com.codahale.metrics.MetricRegistry.register (MetricRegistry.java:156)
at com.codahale.metrics.MetricRegistry.registerAll (MetricRegistry.java/1675)
at com.codahale.metrics.MetricRegistry.register (MetricRegistry.java:150)
at io.github.azagniotov.metrics.reporter.cloudwatch.CloudWatchReporter $ Builder.build (CloudWatchReporter.java:778)

From what I've seen, this is because the Application class in the run(String ... arguments) method executes bootstrap.registerMetrics();

And the Bootstrap class in turn already registers

    getMetricRegistry (). register ("jvm.attribute", new JvmAttributeGaugeSet ());
    getMetricRegistry (). register ("jvm.buffers", new BufferPoolMetricSet (ManagementFactory
                                                                           .getPlatformMBeanServer ()));
    getMetricRegistry (). register ("jvm.classloader", new ClassLoadingGaugeSet ());
    getMetricRegistry (). register ("jvm.filedescriptor", new FileDescriptorRatioGauge ());
    getMetricRegistry (). register ("jvm.gc", new GarbageCollectorMetricSet ());
    getMetricRegistry (). register ("jvm.memory", new MemoryUsageGaugeSet ());
    getMetricRegistry (). register ("jvm.threads", new ThreadStatesGaugeSet ());

Are we doing something wrong?

withJvmMetrics does not work

The reporter works perfectly for me except JvmMetrics.
When I set withJvmMetrics option I get the following error:

java.lang.NoSuchMethodError: com.codahale.metrics.JmxAttributeGauge.<init>(Ljavax/management/MBeanServerConnection;Ljavax/management/ObjectName;Ljava/lang/String;)V

I assume it is probably problem with metrics version but I have exactly the same version as you defined in the library.
So the question is do you also face that error or is it something on my side?

InvalidParameterValueException

ERROR [2017-06-21 21:40:01,884] [io.github.azagniotov.metrics.reporter.cloudwatch.CloudWatchReporter] Error reporting metrics to CloudWatch. The data in this CloudWatch API request may have been discarded, did not make it to CloudWatch.
com.amazonaws.services.cloudwatch.model.InvalidParameterValueException: The value ? for parameter MetricData.member.3.Value is invalid.

According to blacklocus/metrics-cloudwatch#4 this occurs when the sample count is zero. In that implementation those metrics were filtered away. Does this one require a custom MetricFilter to do likewise? Could the same fix be incorporated?

Incompatible with dropwizard metrics 3, due to regression in 4.0.0

dropwizard metrics throws java.lang.IncompatibleClassChangeError: Found interface com.codahale.metrics.Timer$Context, but class was expected when this library is used on project compiled against dropwizard metrics 3.x . This includes Apache Spark. Upgrading to 4.0.1 or higher fixes the problem.

Reporter only reporting gauge metrics

I've added the CloudWatchReporter and it seems to only report gauge metrics. I can't see the timed or metered metrics.

  final CloudWatchReporter cloudWatchReporter =
               CloudWatchReporter.forRegistry(metricRegistry, amazonCloudWatchAsync, "ServiceName")
                       .convertRatesTo(TimeUnit.SECONDS)
                       .convertDurationsTo(TimeUnit.MILLISECONDS)
                       .filter(MetricFilter.ALL)
                       .withPercentiles(CloudWatchReporter.Percentile.P75, CloudWatchReporter.Percentile.P99)
                       .withOneMinuteMeanRate()
                       .withMeanRate()
                       .withArithmeticMean()
                       .withStdDev()
                       .withStatisticSet()
                       .withZeroValuesSubmission()
                       .withReportRawCountValue()
                       .withHighResolution()
                       .withMeterUnitSentToCW(StandardUnit.BYTES)
                       .withJvmMetrics()
                       .withGlobalDimensions("Region=us-west-2", "Instance=alpha")
                       .build();

Add option to set prefix

It would be cool if I could specify a prefix for the reported metrics.

Usecase

If there are running multiple instances of the same namespace I'd like to be able to Separate the reported metrics per instance.

Separating them on namespace level somehow feels wrong since it's the same application reporting the data. Furthermore it's not possible to specify wildcards when querying for a namespace.

Or did I miss something and this is already possible?

Thanks

This isn't really an issue - I just wanted to say thanks for making this available. It has been really helpful to me.

Feel free to close this as done :-)

Thanks,

Anthony.

Send initial zero value

Hey Alexander,

similar to issue #8 I'd like to propose to send a zero value for metrics that haven't been processed before. The reason for this is that I'd like to create a dashboard based on metrics that ideally never have a value greater zero (e.g. a counter for a "this shouldn't happen" exception) but I still want to save some money and not send a zero on every interval.

Let me know what you think and how I might help. Thanks.

Dimitri

Upgrade from 2.0.5 to 2.0.7 does not work like expected

Hi,

I tried to upgrade from 2.0.5 to 2.0.7, which should work, I imagine ...

But our code doesn't compile anymore

The software.amazon.awssdk.services.cloudwatch.CloudWatchAsyncClient type cannot be resolved. It is indirectly referenced from necessary .class files

This is the error of the code below

final CloudWatchAsyncClient amazonCloudWatchAsync = CloudWatchAsyncClient .builder() .region(this.getAWSRegion()) .build();

Artifact missing on maven central

I would like to use this reporter, but couldn't find the artifact on maven central. Would be great to put it there for ease of use.

Make it configurable to send zero values

Hey Alexander,

first of all: Great work, exactly what I was looking for! Thx!

I would love to see a builder option to send zero values, which isn't done at the moment to save costs. That makes sense, but for our use case we would like to see zeros as well as we would interprete missing values as possible error in the monitoring system.

What do you think? Does this make sense for you?

All the best from Germany
Marcel

Histograms statistic sets sending the wrong max, min, and sum values

On logging several histograms to cloudwatch i was looking out for max values to match what i was seeing in my server logs. They were not the same. I investigated and looked at the stageMetricDatum method which is being used to convert both timer and histogram statistic sets. This method does a conversion by a factor depending on the time duration which makes sense for a timer but doesnt seem to make sense for max, min, sum over time as scaling by the duration shouldnt matter in this case. I added a simple test to show this, and a simple fix in the code base. I have a pull request to hopefully fix this if i understand the code correctly

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.