Giter VIP home page Giter VIP logo

jmxfetch's Introduction

Build Status

JMXFetch

JMXFetch is the new tool to collect metrics from JMX Servers in order to be sent to Datadog (http://www.datadoghq.com) It is called by the Datadog Agent (https://github.com/Datadog/datadog-agent) and sends metrics back to the Agent using the Java dogstatsd library.

How to contribute code?

First of all and most importantly, thank you for sharing.

If you want to submit code, please fork this repository and submit pull requests against the master branch.

Please note that the Agent is licensed for simplicity's sake under a simplified BSD license, as indicated in the LICENSE file. Exceptions are marked with LICENSE-xxx where xxx is the component name. If you do not agree with the licensing terms and wish to contribute code nonetheless, please email us at [email protected] before submitting your pull request.

Building from source

JMXFetch uses Maven for its build system. The repo contains a Maven Wrapper, so you don't need to download and install Maven.

In order to generate the JAR artifact, run the ./mvnw clean compile assembly:single command in the cloned directory.

The distribution will be created under target/.

To use this JAR in the Agent, see these docs.

To test your JAR with a local test-server, see below instructions for "Local Testing"

Note

If you want build all the JAR files for JMXFetch, you need to use an older JDK version like JDK 8. There is a known issue where the build can't find javadoc command on modern JDKs. The quickest way to build these JAR files is to use Docker:

docker run -it --rm \
    --name my-maven-project \
    -v "$(pwd)":/usr/src/app \
    -w /usr/src/app \
    eclipse-temurin:8-jdk ./mvnw -DskipTests clean package

Coding standards

JMXFetch uses Checkstyle with Google Java Style to enforce coding standards.

To perform a Checkstyle analysis and outputs violations, run:

./mvnw checkstyle::check

Checkstyle analysis is automatically executed prior to compiling the code, testing.

Generated code

JMXFetch uses Lombok to modify classes and generate additional code at runtime. You may need to enable annotation processors to compile in your IDE.

Useful Developer Settings

JDK version management

sdkman is recommended to manage multiple versions of Java. If you are an sdkman user, there is a config file present in this project with the recommended JDK version for development, use sdk env to activate it.

Logging Options

The logging output is configured in code in CustomLogger.java. The following system properties will affect the format of the log records.

  • -Djmxfetch.filelinelogging=true
    • All log records will include the specific file and line number which emitted that log.
  • -Djmxfetch.millisecondlogging=true
    • All log records will include milliseconds in the timestamp, rather than the default 'second' timestamp resolution.

Local Testing

You can utilize the provided testing server misbehaving-jmx-server with the preset ./docker-compose.yaml file. This runs two containers, one is the test server and the other is the Datadog Agent running your local JAR's version of JMXFetch.

  1. docker compose up -d

Testing

To run unit test, issue the following command:

./mvnw test

Some tests utilize TestContainers which requires a docker client. You can rely on the CI to run these, or if you have docker installed on Linux these should work out of the box.

If you're on macOS or Windows, docker desktop is architected to run a linux VM which then runs all your containers. This makes the networking a bit different and you should use the following command to run the tests.

docker run -it --rm -e TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal -v $PWD:$PWD -w $PWD -v /var/run/docker.sock:/var/run/docker.sock eclipse-temurin:8-jdk ./mvnw test

This version runs the maven jmxfetch tests within a container as well, which works as long as the TEST_CONTAINERS_HOST_OVERRIDE env var is set.

Testing Deployments

You can test the deployment by using the Nexus3 OSS image. To do so you need to:

  • Use the local endpoints in the pom.xml:
...
    <distributionManagement>
        <snapshotRepository>
          <id>nexus</id>
          <url>http://localhost:8081/repository/maven-snapshots/</url>
        </snapshotRepository>
        <repository>
            <id>nexus</id>
            <url>http://localhost:8081/repository/maven-releases/</url>
        </repository>
    </distributionManagement>
...
    <build>
        <plugins>
            ...
            <plugin>
                <groupId>org.sonatype.plugins</groupId>
                ...
                <configuration>
                    <serverId>nexus</serverId>
                    <nexusUrl>http://localhost:8081/repository/maven-releases/</nexusUrl>
                    <autoReleaseAfterClose>false</autoReleaseAfterClose>
                </configuration>
            </plugin>
        </plugins>
    </build>
...
  • Start the Nexus 3 container:
docker run --rm -d \
  --name nexus3 \
  -p 8081:8081 \
  -e INSTALL4J_ADD_VM_PARAMS='-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m -Djava.util.prefs.userRoot=/nexus-data/javaprefs' \
  sonatype/nexus3
docker exec -it nexus3 cat /nexus-data/admin.password
  • Export those values for the build:
export GPG_KEY="<Key ID for signing the artifact>"
export GPG_PASSPHRASE="<Password for the private key of $GPG_KEY>"
export SONATYPE_USER="admin"
export SONATYPE_PASS="<Password for Nexus admin from previous step>"
  • Run the deploy with the appropriate skipStaging flag:
./mvnw -DskipTests -DskipStaging=true -DperformRelease=true --settings settings.xml clean deploy

If you do this correctly, the artifact will be available in the Nexus container at http://localhost:8081/#browse/browse:maven-releases.

Note: If you are trying to redeploy the same version of the artifact to the local Nexus repository, you will need to change the Deployment Policy for maven-releases to Allow redeploy in the UI otherwise the subsequent publishes will fail.

Running:

Get help on usage:
java -jar jmxfetch-0.49.3-SNAPSHOT-jar-with-dependencies.jar --help

Updating Maven Wrapper

To upgrade the Maven Wrapper, you need to run:

./mvn wrapper:wrapper -Dmaven=<Maven Version X.Y.Z>

The easiest way to regenerate the wrapper files (mvnw, mvnw.cmd, .mvn/wrapper/maven-wrapper.jar and .mvn/wrapper/maven-wrapper.properties) is to use Docker:

docker run -it --rm \
    --name my-maven-project \
    -v "$(pwd)":/usr/src/app \
    -w /usr/src/app maven:3 mvn wrapper:wrapper -Dmaven=<Maven Version X.Y.Z>

Leave out the -Dmaven=<Maven Version X.Y.Z> to get the latest version of Maven.

jmxfetch's People

Contributors

albertvaka avatar alexandreyang avatar arbll avatar carlosroman avatar cmetz100 avatar ganeshkumarsv avatar gh123man avatar gmmeyer avatar hantingzhang2 avatar hush-hush avatar ian28223 avatar isaacdd avatar jpbempel avatar labbati avatar mar-kolya avatar nmuesch avatar ogaca-dd avatar olivielpeau avatar prognant avatar rayz avatar remeh avatar remh avatar richardstartin avatar scottopell avatar sgnn7 avatar therve avatar truthbk avatar tylerbenson avatar vickenty avatar yannmh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jmxfetch's Issues

memory leak when jmxfetch can't connect to an instance

After 13 days of failing to connect to a jmx instance:

2014-05-19 19:59:22,832 | ERROR| App | Cannot connect to instance 127.0.0.1:7199 java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
    java.net.ConnectException: Connection refused]

jmxfetch grew to about 6gb:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dd-agent 27364  0.1 75.9 9722184 6218296 ?     Sl   May06  20:56      \_ java -jar /usr/share/datadog/agent/checks/libs/jmxfetch-0.3.0-jar-with-dependencies.jar --check jmx.yaml --check_period 15000 --conf_di

OS and Java version coming soon

Support for String conversions

Need the ability to convert from a string (eg. 'RUNNING') to a metric that we can alert on

I took a stab at adding this functionality. Can you review my approach and let me know if it's suitable for a Pull request.

coupacooke@9aac3b5

What's the right way to tell jmxfetch to refresh its cached beans list?

I have jmxfetch to plug lots of metrics off of a Kafka jvm. Every one a while, for good reasons, I may have to restart my Kafka process. If the running jmxfetch process does a refresh beans while the restart is happening, the number of mbeans it detects is significantly lower than what a normal properly running Kafka process support, and for the next 10 mins the metrics that plug off from this small number of mbeans will cause my Datadog monitors to alert due to "No data" being submitted.

The question is, what is the right way to restart the jmxfetch process? It appears that it is in a supervisor managed set up. Some trial and error suggests that sending a -HUP signal to the jmxfetch jvm will cause the jmxfetch process to restart itself, thus refresh its bean list.

Is this documented somewhere on how I should do this? TIA.

Rotate log file

Rotation is not implemented yet and it should follow a similar pattern as other agent logs.

Access to java.lang.Object attributes

I hit the same issue as #118
No errors an no data:

$>bean org.springframework.boot:name=metricsEndpoint,type=Endpoint
#bean is set to org.springframework.boot:name=metricsEndpoint,type=Endpoint
$>info
#mbean = org.springframework.boot:name=metricsEndpoint,type=Endpoint
#class name = org.springframework.boot.actuate.endpoint.jmx.DataEndpointMBean

attributes

%0 - Data (java.lang.Object, r)
%1 - EndpointClass (java.lang.String, r)
%2 - Sensitive (boolean, r)

operations

%0 - java.lang.Object getData()
%1 - java.lang.String getEndpointClass()
%2 - boolean isSensitive()
#there's no notifications
$>get Data
#mbean = org.springframework.boot:name=metricsEndpoint,type=Endpoint:
Data = {
mem = 893013;
mem.free = 590049;
processors = 4;
instance.uptime = 181587;
uptime = 201589;
systemload.average = 1.78955078125;
heap.committed = 798720;
heap.init = 131072;
heap.used = 208670;
heap = 1864192;
nonheap.committed = 96664;
nonheap.init = 2496;
nonheap.used = 94293;
nonheap = 0;
threads.peak = 24;
threads.daemon = 22;
threads.totalStarted = 31;
threads = 24;
classes = 12646;
classes.loaded = 12646;
classes.unloaded = 0;
gc.ps_scavenge.count = 16;
gc.ps_scavenge.time = 301;
gc.ps_marksweep.count = 3;
gc.ps_marksweep.time = 424;
httpsessions.max = -1;
httpsessions.active = 0;
datasource.primary.active = 0;
datasource.primary.usage = 0.0;
};

$>get Data.mem
#mbean = org.springframework.boot:name=metricsEndpoint,type=Endpoint:

How can i get metrics inside Data structure? Any help are welcome.
Thanks.

Can I exclude default metrics in java.lang.*?

Hi, I have a question.

I defined /etc/dd-agent/conf.d/jmx.yaml like the following. exclude is set for ignoring bean java.lang.*.


---
instances:
- name: jdbc_master
  host: 127.0.0.1
  port: 8086
  tags:
    pool: master
  conf:
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*master.*
    exclude:
      bean_regex:
        - java.lang.*
        - .*PoolConfig.*
- name: jdbc_slave
  host: 127.0.0.1
  port: 8086
  tags:
    pool: slave
  conf:
  - include:
      domain: com.zaxxer.hikari
      bean_regex:
        - .*slave.*
    exclude:
      bean_regex:
        - java.lang.*
        - .*PoolConfig.*

# Nothing to configure here
init_config:

But the config seems to be ignored.

service datadog-agent jmx list_collected_attributes

#####################################
Instance: 127.0.0.1:8086
#####################################

       Matching: 1/350. Bean name: java.lang:type=Threading - Attribute name: ThreadCount  - Attribute type: int
       Matching: 2/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 3/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 7/350. Bean name: java.lang:type=Memory - Attribute name: HeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 11/350. Bean name: java.lang:type=Memory - Attribute name: NonHeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 12/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 13/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 14/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: TotalConnections  - Attribute type: int
       Matching: 15/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: IdleConnections  - Attribute type: int
       Matching: 16/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: ActiveConnections  - Attribute type: int
       Matching: 17/350. Bean name: com.zaxxer.hikari:type=Pool (master) - Attribute name: ThreadsAwaitingConnection  - Attribute type: int

#####################################
Instance: 127.0.0.1:8086
#####################################

       Matching: 1/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: TotalConnections  - Attribute type: int
       Matching: 2/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: IdleConnections  - Attribute type: int
       Matching: 3/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: ActiveConnections  - Attribute type: int
       Matching: 4/350. Bean name: com.zaxxer.hikari:type=Pool (slave) - Attribute name: ThreadsAwaitingConnection  - Attribute type: int
       Matching: 5/350. Bean name: java.lang:type=Threading - Attribute name: ThreadCount  - Attribute type: int
       Matching: 6/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 7/350. Bean name: java.lang:type=GarbageCollector,name=G1 Young Generation - Attribute name: CollectionTime  - Attribute type: long
       Matching: 11/350. Bean name: java.lang:type=Memory - Attribute name: HeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 15/350. Bean name: java.lang:type=Memory - Attribute name: NonHeapMemoryUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 16/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionCount  - Attribute type: long
       Matching: 17/350. Bean name: java.lang:type=GarbageCollector,name=G1 Old Generation - Attribute name: CollectionTime  - Attribute type: long

Any way to ignore java.lang.* metrics?

memory leak if instance unreachable

If an instance is unreachable (from start-up, not sure what might happen if it goes down), there is a pretty bad memory leak in jmxfetch.

Apparently the start-up logic trying to connect to the instance, not being able to do so, and then going over the entire motion in the next iteration causes the leak. Could be other scenarios, but this one is certain and easy to reproduce.

In the scenario described the leak is ~10MB/15 minutes - so bad.

In case of bad configuration, jmxfetch shouldn't log the traceback

It's ugly and scary.
It should also try to give more details about what's wrong in the yaml file.

jmx.yaml here

/var/log/datadog/jmxfetch.log

2014-01-03 16:28:42,477 | INFO | App | Found 1 yaml files.
2014-01-03 16:28:42,478 | INFO | App | Reading /etc/dd-agent/conf.d/jmx.yaml
2014-01-03 16:28:42,485 | ERROR| App | Unable to create instance. Please check your yaml file
java.lang.NullPointerException
        at org.datadog.jmxfetch.Instance.<init>(Instance.java:70)
        at org.datadog.jmxfetch.App.init(App.java:289)
        at org.datadog.jmxfetch.App.init(App.java:208)
        at org.datadog.jmxfetch.App.main(App.java:53)
2014-01-03 16:28:42,489 | WARN | App | No instance could be initiated. Retrying initialization.
2014-01-03 16:28:42,523 | INFO | App | Found 1 yaml files.
2014-01-03 16:28:42,523 | INFO | App | Reading /etc/dd-agent/conf.d/jmx.yaml
2014-01-03 16:28:42,525 | ERROR| App | Unable to create instance. Please check your yaml file
java.lang.NullPointerException
        at org.datadog.jmxfetch.Instance.<init>(Instance.java:70)
        at org.datadog.jmxfetch.App.init(App.java:289)
        at org.datadog.jmxfetch.App._doLoop(App.java:102)
        at org.datadog.jmxfetch.App.main(App.java:59)
   ...done.

100% IO Wait

I don't know why, every a few hours, jmxfetch eats up all of the Disk IO.
It also lasts for several minutes and after that the process dies.

screenshot from 2015-08-21 16-37-37

screenshot from 2015-08-21 16-37-38

NullPointerException when logging unsupported attribute type

java.lang.NullPointerException
at
javax.management.MBeanAttributeInfo.hashCode(MBeanAttributeInfo.java:217)
at java.lang.Object.toString(Object.java:209)
at java.lang.String.valueOf(String.java:2615)
at java.lang.StringBuilder.append(StringBuilder.java:116)
at org.datadog.jmxfetch.Instance._getMatchingAttributes(Instance.java:195)
at org.datadog.jmxfetch.Instance.init(Instance.java:90)
at org.datadog.jmxfetch.App.init(App.java:295)
at org.datadog.jmxfetch.App.main(App.java:82)

Running v 0.2.0 but issue still exists on master

JVM GC metrics in jmx-1.yaml are named wrong

The garbage collector metrics auto-added by jmxfetch in jmx-1.yaml are incorrectly labeled with specific GC types that are not universally used.

e.g. the CollectionCount for all collector types is named jvm.gc.cms.count, even though it is the metric name for counts for all collector types (of which there are many).
Similarly CollectionTime is labeled jvm.gc.parnew.time for all collectors.

Obviously this is just naming, and you can functionally use the metrics, but it's a bit (facepalm) for a Java engineer to see.

These metrics would more appropriately be named jvm.gc.count and jvm.gc.time - the existing tags created for the name part of the bean name allow the different collectors to be identified.

"Unexpected exception while initiating instance" exception

Here is output of 'datadog-agent jmx list_collected_attributes' command:

#####################################
Instance: localhost:9110
#####################################

       Matching: 1/350. Bean name: java.lang:type=GarbageCollector,name=ParNew - Attribute name: CollectionCount  - Attribute type: long
       Matching: 2/350. Bean name: java.lang:type=GarbageCollector,name=ParNew - Attribute name: CollectionTime  - Attribute type: long
       Matching: 3/350. Bean name: java.lang:type=GarbageCollector,name=ConcurrentMarkSweep - Attribute name: CollectionCount  - Attribute type: long
       Matching: 4/350. Bean name: java.lang:type=GarbageCollector,name=ConcurrentMarkSweep - Attribute name: CollectionTime  - Attribute type: long
       Matching: 5/350. Bean name: java.lang:type=Threading - Attribute name: ThreadCount  - Attribute type: int
       Matching: 6/350. Bean name: java.lang:type=OperatingSystem - Attribute name: OpenFileDescriptorCount  - Attribute type: long
jmx[instance:staticcache] - 1433262458 = ERROR
2015-06-02 12:27:38,113 | ERROR | App | Unexpected exception while initiating instance localhost:9110 : 1
java.lang.ArrayIndexOutOfBoundsException: 1
    at org.datadog.jmxfetch.JMXAttribute.getBeanParametersHash(JMXAttribute.java:70)
    at org.datadog.jmxfetch.JMXAttribute.<init>(JMXAttribute.java:59)
    at org.datadog.jmxfetch.JMXSimpleAttribute.<init>(JMXSimpleAttribute.java:24)
    at org.datadog.jmxfetch.Instance.getMatchingAttributes(Instance.java:221)
    at org.datadog.jmxfetch.Instance.init(Instance.java:120)
    at org.datadog.jmxfetch.App.init(App.java:330)
    at org.datadog.jmxfetch.App.main(App.java:91)

Integration doesn't work for us b/c of this. Please let me know if you need more data.

Connection to statsd server can break and is not reset

JMXFetch sends metrics to statsd server properly but dogstatsd doesn't receive them.

There are no errors in the logs but it's the most likely reason.

We need a way to ensure a reliable connection to the dogstastd server.

Not sure if this should be fixed in jmxfetch or in the java statsd library.

use a table to print jmx metric listings

instead of

       Matching: 1/100. Bean name: org.apache.cassandra.internal:type=MiscStage - Attribute name: TotalBlockedTasks  - Attribute type: int
       Matching: 2/100. Bean name: org.apache.cassandra.internal:type=MiscStage - Attribute name: CurrentlyBlockedTasks  - Attribute type: int
       Matching: 3/100. Bean name: org.apache.cassandra.internal:type=MiscStage - Attribute name: CoreThreads  - Attribute type: int
       Matching: 4/100. Bean name: org.apache.cassandra.internal:type=MiscStage - Attribute name: MaximumThreads  - Attribute type: int

a more readable version might be something like (obviously the wrong content) which doesn't repeat the column names on every row.

+-----------+------+------------+-----------------+
| City name | Area | Population | Annual Rainfall |
+-----------+------+------------+-----------------+
| Adelaide  | 1295 |  1158259   |      600.5      |
| Brisbane  | 5905 |  1857594   |      1146.4     |
| Darwin    | 112  |   120900   |      1714.7     |
| Hobart    | 1357 |   205556   |      619.5      |
| Melbourne | 1566 |  3806092   |      646.9      |
| Perth     | 5386 |  1554769   |      869.4      |
| Sydney    | 2058 |  4336374   |      1214.8     |
+-----------+------+------------+-----------------+

we could use:

https://code.google.com/p/prettytable/

or just a csv

Add tests

Not to be "that guy" but this doesn't have any test coverage yet.

It would be nice to have some tests around this so if people change it later on they can know that didn't break anything.

Properly cast port

Originally was:
DataDog/dd-agent#777

Copied here:
In current dadadog 4.x version if yaml (for example tomcat) has option port: set as "12345" (note " instead of just pure number), datadog read config, start jmxfetch but
$datadog-agent info shows
Unexpected exception while initiating instance localhost:11198 : java.lang.String cannot be cast to java.lang.Integer
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
at org.datadog.jmxfetch.Connection.(Connection.java:49)
at org.datadog.jmxfetch.ConnectionManager.getConnection(ConnectionManager.java:43)
at org.datadog.jmxfetch.Instance.init(Instance.java:81)
at org.datadog.jmxfetch.App.init(App.java:298)
at org.datadog.jmxfetch.App.init(App.java:208)
at org.datadog.jmxfetch.App.main(App.java:53)

Previous version behave differently. It seems wize to try casting port to int() or log error\warning.

Fails to get metrics if started while application is initializing

If I start my application and immediately start jmxfetch it fails to get some (not all) metrics and never recovers from this state. It forever complains that the instance is not returning any metrics.

Here's an extract from the log: https://gist.github.com/sschepens/6efa136bf5ae5e06c07a
They key errors are:

2015-08-28 13:27:49,433 | WARN | App | Instance application:7199 didn't return any metrics
2015-08-28 13:27:49,454 | WARN | App | Instance application:7199 didn't return any metrics.Maybe the server got disconnected ? Trying to reconnect.
2015-08-28 13:27:49,455 | INFO | App | Trying to reconnect to: application:7199

After restarting jmxfetch, it logs:

2015-08-28 13:31:10,561 | INFO | App | JMX Fetch has started
2015-08-28 13:31:10,562 | INFO | App | Reading /etc/dd-agent/conf.d/jmx.yaml
2015-08-28 13:31:10,788 | INFO | App | Reading /etc/dd-agent/conf.d/tomcat.yaml
2015-08-28 13:31:10,822 | INFO | App | Found 2 config files
2015-08-28 13:31:10,824 | WARN | Instance | Cannot find a "conf" section in jmx_instance
2015-08-28 13:31:10,875 | INFO | Instance | Trying to connect to JMX Server at application:7199
2015-08-28 13:31:10,876 | INFO | ConnectionManager | Connection closed or does not exist. Creating a new connection!
2015-08-28 13:31:10,876 | INFO | ConnectionManager | Connecting using JMX Remote
2015-08-28 13:31:10,907 | INFO | Connection | Connecting to: service:jmx:rmi:///jndi/rmi://application:7199/jmxrmi
2015-08-28 13:31:11,143 | INFO | Instance | Connected to JMX Server at application:7199
2015-08-28 13:31:12,209 | INFO | Instance | Found 7 matching attributes
2015-08-28 13:31:12,213 | INFO | Instance | Trying to connect to JMX Server at application:7199
2015-08-28 13:31:12,225 | INFO | ConnectionManager | Connection already exists for key: application:7199:null . Using it...
2015-08-28 13:31:12,225 | INFO | Instance | Connected to JMX Server at application:7199
2015-08-28 13:31:12,463 | WARN | Instance | Unable to get metrics from Catalina:type=JspMonitor,name=jsp,WebModule=//localhost/,J2EEApplication=none,J2EEServer=none
2015-08-28 13:31:13,064 | WARN | Instance | Unable to get metrics from Catalina:type=Cache,host=localhost,context=/
2015-08-28 13:31:13,113 | INFO | Instance | Found 53 matching attributes
2015-08-28 13:31:13,133 | INFO | App | Instance jmx_instance is sending 13 metrics to the metrics reporter during collection #1
2015-08-28 13:31:13,362 | INFO | App | Instance tomcat_instance is sending 57 metrics to the metrics reporter during collection #1
2015-08-28 13:31:28,502 | INFO | App | Instance jmx_instance is sending 13 metrics to the metrics reporter during collection #2

And everything magically works

Unable to refresh bean list for instance

I've been trying to setup the JMX datadog integration and running into, I believe, connection issues. I've setup a jar process that runs JMX and I've verified that I can connect to it using jconsole. I then use datadog's JMX integration to collect stats. In the /var/log/datadog/jmxfetch.log I have this output:

2016-12-05 17:54:46,474 | WARN | App | Instance localhost:9999 didn't return any metrics.Maybe the server got disconnected ? Trying to reconnect.
2016-12-05 17:54:46,478 | INFO | App | Trying to reconnect to: localhost:9999
2016-12-05 17:54:46,478 | INFO | Instance | Trying to connect to JMX Server at localhost:9999
2016-12-05 17:54:46,478 | INFO | ConnectionManager | Connection closed or does not exist. Creating a new connection!
2016-12-05 17:54:46,478 | INFO | ConnectionManager | Connecting using JMX Remote
2016-12-05 17:54:46,478 | INFO | Connection | Connecting to: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
2016-12-05 17:54:46,486 | INFO | Instance | Connected to JMX Server at localhost:9999

Which makes it seem like it has successfully connected. However, in the /opt/datadog-agent/run/jmx_status.yaml I get this error:

checks:
  failed_checks: {}
  initialized_checks:
    jmx:
    - {message: 'Unable to refresh bean list for instance localhost:9999', service_check_count: 0,
      status: ERROR, metric_count: 0, instance_name: jmx_formplayer_instance}

Are there any ways to get more verbose output to debug this error? Are there common causes for the Unable to refresh bean list error? Thanks! Any help is appreciated. My jmx config is here:

init_config:
  custom_jar_paths: # optional
    - /home/cchq/www/staging/current/formplayer_build/formplayer.jar

instances:
  - host: localhost
    port: 9999

    name: jmx_formplayer_instance

    tags:
      env: staging
      newTag: formplayer

    conf:
      - include:
          bean_regex:
            - .*Controller.*
            - .*Service.*
            - .*Factory.*
            - .*Aspect.*
            - .*Filter.*

And these are my java jmx jar args: -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9999

3 jmxfetch processes running on i-ef5f658f (staging)

I'm assuming there should be only one?

dd-agent 13596     1  1 Jul31 ?        06:23:15 java -jar /usr/share/datadog/agent/checks/libs/jmxfetch-0.0.1-SNAPSHOT-jar-with-dependencies.jar /etc/dd-agent/conf.d 8125 15000 /var/log/datadog/jmxfetch.log FINE tomcat.yaml,activemq.yaml,solr.yaml,cassandra.yaml,jmx.yaml
dd-agent 13703     1  0 Jul31 ?        06:20:27 java -jar /usr/share/datadog/agent/checks/libs/jmxfetch-0.0.1-SNAPSHOT-jar-with-dependencies.jar /etc/dd-agent/conf.d 8125 15000 /var/log/datadog/jmxfetch.log FINE tomcat.yaml,activemq.yaml,solr.yaml,cassandra.yaml,jmx.yaml
dd-agent 14072     1  0 Jul31 ?        06:14:02 java -jar /usr/share/datadog/agent/checks/libs/jmxfetch-0.0.1-SNAPSHOT-jar-with-dependencies.jar /etc/dd-agent/conf.d 8125 15000 /var/log/datadog/jmxfetch.log FINE tomcat.yaml,activemq.yaml,solr.yaml,cassandra.yaml,jmx.yaml

Improvement: add support for attribute regex

Hi all,

What

I'd like to suggest to add support for MBean attribute's name regex, in the same fashion as the current domain_regex and bean_regex.

The desired behaviour would be that any attribute matching the regex would be included in list of matchingAttributes and tags would be attached to specify the matched value (as for bean_regex)

Why

Some Mbeans expose a variable set of attributes whose name contain information about what is being monitored.

My primary interest here is to be able to monitor Kafka's consumer lag. Kafka consumers expose one MBean for each client id and provide this lag for each partition of each topic they consume with as many MBean attributes, following this naming pattern: {topic}-{partition}.records-lag. See kafka 0.11 consumer monitoring documentation for more details.

In the jconsole, that MBean looks like this:

image

How

The selection of MBean attributes to monitor is happening in this if statement of Instance.getMatchingAtrributes(), which ultimately delegates to either JMXSimpleAttribute.matchAttribute(), JMXComplexAttribute.matchAttribute() or JMXTabularAttribute.

I suggest to refactor those three in order to support matching based on regex.

I also suggest to update SimpleAttribute.addAdditionalTag() to add tags to document the matched value (in my kafka example: the topic name and partition number).

In order to populate those tags we need to have a tag name. I suggest to allow the user to specify them as part of the regex, as documented here in the java 7 regex documentation.

Who

I'm happy to start working on this and submit a PR if the approach above sounds ok to you.

What do you think ?

Connect to JMX via sockets

To configure jmx, users should be able to utilize a socket connection, instead of having to connect to a host and port.

Negative rates in tomcat checks

It's been reported by Org 2912.

It happens either during a restart or a war deployment.

It's hasn't been confirmed on the latest version of JMXFetch as they were using the Beta at some point that had a similar bug but that was fixed for the official release.

process_name_regex cannot see all JVMs

The process_name_regex construct, to enumerate & connect to JVMs using the AttachAPI, can only see JVMs running as the same user account; in the default install of the dd agent this is only the jmxfetch JVM itself, running as dd-agent user - other JVMs aren't visible.

Running jmxfetch as root allows visibility to all JVMs.

Improved tagging capabilities

I have a similar request to #103 where I want to whitelist tags, specifically the name tag that comes from the MBean name property.

The issue I'm having is that we have tagged our ec2 hosts with name and we can not easily get system-wide results when creating graphs with something along the lines of get jmx.domain.attribute1, from attribute_group:group1, sum by name since it splits it by both the bean name and the instance name.

To solve this, you could use something similar to what you have done aliases e.g.

include:
  attribute:
    attribute1:
      alias: jmx.$domain.$attribute
      metric_type: gauge
      tags_include:
        - "attribute_group:group1"
        - "bean_name:$name"
  bean_regex: Domain:name=group1-*

I was also thinking though that what I really want is to put the tags on all attributes of the bean, not just one. The above config becomes repetitive with multiple attributes. It would be great if the config could be simplified further to:

include:
  attribute:
    - attribute1
    - attribute2
  bean_regex: Domain:name=group1-*
  tags_include:
     attribute_group: group1
     bean_name: $name

Maybe I am missing something? Is there a way to achieve this with the current jmxfetch?

Cassandra metrics change starting with version 1.2

Cassandra now exposes both:

org.apache.cassandra.db.ColumnFamilies <<<<<<<<<<< old
org.apache.cassandra.metrics.ColumnFamily <<<<<<<<<<< new

See this for more details on the new style.

One key change is that the old format was:

  1. CF is an MBean
  2. CF has a bunch of attributes that map to metrics
    while the new format has
  3. CF is a scope
  4. the MBean is the metric name
  5. the attribute is named "Value"

How to configure JMX-JBOSS in Datadog

Hi,

I have configured JMX in jboss to get the metrics in datadog.But i am unable to get the metrics.Will you pls give the steps to configure the jboss in datadog

Ability to filter tags out of an attribute?

Hi,
I'm wiring up jmxfetch to a kafka producer JVM. It includes a "client-id" tag(?) that I'd like to filter out, as historically large number of unique tags have been problematic to our datadog install. Is this possible with jmxfetch?

The log line when it discovers the metric:

2016-06-13 21:55:08,313 | DEBUG| Instance | Getting attributes for bean: kafka.producer:type=producer-node-metrics,client-id=9388b6bb-539b-4bc5-8d60-14be420574d6,node-id=node-8
2016-06-13 21:55:08,319 | DEBUG| Instance | Attribute: kafka.producer:type=producer-node-metrics,client-id=9388b6bb-539b-4bc5-8d60-14be420574d6,node-id=node-8 : javax.management.MBeanAttributeInfo[description=The average number of requests sent per second., name=request-rate, type=double, read-only, descriptor={}] has attributeInfo simple type
2016-06-13 21:55:08,322 | DEBUG| Instance | Attribute: kafka.producer:type=producer-node-metrics,client-id=9388b6bb-539b-4bc5-8d60-14be420574d6,node-id=node-8 : javax.management.MBeanAttributeInfo[description=The average size of all requests in the window.., name=request-size-avg, type=double, read-only, descriptor={}] has attributeInfo simple type
2016-06-13 21:55:08,326 | DEBUG| Instance | Attribute: kafka.producer:type=producer-node-metrics,client-id=9388b6bb-539b-4bc5-8d60-14be420574d6,node-id=node-8 : javax.management.MBeanAttributeInfo[description=, name=incoming-byte-rate, type=double, read-only, descriptor={}] has attributeInfo simple type

Fetch metrics for java.lang.Object attributes

I'm trying to fetch attributes that are java.lang.Object which seem to be supported based off of this. The issue is the collector crashes when trying to collect that attribute. Here is the output of a Data attribute for Spring actuator using the jmxterm-1.0-DATADOG-uber.jar:

$>bean
org.springframework.boot:type=Endpoint,name=metricsEndpoint
$>get Data
#mbean = org.springframework.boot:type=Endpoint,name=metricsEndpoint:
Data = {
  mem = 1241887;
  mem.free = 233668;
  processors = 4;
  instance.uptime = 243348342;
  uptime = 243367358;
  systemload.average = 1.95;
  heap.committed = 1159168;
  heap.init = 63168;
  heap.used = 925499;
  heap = 3262464;
  nonheap.committed = 83456;
  nonheap.init = 24000;
  nonheap.used = 82719;
  nonheap = 133120;
  threads.peak = 49;
  threads.daemon = 40;
  threads.totalStarted = 1727;
  threads = 44;
  classes = 12401;
  classes.loaded = 12561;
  classes.unloaded = 160;
  gc.ps_scavenge.count = 6604;
  gc.ps_scavenge.time = 143260;
  gc.ps_marksweep.count = 103;
  gc.ps_marksweep.time = 54414;
  httpsessions.max = -1;
  httpsessions.active = 0;
  datasource.primary.active = 0;
  datasource.primary.usage = 0.0;
  datasource.hq.active = 0;
  datasource.hq.usage = 0.0;
  counter.status.200.answer = 15249;
  counter.status.200.delete-incomplete-form = 1;
  counter.status.200.delete_application_dbs = 57;
  counter.status.200.formatted_questions = 4;
  counter.status.200.get_sessions = 31;
  counter.status.200.incomplete-form = 19;
  counter.status.200.navigate_menu = 1751;
  counter.status.200.serverup = 4128;
  counter.status.200.submit-all = 512;
  counter.status.200.sync-db = 18;
  counter.status.200.validate_form = 241;
  gauge.response.answer = 621.0;
  gauge.response.delete-incomplete-form = 22.0;
  gauge.response.delete_application_dbs = 25.0;
  gauge.response.formatted_questions = 249.0;
  gauge.response.get_sessions = 7679.0;
  gauge.response.incomplete-form = 8206.0;
  gauge.response.navigate_menu = 417.0;
  gauge.response.serverup = 1.0;
  gauge.response.submit-all = 11387.0;
  gauge.response.sync-db = 7330.0;
  gauge.response.validate_form = 17.0;
 };

All of which seem to be doubles that should be parsable by jmxfetch. My conf for this bean is:

- include:
          bean:
            - org.springframework.boot:type=Endpoint,name=metricsEndpoint
          attribute:
            - Data

The jmxfetch process finds the value when I use /etc/init.d/datadog-agent jmx list_matching_attributes, and in the jmxfetch.log I get this error:

2017-01-02 08:24:23,095 | WARN | JMXAttribute | Unable to get metrics from org.springframework.boot:type=Endpoint,name=metricsEndpoint - Data
java.lang.NumberFormatException
        at org.datadog.jmxfetch.JMXAttribute.getValueAsDouble(JMXAttribute.java:250)
        at org.datadog.jmxfetch.JMXSimpleAttribute.getValue(JMXSimpleAttribute.java:103)
        at org.datadog.jmxfetch.JMXSimpleAttribute.getMetrics(JMXSimpleAttribute.java:32)
        at org.datadog.jmxfetch.JMXAttribute.getMetricsCount(JMXAttribute.java:185)
        at org.datadog.jmxfetch.Instance.getMatchingAttributes(Instance.java:256)
        at org.datadog.jmxfetch.Instance.init(Instance.java:133)
        at org.datadog.jmxfetch.App.init(App.java:354)
        at org.datadog.jmxfetch.App.main(App.java:92)

Any insight into how to configure the yaml file to properly read this would be very helpful. Thank you!

Be more flexible on configuration

  • Support wildcards
  • Support list of filters instead of simple filters.
  • ...

Use case:
I want to exclude a few column families for the cassandra integration.

Error parsing list of tags

JMX-fetch only expects tags to be listed as a map in the YAML file but there are cases when they might be arrays. One example is the output of the template parsing in Service Discovery, see https://github.com/DataDog/dd-agent/blob/69d58afeea2126ab2a7e1e1efba6f3cbdafc7f33/utils/service_discovery/sd_docker_backend.py#L410-#L420

In such cases the error looks like this:

java.lang.ClassCastException: java.util.ArrayList cannot be cast to java.util.LinkedHashMap
	at org.datadog.jmxfetch.Instance.<init>(Instance.java:71)
	at org.datadog.jmxfetch.App.init(App.java:485)
	at org.datadog.jmxfetch.App.start(App.java:241)
	at org.datadog.jmxfetch.App.main(App.java:122)

We should fix JMX-fetch so it can accept tags in both formats.

Attempt to cast java.lang.String as an int?

It would be super handy if attributes of type java.lang.String where attempted to be cast as an int before flat out rejecting them.

So many java applications just post all attributes as java.lang.String, even if it's a number. Cutting out a pretty big slice of metrics that would be useful to monitor.

No support for type: java.util.map

Migrating a bunch of hosts from current monitoring solution to Datadog, discovered that this JMX type was not supported by Datadog.

Example output from JTerm:

#mbean = sme:name=ActiveCampaignStats:
CampaignCount = {
  Europe = 4563;
  Asia = 1128;
  All = 82134;
 };
$>

Which in JConsole might look something like:

AttributeName:        AttributeValue:
=======================================
CampaignCount         {Europe=4563,Asia=1128,All=82134}

Add tags to returned metrics

We want to add arbitrary tags (eg. env:prod, role:atom) to our JMX checks on a per-instance level (but still have the instance, server come through).

Is it possible today?

NullPointerException if no attribute alias is specified

The following attribute configuration results in a NullPointerException in 0.12.0:

            - include:
               bean: org.datadog.jmxfetch.test:type=SimpleTestJavaApp,scope=Co|olScope,host=localhost,component=
               attribute:
                    ShouldBe100:
                        metric_type: gauge

I reproduced it in a unit test:
Exception is:

java.lang.NullPointerException
	at org.datadog.jmxfetch.JMXAttribute.getUserAlias(JMXAttribute.java:445)
	at org.datadog.jmxfetch.JMXAttribute.getAlias(JMXAttribute.java:385)
	at org.datadog.jmxfetch.JMXAttribute.getAlias(JMXAttribute.java:462)
	at org.datadog.jmxfetch.JMXSimpleAttribute.getMetrics(JMXSimpleAttribute.java:31)
	at org.datadog.jmxfetch.Instance.getMetrics(Instance.java:164)
	at org.datadog.jmxfetch.App.doIteration(App.java:182)
	at org.datadog.jmxfetch.TestCommon.run(TestCommon.java:99)
	at org.datadog.jmxfetch.TestApp.testBeanTags(TestApp.java:31)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Support for Booleans

Should support Booleans so we can alert on them.

Took a stab at this as well - converts from false/true to 0/1. Can you review and let me know it's suitable for a pull
coupacooke@9fa26e6

WARN not accepted as a valid log_level

Valid log levels in JMX currently include: "ALL", "DEBUG", "ERROR", "FATAL", "INFO", "OFF", "TRACE", "LEVEL", whereas the Python part of the agent accepts "WARN".

dogstatsd errors on cassandra data submissions

from i-ef5f658f (staging), /var/log/datadog/dogstatsd.log

2013-08-26 19:58:18,648 | ERROR | dd.dogstatsd | dogstatsd(dogstatsd:240) | Error receiving datagram
Traceback (most recent call last):
  File "/usr/bin/dogstatsd", line 231, in start
    aggregator_submit(socket_recv(buffer_size))
  File "/usr/share/datadog/agent/aggregator.py", line 370, in submit_packets
    name, value, mtype, tags, sample_rate = self.parse_metric_packet(packet)
  File "/usr/share/datadog/agent/aggregator.py", line 299, in parse_metric_packet
    raise Exception('Metric value must be a number: %s, %s' % (name, metadata[0]))
Exception: Metric value must be a number: cassandra.db.recent_write_latency_micros, ?
2013-08-26 19:58:18,649 | ERROR | dd.dogstatsd | dogstatsd(dogstatsd:240) | Error receiving datagram
Traceback (most recent call last):
  File "/usr/bin/dogstatsd", line 231, in start
    aggregator_submit(socket_recv(buffer_size))
  File "/usr/share/datadog/agent/aggregator.py", line 370, in submit_packets
    name, value, mtype, tags, sample_rate = self.parse_metric_packet(packet)
  File "/usr/share/datadog/agent/aggregator.py", line 299, in parse_metric_packet
    raise Exception('Metric value must be a number: %s, %s' % (name, metadata[0]))
Exception: Metric value must be a number: cassandra.db.key_cache_recent_hit_rate, ?
2013-08-26 19:58:18,650 | ERROR | dd.dogstatsd | dogstatsd(dogstatsd:240) | Error receiving datagram
Traceback (most recent call last):
  File "/usr/bin/dogstatsd", line 231, in start
    aggregator_submit(socket_recv(buffer_size))
  File "/usr/share/datadog/agent/aggregator.py", line 370, in submit_packets
    name, value, mtype, tags, sample_rate = self.parse_metric_packet(packet)
  File "/usr/share/datadog/agent/aggregator.py", line 299, in parse_metric_packet
    raise Exception('Metric value must be a number: %s, %s' % (name, metadata[0]))
Exception: Metric value must be a number: cassandra.db.row_cache_recent_hit_rate, ?

How to configure YAML for nested javax.management.openmbean.CompositeData type?

Migrating a bunch of hosts from current monitoring solution to Datadog, hit a snag when trying to format YAML for a nested CompositeData.

Example output from JTerm:

$>get LastSuccessfulRun
#mbean = mybean:type=test:
LastSuccessfulRun = {
  elapsedTime = 220;
  mapOutputRuntime = {
    records = 37901;
    runtime = 23463;
   };
  cellStats = {
    TypeFilter = {
      Data loader runtime (ms) = -1;
      Filter runtime (ms) = -1;
     };
    allDataLoadersRuntime = 6611;
    allFiltersRuntime = 83921;
   };
  domainLoadRuntimes = {
    ( APPLY_SETS ) = {
      name = APPLY_SETS;
      records = 11923;
      runtime = 12;
     };
    ( RIGHT_SITE ) = {
      name = RIGHT_SITE;
      records = 1612;
      runtime = 25;
     };
   };
 };

Several different layers represented in this snippet, the only thing here I have been successfully able to capture is elapsedTime = 220; This was done successfully using the formatting demonstrated here: https://github.com/DataDog/jmxfetch/blob/master/src/main/resources/jmx-2.yaml#L3-L14

I tried adding another period (.) separator for the nested scenarios as shown below (as well as a few other variations), but it didn't seem to work. Any advice would be appreciated, thanks.

    conf:
      - 
        include: 
          attribute:
            LastSuccessfulRun.mapOutputRuntime.records:
              alias: test.LastSuccessfulRune_mapOutputRuntime_records
              metric_type: gauge 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.