jmxtrans / jmxtrans-agent Goto Github PK
View Code? Open in Web Editor NEWJava Agent based JMX metrics exporter.
License: MIT License
Java Agent based JMX metrics exporter.
License: MIT License
Logging to stdout
could be problematic when chaining multiple programs, since in linux people usually assumes stdout
to be actual output of a problem.
jmxtrans-agent-1.2.4.jar StatsDOutputWriter is indefinitelly failing with "java.nio.BufferOverflowException". It seems error was triggered by random failed DNS resolution, that caused some buffer (lim=91) to fill up.
2017-03-07 03:19:29.614 SEVERE [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Could not send stat java.nio.HeapByteBuffer[pos=0 lim=91 cap=1024] to host telegraf:8125
java.io.IOException: Target address not resolved
at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:425)
at org.jmxtrans.agent.StatsDOutputWriter.flush(StatsDOutputWriter.java:157)
at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:128)
at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2017-03-07 03:19:29.614 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Given data too big (98bytes) for the buffer size (91bytes), skip it: jmxtransagents.xx...
2017-03-07 03:19:29.615 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Exception collecting java.lang:type=OperatingSystem#MaxFileDescriptorCount
java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at java.nio.ByteBuffer.put(ByteBuffer.java:832)
at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:136)
at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
...
2017-03-07 08:26:33.896 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.StatsDOutputWriter - Given data too big (104bytes) for the buffer size (91bytes), skip it: jmxtransagents.xx...
2017-03-07 08:26:33.896 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Exception collecting java.lang:type=OperatingSystem#SystemCpuLoad
java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:183)
at java.nio.ByteBuffer.put(ByteBuffer.java:832)
at org.jmxtrans.agent.StatsDOutputWriter.doSend(StatsDOutputWriter.java:136)
at org.jmxtrans.agent.StatsDOutputWriter.writeQueryResult(StatsDOutputWriter.java:118)
at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.writeQueryResult(OutputWriterCircuitBreakerDecorator.java:90)
at org.jmxtrans.agent.Query.processAttributeValue(Query.java:274)
at org.jmxtrans.agent.Query.collectAndExportAttribute(Query.java:239)
at org.jmxtrans.agent.Query.collectAndExportForObjectName(Query.java:173)
at org.jmxtrans.agent.Query.collectAndExport(Query.java:167)
at org.jmxtrans.agent.TimeTrackingCollector.collectIfEnoughTimeHasPassed(TimeTrackingCollector.java:54)
at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:209)
at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Follow-up on the email thread "Elastichsearch output writer strategies" initiated by @zepouet .
https://groups.google.com/forum/#!topic/jmxtrans/jIyYctTW2go
It would be nice to have an ElasticSearchOutputWriter.
ElasticSearchOutputWriter
have a configurable index name capable of using SimpleDateFormat to add a temporal element in the index name. Default value jmxtrans-%{yyyy.MM.dd}
. See hereFollowing 2 constructors for Query
assume resultAlias
as the attributeName
.
This works fine if we are polling limited set of attributes and attribute names are self-explanatory. However, with the #34 we are allowing capability to export all the attributes and that IMO makes this assumption unnecessary.
If user does not specify the resultAlias
, then we can rely on ResultNameStrategy to generate the appropriate name comprised of <objectname>.<attribute-name>
.
/**
* @see #Query(String, String, String, Integer, String, String, ResultNameStrategy)
*/
public Query(@Nonnull String objectName, @Nullable String attribute, @Nonnull ResultNameStrategy resultNameStrategy) {
this(objectName, attribute, null, null, null, attribute, resultNameStrategy);
}
/**
* @see #Query(String, String, String, Integer, String, String, ResultNameStrategy)
*/
public Query(@Nonnull String objectName, @Nullable String attribute, int position, @Nonnull ResultNameStrategy resultNameStrategy) {
this(objectName, attribute, null, position, null, attribute, resultNameStrategy);
}
Sponsored by: Lookout, Inc.
To make it easier for users to get started with jmxtrans-agent it might be helpful to provide some bundled configuration files. That would make it possible to test jmxtrans-agent without having to go through the hassle creating a configuration file. I can also imagine that for some use cases these bundled files would be good enough.
I'm thinking a configuration that polls the typical MBeans on a vanilla HotSpot JVM like memory, GC, threads, CPU, ...
We could bundle configuration files for the most common output writers and use property expansion with reasonable defaults to allow users to configure the specifics with system properties or a properties file.
Thoughts?
DiscoveryQuery : Used to discover a list of JMX beans matching a specific naming pattern.
Used with the Zabbix server.
For example, the following discovery rule :
<discoveryQuery
objectName="java.lang:type=GarbageCollector,name=*"
attributes="name,type"
resultAlias="discovery[garbageCollector]"
collectIntervalInSeconds="300"/>
<query objectName="java.lang:type=GarbageCollector,name=*"
attributes="CollectionTime,CollectionCount"
resultAlias="discovery[GarbageCollector.%name%.#attribute#]" />
May yield the following discovery output (formatted for readability) :
{"data":[
{"{#NAME}":"PS Scavenge","{#TYPE}":"GarbageCollector"},
{"{#NAME}":"PS MarkSweep","{#TYPE}":"GarbageCollector"}
]}
On the Zabbix side, create a "Discovery Rule" of type "Zabbix trapper"
with a "Key" that matches the result alias. You can then create "Item prototypes" that use the values.
Sample Zabbix configuration that matches the example above :
Discovery rule :
Name : Discover Garbage Collectors
Key : discovery[garbageCollector]
Item Prototype
Name : Object {#TYPE} named {#NAME}
Key : discovery[{#TYPE}.{#NAME}.CollectionTime]
Item Prototype
Name : Object {#TYPE} named {#NAME}
Key : discovery[{#TYPE}.{#NAME}.CollectionCount]
Hello mate,
This tool is awsome in our use case where we retrieving metric data from application jvm and feed into graphite.
Just wondering if you could provide us an example for querying ActiveMQ? such as listing every queue sizes (Enqueue count?) .
Thanks again
James
This is an enhancement request.
It would be so much easier to further develop the API if the OutputWriter interface method
/**
* @param metricName
* @param metricType see {@link Query#type}
* @param value
* @throws IOException
*/
void writeQueryResult(@Nonnull String metricName, @Nullable String metricType, @Nullable Object value) throws IOException;
Could be replaced with
/**
* @param metricName
* @param queryResult see {@link QueryResult}
* @param value
* @throws IOException
*/
void writeQueryResult(@Nonnull String metricName, @Nullable QueryResult queryResult) throws IOException;
In that case the backward incompatible changes to the API (such as including extra fields, which are required by the new Writers) would be so much easier to implement.
Happy to assist with rewriting.
Hello,
I'm trying to add jmxtrans-agent to internal java with springboot application.
It's started with :
/opt/jdk1.8/bin/java -javaagent:/servers/jmxtrans-agent/jmxtrans-agent.jar=/servers/jmxtrans-agent/jmxtrans-agent.xml -Dinfluxdb_instance=dots-server_01 -Dorg.jmxtrans.agent.JmxTransAgent.diagnostic=true -Dorg.jmxtrans.agent.util.logging.Logger.level=TRACE -Djavax.net.ssl.trustStore=/etc/digiplug/cacerts -Djavax.net.ssl.trustStorePassword=changeit -jar /servers/dgpapps/dots/dots-server.jar --slot.id=01
It correctly generates some point and finishes with a stack :
sept. 19 16:27:56 java[79309]: 2017-09-19 16:27:56.965 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.JmxTransExporter - Ignore exception flushing metrics
sept. 19 16:27:56 java[79309]: java.net.SocketTimeoutException: Connect timed out
sept. 19 16:27:56 java[79309]: at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:126)
sept. 19 16:27:56 java[79309]: at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:502)
sept. 19 16:27:56 java[79309]: at java.net.Socket.connect(Socket.java:589)
sept. 19 16:27:56 java[79309]: at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
sept. 19 16:27:56 java[79309]: at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
sept. 19 16:27:56 java[79309]: at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
sept. 19 16:27:56 java[79309]: at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
sept. 19 16:27:56 java[79309]: at sun.net.www.http.HttpClient.New(HttpClient.java:339)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1209)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1157)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
sept. 19 16:27:56 java[79309]: at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.writeMetrics(InfluxDbOutputWriter.java:192)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.sendMetrics(InfluxDbOutputWriter.java:160)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.sendMetrics(InfluxDbOutputWriter.java:153)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.influxdb.InfluxDbOutputWriter.postCollect(InfluxDbOutputWriter.java:147)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.OutputWriterCircuitBreakerDecorator.postCollect(OutputWriterCircuitBreakerDecorator.java:124)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.OutputWritersChain.postCollect(OutputWritersChain.java:70)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.JmxTransExporter.collectAndExport(JmxTransExporter.java:214)
sept. 19 16:27:56 java[79309]: at org.jmxtrans.agent.JmxTransExporter$2.run(JmxTransExporter.java:131)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
sept. 19 16:27:56 java[79309]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
sept. 19 16:27:56 java[79309]: at java.lang.Thread.run(Thread.java:748)
My xml :
<?xml version="1.0" encoding="UTF-8"?>
<jmxtrans-agent>
<queries>
<query objectName="java.lang:type=Threading" attributes="DaemonThreadCount,ThreadCount,PeakThreadCount" resultAlias="#attribute#" />
<query objectName="java.lang:type=Runtime" attribute="Uptime" resultAlias="Uptime" />
<query objectName="java.lang:type=OperatingSystem" attributes="OpenFileDescriptorCount,ProcessCpuTime" resultAlias="#attribute#" />
<query objectName="java.lang:type=ClassLoading" attribute="LoadedClassCount" resultAlias="LoadedClassCount" />
<query objectName="java.lang:type=Compilation" attribute="TotalCompilationTime" resultAlias="TotalCompilationTime" />
<query objectName="java.lang:type=GarbageCollector,name=*" attributes="CollectionCount,CollectionTime" resultAlias="%name%.#attribute#" />
<query objectName="java.lang:type=Memory" attributes="HeapMemoryUsage,NonHeapMemoryUsage" resultAlias="#attribute#.#key#" />
<query objectName="java.lang:type=MemoryPool,name=*" attributes="Usage,PeakUsage" resultAlias="%name%.#attribute#.#key#" />
<query objectName="Catalina:type=GlobalRequestProcessor,name=*" attributes="bytesReceived,bytesSent,requestCount,processingTime" resultAlias="%name%.#attribute#" />
<query objectName="Catalina:type=RequestProcessor,name=*" attributes="bytesReceived,bytesSent,requestCount,processingTime" resultAlias="%name%.#attribute#" />
<query objectName="Catalina:type=ThreadPool,name=*" attribute="currentThreadCount,currentThreadsBusy" resultAlias="%name%.#attribute#" />
<query objectName="Catalina:type=Manager,context=*,host=*" attribute="activeSessions" resultAlias="activeSessions" />
</queries>
<outputWriter class="org.jmxtrans.agent.influxdb.InfluxDbOutputWriter">
<url>http://xx.yy.zz.aa:8086</url>
<database>java_db</database>
<tags>host=#hostname#,instance=${influxdb_instance}</tags>
<reloadConfigurationCheckIntervalInSeconds>60</reloadConfigurationCheckIntervalInSeconds>
<connectTimeoutMillis>3000</connectTimeoutMillis>
<readTimeoutMillis>5000</readTimeoutMillis>
</outputWriter>
<outputWriter class="org.jmxtrans.agent.ConsoleOutputWriter">
<enabled>false</enabled>
</outputWriter>
<collectIntervalInSeconds>10</collectIntervalInSeconds>
</jmxtrans-agent>
On the same server, I have a logstash that export correctly with the same xml and jar. I can graph it.
I see the "Ignore exception flushing metrics" exception at https://github.com/jmxtrans/jmxtrans-agent/blob/master/src/main/java/org/jmxtrans/agent/JmxTransExporter.java
But, I have no idea what it may mean.
I tried to debug by myself, but couldn't progress further.
Gerard.
For example, I am querying activemq and want to have a list of queues back, and in graphite, I want to see them in their respective treeview folders with meaningful hierarchy.
is this possible to do with jmxtrans-agent?
Regards
Found it in the code, but thought it might be useful for the docs since i have my graphite resolution at 1 minute
60
thanks for doing an amazing job with this project.
Introduce query.position
configuration for multi-valued attributes (arrays and iterables) to specify which entry will be outputted in the writers.
If position
is not specified, all the values of the attribute are outputted with the name ${resultAlias}_${position}
.
We have a DNS endpoint that contains multiple IP addresses to our KairosDB servers. These servers can change at any given time, but will update DNS as new ones come online.
Taking Cassandra as an example, when cassandra starts it checks the dns entry and gets an IP address and uses that to connect jmxtrans-agent. If that IP becomes no longer available, I get a "broken pipe" and it retries that same IP forever. Is it possible to have it try the connection over (playing by the TTL rules of DNS and getting an updated IP address in the process?
I don't mind forking and adding this in myself, just wanted to make sure there wasn't a flag or anything I could set.
Thanks!
The io.resource framework introduced in jmxtrans-agent 1.2.3 causes a load-time failure if the classpath URI is not hierarchical. For example:
java -javaagent:org.jmxtrans.agent.jmxtrans-agent-1.2.3.jar=classpath:jmxtrans-config.xml ...
SEVERE [main] org.jmxtrans.agent.JmxTransAgent - Exception loading JmxTransExporter from 'classpath:jmxtrans-config.xml'
java.lang.IllegalArgumentException: URI is not hierarchical
at java.io.File.<init>(File.java:418)
at org.jmxtrans.agent.util.io.ClasspathResource.getFile(ClasspathResource.java:68)
at org.jmxtrans.agent.util.io.IoUtils.getFileAsDocument(IoUtils.java:153)
at org.jmxtrans.agent.JmxTransConfigurationXmlLoader.loadConfiguration(JmxTransConfigurationXmlLoader.java:80)
at org.jmxtrans.agent.JmxTransExporter.loadNewConfiguration(JmxTransExporter.java:74)
at org.jmxtrans.agent.JmxTransExporter.<init>(JmxTransExporter.java:70)
at org.jmxtrans.agent.JmxTransAgent.initializeAgent(JmxTransAgent.java:99)
at org.jmxtrans.agent.JmxTransAgent.premain(JmxTransAgent.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
When building the current master branch the build fails on LibratoMetricsIntegrationTest
. Both tests in that class fail in the @Before
method with the following error:
java.lang.AssertionError: File src/test/resources/my-librato-config.properties is missing. See librato-config.template.properties for details
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at org.jmxtrans.agent.LibratoMetricsIntegrationTest.before(LibratoMetricsIntegrationTest.java:48)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
....
In librato-config.template.properties
it says:
Create a file my-librato-config.properties with the following properties
Am I expected to do that myself to get the build to pass? If so, I suggest changing that behavior so that the project can be built from source without modifications.
It would be awesome if the configuration could be dynamically reloaded. It would make the agent much more flexible in long-running applications.
I suggest adding a <reloadConfigurationOnChanges>true/false</reloadConfigurationOnChanges>
element. Defaults to false
.
Enabling this would re-initialize the JmxTransExporter
when changes are detected in the configuration file.
I can see some issues where stuff like output writer connections are not cleaned up properly, but that should not be too hard to fix.
If you agree that it would be a desired feature we can look into implementing it.
I'm a big fan of jmxtrans-agent & Zabbix, so I implemented an output writer that supports the Zabbix protocol :
<outputWriter class="org.jmxtrans.agent.zabbix.ZabbixTcpOutputWriter">
<host>zabbixserver</host>
<port>10051</port>
<serverName>myhostname</serverName>
</outputWriter>
Hoping to integrate this change in the master branch.
It would be great if you guys supposed influxDB
http://influxdb.org/
https://github.com/novaquark/metrics-influxdb
I added jmxtrans-agent to a Java process (a Confluence install) and I had as part of the configuration the following small typo in the configuration:
<jmxtrans-agent>
<queries>
<!-- 8< snip 8< -->
<query objectname="java.nio:type=BufferPool,name=*" attribute="TotalCapacity" resultAlias="jvm.buffers.%name%.capacity"/>
</queries>
<!-- 8< snip 8< -->
</jmxtrans-agent>
i.e. I had objectname
instead of objectName
. This resulted in lots of log noise like the following:
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:j2eeType=Servlet,WebModule=//localhost/,name=status-servlet,J2EEApplication=none,J2EEServer=none'#TotalCapacity, exception: Cannot find attribute TotalCapacity for StandardEngine[Standalone].StandardHost[localhost].StandardContext[].StandardWrapper[status-servlet]
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:type=MBeanFactory'#TotalCapacity, exception: Cannot find attribute TotalCapacity for org.apache.catalina.mbeans.MBeanFactory@516ed85c
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Confluence:name=MailTaskQueue'#TotalCapacity, exception: getAttribute failed: ModelMBeanAttributeInfo not found for TotalCapacity
2017-06-08 10:05:36.505 WARNING [jmxtrans-agent-1] org.jmxtrans.agent.Query - Failed to fetch attribute for 'Standalone:j2eeType=Servlet,WebModule=//localhost/,name=default,J2EEApplication=none,J2EEServer=none'#TotalCapacity, exception: Cannot find attribute TotalCapacity for StandardEngine[Standalone].StandardHost[localhost].StandardContext[].StandardWrapper[default]
As it tries to find the TotalCapacity
attribute for every bean. This gets repeated every collectIntervalInSeconds
and rapidly fills the logs up.
Is there a need where the objectName
attribute is ever optional? Or is it possible to have some XML validation performed on the configuration file so any unknown attributes cause an error/exception?
We would like to collect some of our metrics more frequently than others. Hence we would like support for specifying different collect intervals.
If we can agree on how it should work we are prepared to implement it. The most simple solution I see is to allow for an override on query
and invocation
elements:
<query objectName="java.lang:type=Threading" attribute="ThreadCount" resultAlias="jvm.thread"
collectIntervalInSeconds="10"/>
This override would be used when specified. The default would be used for queries which do not have an override.
Another option would to add support for groups of queries/invocations where some configuration parameters could be overridden on group level. Something like:
<queries>
<group name="foo">
<query ... />
<invocation ... />
<collectIntervalInSeconds>10</collectIntervalInSeconds>
</group>
<group name="bar">
...
</group>
<!-- defaults -->
<collectIntervalInSeconds>20</collectIntervalInSeconds>
</queries>
While I can see the benefits of groups in overriding other parameters (e.g. distinct output writers per group), it is a larger change which adds a bit of complexity.
Any thoughts on this?
InfluxDB has a concept of multiple fields and tags per measurement for storing time series metrics in an optimized way.
The current implementation of InfluxDbOutputWriter is not taking advantage of that, it is only possible to create one field value per metric with fixed tags defined at the outputWriter declaration.
But it is not a limitation imposed by the outputWriter but at the query API.
I'd like to propose a more flexible but generic way to have more properties per query, so we could expand the implementation options in the outputWriters so not only InfluxDbOutputWriter would benefit from it. The ideia would be something like adding a propert list inside query definition as follows:
<query objectName="sample:instance=*,name=rollingCount,type=business_counters" attribute="value" resultAlias="business_counters">
<properties>
<property name="host" type="tag" value="${HOSTNAME}"/>
<property name="%instance%" type="field" value="%attribute%">
</properties>
</query>
Does it make sense to you guys? I'd like to hear your thoughts.
@cyrille-leclerc At the following location, resultName is artificially modified after it is computed by the ResultNameStrategy:
In reality, we should let the resultName strategy take care of it.
It was one of the purposes of issue 39.
Copy of jmxtrans/embedded-jmxtrans#85
@mheffner said
I'm from the Librato team. It would help us to include a descriptive User-Agent header in the LibratoWriter that includes "jmxtrans" and the version number of jmxtrans, eg. "jmxtrans/1.2.3". This would assist in our debugging when working with users that are sending data with jmxtrans to Librato.
We have a full write-up here: http://support.metrics.librato.com/knowledgebase/articles/175825-how-to-set-a-custom-user-agent-when-integrating-wi
Hi,
I am trying to send some beans related to the memory pool, but I do not get them, so my guess is I am doing something wrong in the jmxtrans-agent config file (other beans does received): The beans are in the form of:
where XXX is one of {"Eden Space", "Survivor Space", "Tenured Gen" ... }
My guess is I am doing something wrong with 'objectName' value. Does anyone see the problem?
Thanks.
The use case is as follows:
When polling values it is easy to miss spikes for things such as CPU utilization or memory usage. The obvious fix when using polling is to poll more frequently. However, as jmxtrans-agent is built now, that also means metrics are sent to the backend more frequently, which results in increased load and storage requirements.
Hence it would be useful to be able to poll values frequently (think every second) but only send a max/min/avg/median aggregate more rarely.
While the functionality would be useful I'm not sure it fits within jmxtrans-agent. Other solutions would be to use statsd (although that introduces another moving part). Another solution would to instead build a separate agent that exposes new MBeans that keep a rolling min/max/avg and query those beans from jmxtrans-agent.
What are your thoughts on this? Should we look into implementing this in jmxtrans-agent or should we try to limit the scope to what it currently does? @cyrille-leclerc do you have any thoughts?
The following query exports all the attributes of the given objectName:
<query objectName="java.lang:type=Memory" resultAlias="my.localmachine"/>
This query, generates the following output:
my.localmachine false 1432740403
my.localmachine 0 1432740403
my.localmachine 257425408 1432740403
my.localmachine 268435456 1432740403
my.localmachine 3817865216 1432740403
my.localmachine 79592824 1432740403
my.localmachine 50003968 1432740403
my.localmachine 24576000 1432740403
my.localmachine 136314880 1432740403
my.localmachine 49349944 1432740403
my.localmachine java.lang:type=Memory 1432740403
It uses the resultAlias
for all the exported attributes, which makes it impossible to distinguish:
Instead the jmxtrans-agent should append the attribute and compositeDataKey (if applicable) to the resultName to make it self-explanatory. E.g. :
my.localmachine.Verbose false 1432742044
my.localmachine.ObjectPendingFinalizationCount 0 1432742044
my.localmachine.HeapMemoryUsage.committed 257425408 1432742044
my.localmachine.HeapMemoryUsage.init 268435456 1432742044
my.localmachine.HeapMemoryUsage.max 3817865216 1432742044
my.localmachine.HeapMemoryUsage.used 79746712 1432742044
my.localmachine.NonHeapMemoryUsage.committed 50003968 1432742044
my.localmachine.NonHeapMemoryUsage.init 24576000 1432742044
my.localmachine.NonHeapMemoryUsage.max 136314880 1432742044
my.localmachine.NonHeapMemoryUsage.used 49385432 1432742044
my.localmachine.ObjectName java.lang:type=Memory 1432742044
Yes. The expressionLanguageEngine is available, but it does not allow us to add attribute or compositeData key to the resultName. It simple enhances the existing resultAlias.
Sponsored by: Lookout, Inc.
As discussed in pull request #29 by @cyrille-leclerc and @petermodzelewski, creating an issue to allow support for: Builds names with general rules like JConsole / VisualVM do.
i.e. .
E.g. For objectName "type:name=metric,value=bar" => getResultName() will return resultName = "type.metric.bar"
Sponsored by: Lookout, Inc.
Currently all the files/modules are located in top package level agent
directory.
@cyrille-leclerc let me know if you are ok if I restructure the modules into separate directories? It will really help with manageability.
E.g.
com.jmxtrans.agent
Hello Guys,
I encountered a problem with the InfluxDbOutputWriter. The floating point metrics, for example :
java.lang:type=OperatingSystem > ProcessCpuLoad
causes jmxtrans-agent to crash.
Try this :
<query objectName="java.lang:type=OperatingSystem" attribute="ProcessCpuLoad" resultAlias="ProcessCpuLoad"/>
You will obtain :
WARNING [jmxtrans-agent-1] org.jmxtrans.agent.JmxTransExporter - Ignore exception flushing metrics java.lang.RuntimeException: Failed to write metrics, response code: 400, response message: Bad Request ...
My environment :
jmxtrans-agent : 1.2.4
InfluxDB : 1.1
Tomcat : 7.0.73
JRE : Oracle 1.8.0_72-b15
Thank you for your help
I want to get the processingTime metrics for only HTTP connector from the Globalrequestprocessor mbean. Hence I had given the query in jmxtransagent.xml like
query objectName="Catalina:type=GlobalRequestProcessor,name=http-nio-8080" attribute="processingTime" resultAlias="tomcat.processingTime"
instead of
query objectName="Catalina:type=GlobalRequestProcessor,name=*" attribute="processingTime"
resultAlias="tomcat.processingTime"
I am not getting any data related to processingTime in the outputwriter(influxdb, console or fileoutputwriter).
Can anyone let me know what wrong am I doing ?
jaspserserver reports allows access to diagnostic information via jmx through a custom URL
service:jmx:rmi://localhost/jndi/rmi://:10990/jasperserver
when using this via jconsole or java mission control you need to provide a username and password for the application.
Is there a way to add this into the jmxtrans config to access this type of data?
thanks we really love this agent
so I have an mBean in a proprietary app that has a value of long[60], When I poll this with jmxtrans I get the whole array, which is not ideal, but ok. Using the jmxtrans-agent the output to the graphite writer looks like:
query.test [J@6987d334 1375372603
on a side note, sending one value of the array would be super for both.
For metrics based on CompositeData keys, the key name is appended to the alias specified with resultAlias configuration.
It would be useful to have the sensuwriter available in the agent
Statsd allows for collection of stats in a few different data types: https://github.com/etsy/statsd/blob/master/docs/metric_types.md
I propose adding a bucketType field to the statsd output writer to allow configuring these different types of buckets - particularly the GAUGE type.
Is there any reason this hasn't been implemented? I can see in the code it is hardcoded to COUNT type. One issue I foresee is you inherently have the same bucketType for all queries, which will likely not be the case ie. you are collecting gauges and times but you've configured your output for a gauge.
I have implemented and semi-tested the GAUGE type so far.
Google Metric Writer should be able to source authentication token from http://metadata.google.internal when run in Google Container Engine.
Token should have preference over GOOGLE_APPLICATION_CREDENTIALS for authentication purposes.
Hey,
I wrote a RollingFileOutputWriter. It acts very much like a log4j log. It has the following parameters:
<fileName>
is the file you want to write to.<maxFileSize>
is the maximium size in mb before file rolls over. It is optional. Default value is 10mb and maximum file size is 10mb.<maxBackupIndex>
is the number of files you want to keep. It is an optional value that defaults to 5. When a file becomes larger the <maxFileSize>
it then becomes <fileName>.1
this continues up until <maxbackupIndex>
(default <fileName>.4
). Much like log4j rolling file appendenderHere is a sample config:
<outputWriter class="org.jmxtrans.agent.RollingFileOutputWriter">
<fileName>rollingJMXOutputFile</fileName>
<maxFileSize>10</maxFileSize>
<maxBackupIndex>4</maxBackupIndex>
</outputWriter>
Sample output looks like below. Time is in GMT:
[May 16, 2014 1:19:44 PM] os.systemLoadAverage 1.35400390625
[May 16, 2014 1:19:44 PM] jvm.heapMemoryUsage.used 246736032
[May 16, 2014 1:19:44 PM] jvm.heapMemoryUsage.committed 389021696
[May 16, 2014 1:19:44 PM] jvm.nonHeapMemoryUsage.used 30183280
[May 16, 2014 1:19:44 PM] jvm.nonHeapMemoryUsage.committed 30867456
[May 16, 2014 1:19:44 PM] jvm.loadedClasses 4161
[May 16, 2014 1:19:44 PM] jvm.thread 25
[May 16, 2014 1:19:44 PM] tomcat.requestCount 1
[May 16, 2014 1:19:44 PM] tomcat.errorCount 0
[May 16, 2014 1:19:44 PM] tomcat.processingTime 261
[May 16, 2014 1:19:44 PM] tomcat.bytesSent 47142
[May 16, 2014 1:19:44 PM] tomcat.bytesReceived 0
[May 16, 2014 1:19:44 PM] application.activeSessions 0
My company deploys this with every tomcat instance deployed in AWS then consume the logs using AWS CloudWatch, its working very well. It requires java 1.7 though since I uses java.nio.File (but that can be changed to work with 1.6). Let me know if you want a pull request, I just have to undo a change int he FileOverwriter that prints the timestamp.
As discussed in pull request #29 by @cyrille-leclerc and @petermodzelewski, creating an issue to allow support for:
Adding more power for XML configuration
Following changes were made as a result:
getResultName()
in ResultNameStrategy
was modified to pass attributeName, because query.getAttributeName()
could be nullIt will allow support for following configurations:
<queries>
<query/>
</queries>
<queries>
<query objectName="java.lang:type=Memory"/>
</queries>
Sponsored by: Lookout, Inc.
hey there,
This is the config file we use:
<jmxtrans-agent>
<queries>
<query objectName="Catalina:type=GlobalRequestProcessor,name=*" attributes="bytesReceived,bytesSent,errorCount,processingTime,requestCount" resultAlias="tomcat.%name%.#attribute#"/>
<query objectName="Catalina:type=ThreadPool,name=*" attributes="currentThreadCount,currentThreadsBusy,maxThreads" resultAlias="tomcat.%name%.#attribute#"/>
<query objectName="java.lang:type=ClassLoading" attributes="LoadedClassCount,TotalLoadedClassCount,UnloadedClassCount" resultAlias="tomcat.ClassLoading.#attribute#"/>
<query objectName="java.lang:type=Compilation" attribute="TotalCompilationTime" resultAlias="tomcat.Compilation.TotalCompilationTime"/>
<query objectName="java.lang:type=GarbageCollector,name=*" attributes="CollectionCount,CollectionTime" resultAlias="tomcat.%name%.#attribute#"/>
<query objectName="java.lang:type=Memory" attributes="HeapMemoryUsage,NonHeapMemoryUsage" resultAlias="jvm.#attribute#.#key#"/>
<query objectName="java.lang:type=Memory" attributes="ObjectPendingFinalizationCount" resultAlias="tomcat.Memory.ObjectPendingFinalizationCount"/>
<query objectName="java.lang:type=MemoryPool,name=*" attribute="Usage" resultAlias="tomcat.%name%.Usage.#key#"/>
<query objectName="java.lang:type=OperatingSystem" attributes="MaxFileDescriptorCount,OpenFileDescriptorCount,ProcessCpuLoad,SystemCpuLoad,SystemLoadAverage" resultAlias="tomcat.OperatingSystem.#attribute#"/>
<query objectName="java.lang:type=Runtime" attribute="Uptime" resultAlias="tomcat.Runtime.Uptime"/>
<query objectName="java.lang:type=Threading" attributes="DaemonThreadCount,PeakThreadCount,ThreadCount,TotalStartedThreadCount" resultAlias="tomcat.Threading.#attribute#"/>
<query objectName="java.nio:type=*,name=*" attributes="Count,TotalCapacity,MemoryUsed" resultAlias="tomcat.%type%.%name%.#attribute#"/>
<query objectName="com.zaxxer.hikari:type=*" resultAlias="hikari.%type%.#attribute#"/>
</queries>
<outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
<host>XXX</host>
<port>2003</port>
<namePrefix>jmx.undefined-env.undefined-service.#hostname#.</namePrefix>
</outputWriter>
<collectIntervalInSeconds>60</collectIntervalInSeconds>
<reloadConfigurationCheckIntervalInSeconds>30</reloadConfigurationCheckIntervalInSeconds>
</jmxtrans-agent>
We use this script as a cronjob to update the config:
#!/bin/sh
cd /opt/jmxtrans-config
git reset --hard
git pull
sed -i.bak \
-e "s#<namePrefix>.*#<namePrefix>jmx.${ECS_ENVIRONMENT}.${SERVICENAME}.${HOSTNAME}.</namePrefix>#" \
/opt/jmxtrans-config/jmxtrans-agent.xml
touch /opt/jmxtrans-config/jmxtrans-agent.xml
Problem is that the config file does not get reloaded or is not adding new queries to be reported to our influxdb. Re-deploying the service makes reporting work. The config file definitely gets updated by the cronjob.
What could be the problem here?
I have jmxtrans-agent sending graphite to influxdb, but if the influxdb server ever goes away or is switched out, jmxtrans-agent crashes and has to be restarted to start sending data again. It would be nice if it tried reconnecting and possibly even buffering data until it could send graphite data again.
Introduce expression language with:
#hostname#
)%name%
)Docs: https://github.com/jmxtrans/jmxtrans-agent/wiki/Expression-Language
Sample:
<query objectName="Catalina:type=DataSource,context=/,host=localhost,class=javax.sql.DataSource,name=*"
attribute="numActive" resultAlias="tomcat.datasource.%name%.numActive" />
Should match ObjectName:
Catalina:type=DataSource,context=/,host=localhost,class=javax.sql.DataSource,name="jdbc/my-datasource"
And resultAlias
should be:
tomcat.datasource.jdbc_my_datasource.numActive
objectName=“org.springframework.boot:type=Endpoint,name=metricsEndpoint”
type of attributeData
is LinkedHashMap
I'm considering writing a single line per collection rolling file write in the format:
key1=value1, key2=value2, key3=value3 . . .
This allows for improved parsing by Splunk (http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6) and possibly by other (file based) log collectors.
Would you like this writer contributed? If so, do you have a preferred Java class name?
or would you possibly like to change the formatting to single line per collection for:
org\jmxtrans\agent\FileOverwriterOutputWriter.java
org\jmxtrans\agent\RollingFileOutputWriter.java
Please comment. Thanks
I was wondering if converting all the dots in a bean name (using %name%
in the configuration) being transformed to underscores is a specific design decision?
At this line adding the optional second parameter to false
would keep the dots.
It made querying Graphite unpractical since names were really long instead of having easy namespaces, for example service.http-server.akka-http-server_200_count
instead of service.http-server.akka-http-server.200.count
Edit: here are the type of beans I am currently exporting:
I've configured my outputWriter
to:
<outputWriter class="org.jmxtrans.agent.GraphitePlainTextTcpOutputWriter">
<host>graphite.domain.tld</host>
<namePrefix>server.#escaped_canonical_hostname#.${jmxtrans_application_name:}</namePrefix>
</outputWriter>
but what I receive is:
server.#escaped_canonical_hostname#.activemq.CurrentConnectionsCount 3 1457966635
Are basic functions supposed to work in the outputWriter
config?
It would be useful to be able to use statsd tags in metrics (see the Metric Tagging section here), for example:
<query objectName="..." attribute="..." resultAlias="metric-name#tag=value"/>
Currently this will be rejected by jmxtrans-agent's expression engine:
Invalid expression 'metric-name#tag=value', no ending '#' after beginning '#' at position ...
I use JMX-Agent GraphitePlainTextTcpOutputWriter.java writing values into influxDB, but I find just insert one column, and I see the Graphite demands the Message Format is "metric_path value timestamp\n". So how could I insert multi column into influxDB.
JMXtrans Java Agent fails with JDK9:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:500)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:512)
Caused by: java.lang.NoClassDefFoundError: java/sql/Timestamp
at org.jmxtrans.agent.util.logging.Logger.log(Logger.java:105)
at org.jmxtrans.agent.util.logging.Logger.log(Logger.java:98)
at org.jmxtrans.agent.util.logging.Logger.info(Logger.java:124)
at org.jmxtrans.agent.JmxTransAgent.initializeAgent(JmxTransAgent.java:116)
at org.jmxtrans.agent.JmxTransAgent.premain(JmxTransAgent.java:85)
... 6 more
Caused by: java.lang.ClassNotFoundException: java.sql.Timestamp
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496)
... 11 more
FATAL ERROR in native method: processing of -javaagent failed
In our applications, we use the Codahale/Dropwizard metrics library to publish our metrics to JMX. Rather than configure each metric, I'd like to grab everything under a particular domain. For example, something like this:
<query objectName="MyApplication-metrics:type=*,group=*,name=*"
resultAlias="app.%type%.%group%.%name%.#attribute#"/>
However, because of the way the metrics are published to JMX, I end up with metrics that aren't really metrics. For example:
myapplication.app.API.ALL_statusSeries_5xx.responseTime.RateUnit events/second 1505500740
myapplication.app.API.ALL_statusSeries_5xx.responseTime.DurationUnit milliseconds 1505500740
What I'd really like to do is specify a set of attributes not to include. The configuration might look something like this:
<query objectName="MyApplication-metrics:type=*,group=*,name=*"
excludeAttributes="RateUnit,DurationUnit"
resultAlias="app.%type%.%group%.%name%.#attribute#"/>
I'm guessing that some might want to do something similar when processing CompositeData, though I don't personally have that need at the moment.
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.