cinimex / mq-java-exporter Goto Github PK

https://sonarcloud.io/project/issues?id=Cinimex-Informatica_mq-java-exporter&issues=AWmVnep95QE9aC-CC_EF&open=AWmVnep95QE9aC-CC_EF

Check if it is at least V9 so that monitoring will work. Otherwise println error message and expose only additional metrics. Update documentation with this limitation.

Provide extended description, especially "specific" feature - AIX support, extended metrics collection

Create a "how-to" guide, explaining how to register exporter as MQ Service

Add "STOPPED" status for listener. This issue was created from #72

A new metrics should be added, that reflects queue fill percentage as part of "MQ PCF API specific statistics". It should calculated on the exporter's side by deviding "mqobject_get_queue_depth_messages" by "mqobject_queue_max_depth_messages".

Depends on #16
Check, if there is something, that has to be down by the exporter, which works as mq service, when MQ itself if shut down.

Need to make "smart" maximum\minimum gauges.

For example, a metric "mq_subscribe_non_durable_subscriber_high_water_mark_subscriptions" can be thrown by MQ every ten seconds with values 10, 15, 5. If prometheus scrabs metrics every 10 seconds, it will get the last value - 5. In oder to "cover" all the values, maximum\minimum gauges have to analyse recieved metrics and overwrite currect saved "max" metric, if currect saved value has lower value (overwrite currect saved "min" metric, if currect saved value has higher value).

It will allow prometheus to get value "15" in the example above.

With "sendPCFCommands: false" flag exporter doesn't scrape queue-related metrics. But this flag should only disable additional metrics.

I guess, the most efficient way will be to use existing codestyles:

• Google Java Style Guide
• Java Code Conventions

This issue solution should include:

1. Intelij IDEA file with codestyle settings.
2. Massive code reformatting.
3. Readme updates, which include codestyle description, rules, IDE profile.

Connection configuration section have "Use MQCSP for connection?" property.
It should be described (or there should be a reference to documentation), what MQCSP is for, in what cases it has to be used.

Sonar bugs:
1, 2, 3, 4, 5

Metrics from section "Disk usage - platform wide" should have "system" domain:
mq_disk_trace_file_system_in_use_megabytes -> system_disk_trace_file_system_in_use_megabytes

• Change header style: current style doesn't support all levels of nesting
• add "back to toc" links
• add references from PCF requests description to MQ constants mapping
• change section names in "Metrics list" - they have to reflect the word "metric"
• add small descriptions for tables

depends on #7 and #47

Provide correct step-by-step description how to configure exporter

List of all code smells here

Add new MQ 9.1.0.1 metrics:

Log - current primary space in use
Log - workload primary space utilization
Log - bytes required for media recovery
Log - bytes occupied by reusable extents
Log - bytes occupied by extents waiting to be archived
Log - write size
Log - current primary space in use

Add info for all metrics (including old metrics) - "Supported version" or "since version" between columns "Short description" and "MQ metric element"

subj

Provide brief instructuion, how to turn on\turn off\configure mq metrics in general, and for specific objects.
Leave a note, that a full guide can be found at IBM Knowledge center - https://www.ibm.com/support/knowledgecenter/

Create a run section right under "Build" section.
Current info "how to run exporter" should be replaced to the newly created section "run".

Describe what does a metric name consist of.

Part of #17 & add timeout in the loop

https://sonarcloud.io/project/issues?id=Cinimex-Informatica_mq-java-exporter&issues=AWlSW906e1QvbR9zifNY&open=AWlSW906e1QvbR9zifNY

Change exporter name in description to "IBM MQ Exporter"

Describe how exporter works

Assuming that mq metric type "delta" maps to prometheus metric type "counter" is incorrect. Check if all counters are correct.
For example, "mq_subscribe_non_durable_subscriber_high_water_mark_subscriptions" seems not to be a counter metric.

"mq_cpu_ram_total_estimate_megabytes" -> "mq_ram_ram_total_estimate_megabytes"
Changes should be reflected in readme.md in "metrics list" and "domains and subdomains" (depends on #7)

subj

Describe the best way to deal with counters

Enhance exporter with logging

We have found out that MQ publishes percentages as 4-digit decinal integer. For example, "system_ram_ram_free_percentage" can return value 3300 which is equal to 33%.
In order to simplify working with metrics we have to process that info on the exporter's side.
For example, metrics:
system_ram_ram_free_percentage
mq_cpu_user_cpu_time_estimate_percentage
Should be devided by 100 in order to get the "percentage" value.

Moreover, we have to check all the metrics, to be sure, that the presented value is correct.
Metric return types, that need to be checked:

• percentage
• hundredths
• megabytes
• files
• bytes
• microseconds
• totalcalls
• totalconnections
• totalmessages
• totalbytes
• totalbrowses
• subscriptions
• totalattempts
• totalqueues
• messages

Current config info states:

Use wildcards? If yes, only one PCF command will be send, matching all objects on queue manager. Otherwise, each object will be monitored by separate PCF command.
usePCFWildcards: true

The explanation should state clearly, why is it practical/impracticle to use wildcards and in what cases. Examples would be great.

Provide handling for expected and unexpected shutdowns

Metrics mqobject_get_average_queue_time_microseconds, mqobject_get_queue_depth_messages have to have different subdomain - queue.

Moreover, all the metrics in readme file have to be checked if they belong to correct section. Specific sections can be created if they don't exist.

Current exporter implementation manages counters as simple as possible. Every mq counter is a prometheus counter, which can only increase. There are pros and cons of such implementation:
pros:

• exporter only updates counters. There is no additional work required from exporter.

cons:

• query format, which should be used, to display diff between counters (metric - metric offset 10s). Every grafana dashboard will depend on prometheus scrape interval.
• actually, the way we expose counters differs from the way mq exposes them.

The idea is to implement mq counters via prometheus gauges (that's how counters become smart). They will increase between scrapes, but after each scrape all "smart" counters will be reset to 0.
This implementation also have pros and cons:

pros:

• grafana dashboard doesn't depend on prometheus scrape interval.
• exporter exposes metrics the way mq exposes them. From my point it is more obvious implementation.

cons:

• need to reset counters after each scrape.

Need to be discussed.

There is a chance people don't want to save all the metrics to Prometheus. If so, exporter should be able to control list of transferred metrics.

"Exposes API of IBM MQ and system metrics of it's host machine" -> "Exposes API metrcs of IBM MQ and system metrics of it's host machine"

Change exporter name in description to "IBM MQ Exporter"

https://sonarcloud.io/project/issues?id=Cinimex-Informatica_mq-java-exporter&issues=AWlSW91Je1QvbR9zifN4&open=AWlSW91Je1QvbR9zifN4

"MQ metric elemen" have to be replaced by "MQ metric element"

Table with mapping for listener statuses to prometheus metrics has 2 rows with the same status.

Current sonar check result in a bug:

[ERROR] Failed to execute goal on project mq-java-exporter: Could not resolve dependencies for project ru.cinimex:mq-java-exporter:jar:1.0-SNAPSHOT: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project mq-java-exporter: Could not resolve dependencies for project ru.cinimex:mq-java-exporter:jar:1.0-SNAPSHOT: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project ru.cinimex:mq-java-exporter:jar:1.0-SNAPSHOT: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:208)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:342)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:202)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:413)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:215)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:325)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:202)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Failure to find com.ibm.mq:com.ibm.mq.allclient:jar:9.0.0.1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced
at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.newException (DefaultUpdateCheckManager.java:217)
at org.eclipse.aether.internal.impl.DefaultUpdateCheckManager.checkArtifact (DefaultUpdateCheckManager.java:192)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.gatherDownloads (DefaultArtifactResolver.java:554)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:472)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:390)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:215)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:325)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:202)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

https://sonarcloud.io/project/issues?id=Cinimex-Informatica_mq-java-exporter&issues=AWlSW91Ue1QvbR9zifOC&open=AWlSW91Ue1QvbR9zifOC

Both mq-exporter and IBM amqsrua could show negative value of "RAM total bytes".
On the picture you coould see, that at 7:00 a.m value was near 200, but at the 11:30 a.m it became minus 300

Command for amqsrua: ./amqsrua -m -c CPU -t QMgrSummary

Check if a listener doesn't return "stopped" or "inactive" status similarly to channels.
If so, such status has to be returned in a similar way as inactive statuses are returned for channels.

subj

Change exporter name in description to "IBM MQ Exporter"

Provide a guide to fill configuration file

It's recommended to use "domain" prefix and unit description suffix

Need to add scrapping of additional metrics:

1. Listeners statuses.
2. Channels statuses.
3. Queues max depth.

MQ doesn't publish this info to it's topic. That is why we should use direct PCF commands.

subj