dcos2humio's Issues

Parse @type field

Short lived tasks (<10s) might not be harvested

Due to the update latency of the configuration, tasks that goes in and out of TASK_RUNNING state in less than 10 seconds might not be picked up by the agent.

In particular jobs that are less than 10s are at risk

ArrayIndexOutOfBoundsException in Elastic Beat configurator

java.lang.ArrayIndexOutOfBoundsException: -1
	at java.util.ArrayList.elementData(ArrayList.java:422) ~[na:1.8.0_162]
	at java.util.ArrayList.get(ArrayList.java:435) ~[na:1.8.0_162]
	at com.humio.mesos.dcos2humio.scheduler.service.ElasticBeatConfigurationGenerator.wasTaskRecentlyRunning(ElasticBeatConfigurationGenerator.java:118) ~[classes!/:na]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_162]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_162]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270) ~[na:1.8.0_162]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_162]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_162]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270) ~[na:1.8.0_162]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:270) ~[na:1.8.0_162]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_162]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_162]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_162]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_162]
	at com.humio.mesos.dcos2humio.scheduler.service.ElasticBeatConfigurationGenerator.extractTaskDetailsPerSlave(ElasticBeatConfigurationGenerator.java:110) ~[classes!/:na]
	at com.humio.mesos.dcos2humio.scheduler.service.ElasticBeatConfigurationGenerator.pushState(ElasticBeatConfigurationGenerator.java:76) ~[classes!/:na]
	at com.humio.mesos.dcos2humio.scheduler.service.ElasticBeatConfigurationGenerator.updateConfig(ElasticBeatConfigurationGenerator.java:70) ~[classes!/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_162]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_162]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_162]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_162]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_162]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_162]

Harvest dcos/mesos logs

Mainly through journalctl

http 400 error on teardown

$ dcos package uninstall humio-agent --app-id=/humio-agent
WARNING: This action cannot be undone. This will uninstall [/humio-agent] and delete all of its persistent (logs, configurations, database artifacts, everything).
Please type the name of the service to confirm: /humio-agent
Unexpected down stream http error: POST /master/teardown 400

Configure parsers for Mesos agents and scheduler

Mesos agent

timestamp: MMdd HH:mm:ss[.SSSSSS]
regex: (?<level>\w)(?<@timestamp>\d{4}\s\S+)\s+(?<thread_id>\S+)\s(?<file>\S+):(?<line>\d+)]

Edit: Updated regex, according to #13 (comment)

Scheduler stops working after a while

For some reason the leader.mesos cannot be resolved

org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://leader.mesos:5050/state": Network is unreachable (connect failed); nested exception is java.net.SocketException: Network is unreachable (connect failed)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:713) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:657) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.web.client.RestTemplate.getForEntity(RestTemplate.java:346) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at com.humio.mesos.dcos2humio.scheduler.service.FilebeatConfigurationGenerator.updateConfig(FilebeatConfigurationGenerator.java:61) ~[classes!/:na]
	at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_144]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_144]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_144]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_144]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_144]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]
Caused by: java.net.SocketException: Network is unreachable (connect failed)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_144]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_144]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_144]
	at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_144]
	at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_144]
	at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[na:1.8.0_144]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_144]
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_144]
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:1.8.0_144]
	at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_144]
	at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[na:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[na:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[na:1.8.0_144]
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[na:1.8.0_144]
	at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:78) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:104) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.boot.actuate.metrics.web.client.MetricsClientHttpRequestInterceptor.intercept(MetricsClientHttpRequestInterceptor.java:71) ~[spring-boot-actuator-2.0.0.M5.jar!/:2.0.0.M5]
	at org.springframework.http.client.InterceptingClientHttpRequest$InterceptingRequestExecution.execute(InterceptingClientHttpRequest.java:92) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.InterceptingClientHttpRequest.executeInternal(InterceptingClientHttpRequest.java:76) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:699) ~[spring-web-5.0.0.RELEASE.jar!/:5.0.0.RELEASE]
	... 15 common frames omitted

Configuration for multiline parsing of logfiles

Harvest task logs

Tasks

Write base Maraton config
Write scheduler that starts filebeats instance on every node
Write executor that received update configurations
Submit mesosagent parser #13
Write Universe package

Missing host field for beats logs

Somehow the mesos_slave_id went out with the tub water

Write DCOS Commons framework

Tasks

Migrate basic Marathon config. (Deploy a standard configured Filebeat on N number of hosts)
Automatic deployment of new Filebeats on new nodes.
Blocked 🚫: By the looks of things the pod count is always fixed until configuration is updated. @joeljacobson is looking into this
Push Filebeats configurations to executors on new frameworks