erikerlandson / spark-kafka-sink Goto Github PK

Task failed with this sink, but same job worked when switched to console sink.

local mode also has no exception. Only on yarn mode didnt work out.

17/05/09 09:41:29 INFO EventLoggingListener: Logging events to hdfs:///spark-history/application_1494223133661_0016
17/05/09 09:41:32 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_e30_1494223133661_0016_01_000002 on host: hadoop3. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e30_1494223133661_0016_01_000002
Exit code: 1
Stack trace: org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException: Launch container failed
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DefaultLinuxContainerRuntime.launchContainer(DefaultLinuxContainerRuntime.java:109)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DelegatingLinuxContainerRuntime.launchContainer(DelegatingLinuxContainerRuntime.java:89)
at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:392)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:317)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:83)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Add Timestamps to messages

The newest version of kafka supports timestamps with every message however we currently cannot see any timestamps. We would like to have a timestamp for every time a metric is "recorded" by spark.