jruby-gradle / jruby-gradle-storm-plugin Goto Github PK
View Code? Open in Web Editor NEWJRuby Gradle plugin to manage creating Storm topology jars
License: MIT License
JRuby Gradle plugin to manage creating Storm topology jars
License: MIT License
these lines cause the classpath for the JRubyStormLocal
task type to only have their classpath set properly if a parentTask has been provided.
In the case of a stand-alone task definition (i.e. not auto-created by jrubyStorm
) this will result in an missing classpath and:
:runMyFancyTopology
Error: Could not find or load main class redstorm.TopologyLauncher
:runMyFancyTopology FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runMyFancyTopology'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
Perhaps we should just inherit the jruby.defaultRepositories
setting from the base plugin.
Either way, forcing me to use jcenter and clojars causes my builds to fail if any of those third party providers has issues.
* Where:
Build file '/home/tyler/source/lookout/REDACTED/build.gradle' line: 21
* What went wrong:
A problem occurred evaluating root project 'REDACTED'.
> Failed to apply plugin [id 'com.github.jruby-gradle.storm']
> Could not find method jruby() for arguments [com.github.jrubygradle.storm.JRubyStormPlugin$_apply_closure2_closure8@616a00ab] on task ':jrubyStorm'.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
BUILD FAILED
Total time: 4.408 secs
If you include a directory with into()
in JRubyStorm
, that directory should have the jrubydir
file(s) created
The configuration management code in JRubyStormLocal
needs to be retooled a bit and better tested to make sure that:
jrubyStormLocal
by default (which should extend jrubyStorm)Right now jruby-mains.jar gets stuck into the topology jar, that's not necessary so we should prevent it from being included
In my local testing, when I'm assembling the topology over and over it seems to be installing the gems over and over again.
It's not clear whether this is a storm plugin bug or a bug with the JRubyPrepare
task
assembleJRubyStorm
should depend on whatever has been declared in:
jrubyStorm {
dependsOn foo
}
I'm tired of seeing this in build.gradle
files:
artifacts {
archives jrubyStorm
}
Working with the pre-release version of 0.4.0, and this line doesn't seem to want to cooperate
@ysb33r this ring any bells? :/
Caused by: groovy.lang.MissingMethodException: No signature of method: com.github.jrubygradle.storm.internal.JRubyStormJar_Decorated.getJrubyVersion() is applicable for argument types: () values: []
Possible solutions: getJrubyVersion(), getJrubyVersion(), getJrubyVersion(), jrubyVersion(java.lang.String), getVersion(), getJrubyMainsVersion()
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:130)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at com.github.jrubygradle.storm.internal.JRubyStormJar.getJrubyVersion(JRubyStormJar.groovy:24)
at com.github.jrubygradle.storm.internal.JRubyStormJar_Decorated.getJrubyVersion(Unknown Source)
at com.github.jrubygradle.jar.JRubyJar.addJRubyDependency(JRubyJar.groovy:209)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Right now we're just using the default jrubyExec
configuration for the JRubyStormLocal
tasks which is definitely not correct.
All Storm tasks should be in the "Storm" group and should have a description. Right now neither jrubyStorm
nor runJRubyStorm
has a description.
In an older Storm cluster, where storm-kafka isn't already baked in, I'm having trouble getting a topology which includes the storm-kafka jar to start properly on supervisor nodes.
The classpath and everything is correct enough to submit and load successfully (on-submit does resolve and make use of storm-kafka).
java.lang.RuntimeException: java.lang.ClassNotFoundException: storm.kafka.KafkaSpout
at backtype.storm.utils.Utils.deserialize(Utils.java:95) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.Utils.getSetComponentObject(Utils.java:235) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.task$get_task_object.invoke(task.clj:73) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.task$mk_task_data$fn__3061.invoke(task.clj:180) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$assoc_apply_self.invoke(util.clj:816) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.task$mk_task_data.invoke(task.clj:173) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.task$mk_task.invoke(task.clj:184) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.executor$mk_executor$fn__5510.invoke(executor.clj:321) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.core$map$fn__4207.invoke(core.clj:2485) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__6026.invoke(protocols.clj:54) ~[clojure-1.5.1.jar:na]
at clojure.core.protocols$fn__5979$G__5974__5992.invoke(protocols.clj:13) ~[clojure-1.5.1.jar:na]
at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:321) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.daemon.worker$fn__5940$exec_fn__1396__auto____5941$iter__5946__5950$fn__5951.invoke(worker.clj:375) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.5.1.jar:na]
at clojure.lang.RT.next(RT.java:598) ~[clojure-1.5.1.jar:na]
at clojure.core$next.invoke(core.clj:64) ~[clojure-1.5.1.jar:na]
at clojure.core$dorun.invoke(core.clj:2781) ~[clojure-1.5.1.jar:na]
at clojure.core$doall.invoke(core.clj:2796) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__5940$exec_fn__1396__auto____5941.invoke(worker.clj:375) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$fn__5940$mk_worker__5996.doInvoke(worker.clj:347) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker$_main.invoke(worker.clj:454) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na]
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.2-incubating.jar:0.9.2-incubating]
Caused by: java.lang.ClassNotFoundException: storm.kafka.KafkaSpout
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) ~[na:1.7.0_75]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) ~[na:1.7.0_75]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_75]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) ~[na:1.7.0_75]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) ~[na:1.7.0_75]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) ~[na:1.7.0_75]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_75]
at java.lang.Class.forName0(Native Method) ~[na:1.7.0_75]
at java.lang.Class.forName(Class.java:274) ~[na:1.7.0_75]
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:625) ~[na:1.7.0_75]
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612) ~[na:1.7.0_75]
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) ~[na:1.7.0_75]
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) ~[na:1.7.0_75]
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ~[na:1.7.0_75]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_75]
at backtype.storm.utils.Utils.deserialize(Utils.java:89) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 36 common frames omitted
The new nuansces of the JRubyStorm
task type should be documented
This is confusing behavior and breaks the expectation that existing plugin users will have
There's changes in the base plugin now that properly filter GEM_HOME
and GEM_PATH
without totally nuking the environment.
We should be using that code instead of calling set Environment
in JRubyStormLocal
task runLocalTopology(type: JRubyStormLocal) {
description 'Run the Topology in local-topology mode'
topology 'topologies/deferred_url_topology'
standardInput System.in
}
Results in:
:runLocalTopology FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runLocalTopology'.
> java.lang.NullPointerException (no error message)
* Try:
Run with --info or --debug option to get more log output.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':runLocalTopology'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:42)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:306)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.executeTask(AbstractTaskPlanExecutor.java:79)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:63)
at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:51)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:23)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:88)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62)
at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:68)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:55)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:149)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:80)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:36)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:51)
at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:169)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
at org.gradle.launcher.Main.doAction(Main.java:33)
at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:127)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:56)
Caused by: java.lang.NullPointerException
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.createGroovyObjectGetPropertySite(AbstractCallSite.java:254)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.acceptGroovyObjectGetProperty(AbstractCallSite.java:239)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:231)
at com.github.jrubygradle.storm.JRubyStormLocal.exec(JRubyStormLocal.groovy:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:63)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.doExecute(AnnotationProcessingTaskFactory.java:218)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:211)
at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:200)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:585)
at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:568)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
... 55 more
it kind of difficult to load some jars via jar-dependencies and some other jars should not be loaded.
jruby comes with couple of jars with its default gems: jline-2.11.jar, those bouncy-castle jars. jruby-9k will come with snakeyaml-1.14.jar and maybe one or more other jars soon.
AND there is no way to find out which jars jruby bundles with those default gems (any ideas on how to expose those extra dependencies formally are welcome).
now jar-dependencies loads those jars. it will look on "classpath" or "classloader" first.
with maven I just use http://maven.apache.org/plugins/maven-dependency-plugin/copy-dependencies-mojo.html, set useRepositoryLayout=true and outputDirectory=target/classes and the artifact will have those jars where jar-dependencies is going to look for them.
adding the dependency jruby-complete 1.7.22 to jar
unzipping /home/tyler/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/1.7.22/42dc9be2f706774c24ef61ec2981d05e2c79a9e2/jruby-complete-1.7.22.jar in the jar
unzipping /home/tyler/.gradle/caches/modules-2/files-2.1/org.jruby.mains/jruby-mains/0.4.0/c767732c802c52592a949b57e68023e66b8b860e/jruby-mains-0.4.0.jar in the jar
task ':jrubyJar' including files in /home/tyler/source/lookout/REDACTED/build/dirinfo/jrubyJar
adding the dependency jruby-complete 1.7.21 to jar
unzipping /home/tyler/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/1.7.21/e061b9f399a5e8d5cfcca84d4a6baf879111e83c/jruby-complete-1.7.21.jar in the jar
unzipping /home/tyler/.gradle/caches/modules-2/files-2.1/org.jruby.mains/jruby-mains/0.4.0/c767732c802c52592a949b57e68023e66b8b860e/jruby-mains-0.4.0.jar in the jar
task ':assembleJRubyStorm' including files in /home/tyler/source/lookout/REDACTED/build/dirinfo/jrubyJar
Looks like my attempt to set a default version to 1.7.21 as a baseline didn't work :(
The two methods we need to delegate to the assemble task are:
into(destPath)
into(destPath, configureClosure)
The JRubyStormLocal
task sets the GEM_HOME
and GEM_PATH
variables to where the gems are being installed for thep roject, but if those dependencies include jar-dependencies, they will not be properly added to the classpath.
A temporary work-around right now is:
configurations {
jrubyStormLocal.extendsFrom gems
}
It's currently using jrubyJar, blech
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.