Comments (10)
Does this example work with Cloudflow 2.2.0
+ Spark 3.1.2
+ Avro ?
from cloudflow-contrib.
shading should fix it .. https://stackoverflow.com/a/41609653/16405999. Since we don't use sbt assembly
, we can use https://github.com/coursier/sbt-shading. But in that case do we need to use the sbt runner from https://github.com/coursier/sbt-shading/blob/master/sbt ?
from cloudflow-contrib.
Tried to shade the module as per https://scalapb.github.io/docs/sparksql/ but ran into the following in the logs for spark-sensors-proto-process-6ca7837b719373ca-driver
..
"Exception in thread "main" java.lang.AbstractMethodError: Method shadeproto/descriptor/FileDescriptorProto$.parseFrom(Lcom/google/protobuf/CodedInputStream;)Lscalapb/GeneratedMessage; is abstract
at shadeproto.descriptor.FileDescriptorProto$.parseFrom(FileDescriptorProto.scala)
at scalapb.GeneratedMessageCompanion.parseFrom(GeneratedMessageCompanion.scala:185)
at scalapb.GeneratedMessageCompanion.parseFrom$(GeneratedMessageCompanion.scala:185)
at shadeproto.descriptor.FileDescriptorProto$.parseFrom(FileDescriptorProto.scala:258)
at shadeproto.wrappers.WrappersProto$.scalaDescriptor$lzycompute(WrappersProto.scala:35)
at shadeproto.wrappers.WrappersProto$.scalaDescriptor(WrappersProto.scala:34)
at shadeproto.wrappers.DoubleValue$.scalaDescriptor(DoubleValue.scala:114)
at scalapb.spark.SchemaOptions$.<init>(SchemaOptions.scala:51)
at scalapb.spark.SchemaOptions$.<clinit>(SchemaOptions.scala)
at scalapb.spark.ProtoSQL$.<init>(ProtoSQL.scala:198)
at scalapb.spark.ProtoSQL$.<clinit>(ProtoSQL.scala)
at scalapb.spark.Implicits$$anon$1.protoSql$lzycompute(TypedEncoders.scala:129)
at scalapb.spark.Implicits$$anon$1.protoSql(TypedEncoders.scala:129)
at scalapb.spark.Implicits$$anon$1.protoSql(TypedEncoders.scala:127)
at scalapb.spark.ToCatalystHelpers.messageToCatalyst(ToCatalystHelpers.scala:27)
at scalapb.spark.ToCatalystHelpers.messageToCatalyst$(ToCatalystHelpers.scala:23)
at scalapb.spark.Implicits$$anon$1.messageToCatalyst(TypedEncoders.scala:127)
at scalapb.spark.TypedEncoders$MessageTypedEncoder.toCatalyst(TypedEncoders.scala:48)
at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
at scalapb.spark.Implicits.typedEncoderToEncoder(TypedEncoders.scala:123)
at scalapb.spark.Implicits.typedEncoderToEncoder$(TypedEncoders.scala:120)
at scalapb.spark.Implicits$.typedEncoderToEncoder(TypedEncoders.scala:126)
at sensors.proto.MovingAverageSparklet$$anon$1.buildStreamingQueries(MovingAverageSparklet.scala:38)
at cloudflow.spark.SparkStreamlet.run(SparkStreamlet.scala:91)
at cloudflow.spark.SparkStreamlet.run$(SparkStreamlet.scala:82)
at sensors.proto.MovingAverageSparklet.run(MovingAverageSparklet.scala:30)
at sensors.proto.MovingAverageSparklet.run(MovingAverageSparklet.scala:30)
at cloudflow.streamlets.Streamlet.run(Streamlet.scala:107)
at cloudflow.runner.Runner$.run(Runner.scala:67)
at cloudflow.runner.Runner$.main(Runner.scala:45)
at cloudflow.runner.Runner.main(Runner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Code in branch
Looks somewhat similar to scalapb/ScalaPB#1122 ..
from cloudflow-contrib.
@debasishg , after further investigation the result is:
- we can't use
protobuf-java
2.5.0 since ScalaPB doesn't support it and we are using relevant code - shading is, really, the only option
taking a look at what you did, probably the missing steps to hack together something are:
- remove all the libraries explicitly, since they are going to be included in the jar, e.g.:
extraDockerInstructions := Seq(
Instructions.Run.shell(Seq("rm", "/opt/cloudflow/*.jar"))
),
- add back the generated uber-jar in
/opt/cloudflow
and rename it/opt/cloudflow/cloudflow-runner.jar
if the resulting Docker image works we can cleanup the process.
from cloudflow-contrib.
@andreaTP also spark cannot work with protobuf-java
3.5.x
- hence I guess the only option is to ..
- shade
protobuf-java
withinscalapb
so that it uses the shaded version - let spark use the
2.5.0
version (it uses APIs which are incompatible with3.5.x
- not sure what version
akka-grpc
needs though - will have to check
Hence I don't think we can delete all versions other than 3.5.x
from cloudflow-contrib.
@debasishg all good but:
Hence I don't think we can delete all versions other than 3.5.x
We strictly have to remove all versions but 2.5.0
, which is the one used by Spark itself, 3.X will be shaded in the resulting application artifact and be used by akka-grpc
, scalapb and Cloudflow.
from cloudflow-contrib.
@andreaTP Ok, so I shaded akka-protobuf
as well, so that it also uses 3.15.8
, which it requires. The problem is if I keep only 2.5.0
unshaded, the protobuf compiler complains of missing class ..
Compiling 2 protobuf files to /Users/debasishghosh/lightbend/cloudflow-contrib/examples/spark-sensors-proto/target/scala-2.12/akka-grpc/main
[error] java.lang.NoClassDefFoundError: com/google/protobuf/compiler/PluginProtos$CodeGeneratorRequest
from cloudflow-contrib.
The problem is if I keep only 2.5.0 unshaded, the protobuf compiler complains of missing class ..
You probably need an extra step, move the compilation of protobuf files to a separate sub-project, depends on the artifact, and shade the proto dependency.
from cloudflow-contrib.
Here's the latest status on this ..
After exploring lots of options, I figured out that the only way to achieve the goal is to do the following steps:
- shade rules for
protobuf-java
(version3.15.8
) coming fromscalapb
- shade rules for
protobuf-java
(version3.15.8
) coming fromakka-protobuf
- shade rules for other sources of
protobuf-java
for any version other than2.5.0
- prepare an assembly of all jars except Spark so that all instances of
protobuf-java
v3.15.8
are shaded and their transitive dependencies changed accordingly. This will result in an uber-jar + the Spark jars - use
dependencyOverrides
to base the version ofprotobuf-java
to2.5.0
, the one that Spark uses
This implies a redesign of how we package the image in cloudflow today for spark plugin. This is a significant effort I think. Should we invest time and effort on this ? @RayRoestenburg @andreaTP
from cloudflow-contrib.
This should be retried with 0.2.0, which upgrades to Spark 3.2.0. Closing this ticket for that reason.
from cloudflow-contrib.
Related Issues (7)
- where and how publish the docs HOT 13
- Review doc flow for usability and standard construction HOT 2
- Flink scripts. Documentation or CR retrieval HOT 1
- Spark 3.1.1 is not available for download anymore. HOT 1
- Flink 1.13.1 is not available anymore. HOT 2
- The Flink Streamlet integration tests doesn't work. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cloudflow-contrib.