Giter VIP home page Giter VIP logo

cloudflow-contrib's Introduction

Deprecated - The Cloudflow project is deprecated

If you are an existing customer of Lightbend and we have not contacted, please reach out to support.
All existing contracts will be honored and assistance with migration to new tools is available.

Cloudflow Community Supported integrations

Maven Central

Cloudflow Contrib is the project for Cloudflow's extension and integration with popular streaming frameworks.

Documentation

User documentation is available here.

License

This software is licensed under the Apache 2 license.

Maintenance notes

Cloudflow Contrib is NOT supported under the Lightbend subscription.

The project is community maintained by the contributors. Pull requests are very welcome–thanks in advance!

cloudflow-contrib's People

Contributors

andreatp avatar debasishg avatar fieldera-da avatar franciscolopezsancho avatar rayroestenburg avatar scala-steward avatar sebastian-alfers avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloudflow-contrib's Issues

Spark 3.1.2 support does not work with protobuf (sparksql-scalapb)

Trying out a spark project with protobuf, when the executor runs:

Caused by: java.lang.NoSuchMethodError: com.google.protobuf.CodedInputStream.readStringRequireUtf8()Ljava/lang/String;
	at sensors.proto.Data$.parseFrom(Data.scala:140)
	at sensors.proto.Data$.parseFrom(Data.scala:126)
	at scalapb.GeneratedMessageCompanion.parseFrom(GeneratedMessageCompanion.scala:185)
	at scalapb.GeneratedMessageCompanion.parseFrom$(GeneratedMessageCompanion.scala:185)
	at sensors.proto.Data$.parseFrom(Data.scala:126)
	at cloudflow.streamlets.proto.ProtoCodec.$anonfun$decode$1(ProtoCodec.scala:27)

Probably because of many protobuf libs on the classpath,
looking at sparksql-scalapb, hadoop contains an outdated protobuf library that needs to be shaded:
https://scalapb.github.io/docs/sparksql/#setting-up-your-project
https://github.com/thesamet/sparksql-scalapb-test/blob/master/build.sbt
But cloudflow does not use sbt-assembly, so we need to find another way to ensure that the right protobuf library is used.
(could also have another reason, but we should be able to see from dependencies)
Should probably try https://github.com/coursier/sbt-shading to shade the dependencies.

Flink scripts. Documentation or CR retrieval

I think it would be useful to have some explanation on how data is retrieved from the Cloudflow CR and the configuration.

As I understand the scripts work in two ways they read and then write. They read the current deployment from the CR and config from ConfigMaps. Based on that then they create a deployment to be executed with Flink CLI. The write side is Flink-related and requires reading Flink docs but not the read side.

I think there is value in making explicit through documentation how and what needs to be read from the Cloudflow CR and the configuration to learn what needs to be deployed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.