Giter VIP home page Giter VIP logo

rohankumardubey / stream-reactor Goto Github PK

View Code? Open in Web Editor NEW

This project forked from lensesio/stream-reactor

0.0 0.0 0.0 11.61 MB

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

Home Page: https://lenses.io

License: Apache License 2.0

Shell 0.05% Java 1.02% Scala 98.69% ANTLR 0.24%

stream-reactor's Introduction

Actions Status

Join us on slack Alt text

Lenses Connectors for Apache Kafka

Lenses.io is the leader in offering Apache 2 licensed Kafka Connectors (Stream Reactor) since 2016.

Enterprise Support for Kafka Connectors

Lenses offers the leading Developer Experience solution for engineers building real-time applications on any Apache Kafka (lenses.io). Subscribed customers are entitled to full 24x7 support for selected Kafka Connectors. This includes priority over feature requests and security incident SLAs. Email [email protected] for more information.

Engage with the Community

Speak to us on our Community Slack channel (Register at https://launchpass.com/lensesio) or ask the Community a question in our Ask Marios forum.

Kafka Connectors Roadmap

A series of next-generation Connectors are in active development. Give us your feedback of which connectors we should be working on or to to get the latest information, send us an email at [email protected]

Stream Reactor Kafka Connectors

Alt text FOSSA Status

A collection of components to build a real time ingestion pipeline.

Kafka Compatibility

  • Kafka 2.8 -> 3.5 (Confluent 6.2 -> 7.5) - Stream Reactor 4.1.0+
  • Kafka 3.1 (Confluent 7.1) - Stream Reactor 4.0.0 (Kafka 3.1 Build)
  • Kafka 2.8 (Confluent 6.2) - Stream Reactor 4.0.0 (Kafka 2.8 Build)
  • Kafka 2.5 (Confluent 5.5) - Stream reactor 2.0.0+
  • Kafka 2.0 -> 2.4 (Confluent 5.4) - Stream reactor 1.2.7

DEPRECATION NOTICE

The following connectors have been deprecated and are no longer included in future releases:

  • Kudu
  • Hazelcast
  • HBase
  • Hive
  • Pulsar

Connectors

Please take a moment and read the documentation and make sure the software prerequisites are met!!

Connector Type Description Docs
AWS S3 Sink Copy data from Kafka to AWS S3. Docs
AWS S3 Source Copy data from AWS S3 to Kafka. Docs
Azure Data Lake (Beta) Sink Copy data from Kafka to Azure Data Lake Docs
AzureDocumentDb Sink Copy data from Kafka and Azure Document Db. Docs
Cassandra Source Copy data from Cassandra to Kafka. Docs
*Cassandra Sink Certified DSE Cassandra, copy data from Kafka to Cassandra. Docs
Elastic 6 Sink Copy data from Kafka to Elastic Search 6.x w. tcp or http Docs
FTP/HTTP Source Copy data from FTP/HTTP to Kafka. Docs
Google Cloud Storage (Beta) Sink Copy data from Kafka to Google Cloud Storage. Docs
InfluxDb Sink Copy data from Kafka to InfluxDb. Docs
JMS Source Copy data from JMS topics/queues to Kafka. Docs
JMS Sink Copy data from Kafka to JMS. Docs
MongoDB Sink Copy data from Kafka to MongoDB. Docs
MQTT Source Copy data from MQTT to Kafka. Docs
MQTT Sink Copy data from Kafka to MQTT. Docs
Redis Sink Copy data from Kafka to Redis. Docs

Release Notes

Please see the Stream Reactor Release Notes at Lenses Documentation.

Building

To build:

sbt clean compile

To test:

sbt test

To create assemblies:

sbt assembly

To build a particular project:

sbt "project cassandra" compile

To test a particular project:

sbt "project cassandra" test

To create a jar of a particular project:

sbt "project cassandra" assembly

Running E2E tests

If not already built, you must first build the connector archives:

sbt "project cassandra" assembly
sbt "project elastic6" assembly 
sbt "project mongodb" assembly
sbt "project redis" assembly

To run the tests:

sbt e2e:test

Contributing

We'd love to accept your contributions! Please use GitHub pull requests: fork the repo, develop and test your code, semantically commit and submit a pull request. Thanks!

License

FOSSA Status

stream-reactor's People

Contributors

stheppi avatar davidsloan avatar antwnis avatar andrewstevenson avatar codesmell avatar johnathana avatar andmarios avatar pedrobento988 avatar goncaloccastro avatar caiooliveiraeti avatar carolinaloureiro avatar fartariatomas avatar sotirissl avatar richardlundsky avatar spirosoik avatar mbarlot avatar matthedude avatar lauramorab avatar sksamuel avatar rollulus avatar michaelandrepearce avatar jbruggem avatar boeboe avatar orbiran88 avatar rdebokx avatar afausti avatar v-gerasimov avatar sdlcwangsong avatar invadergir avatar sandonjacobs avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.