Apache SeaTunnel (Incubating)

SeaTunnel was formerly named Waterdrop , and renamed SeaTunnel since October 12, 2021.

SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform that supports real-time synchronization of massive data. It can synchronize tens of billions of data stably and efficiently every day, and has been used in the production of nearly 100 companies.

Why do we need SeaTunnel

SeaTunnel will do its best to solve the problems that may be encountered in the synchronization of massive data:

Data loss and duplication
Task accumulation and delay
Low throughput
Long cycle to be applied in the production environment
Lack of application running status monitoring

SeaTunnel use scenarios

Mass data synchronization
Mass data integration
ETL with massive data
Mass data aggregation
Multi-source data processing

Features of SeaTunnel

Easy to use, flexible configuration, low code development
Real-time streaming
Offline multi-source data analysis
High-performance, massive data processing capabilities
Modular and plug-in mechanism, easy to extend
Support data processing and aggregation by SQL
Support Spark structured streaming
Support Spark 2.x

Workflow of SeaTunnel

Source[Data Source Input] -> Transform[Data Processing] -> Sink[Result Output]

The data processing pipeline is constituted by multiple filters to meet a variety of data processing needs. If you are accustomed to SQL, you can also directly construct a data processing pipeline by SQL, which is simple and efficient. Currently, the filter list supported by SeaTunnel is still being expanded. Furthermore, you can develop your own data processing plug-in, because the whole system is easy to expand.

Plugins supported by SeaTunnel

Connector

Connector Type	Source	Sink
Clickhouse		doc
Doris		doc
Druid	doc	doc
ElasticSearch	doc	doc
Email		doc
Fake	doc
File	doc	doc
Hbase	doc	doc
Hive	doc	doc
Hudi	doc	doc
Iceberg	doc	doc
InfluxDb	doc	doc
Jdbc	doc	doc
Kafka	doc	doc
Kudu	doc	doc
MongoDB	doc	doc
Neo4j	doc
Phoenix	doc	doc
Redis	doc	doc
Socket	doc
Tidb	doc	doc

Transform

Transform Plugins
Add
CheckSum
Convert
Date
Drop
Grok
Json
Kv
Lowercase
Remove
Rename
Repartition
Replace
Sample
Split
Sql
Table
Truncate
Uppercase
Uuid

Environmental dependency

java runtime environment, java >= 8
If you want to run SeaTunnel in a cluster environment, any of the following Spark cluster environments is usable:

Spark on Yarn
Spark Standalone

If the data volume is small, or the goal is merely for functional verification, you can also start in local mode without a cluster environment, because SeaTunnel supports standalone operation. Note: SeaTunnel 2.0 supports running on Spark and Flink.

Compiling project

Follow this document.

Downloads

Download address for run-directly software package : https://seatunnel.apache.org/download

Quick start

Spark https://seatunnel.apache.org/docs/deployment

Flink https://seatunnel.apache.org/docs/deployment

Detailed documentation on SeaTunnel https://seatunnel.apache.org/docs/intro/about

Application practice cases

Weibo, Value-added Business Department Data Platform

Weibo business uses an internal customized version of SeaTunnel and its sub-project Guardian for SeaTunnel On Yarn task monitoring for hundreds of real-time streaming computing tasks.

Sina, Big Data Operation Analysis Platform

Sina Data Operation Analysis Platform uses SeaTunnel to perform real-time and offline analysis of data operation and maintenance for Sina News, CDN and other services, and write it into Clickhouse.

Sogou, Sogou Qiqian System

Sogou Qiqian System takes SeaTunnel as an ETL tool to help establish a real-time data warehouse system.

Qutoutiao, Qutoutiao Data Center

Qutoutiao Data Center uses SeaTunnel to support mysql to hive offline ETL tasks, real-time hive to clickhouse backfill technical support, and well covers most offline and real-time tasks needs.

Yixia Technology, Yizhibo Data Platform
Yonghui Superstores Founders' Alliance-Yonghui Yunchuang Technology, Member E-commerce Data Analysis Platform

SeaTunnel provides real-time streaming and offline SQL computing of e-commerce user behavior data for Yonghui Life, a new retail brand of Yonghui Yunchuang Technology.

Shuidichou, Data Platform

Shuidichou adopts SeaTunnel to do real-time streaming and regular offline batch processing on Yarn, processing 3~4T data volume average daily, and later writing the data to Clickhouse.

Tencent Cloud

Collecting various logs from business services into Apache Kafka, some of the data in Apache Kafka is consumed and extracted through Seatunnel, and then store into Clickhouse.

For more use cases, please refer to: https://seatunnel.apache.org/blog

Code of conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please follow the REPORTING GUIDELINES to report unacceptable behavior.

Developer

Thanks to all developers!

Contact Us

Mail list: [email protected]. Mail to [email protected], follow the reply to subscribe the mail list.
Slack: https://join.slack.com/t/apacheseatunnel/shared_invite/zt-123jmewxe-RjB_DW3M3gV~xL91pZ0oVQ
Twitter: https://twitter.com/ASFSeaTunnel
Bilibili (for Chinese users)

Landscapes

SeaTunnel enriches the CNCF CLOUD NATIVE Landscape.

Our Users

Various companies and organizations use SeaTunnel for research, production and commercial products. Visit our website to find the user page.

License

Apache 2.0 License.

[Bug] [Module Name] Bug title

Search before asking

I had searched in the issues and found no similar issues.

What happened

bug

SeaTunnel Version

1.2.3

SeaTunnel Config

env {
  # Waterdrop defined streaming batch duration in seconds
#   spark.streaming.batchDuration = 5

#   spark.app.name = "cnb"
#   spark.ui.port = 13000
}

source {
     FakeSourceStream {
      result_table_name = "fake"
      field_name = "name,age"
    }
}

transform {
}

sink {
  ConsoleSink {}
}

Running Command

./bin/start-waterdrop-flink.sh --config ./companies.conf

Error Exception

java.lang.NoClassDefFoundError: org/apache/flink/table/api/java/StreamTableEnvironment
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.createStreamTableEnvironment(FlinkEnvironment.java:90)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:63)
at io.github.interestinglab.waterdrop.flink.FlinkEnvironment.prepare(FlinkEnvironment.java:24)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.createEnv(ConfigBuilder.java:173)
at io.github.interestinglab.waterdrop.config.ConfigBuilder.(ConfigBuilder.java:38)
at io.github.interestinglab.waterdrop.Waterdrop.entryPoint(Waterdrop.java:82)
at io.github.interestinglab.waterdrop.Waterdrop.run(Waterdrop.java:50)
at io.github.interestinglab.waterdrop.WaterdropFlink.main(WaterdropFlink.java:14)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.table.api.java.StreamTableEnvironment
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64)
at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65)
at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 21 more

Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

zhongjiajie / seatunnel Goto Github PK

seatunnel's Introduction

Apache SeaTunnel (Incubating)

Why do we need SeaTunnel

SeaTunnel use scenarios

Features of SeaTunnel

Workflow of SeaTunnel

Plugins supported by SeaTunnel

Connector

Transform

Environmental dependency

Compiling project

Downloads

Quick start

Application practice cases

Code of conduct

Developer

Contact Us

Landscapes

Our Users

License

seatunnel's People

Contributors

Watchers

seatunnel's Issues

Search before asking

What happened

SeaTunnel Version

SeaTunnel Config

Running Command

Error Exception

Flink or Spark Version

Java or Scala Version

Screenshots

Are you willing to submit PR?

Code of Conduct

Recommend Projects

Recommend Topics

Recommend Org