Distributed system and engine for stateful big data streaming

distributed system => runs on a cluster engine => can execute any kind of computation stateful => can keep & update internal information while data being processed big data => can scale beyond what fits on standard computer streaming => can process an unbounded amount of data as it arrives

Data synchronization with warehouses is a pain

• need to rerun computations on the entire data

Results need to be updated along with the data

• incremental pipelines & micro-batches have unacceptable latency

Need to provide data guarantees

• distributed systems are inherently unreliable

Need to run arbitrary computation on incoming data

• SQL is not general enough
• need to keep internal information (state) as new data arrives

Distribution and scalability

• can scale out to thousands of machines, process an unbounded amount of data

Real-time results

• provides very low latency

Strong data consistency

• can be controlled to offer exactly-once guarantees

Stateful, arbitrary computation

• can run any code as soon as a new element of data arrives
• can keep state as data is being processed

Event-time and processing-time

• Dealing with data elements in the time they were created or processed

Low latency, high throughput

• Excellent solutions to latency vs throughput tradeoffs: millisecond latency, millions of events/s

Exactly-once consistency guarantees

• Also configurable for at-least-once, at-most-once

Connectors available for a bunch of tools

• Kafka, Cassandra, JDBC, ES, HDFS, S3, Pulsar, etc

Layerd, simple to use APIs

• high control vs easy of use

Other features

• high availability, integration with k8s/YARN/Mesos, dynamic scaling
• easy to update applications without losing state

Event-Driven Applications

• react to data elements as events: trigger computations, external actions, updates, etc... ex: recommandation systems, fraud detection, alerting & monitoring

Low-latency data pipelines

• transform/enrich/transport data as it arrives
• much lower latency than regular ETL scheduled jobs
• ex: real-time data aggregation (search indices)

Real-time data analytics

• process data, extract insights and make decisions as soon as new data arrives
• ex: data-driven business decisions, IoT apps, measuring data from customer applications

Spark - Flink is better for low-latency, stateful, arbitrary computations - Spark is better suited to (micro-)batch processing

Kafka - Flink is better for data analytics & data aggregation - Kafka is better used as a message bus between services

Beam - Flink is far more powerful, with more expressice APIs - Beam is a general data API, Flink can run as a backend of Beam

Akka Streams - Flink is better for purely data-related outcomes - Akka Steams is better as a tol to implement reactive services themselves

Airflow - Flink is a data processing enfine - Airflow is simply a workflow manager, has no pocessing capabbilities of its own

Dataflow: description of data processing steps in the application - made of operators

Operator: an independent logical processing step - can be parallelized tasks

Task "operator task": an independent instance of an operator - works on a data partition: a portion of the data to be independently processed - runs on a physical machine

Latency vs throughput - latency -> time between event and result - throughput -> number of processed events per unit of time (seconds) - lowering latency increases thourghtput - higher throughput is possible at the expense of latency (e.g. micro-batches in Spark)

Event time vs processing time - all events in Flink are timestamped - event time -> time when the event was created in the source system - processing time -> time when the event arived at the Flink processor choosing event time vs processing time can lead to different results

• SQL -> Flink SQL
• High-level -> Table API
• Core APIs -> DataStream API
• Low-levels APIs -> ProcessFunction, sources, sinks, state, time