The goal is to create a specification and a sample implementation for piping and transforming data.
None of the current open source and commercial products provides the full set of features of Better ETL.
- Language, OS and cloud agnostic
- Pluggable: a data pipeline consists of steps where each step implements a single interface. This allows anyone to implement data sources, outputs, transformations, logic etc. in various languages, mix them in a single pipeline, deploy each step independently. All this together allows greater freedom of development, deployment and scalability.
- A pipeline is a DAG
- Each step of a pipeline runs independently. This allows continuous flow of data.
- API. Can be used to create dependencies among pipelines.
- Scheduler.
- Persistence: data coming into a pipeline step is persisted until successfully output into a next step.
- A step can output data, metadata, status, message, event. Even listener can start a DAG.
- A step that has multiple following steps can serve as a condition clause. (named outputs/connections/links)
- Map/reduce
- Centralized logging. This allows centralized monitoring.
Here is another more specific list of featuers.
This diagram shows several things:
- A pipeline step can output 4 types of data:
- status: status of the step such as success or failure.
- data: raw data in any format.
- metadata: description of the data, such as its format.
- message: a message from this step for the next steps
- A condition step used for routing messages
- Examples of steps: data source, condition, transformation, output, notification, sending message to another pipeline
- Data Dagster infrastructure consists of
- a unified logging layer and a control module for controlling all pipelines
- API layer on top of controller and logging. The API serves to communicate with Data Dagster.
- UI for monitoring and controlling Data Dagster. The UI uses the API to communicate with the logging and controller.
- SDK that uses the API to communicate with the logging and controller.
- CLI on top of the SDK to communicate with the logging and controller.
- Code that uses the SDK to communicate with the logging and controller.
- Install Python 3
- Run
. ./setup.sh