Topic: data-pipelines Goto Github

👇 Here are 192 public repositories matching this topic...

apache / airflow
data-pipelines,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / dolphinscheduler
data-pipelines,Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Organization: apache
Home Page: https://dolphinscheduler.apache.org/
workflow-schedule azkaban airflow task-scheduler job-scheduler cloud-native data-pipelines orchestration workflow workflow-orchestration

artie-labs / transfer
data-pipelines,Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
Organization: artie-labs
Home Page: https://artie.so
snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database

bakdata / streams-explorer
data-pipelines,Explore Apache Kafka data pipelines in Kubernetes.
Organization: bakdata
Home Page: https://medium.com/bakdata/exploring-data-pipelines-in-apache-kafka-with-streams-explorer-8337dd11fdad
python data-pipelines hacktoberfest kubernetes react apache-kafka data-stream kafka-connect kafka-streams

beneath-hq / beneath
data-pipelines,Beneath is a serverless real-time data platform ⚡️
Organization: beneath-hq
Home Page: https://beneath.dev
dataops mlops data-science data-engineering data-pipelines developer-tools data-warehouse streaming etl analytics

bruin-data / bruin
data-pipelines,Bruin is a data pipeline tool that is designed to be easy-to-use. It allows building data pipelines using SQL and Python, and has built-in data quality checks.
Organization: bruin-data
Home Page: https://getbruin.com
analytics bigquery data-modeling data-pipelines python snowflake sql data-analysis data-transformation

cogstack / cogstack-nifi
data-pipelines,Building data processing pipelines for documents processing with NLP using Apache NiFi and related services
Organization: cogstack
Home Page: https://hub.docker.com/r/cogstacksystems/cogstack-nifi/
apache-nifi nifi elasticsearch kibana data-pipelines nlp rest electronic-health-records data-integration

combust / mleap
data-pipelines,MLeap: Deploy ML Pipelines to Production
Organization: combust
Home Page: https://combust.github.io/mleap-docs/
scikit-learn spark data-pipelines transformers tensorflow scala python

conductor-sdk / conductor-python
data-pipelines,Conductor OSS SDK for Python programming language
Organization: conductor-sdk
python conductor workflow data-pipelines durable-computing durable-execution etl-pipeline

confluentinc / learn-kafka-courses
data-pipelines,Learn the basics of Apache Kafka® from leaders in the Kafka community with these video courses covering the Kafka ecosystem and hands-on exercises.
Organization: confluentinc
Home Page: https://developer.confluent.io/
apache-kafka kafka kafka-streams data-pipelines data-pipeline ksqldb stream-processing

dagster-io / dagster
data-pipelines,An orchestration platform for the development, production, and observation of data assets.
Organization: dagster-io
Home Page: https://dagster.io
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

danilbaibak / ml-in-production
data-pipelines,The practical use-cases of how to make your Machine Learning Pipelines robust and reliable using Apache Airflow.
User: danilbaibak
machine-learning data-pipelines apache-airflow data-engineering

data-engineering-community / data-engineering-wiki
data-pipelines,The best place to learn data engineering. Built and maintained by the data engineering community.
Organization: data-engineering-community
Home Page: https://dataengineering.wiki
data database data-engineering data-engineer sql data-modeling data-pipelines etl

datacater / datacater
data-pipelines,The developer-friendly ETL platform for transforming data in real-time. Based on Apache Kafka® and Kubernetes®.
Organization: datacater
Home Page: https://datacater.io
cloud-native data-pipelines kafka kubernetes etl apache-kafka python

dataflint / spark
data-pipelines,Performance Observability for Apache Spark
Organization: dataflint
apache-spark big-data observability optimization data-pipeline data-pipelines databricks dataproc emr etl

dataform-co / dataform
data-pipelines,Dataform is a framework for managing SQL based data operations in BigQuery
Organization: dataform-co
Home Page: https://cloud.google.com/dataform/docs
data-pipelines elt data-engineering business-intelligence analytics etl hacktoberfest

datajoint / datajoint-python
data-pipelines,Relational data pipelines for the science lab
Organization: datajoint
Home Page: https://datajoint.com/docs
datajoint scientific-computing databases data-analysis pipeline-framework python relational-databases relational-algebra relational-model mysql

dataplane-app / dataplane
data-pipelines,Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Organization: dataplane-app
Home Page: https://dataplane.app
golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering

didacthq / didact
data-pipelines,The open source, standalone, fullstack .NET job orchestrator that we've been missing.
Organization: didacthq
Home Page: https://www.didact.dev
background-jobs background-service data-engineering data-orchestrator data-pipelines orchestration orchestration-framework orchestrator pipelines workflow-engine

didacthq / didact-engine
data-pipelines,The REST API and execution engine for the Didact Platform.
Organization: didacthq
Home Page: https://www.didact.dev
data-engineering data-pipelines dotnet dotnet-standard orchestrator parallel-processing pipelines rest-api workflow workflow-engine

elementary-data / dbt-data-reliability
data-pipelines,dbt package that is part of Elementary, the dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Organization: elementary-data
Home Page: https://www.elementary-data.com/
data dbt-packages data-reliability data-pipeline-monitoring data-pipelines analytics analytics-engineering data-observability data-lineage dbt

elementary-data / elementary
data-pipelines,The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Organization: elementary-data
Home Page: https://www.elementary-data.com/
data-lineage data-governance data-warehouse snowflake bigquery data-analysis data-pipelines data-pipeline lineage data-reliability

flipkart-incubator / spark-transformers
data-pipelines,Spark-Transformers: Library for exporting Apache Spark MLLIB models to use them in any Java application with no other dependencies.
Organization: flipkart-incubator
spark mllib export machine-learning-library machine-learning-algorithms machine-learning apache-spark data-pipelines transformers scala java

fmind / mlops-python-package
data-pipelines,Kickstart your MLOps initiative with a flexible, robust, and productive Python package.
User: fmind
Home Page: https://fmind.github.io/mlops-python-package/
mlops python automation data-pipelines data-science machine-learning mlflow pandera pydantic

galileo-galilei / kedro-pandera
data-pipelines,A kedro plugin to use pandera in your kedro projects
User: galileo-galilei
Home Page: https://kedro-pandera.readthedocs.io/en/latest/
data-contracts data-pipelines data-schemas kedro kedro-plugin pandera pipelines-testing

googlecloudplatform / public-datasets-pipelines
data-pipelines,Cloud-native, data onboarding architecture for Google Cloud Datasets
Organization: googlecloudplatform
Home Page: https://cloud.google.com/solutions/datasets
datasets data-pipelines airflow google-cloud cloud-composer open-data data-engineering data-architecture cloud-native bigquery

immu0001 / udacity-data-engineer-nanodegree
data-pipelines,Classwork projects and home works done through Udacity data engineering nano degree
User: immu0001
spark data-analysis big-data etl data-pipelines classwork airflow-dags data-science data-lake-analytics s3-bucket

infiniflow / ragflow
data-pipelines,RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
Organization: infiniflow
Home Page: https://ragflow.io
document-understanding llm ocr rag table-structure-recognition data-pipelines deep-learning document-parser information-retrieval llmops

infinyon / fluvio
data-pipelines,Lean and mean distributed stream processing system written in rust and web assembly.
Organization: infinyon
Home Page: https://www.fluvio.io/
cloud-native streaming rust real-time serverless stateful stream-processing webassembly data-integration data-flow

kenthsu / udacity-data-engineering-nanodgree
data-pipelines,Udacity Data Engineering Nanodegree Program
User: kenthsu
data-engineering postgresql apache-cassandra aws-redshift aws-s3 apache-spark apache-airflow data-pipelines data-warehouses data-lake

kevin-hanselman / dud
data-pipelines,A lightweight CLI tool for versioning data alongside source code and building data pipelines.
User: kevin-hanselman
Home Page: https://kevin-hanselman.github.io/dud/
data-engineering data-pipelines data-science dataset dvcs machine-learning mlops

koolreport / core
data-pipelines,An Open Source PHP Reporting Framework that helps you to write perfect data reports or to construct awesome dashboards in PHP. Working great with all PHP versions from 5.6 to latest 8.0. Fully compatible with all kinds of MVC frameworks like Laravel, CodeIgniter, Symfony.
User: koolreport
Home Page: https://www.koolreport.com/
php reporting framework data-visualization data-viz data-analysis data-summarization reporting-engine reporting-tool mysql-reporting-tools

linkedin / hoptimator
data-pipelines,Multi-hop declarative data pipelines
Organization: linkedin
brooklin cdc data-pipelines flink kafka kafka-connect

mage-ai / mage-ai
data-pipelines,🧙 Build, run, and manage data pipelines for integrating and transforming data.
Organization: mage-ai
Home Page: https://www.mage.ai/
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

mdh266 / airflowdatapipeline
data-pipelines,Example of an ETL Pipeline using Airflow
User: mdh266
airflow data-engineering data-pipelines etl postgresql python

meltano / meltano
data-pipelines,Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Organization: meltano
Home Page: https://meltano.com/
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk

minhadona / data_engineer_interview_challenges
data-pipelines,Found a data engineering challenge or participated in a selection process ? Share with us!
User: minhadona
beginner beginner-friendly beginner-project beginners code-challenge-practice code-challenges data-engineering data-pipeline data-pipelines dataengineering

mycelial / mycelial
data-pipelines,Move your data with ease.
Organization: mycelial
Home Page: https://mycelial.com
data-pipelines edge-computing etl etl-pipeline rust

opendatadiscovery / odd-platform
data-pipelines,First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Organization: opendatadiscovery
Home Page: https://opendatadiscovery.org
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery

orchest / orchest
data-pipelines,Build data pipelines, the easy way 🛠️
Organization: orchest
Home Page: https://orchest.readthedocs.io/en/stable/
data-science machine-learning pipelines ide jupyter cloud self-hosted jupyterlab notebooks docker

patterns-app / patterns-devkit
data-pipelines,Data pipelines from re-usable components
Organization: patterns-app
data-science data-analysis pipelines etl etl-pipeline etl-framework functional-reactive-programming data-engineering sql immutability

raystack / optimus
data-pipelines,Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Organization: raystack
Home Page: https://raystack.github.io/optimus
airflow etl workflows automation golang bigquery data-warehouse analytics data-modelling analytics-engineering

recap-build / recap
data-pipelines,Work with your web service, database, and streaming schemas in a single format.
Organization: recap-build
Home Page: https://recap.build
data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl

sciphi-ai / r2r
data-pipelines,The framework for fast development and deployment of RAG systems.
Organization: sciphi-ai
Home Page: https://r2r-docs.sciphi.ai/
artificial-intelligence large-language-models retrieval retrieval-augmented-generation search chatbot data-pipelines deep-learning langchain llama-index

shravan-kuchkula / udacity-data-eng-proj-1
data-pipelines,Developed a data pipeline to automate data warehouse ETL by building custom airflow operators that handle the extraction, transformation, validation and loading of data from S3 -> Redshift -> S3
User: shravan-kuchkula
airflow aws-redshift data-pipelines docker

smart-data-lake / smart-data-lake
data-pipelines,Smart Automation Tool for building modern Data Lakes and Data Pipelines
Organization: smart-data-lake
Home Page: https://www.smartdatalake.io
data-lake scala spark smart-data-lake hadoop hive deltalake transform-data data-pipelines

terrytangyuan / awesome-kubeflow
data-pipelines,A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)
User: terrytangyuan
Home Page: https://terrytangyuan.github.io/awesome-kubeflow/
awesome awesome-list awesome-lists cloud-native cncf deep-learning devops kubeflow kubeflow-pipelines kubernetes machine-learning mlops workflow-orchestration data-pipelines data-science mle-pipelines

tuva-health / tuva
data-pipelines,Main repo including core data model, data marts, reference data, terminology, and the clinical concept library
Organization: tuva-health
Home Page: https://thetuvaproject.com/
analytics-engineering bigquery data-analytics data-governance data-lineage data-pipelines data-warehouse dbt dbt-packages healthcare healthcare-analysis healthcare-data open-source redshift snowflake sql terminology

unstructured-io / unstructured
data-pipelines,Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
Organization: unstructured-io
Home Page: https://www.unstructured.io/
deep-learning document-parsing machine-learning nlp ocr information-retrieval data-pipelines ml preprocessing pdf-to-text

vmware / versatile-data-kit
data-pipelines,One framework to develop, deploy and operate data workflows with Python and SQL.
Organization: vmware
data-science data-engineering sql trino data-lineage etl elt data-pipelines data-engineer data-warehouse