Topic: dataengineering Goto Github

👇 Here are 489 public repositories matching this topic...

514-labs / moose
dataengineering,The developer framework for your data & analytics stack
Organization: 514-labs
Home Page: https://www.moosejs.com
data dataengineering deployment framework insights metrics analytics

aakashnand / trino-ranger-demo
dataengineering,Tutorial on how to setup Trino and Apache Ranger using docker
User: aakashnand
docker ranger trino dataengineering datagovernance tutorial

abhishek-ch / data-machinelearning-the-boring-way
dataengineering,Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.
User: abhishek-ch
data-infrastructure dataengineering datascience kubernetes machine-learning mlops

anuran-roy / serpytor
dataengineering,A distributed, low-code, end-to-end data collection and analysis tool for data folks. Take the pain out of data collection from your pipeline!
User: anuran-roy
Home Page: https://anuran-roy.github.io/serpytor
data dataengineering datascience low-code lowcode pipeline python python3 distributed-computing distributed-systems

awslabs / aws-ddk
dataengineering,An open source development framework to help you build data workflows and modern data architecture on AWS.
Organization: awslabs
Home Page: https://awslabs.github.io/aws-ddk/
aws dataops dataengineering python

awslabs / aws-orbit-workbench
dataengineering,A Data Platform built for AWS, powered by Kubernetes.
Organization: awslabs
Home Page: https://awslabs.github.io/aws-orbit-workbench/
kubernetes eks jupyter jupyterhub data-analysis datalake mach orbit-workbench gpu eks-cluster aws dataengineering redshift analytics workbench

balajirvp / de-zoomcamp
dataengineering,Code/Notes for the Data Engineering Zoomcamp by DataTalksClub
User: balajirvp
batchprocessing bigquery dataengineering datalake datamodeling datawarehouse dbt docker gcp prefect pyspark python rdd spark sql streaming terraform

chollinger93 / bridgefour
dataengineering,Bridge Four is a simple, functional, effectful, single-leader, multi worker, distributed compute system optimized for embarrassingly parallel workloads.
User: chollinger93
Home Page: https://chollinger.com/blog/2023/06/building-a-functional-effectful-distributed-system-from-scratch-in-scala-3-just-to-avoid-leetcode-part-1/
data dataengineering distributed-systems scala

cynthiakoopman / forecasting-solar-energy
dataengineering,Forecasting Solar Power: Analysis of using a LSTM Neural Network
User: cynthiakoopman
deeplearning machinelearning forecasting lstm neuralnetworks data-science dataengineering solarpower solar-energy renewable-energy

daihuynh / dagster_dbt_metabase_simple_solution
dataengineering,A simple Data Engineering solution for testing or education purposes. You only need to know SQL and Python to understand this project. Dagster is the data orchestration, while DBT is for data transformation. The BI tool is Metabase.
User: daihuynh
dagster dataengineering dbt metabase sqlite

danielsaban / data-scraping-sofascore
dataengineering,Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.
User: danielsaban
dataengineering datamining webscraping relational-databases parser etl-pipeline rest-api

data-burst / data-engineering-roadmap
dataengineering,
Organization: data-burst
Home Page: https://databurst.tech
data-engineering dataengineering roadmap data-engineering-roadmap

dataexpert-io / data-engineer-handbook
dataengineering,This is a repo with links to everything you'd ever want to learn about data engineering
Organization: dataexpert-io
apachespark awesome bigdata data dataengineering sql

datafold / data-diff
dataengineering,Compare tables within or across databases
Organization: datafold
Home Page: https://docs.datafold.com
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring

datavault-uk / automate-dv
dataengineering,A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
Organization: datavault-uk
Home Page: https://www.automate-dv.com
datavault20 dbt datawarehousing datavault etl snowflake dataengineering datawarehouse datalake elt

deganza / jupyter_pandas_cheat_sheet
dataengineering,Learn the basic commands to use Pandas in Jupyter-Notebook to accomplish the most important Data Enginnering tasks. Read the underlying article on Medium:
User: deganza
Home Page: https://deganza11.medium.com/the-basics-of-a-jupyter-notebook-pandas-cheat-sheet-for-data-engineers-c1099791a93c
cheatsheet pandas dataengineering jupyter-notebook

eldar1205 / awesome-python-backend
dataengineering,Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers
User: eldar1205
backend backendengineering dataengineering python

erdemozgen / data-engineering-roadmap
dataengineering,Roadmap for Data Engineering
User: erdemozgen
awesome ci-cd cloud data-science dataengineering development devops machine-learning mlops roadmap

finance-and-ml / us-stock-prediction-using-ml-and-spark
dataengineering,Predict stock price based on financial news feeds
Organization: finance-and-ml
stock-price-prediction spark scala tableau webscraping dataengineering

franloza / coches-net-dashboard
dataengineering,Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market
User: franloza
dagster data dataengineering duckdb elt pipeline scraping visualization

fvaleye / metadata-guardian
dataengineering,Provide an easy way with Python to protect your data sources by searching its metadata. 🛡️
User: fvaleye
Home Page: https://fvaleye.github.io/metadata-guardian/python
metadata dataengineering pii-detection data dataset datastructures metadata-extraction metadata-management metadata-parser metadata-driven

grai-io / grai-core
dataengineering,
Organization: grai-io
Home Page: https://www.grai.io
hacktoberfest data datalineage dbt open-source postgresql python mssql mysql snowflake

josephmachado / e2e_datapipeline_test
dataengineering,Example repo to create end to end tests for data pipeline.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/setting-up-e2e-tests/
aws dataengineering moto pytest python3 testing

josephmachado / socialetl
dataengineering,Project for "Data pipeline design patterns" blog.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/code-patterns/
dataengineering design-patterns etl-pipeline makefile python social-media-data sqllite3 reddit

kevinheavey / modern-polars
dataengineering,Code and data for the Modern Polars book
User: kevinheavey
Home Page: https://kevinheavey.github.io/modern-polars/
data-analytics data-engineering data-science dataengineering pandas polars python

kirilldikalin / myknowledge
dataengineering,Все, о чем меня когда-либо спрашивали на собеседованиях, и другие полезные знания в кратком формате
User: kirilldikalin
Home Page: https://kirilldikalin.github.io/kirilldikalin.io/knowlege_base/iKnowledge_base.html
data-science python dataengineering sql

kislerdm / data-engineering-interviews
dataengineering,Data engineering interviews Q&A for data community by data community
User: kislerdm
Home Page: https://data-engineering-interviews.org
dataengineering interview-questions opensource python sql kafka spark linux

marcosmjd / ghcn-d
dataengineering,Data Pipeline from the Global Historical Climatology Network DataSet
User: marcosmjd
airflow bigquery dataengineering elt etl dbt googlecloudstorage googledatastudio iac terraform

mehd-io / pypi-duck-flow
dataengineering,end-to-end data engineering project to get insights from PyPi using python and duckdb
User: mehd-io
dataengineering duckdb etl python

mikestack15 / orangutan-stem
dataengineering,An open-source project dedicated to constructing robust data pipelines and scalable software infrastructure. We leverage industry-standard tools favored by developers to enhance efficiency and reliability. Uniquely, these pipelines are field-tested on farms across Sumatra, Indonesia, ensuring real-world applicability and resilience.
User: mikestack15
Home Page: https://orangutan-stem.com
dataengineering portfolio stem airflow-docker computer-vision django-application software-engineering django-docker traefik ultralytics

minhadona / data_engineer_interview_challenges
dataengineering,Found a data engineering challenge or participated in a selection process ? Share with us!
User: minhadona
beginner beginner-friendly beginner-project beginners code-challenge-practice code-challenges data-engineering data-pipeline data-pipelines dataengineering

noahgift / data-engineering-and-dataops
dataengineering,Duke MIDS: Data Engineering and DataOps Course
User: noahgift
Home Page: https://noahgift.github.io/data-engineering-and-dataops/
duke cloud dataops dataengineering data data-science software-engineering mlops book course

ogbinar / dataengineeringpilipinas
dataengineering,Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.
User: ogbinar
Home Page: https://dataengineering.ph
community data database dataengineering datascience datasets engineering etl philippines python sql

olist / work-at-olist-data
dataengineering,Apply for a job at Olist's Data Team: https://olist.gupy.io/
Organization: olist
Home Page: https://olist.gupy.io/
python pandas sql datascience data analytics dataengineering machinelearning r julia dataset

open-metadata / openmetadata
dataengineering,Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
Organization: open-metadata
Home Page: https://open-metadata.org
metadata datadiscovery data-science dataquality data-profiling metadata-management dataengineering data-catalog data-observability dbt

prodmodel / prodmodel
dataengineering,Build, test, deploy, iterate - Dev and prod tool for data science pipelines
Organization: prodmodel
build-tool build-automation build-system data-science data datascience production productivity dataengineering dataeng

sarahmk125 / airflow-docker-metrics
dataengineering,
User: sarahmk125
dataengineering airflow-metrics statsd-format grafana airflow-containers statsd-metrics data orchestration workflow-orchestration

snowflake-labs / snowpark-python-demos
dataengineering,This repository provides various demos/examples of using Snowpark for Python.
Organization: snowflake-labs
python snowpark dataengineering datascience machine-learning

sparsh-ai / recohut
dataengineering,Recohut - Learn data engineering, data science
User: sparsh-ai
Home Page: https://www.recohut.in/docs/introduction
data-engineering dataengineering datascience

spratiher9 / sparkdataset
dataengineering,Instant search for and access to many datasets in Pyspark.
User: spratiher9
Home Page: https://pypi.org/project/sparkdataset/
standard datasets pyspark python3 python spark easy-access-application instantsearch r benchmark-framework

tirendazacademy / awesome-data-science-resources
dataengineering,Resources about data science, machine learning, deep learning, data engineering, and SQL.
User: tirendazacademy
ai artificial-intelligence data-analysis data-engineering data-science dataengineering datascience deep-learning deeplearning machine-learning

tobikodata / sqlmesh
dataengineering,Efficient data transformation and modeling framework that is backwards compatible with dbt.
Organization: tobikodata
Home Page: https://sqlmesh.com
dataops elt etl sql python dataengineering transformation dbt

waylonwalker / kedro-action
dataengineering,A GitHub Action to lint, test, build-docs, package, and run your kedro pipelines. Supports any Python version you'll give it (that is also supported by pyenv).
User: waylonwalker
kedro dataengineering datapipeline actions

waylonwalker / kedro-static-viz
dataengineering,kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
User: waylonwalker
Home Page: https://static-viz.kedro.dev
data datapipeline dataengineering kedro python kedro-plugin

wittline / apache-spark-docker
dataengineering,Dockerizing an Apache Spark Standalone Cluster
User: wittline
Home Page: https://wittline.github.io/apache-spark-docker/
pyspark apache-spark docker-compose hive-metastore hive hdfs hue dataengineering dataengineer hadoop-cluster

wittline / data-engineer-challenge
dataengineering,Challenge Data Engineer
User: wittline
Home Page: https://wittline.github.io/data-engineer-challenge/
data-engineering data-pipeline dataengineering docker docker-compose fastapi postgresql

wittline / pydag
dataengineering,Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag
User: wittline
Home Page: https://wittline.github.io/pyDag/
big-data bigquery cloud dag data-engineering data-pipeline dataengineering dataproc dataproc-cluster directed-acyclic-graph google-cloud google-cloud-platform parallel-processing task-scheduler task-scheduling workflow-engine

wittline / pyspark-on-aws-emr
dataengineering,The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.
User: wittline
Home Page: https://wittline.github.io/pyspark-on-aws-emr/
aws emr-cluster aws-emr python spark pyspark big-data-analytics big-data dataengineering wordcloud-generator

zacharyt-cs / reddit-data-engineering
dataengineering,An end-to-end data engineering pipeline to create a dashboard for the latest content on the r/Stocks subreddit
User: zacharyt-cs
dataengineering google-cloud pyspark batch-processing airflow bigquery terraform

zinggai / zingg
dataengineering,Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Organization: zinggai
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution