spotify-data-engineering's People
Contributors
Watchers
spotify-data-engineering's Issues
refactory: sql문 깔끔하게 하기
Summary
sql문 깔끔하게 고침
To Do List
- dbkit.py 만들고 sql문 변수로 바꾸기
- data-catalog-query.py에 import
ETL 데이터전처리 Spark
Summary
Azure Databricks PySpark로 Azure Data Factory ETL 파이프라인에서 전처리하는 코드 작성
To Do List
- PySpark로 ETL 파이프라인에서 전처리하는 코드 작성 (f3ac6be)
- Azure Data Factory ETL 관련 파이프라인 연결 구성
ruff action 추가
Summary
ruff action 추가
To Do List
- ruff action 추가
Scraper Server 제작
Summary
Spotify API를 사용하여 Scraper 서버 제작.
FastAPI는 끝, 이제 Airflow할 차례
To Do List
- Docker Hub CI
- Azure Container App CD (아직은 약간 수동)
- FastAPI Scraper - 국내 Top 50 플레이리스트, 유명 여자아이돌 인기 트랙 & 아티스트 정보
Airflow DAG 작성
Summary
Airflow DAGs 작성
To Do List
- scraper api call DAG
- azure data factory run pipelines DAG
- postgresql
rawschema query DAG - azure databricks data warehouse ELT DAG
1/21 todo
Summary
github 기본적인 거 배움
branch 만들기, pr 하기, merge 하기 등등
To Do List
- 굿
Scraper Server 추가 제작 완료
Summary
Azure Container App에서 실행 중인 모든 FastAPI 서버 관련 코드 완료
To Do List
- FastAPI Scraper - 국내 Top 50 플레이리스트, 유명 남자아이돌 인기 트랙 & 아티스트 정보 추가
- FastAPI Data Catalog Query - Azure Data Factory
data_catalog파이프라인에서 사용할 Web 서버 제작
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.