khuyentran1401 / data-science-template Goto Github PK

Template for a data science project

Makefile 26.30% Python 73.70%

data-science-template's Introduction

Data Science Cookie Cutter

Why?

It is important to structure your data science project based on a certain standard so that your teammates can easily maintain and modify your project.

This repository provides a template that incorporates best practices to create a maintainable and reproducible data science project.

Tools used in this project

hydra: Manage configuration files - article
pdoc: Automatically create an API documentation for your project
pre-commit plugins: Automate code reviewing formatting
Poetry: Dependency management - article

How to use this project

Install Cookiecutter:

pip install cookiecutter

Create a project based on the template:

cookiecutter https://github.com/khuyentran1401/data-science-template

Resources for a detailed explanation of this template:

data-science-template's People

Contributors

Stargazers

Watchers

Forkers

joseferncruz vpachava adbmd jwlai-cloud junhwanjang adamazoulay uberkeil cbuton rimij405 dusanbaek bhatkishan42 dtheod galvin-mj jjneid psxz unquietmind youngxu06 fredmontet datalabs-apps-tools-docs-tips ioannismesionis andrew-siu12 tomblackwoodttm pstanisl marantmir onegneissguy hasnainrsyed oort77 shadynagy232 edoch94 chriskununu carloscaravantsz tariqmusa leeaurele satfail rodrigo-defined infoslack carloskin96 banditelol tfha jonatasmarqs aleksei-mashlakov sgouda0412 pkruskal lealdino rodrigocesar jnsofini wzhao5 adevgarcia sophia0509 f1se4 data-verso kaidduong andrewdo25 cuevash m4noj thanhbast sudheer554 cdfrdmn cryptonome torrikar chooonth marxmt research-ai-templates jhanvi0905 homodigitus angel-rc owini yaniela garcialn ajayarunachalam blurrydev tusharkalecam michalnawrot ganesh3 atworkdante mrdo19111911 tariqueakhtar mbongenimm biojack-ds alekseimikhalev amgfernandes wedeueisbraz wedeueis hiyojoon fiqihfathorr thiwankajayasiri baoson1110 sgmil-mdfi takeyama0 kabilan108 maniklem spsarkar joseestevan rake93 iserralv mattobad nhatrio wagnerchagas rudzanimulaudzi ianalrahwan

data-science-template's Issues

Invalid project setup with poetry

Hey Khuyen,

Thanks for creating this easy-to-use template. I really like the simple approach.
I tried creating a template based on the instructions provided. Thus far, it works but with some issues around this line (I think):
python = "{{ cookiecutter.author_name }}" in the project.toml file.

It's referencing the author's name as opposed to the compatible_python_versions.

Refactorize Cookie-cutter command prompt to create a heavily modularized data science template

Hi @khuyentran1401 !

I just saw in cookiecutter/cookiecutter#1881 that cookiecutter finally has the feature of adding human-readable prompts to the different variables. This enables us to create a more sophisticated data science template.

My initial thoughts is to make a step further in what I did in #18 (I didn't check exactly how it looked like since you made some modifications). My initial idea is to categorize all this giant universe of Machine Learning tools regarding its functionalities (logging, orchestration, data storage, Python linter and code formatter, etc), and then list all tools so that the user may choose one. Therefore, my initial idea is: to create a heavily modularized data science template, in which the final template structure depends on the tools opted by the user, but at the same time to ensure that the directory structure don't vary too much.

However, I am not sure if you share the same goal as me. I just saw that you removed DVC, so you may have some considerations to do regarding this goal.

What do you think about it?

Angreal as an alternative to cookiecutter + Makefile

First, awesome template. I've been doing some research on what's changed on templating data science projects over the last few years and came across yours.

Wasn't sure what the best route to reach out was so thought I'd just present it to you in an issue, I built a templating engine very similar to cookie cutter but includes the ability to include python functions as plugins for a command line interface. Thought you mind find it interesting / useful if you find maintaining / extending the Makefile annoying.

https://angreal.github.io/angreal/

Again - awesome work !

Update hydra to version 1.2

Update hydra to version 1.2 to stop changing the runtime working directory to the job's output directory by default.

Uniforming the tools used in each branch

Hi there!

Besides dvc, pip, and poetry, I've also noticed that some tools that are used in some branches are simply not used in others. I honestly didn't see the reason for that since there isn't an alternative being used for a replacement of them...

	`hydra`	`flake8`	`prefect`
`dvc-poetry`	✅	❌	❌
`dvc-pip`	✅	❌	❌
`prefect-poetry`	❌	✅	✅

Why doesn't prefect-poetry use hydra?
Why don't dvc-poetry and dvc-pip use prefect or flake8?

I understood the reason of having dvc-poetry and dvc-pip as pip and poetry are two different strategies to manage packages, and therefore two different templates are needed. However, prefect-poetry seems unnecessary as prefect doesn't overlap the other tools' goals.

It seems that you started with one model and then you recreated it, thus abandoning some tools and adopting others. I am not sure... If PR are welcoming, I propose to adopt the same set of tools, unless they are conflicting or overlapping. Let me understand what is your idea so that I can collaborate with your project :)

Mkdocs template with material

Hello, have you tried mkdocs for documentation?

I think it's more beautiful and complete. Are you accepting PR for this repo?

Thanks.