nogibjj / mlops-template-shunian-chen Goto Github PK

Template for MLOPs projects with GPU

1. First thing to do on launch is to open a new shell and verify virtualenv is sourced.

Things included are:

• Makefile

• Pytest

• pandas

• Pylint

• Dockerfile

• GitHub copilot

• jupyter and ipython

• Most common Python libraries for ML/DL and Hugging Face

• githubactions

Verify GPU works

The following examples test out the GPU

• run pytorch training test: python utils/quickstart_pytorch.py
• run pytorch CUDA test: python utils/verify_cuda_pytorch.py
• run tensorflow training test: python utils/quickstart_tf2.py
• run nvidia monitoring test: nvidia-smi -l 1 it should show a GPU
• run whisper transcribe test ./utils/transcribe-whisper.sh and verify GPU is working with nvidia-smi -l 1

Additionally, this workspace is setup to fine-tune Hugging Face

python hf_fine_tune_hello_world.py

Used in Following Projects

Used as the base and customized in the following Duke MLOps and Applied Data Engineering Coursera Labs:

• MLOPs-C2-Lab1-CICD
• MLOps-C2-Lab2-PokerSimulator
• MLOps-C2-Final-HuggingFace
• Coursera-MLOps-C2-lab3-probability-simulations
• Coursera-MLOps-C2-lab4-greedy-optimization

References

• Building Cloud Computing Solutions at Scale Specialization
• Python, Bash and SQL Essentials for Data Engineering Specialization
• Implementing MLOps in the Enterprise
• Practical MLOps: Operationalizing Machine Learning Models
• Coursera-Dockerfile