A few projects I found that we can use to improve the experience:

• tabs: https://github.com/executablebooks/sphinx-tabs
• toggle: https://github.com/executablebooks/sphinx-togglebutton
• exercises: https://github.com/executablebooks/sphinx-exercise
• comments: https://github.com/executablebooks/sphinx-comments
• runnable snippets: https://github.com/executablebooks/thebe

This site is deployed here.

The landing page needs some work.

• add some CSS, so the div for subscribing to the newsletter looks good
• Change the links to Twitter and LinkedIn for the actual logos and center them

We're using jupyter-book so check out their docs to see how to do this.

• finish code examples
• finish lesson summary
• ensure building the website does not execute the example notebooks (but still indexes them)
• record video
• upload to youtube
• publish
• announce

Notebooks Academy: Write Production-Ready Code From Jupyter [DRAFT]

The course teaches how to use Jupyter to develop maintainable and production-ready code.

Please comment on this issue with your feedback! Are we missing any topics?

Lessons

1. Why? The prototype, then refactor problem
2. Writing clean notebooks
3. Version control
4. Hidden state
5. Modularization
6. Refactoring legacy pipelines
7. Building data pipelines
8. Integration testing
9. Debugging
10. Running pipelines in the cloud
11. Notebook meta-analysis
12. Using SQL in Jupyter
13. Deployment

Format

11 video lessons, 20-30 minutes each.

I'm thinking of doing this a project-based course, so by the end of it, students have a pipeline up and running this dataset looks interesting.

Pre-requisites

• Experience working with standard open-source tools: Jupyter, pandas, and scikit-learn

Syllabus

1. Why? The prototype, then refactor problem

Introduction to the problem: Developing projects in single notebooks cause a lot of trouble. They are hard to maintain, test and review. However, if we follow some best practices, and with the help of some open-source tools, we can implement a workflow that allows us to go from Jupyter to production instantly.

Related material

• Why (and how) to put notebooks in production
• From Jupyter to production presentation

2. Writing clean notebooks

This lesson shows best practices for writing clean notebooks (it takes most of its content from the blog post).

Related material

• On Writing Clean Notebooks
• Soorgeon (for linting and formatting)
• https://github.com/jupyter-lsp/jupyterlab-lsp
• micro pipelines

3. Version control

It's challenging to version control Jupyter Notebooks because the .ipynb format is JSON. This lesson shows how to change the underlying format to .py and still interact with those files as notebooks.

Notes

• Show other alternatives such as nbdime and the Jupyterlab-git plugin
• Discuss jupytext's pairing feature to store the output in a separate file

Related material

• Why (and how) to put notebooks in production
• On the Myths and Problems of Jupyter Notebooks

4. Hidden state

Since notebooks are developed interactively, excessive editing often leads to broken notebooks. This lesson introduces notebooks smoke testing: we execute them with a sample of the data on each git push using papermill. It also shows how to set up GitHub Actions.

Related material

• Why (and how) to put notebooks in production
• On the Myths and Problems of Jupyter Notebooks

5. Modularization

Modularizing code is critical to developing maintainable and testable software. This lesson shows how to create a package to modularize our work, define functions in Python modules, and unit test those functions using pytest.

Notes

• Show how IPython auto-reloading works
• Covers pytest basic features: fixtures, parametrizing, testing exceptions

Related material

• Rethinking Software Testing for Data Science @ PyData Global 2020
• Rethinking Continuous Integration for Data Science
• Training-serving skew
• ML testing blog post

6. Refactoring legacy pipelines

A lot of existing pipelines live in notebooks. This lesson shows how to refactor a monolithic notebook-based project into a data pipeline.

Related material

• Refactoring a Jupyter notebook into a maintainable pipeline series
• Soorgeon

7. Building data pipelines

Long notebooks are hard to manage because many variables and code are involved. Breaking down our analysis in multiple steps allows us to collaborate better and test our notebooks.

Notes

• Why is structure important?
• Mention advantages of building a data pipeline: can do integration testing, run tasks in parallel

Related material

• On Writing Clean Notebooks
• The task is a quantum of workflow

8. Integration testing

Garbage in, garbage out. Testing for data quality at each stage of our pipeline ensures that we meet a minimum level of data quality. This lesson shows how to do integration testing after executing each notebook.

Related material

• [Same as lesson 4. Modularization]
• Effective Testing for ML projects recording
• Effective Testing for ML series
• ml-testing repository

9. Debugging

Debugging data pipelines is challenging; however, having a robust unit and integration test suite helps us debug more effectively. This lesson shows how to debug data pipelines by conducting root cause analysis using pytest and the Python debugger.

Notes

• Show how to debug failing tests
• Cover Jupyter's visual debugger
• Show how to use ipdb
• Debugging with IPython.embed()
• Debugging code with breakpoints
• debuglater feature in Ploomber

Related material

• Rethinking Software Testing for Data Science @ PyData Global 2020

10. Running pipelines in the cloud

When working with large datasets, we may want to run our pipeline in the cloud. This lesson shows how to use Ploomber to run a pipeline in AWS and Kubernetes and retrieve results.

• Soopervisor

11. Notebook meta-analysis

The .ipynb format is self-contained to store code and output. Such output can be anything from text, tables, or images. This lesson shows how to analyze the content of a Jupyter notebook to extract its output to evaluate and compare model experiments.

Related materials

• sklearn-evaluation
• Scrapbook

12. Using SQL in Jupyter

Related materials

• Effective SQL
• JupySQL
• SQL pipelines in Ploomber

13. Deployment

This lesson shows how to generate a deployment artifact using a previously trained model to serve predictions.

Notes

• Show how to use Ploomber's pipeline composition capabilities to create a serving pipeline
• Discuss the importance of dependency locking
• Generating a source distribution
• When to use Docker (and when not to)

Related material

• Developing custom scikit-learn transformers and estimators
• Soopervisor

Basic materials

I may record these additional short lessons to cover the basics of dependency management and virtual environments.

• Managing software dependencies for Data Science projects
• Setting up reproducible Python environments for Data Science

Lessons to be considered

• Profiling notebooks (memory, CPU, GPU)
• Report generation (nbconvert, quarto)
• Dashboards (voilá)
• Technical blogging (Jupyblog)

Optional lessons

Optional lessons I may record.

• Data lineage using parquet's metadata
• Versioning deployment artifacts and models with python-versioneer
• Managing dependencies and virtual environments

this looks useful: https://github.com/wpilibsuite/sphinxext-opengraph