[v0.2] Embed jupyter notebook examples into SPHINX. about atlite HOT 4 CLOSED

I started looking into this (because it is a great thing and I wanted it for the atlite examples).
Writing it up here for reuse later.

The package

• Documentation for the SPHINX addon nbsphinx

Quick start from the doc

1. Installation: conda install -c conda-forge nbsphinx or pip install nbsphinx
2. in doc/conf.py
   a) Add nbsphinx and sphinx.ext.mathjax (if not already there) as extension
   b) Check that the checkpoints from jupyter are included here:
   exclude_patterns = ['_build', '**.ipynb_checkpoints']
3. Now *.ipynb files from doc and sub directories can be included and are rendered.

Important facts to remember

• For long-running cells / cells with download:
  a) If you clear all cell output, then the notebook is run by SPHINX before being displayed.
  b) If there is >= 1 cell output, then the notebook is not run and only displayed. (I leveraged this for
  not downloading cutouts again and again)
  c) Behaviour can be explicitly set, but is complicated, involves editing notebook metadata
• Do not include large pictures (because of git tracking, diffs and size)
• Markdown cells > Comments for rendering notebooks

NB: I am currently experiencing problems when nbconvert wants to run notebooks without cell output (see this issue). To prevent this, I set nbsphinx_execute = 'never' in the RTD conf.py.

ToDo / Check

• What else to look out for when using git and nbsphinx, see
  https://nbsphinx.readthedocs.io/en/0.4.2/usage.html#Using-Notebooks-with-Git

Including examples/*.ipynb into docs

I wanted the Examples to be listed in the toctree on readthedocs.org.

Problem

By default, SPHINX only allows for files in doc and subdirectories to be included.
For RELEASE_NOTES.rst this can be circumvented using .. include:: black magic (will set this up for atlite tomorrow), but this does not work for notebooks.

One approach would be to use Unix sym-links, but that breaks cross-plattform compatability.

Solution

Another addon called nbsphinx_link.
I installed it locally and it works nicely and does exactly what I want.

Setting this up with it installed is straight forward:

• Create your foo.ipynb e.g. in examples/foo.ipynb
• Create a linking file where you want it to be, I choose `doc/examples/foo.nblink.
• Edit the foo.nblink to contain

{
    "path": "../../examples/foo.ipynb"
}

• Reference doc/examples/foo.nblink in the doc/index.rst as a normal notebook / .rst file.

Clearing output of Jupyter notebooks

Problem

Usually you don't want the output of Jupyter notebooks commited to the repository,
as they make things harder / larger to diff / edit / store.

Recommendation

Locally install the precommit hook from here into your <repository>/.git/hooks/pre-commit file:
https://jamesfolberth.org/articles/2017/08/07/git-commit-hook-for-jupyter-notebooks/

If you are in the process of comitting an .ipynb file with output, it will ask you to confirm (y/n).
"y" if you want the output online, "n" to get another chance to delete it manually.

Other solutions exist (auto-purging, commit blocking), but this one is the simpliest and I think most flexible one for our small projects.

from atlite.

Recommended Workflow

Preparation

• Locally install this precommit hook.
• Install the SPHINX extensions in conf.py.

Workflow

1. Write a foo.ipynb with your desired content.
2. Do you need the output in the notebook?

• Yes: (for e.g. long running cells, large data ops) Keep it.
• No: "Edit -> Clear all output".

3. Is the notebook in the doc subdir?

• Yes: Good.
• No: Create an foo.nblink with this content and the "path" pointing to your foo.ipynb:
  {"path": "../../examples/foo.ipynb"}

4. Link to foo.ipynb or foo.nblnk in the documentation.
5. Commit; if the notebook contains the hook kick in: Confirm if you answered 2. with "Yes" or delete the output.

from atlite.

This nicely prepared workflow description should find its own place in the documentation, as a subsection of contributing.rst or in a new developernotes.rst, maybe.

from atlite.

I included it into the documentation at
https://atlite.readthedocs.io/en/latest/contributing.html#contributing-examples

Also SPHINX (more precise: Jinja2) now adds a note on top of each documentation file generated from notebooks linking to the original.
Realised using these lines:

Tagging and closing this issue.

from atlite.