jckantor / nbpages Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 4.0 7.42 MB

Command line tool to maintain a repository of Jupyter notebooks.

Home Page: https://jckantor.github.io/nbpages/

License: MIT License

Python 14.41% Jupyter Notebook 84.00% Smarty 1.58%

nbpages's Introduction

Welcome!

Here's my Mastodon feed.

nbpages's People

Contributors

Stargazers

Watchers

Forkers

fagan2888 simrit1 kylajones

nbpages's Issues

Critical: error with loading data files in Colab

Take a look at this notebook. Open it in Colab.
https://ndcbe.github.io/cbe67701-uncertainty-quantification/01.01-Contributed-Example.html

Running this line:

stock_data = pd.read_csv('./data/Stock_Data.csv')

Gave the following error:

---------------------------------------------------------------------------

FileNotFoundError                         Traceback (most recent call last)

<ipython-input-4-4072e866c75c> in <module>()
----> 1 stock_data = pd.read_csv('./data/Stock_Data.csv')

4 frames

/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
   1889         kwds["usecols"] = self.usecols
   1890 
-> 1891         self._reader = parsers.TextReader(src, **kwds)
   1892         self.unnamed_cols = self._reader.unnamed_cols
   1893 

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()

pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()

FileNotFoundError: [Errno 2] File ./data/Stock_Data.csv does not exist: './data/Stock_Data.csv'

Here is the problem: When writing to HTML, we need to point to the text file in GitHub.

@jckantor I would like to share this notebook with the class on Tuesday.

Embedded figures (likely) missing in Colab

Suggestion: Replace figure paths with urls to GitHub when rewriting the notebook

How to handle references?

End goal: extend "tags" with chapter and section.

Duplicate entries in data index

https://ndcbe.github.io/cbe67701-uncertainty-quantification/data_index.html

This is because I both give the name of the text file markdown description and I use it in the code. This is a low priority.

Document the style guide and lint feature

The header needs to be the first line in a cell. Lint checks for this.
Don't use HTML image tags

Missing Level one header leads to error message

Line 449 ... handle edge case of a missing level one header.
Add a corresponding test for this case
Add a lint check for missing level one header.

Fix broken links in the nbpages documentation

Table on Contents
Keyword Index

Tags are sorted alphabetically. But the prefix is header section number which needs to be parsed to correct sort. This is a problem if any of the section number components is becomes more than on digit.

Add comments to the initial templates

Links to notebooks in repos owned by an org do not work

For example, see the links to notebooks here: https://ndcbe.github.io/cbe67701-uncertainty-quantification/

This may be related to issue #13.

Special formatting for cells with special tags

In my current CBE 20258 notebooks, I use some HTML code to put a colored box around certain markdown cells. For example:

<div style="background-color: rgba(0,255,0,0.05) ; padding: 10px; border: 1px solid darkgreen;"> 
<b>Home Activity</b>: In the cell below, assign the value of 5 to the variable x. Then type Shift+Enter to execute the line.
</div>

This box is solid dark green which I use for all "home activities". I used a blue box for "class activities" and a red box for "important notes". Would it be possible to add a border and set the background color accordingly for cells with "home-activity", "class-activity", and "important-note" tags when publishing to the HTML pages? Currently, nbpages strips out this formatting, which I think is fine. I would much rather use tags.

Make the default behavior for nbpages to print the help

This will make it a little safer by requiring one to explicitly say publish.

Error setting up dual repo

@jckantor I am trying to set up https://github.com/ndcbe/cbe-xx258 to be the public version of https://github.com/ndcbe/CBExx258-instructor. After creating the public repo, I tried to run nbpages --setup, but got the following error message:

Traceback (most recent call last):
  File "/anaconda3/bin/nbpages", line 11, in <module>
    load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
    return ep.load()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
    return self.resolve()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
    from .nbcollection import Nb, NbCollection
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 20, in <module>
    sys.exit(1)
NameError: name 'sys' is not defined

I then manually made nbpages.cfg using the example here:

https://github.com/ndcbe/nbpages-testing-public/blob/master/nbpages.cfg

When I run nbpages --setup, I get the following error message:

Traceback (most recent call last):
  File "/anaconda3/bin/nbpages", line 11, in <module>
    load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
    return ep.load()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
    return self.resolve()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
    from .nbcollection import Nb, NbCollection
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 24, in <module>
    config = config["nbpages"]
  File "/anaconda3/lib/python3.7/configparser.py", line 958, in __getitem__
    raise KeyError(key)
KeyError: 'nbpages'

I am using the master branch of nbpages.

Document the metdata feature

This helps screen that all of the notebooks use the same version of Python.

Possible enhancements:

Specify a minimum version for Python

Setup fails using a local installation from master of git repo

Installation of nbpages:

(teaching) esc306856:nbpages adowling$ pip install -e .
Obtaining file:///Users/adowling/DowlingLab/Teaching/nbpages
Requirement already satisfied: nbconvert in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (5.4.1)
Requirement already satisfied: nbformat in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (4.4.0)
Collecting notedown (from nbpaages==0.0.1.dev12)
  Downloading https://files.pythonhosted.org/packages/d3/3a/d7c1817f3acb2e958b32fe85a35f52d270820fb9cebf1364b36c772cf3d0/notedown-1.5.1-py3-none-any.whl
Requirement already satisfied: mistune>=0.8.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.8.4)
Requirement already satisfied: jinja2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.10)
Requirement already satisfied: pygments in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.3.1)
Requirement already satisfied: traitlets>=4.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.3.2)
Requirement already satisfied: jupyter_core in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: entrypoints>=0.2.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.3)
Requirement already satisfied: bleach in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (3.1.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (1.4.2)
Requirement already satisfied: testpath in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.4.2)
Requirement already satisfied: defusedxml in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.5.0)
Requirement already satisfied: ipython-genutils in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (0.2.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (3.0.1)
Requirement already satisfied: six in /anaconda3/lib/python3.7/site-packages (from notedown->nbpaages==0.0.1.dev12) (1.12.0)
Collecting pandoc-attributes (from notedown->nbpaages==0.0.1.dev12)
  Downloading https://files.pythonhosted.org/packages/c2/0a/442cc9237dc997cd88155bdcb54bf86e703e699881f4134ecb733ccd670c/pandoc-attributes-0.1.7.tar.gz
Requirement already satisfied: MarkupSafe>=0.23 in /anaconda3/lib/python3.7/site-packages (from jinja2->nbconvert->nbpaages==0.0.1.dev12) (1.1.1)
Requirement already satisfied: decorator in /anaconda3/lib/python3.7/site-packages (from traitlets>=4.2->nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: webencodings in /anaconda3/lib/python3.7/site-packages (from bleach->nbconvert->nbpaages==0.0.1.dev12) (0.5.1)
Requirement already satisfied: attrs>=17.4.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (19.1.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (0.14.11)
Requirement already satisfied: setuptools in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (40.8.0)
Building wheels for collected packages: pandoc-attributes
  Building wheel for pandoc-attributes (setup.py) ... done
  Stored in directory: /Users/adowling/Library/Caches/pip/wheels/0f/43/b0/252730521a965a687748f0be111073e67ba229e640d147e65f
Successfully built pandoc-attributes
Installing collected packages: pandoc-attributes, notedown, nbpaages
  Running setup.py develop for nbpaages
Successfully installed nbpaages notedown-1.5.1 pandoc-attributes-0.1.7

Attempting to configure my first repo:

(teaching) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages --setup
Traceback (most recent call last):
  File "/anaconda3/bin/nbpages", line 11, in <module>
    load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
    return ep.load()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
    return self.resolve()
  File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 7, in <module>
    from .nbcollection import NbCollection
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 30, in <module>
    assert os.path.exists(config_file), f"configuration file {config_file} not found"
AssertionError: configuration file nbpages.cfg not found

Add Traceback to lint

Search and report 'Traceback' in order to detect faulty code cells.

nbpages -h only works out of the top directory

Need to either find nbpages.cfg or warn it needs to be run for the main directory.

Make the templates consistent with nbpages.cfg

Extend --remove_solution_code to include autograder tests

Mimic the syntax of Vocareum.

Mistake in regular expression used to generate nbpages.cfg

[NBPAGES]
github_repo_url = [email protected]:ndcbe/cbe67701-uncertainty-quantification.git
github_user_name = [email protected]:ndcbe
github_repo_name = cbe67701-uncertainty-quantification
github_pages_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification
repository = [email protected]:ndcbe/cbe67701-uncertainty-quantification
page_title = cbe67701-uncertainty-quantification
page_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification

Bug: Extra prefix in github_pages_url. Perhaps this is because the repo is in an organization.

Create stubs for Chapter 1 and Section 1.1 and 1.2 notebooks

This would be the most helpful to a new user.

IPython3 lexer unavailable warning

I installed nbpages from the master branch this morning on a second computer (macOS X).

I am seeing the following error messages:
ModuleNotFoundError: No module named 'jupyter_contrib_nbextensions'

UserWarning: IPython3 lexer unavailable, falling back on Python 3

Are these missing dependencies for nbpages?

I installed nbpages by cloning the git repo and running pin install -e .

Index class needed

Indexing is turning out to be one of the most useful features of nbpages. So far there are tag, python module, figure, and data indexes. Additional indexes that might be useful for notebook development are

index of python error messages
index of bibliographic references
index resulting from a user search

Currently some of the indexes are cached, which makes it problematic to do rewriting for 'under-construction' tags. (issue #28).

Propose a new index class created with

tag_index = Index()
tag_index.index(pattern)
tag_index.write_html(title)

document --search feature

Open in Colab link is broken for repos hosted by orgs

Perhaps #26 will also fix this. I am opening a separate issue so we remember to test it.

python library usage index

Index usage of python libraries by search code cells for "from ... import" and "import ... " expressions. Display as "python_library_index" sorted by library, sublibrary with links to each instance of usage. Add link to the index file in index.html.

can nbpages.cfg be hard coded?

Can we hard code the name 'nbpages.cfg'? Currently there is a --config option to allow the possibility of an alternate or multiple config file. But would that ever be necessary? For the case of multiple destinations, nbpages could always be run from the destination directory with a unique nbpages.cfg.

Automatically run nbpages from GitHub

We are using nbpages for this class: https://github.com/ndcbe/cbe67701-uncertainty-quantification/

Students are each going to contribute an example notebook (I gave each one a template) via a pull request. Is there a way I can trigger nbpages to run automatically from GitHub on the master branch after a pull request? This would be convenient but is not critical.

https://help.github.com/en/actions/reference/events-that-trigger-workflows

incorrect parsing of import statements

The python index doesn't match and parse comma separated lists of libraries from the import statement.

Test nbpages using an organization repo

All of the testing so far has been with a repo owned by an individual user.

read_config.py is redundant

read_config.py was originally used to read the nbpages config. That functionality has been moved into main.py and nbcollection.py. Need to remove file, then test.

Reduce verbosity of --publish

remove excessive status messaging during --publish

Data index for the table of contents

Challenge: no clear format for data

Error when running nbpages for a few stub notebooks

(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages 
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
Traceback (most recent call last):
  File "/anaconda3/bin/nbpages", line 11, in <module>
    load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 92, in main
    notebooks.insert_headers()
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 365, in insert_headers
    self.nbheader.insert(nb)
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 307, in insert
    print('- inserting header for {0}'.format(nb.filenapatth_srme))
AttributeError: 'Chapter' object has no attribute 'filenapatth_srme'

I will commit the offending notebooks in https://github.com/ndcbe/cbe67701-uncertainty-quantification in a few minutes.

Complete lack of testing.

Need testing classes.

Implement "easier"/"poor man's" testing

Create three repositories owned by ndcbe:

nbpages-testing (completely hosted in 1 repo)
nbpages-testing-dev (private, publishes to nbpages-testing-public)
nbpages-testing-public (public)

The repos would contain just a few example notebooks and a script to completely reset /docs.

enhancements for lint

check and report dead links
check and report orphaned figure files
check and report orphaned data files

Add a few lines of context for each appearance for a search result

Or perhaps print the section and subsection name.

Possible bug in nbpages.tpl

Below is output after adopting the changes in PR #24:

(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages 
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting header for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting header for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- writing table of contents file
[NbConvertApp] Converting notebook docs/toc.ipynb to html
[NbConvertApp] Writing 274340 bytes to docs/toc.html
- writing tag index file
[NbConvertApp] Converting notebook docs/tag_index.ipynb to html
[NbConvertApp] Writing 273413 bytes to docs/tag_index.html
- writing index.md
Traceback (most recent call last):
  File "/anaconda3/bin/nbpages", line 11, in <module>
    load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 98, in main
    notebooks.write_html(NOTEBOOK_DST_DIR, os.path.join("templates", 'nbpages.tpl'))
  File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 464, in write_html
    (body, resources) = html_exporter.from_notebook_node(nb.content)
  File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/html.py", line 96, in from_notebook_node
    output, resources = super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
  File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 315, in from_notebook_node
    output = self.template.render(nb=nb_copy, resources=resources)
  File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 113, in template
    self._template_cached = self._load_template()
  File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 286, in _load_template
    return self.environment.get_template(template_file)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 405, in load
    return loader.load(environment, name, globals)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 125, in load
    code = environment.compile(source, name, filename)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 591, in compile
    self.handle_exception(exc_info, source_hint=source_hint)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/Users/adowling/DowlingLab/Teaching/cbe67701-uncertainty-quantification/templates/nbpages.tpl", line 15, in template
    {% endblock any
  File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 497, in _parse
    return Parser(self, source, name, encode_filename(filename)).parse()
  File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 901, in parse
    result = nodes.Template(self.subparse(), lineno=1)
  File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 888, in subparse
    self.stream.expect('block_end')
  File "/anaconda3/lib/python3.7/site-packages/jinja2/lexer.py", line 384, in expect
    self.name, self.filename)
jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got 'any'

Here are the contents of templates/nbpages.tpl:

<!-- jinja2 template that extendes `full` to add cell tags in the html rendering of notebooks -->
{% extends 'full.tpl'%}
{% block any_cell %}
{% if cell['metadata'].get('tags', []) %}
    <div style="background-color:white; border:thin solid grey; margin-left:95px; margin-right:6px">
    {% for tag in cell['metadata'].get('tags', []) %}
        &nbsp; <a href="https://{github_user_name}.github.io/{github_repo_name}/tag_index.html#{{ tag }}">{{ tag }}</a>
    {% endfor %}
    </div>
    {{ super() }}
{% else %}
    {{ super() }}
{% endif %}
{% endblock any

readthedocs

Need to complete integration with readthedocs

cleanup before write

The docs directory accumulates unused files after repeated publishing runs. Do a cleanup of

/docs
/docs/figures
/docs/data

before publish. But don't touch _config.yml or any file listed inside _config.yml

Create /data and /figures folders with setup.

This means it will work the same for /notebooks and /docs.

How to handle the case with two repos? It would need to copy only the files that are used.

How should --publish handle /data and /figures

Should there be a destination directory for /data and /figures which, if specified, causes --publish to copy those, too?

Chapter numbers missing, error in nbpages.cfg, TOC links broken

https://ndcbe.github.io/cbe67701-uncertainty-quantification/

Support for two repos

Need to verify support for two repositories:

Repo 1: private, for instructors
Repo 2: public, for everyone

Document the steps for setting up a new repo

Steps including:

Editing templates/index.md.tpl
Editing templates/notebook_header.tpl
Editing templates/nbpages.tpl (advanced users only)
Editing nbpages.cfg (advanced users only)

Lint extension: verify the appreance of data and figures in a notebook

This helps find "orphaned" data and figures. It is also a precursor to data and figure TOC.

Special processing for tag under-construction

Proposal: With a special command-line option, if nbpages finds a header cell with the tag under-construction, it will strip out replace all of the subsequent non-header cells that under it with custom text. We can make the default "This notebook section is under construction". Check again soon."

Example with three markdown cells:

## Some time here

blah A blah A blab A

### Subtle point 1

blah B blah B blah B

### Subtle point 2

blah C blah C blah C

If I tagged the first cell with under-construction, then the contents "blah B blah B, blah B" and "blah C blah C, blah C" would get replaced with "This notebook section is under construction".

Why is this helpful? I want to start migrating old class notebooks but I do not want to make the contents, besides the header structure, public. Some notebooks require a major text or example rewrite. I'd prefer to get everything online and "under construction" then systematically clean notebooks up one section at a time.

put quick-start documentation in index.md template

and clean out REAME.rst for notes to developers.