jckantor / nbpages Goto Github PK
View Code? Open in Web Editor NEWCommand line tool to maintain a repository of Jupyter notebooks.
Home Page: https://jckantor.github.io/nbpages/
License: MIT License
Command line tool to maintain a repository of Jupyter notebooks.
Home Page: https://jckantor.github.io/nbpages/
License: MIT License
https://ndcbe.github.io/cbe67701-uncertainty-quantification/data_index.html
This is because I both give the name of the text file markdown description and I use it in the code. This is a low priority.
All of the testing so far has been with a repo owned by an individual user.
In my current CBE 20258 notebooks, I use some HTML code to put a colored box around certain markdown cells. For example:
<div style="background-color: rgba(0,255,0,0.05) ; padding: 10px; border: 1px solid darkgreen;">
<b>Home Activity</b>: In the cell below, assign the value of 5 to the variable x. Then type Shift+Enter to execute the line.
</div>
This box is solid dark green which I use for all "home activities". I used a blue box for "class activities" and a red box for "important notes". Would it be possible to add a border and set the background color accordingly for cells with "home-activity", "class-activity", and "important-note" tags when publishing to the HTML pages? Currently, nbpages strips out this formatting, which I think is fine. I would much rather use tags.
This helps screen that all of the notebooks use the same version of Python.
Possible enhancements:
Installation of nbpages:
(teaching) esc306856:nbpages adowling$ pip install -e .
Obtaining file:///Users/adowling/DowlingLab/Teaching/nbpages
Requirement already satisfied: nbconvert in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (5.4.1)
Requirement already satisfied: nbformat in /anaconda3/lib/python3.7/site-packages (from nbpaages==0.0.1.dev12) (4.4.0)
Collecting notedown (from nbpaages==0.0.1.dev12)
Downloading https://files.pythonhosted.org/packages/d3/3a/d7c1817f3acb2e958b32fe85a35f52d270820fb9cebf1364b36c772cf3d0/notedown-1.5.1-py3-none-any.whl
Requirement already satisfied: mistune>=0.8.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.8.4)
Requirement already satisfied: jinja2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.10)
Requirement already satisfied: pygments in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (2.3.1)
Requirement already satisfied: traitlets>=4.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.3.2)
Requirement already satisfied: jupyter_core in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: entrypoints>=0.2.2 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.3)
Requirement already satisfied: bleach in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (3.1.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (1.4.2)
Requirement already satisfied: testpath in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.4.2)
Requirement already satisfied: defusedxml in /anaconda3/lib/python3.7/site-packages (from nbconvert->nbpaages==0.0.1.dev12) (0.5.0)
Requirement already satisfied: ipython-genutils in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (0.2.0)
Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /anaconda3/lib/python3.7/site-packages (from nbformat->nbpaages==0.0.1.dev12) (3.0.1)
Requirement already satisfied: six in /anaconda3/lib/python3.7/site-packages (from notedown->nbpaages==0.0.1.dev12) (1.12.0)
Collecting pandoc-attributes (from notedown->nbpaages==0.0.1.dev12)
Downloading https://files.pythonhosted.org/packages/c2/0a/442cc9237dc997cd88155bdcb54bf86e703e699881f4134ecb733ccd670c/pandoc-attributes-0.1.7.tar.gz
Requirement already satisfied: MarkupSafe>=0.23 in /anaconda3/lib/python3.7/site-packages (from jinja2->nbconvert->nbpaages==0.0.1.dev12) (1.1.1)
Requirement already satisfied: decorator in /anaconda3/lib/python3.7/site-packages (from traitlets>=4.2->nbconvert->nbpaages==0.0.1.dev12) (4.4.0)
Requirement already satisfied: webencodings in /anaconda3/lib/python3.7/site-packages (from bleach->nbconvert->nbpaages==0.0.1.dev12) (0.5.1)
Requirement already satisfied: attrs>=17.4.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (19.1.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (0.14.11)
Requirement already satisfied: setuptools in /anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbpaages==0.0.1.dev12) (40.8.0)
Building wheels for collected packages: pandoc-attributes
Building wheel for pandoc-attributes (setup.py) ... done
Stored in directory: /Users/adowling/Library/Caches/pip/wheels/0f/43/b0/252730521a965a687748f0be111073e67ba229e640d147e65f
Successfully built pandoc-attributes
Installing collected packages: pandoc-attributes, notedown, nbpaages
Running setup.py develop for nbpaages
Successfully installed nbpaages notedown-1.5.1 pandoc-attributes-0.1.7
Attempting to configure my first repo:
(teaching) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages --setup
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 7, in <module>
from .nbcollection import NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 30, in <module>
assert os.path.exists(config_file), f"configuration file {config_file} not found"
AssertionError: configuration file nbpages.cfg not found
This will make it a little safer by requiring one to explicitly say publish.
Indexing is turning out to be one of the most useful features of nbpages. So far there are tag, python module, figure, and data indexes. Additional indexes that might be useful for notebook development are
Currently some of the indexes are cached, which makes it problematic to do rewriting for 'under-construction' tags. (issue #28).
Propose a new index class created with
tag_index = Index()
tag_index.index(pattern)
tag_index.write_html(title)
Need to verify support for two repositories:
This means it will work the same for /notebooks and /docs.
How to handle the case with two repos? It would need to copy only the files that are used.
Search and report 'Traceback' in order to detect faulty code cells.
remove excessive status messaging during --publish
Take a look at this notebook. Open it in Colab.
https://ndcbe.github.io/cbe67701-uncertainty-quantification/01.01-Contributed-Example.html
Running this line:
stock_data = pd.read_csv('./data/Stock_Data.csv')
Gave the following error:
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
<ipython-input-4-4072e866c75c> in <module>()
----> 1 stock_data = pd.read_csv('./data/Stock_Data.csv')
4 frames
/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py in __init__(self, src, **kwds)
1889 kwds["usecols"] = self.usecols
1890
-> 1891 self._reader = parsers.TextReader(src, **kwds)
1892 self.unnamed_cols = self._reader.unnamed_cols
1893
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.__cinit__()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._setup_parser_source()
FileNotFoundError: [Errno 2] File ./data/Stock_Data.csv does not exist: './data/Stock_Data.csv'
Here is the problem: When writing to HTML, we need to point to the text file in GitHub.
@jckantor I would like to share this notebook with the class on Tuesday.
[NBPAGES]
github_repo_url = [email protected]:ndcbe/cbe67701-uncertainty-quantification.git
github_user_name = [email protected]:ndcbe
github_repo_name = cbe67701-uncertainty-quantification
github_pages_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification
repository = [email protected]:ndcbe/cbe67701-uncertainty-quantification
page_title = cbe67701-uncertainty-quantification
page_url = https://[email protected]:ndcbe.github.io/cbe67701-uncertainty-quantification
Bug: Extra prefix in github_pages_url
. Perhaps this is because the repo is in an organization.
Need to complete integration with readthedocs
Can we hard code the name 'nbpages.cfg'? Currently there is a --config option to allow the possibility of an alternate or multiple config file. But would that ever be necessary? For the case of multiple destinations, nbpages could always be run from the destination directory with a unique nbpages.cfg.
Suggestion: Replace figure paths with urls to GitHub when rewriting the notebook
Create three repositories owned by ndcbe:
nbpages-testing
(completely hosted in 1 repo)nbpages-testing-dev
(private, publishes to nbpages-testing-public
)nbpages-testing-public
(public)The repos would contain just a few example notebooks and a script to completely reset /docs
.
The python index doesn't match and parse comma separated lists of libraries from the import statement.
Need to either find nbpages.cfg
or warn it needs to be run for the main directory.
Or perhaps print the section and subsection name.
Challenge: no clear format for data
Steps including:
templates/index.md.tpl
templates/notebook_header.tpl
templates/nbpages.tpl
(advanced users only)nbpages.cfg
(advanced users only)For example, see the links to notebooks here: https://ndcbe.github.io/cbe67701-uncertainty-quantification/
This may be related to issue #13.
Mimic the syntax of Vocareum.
End goal: extend "tags" with chapter and section.
I installed nbpages
from the master branch this morning on a second computer (macOS X).
I am seeing the following error messages:
ModuleNotFoundError: No module named 'jupyter_contrib_nbextensions'
UserWarning: IPython3 lexer unavailable, falling back on Python 3
Are these missing dependencies for nbpages
?
I installed nbpages
by cloning the git repo and running pin install -e .
@jckantor I am trying to set up https://github.com/ndcbe/cbe-xx258 to be the public version of https://github.com/ndcbe/CBExx258-instructor. After creating the public repo, I tried to run nbpages --setup
, but got the following error message:
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
from .nbcollection import Nb, NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 20, in <module>
sys.exit(1)
NameError: name 'sys' is not defined
I then manually made nbpages.cfg
using the example here:
https://github.com/ndcbe/nbpages-testing-public/blob/master/nbpages.cfg
When I run nbpages --setup
, I get the following error message:
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2793, in load_entry_point
return ep.load()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2411, in load
return self.resolve()
File "/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2417, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__init__.py", line 4, in <module>
from .nbcollection import Nb, NbCollection
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 24, in <module>
config = config["nbpages"]
File "/anaconda3/lib/python3.7/configparser.py", line 958, in __getitem__
raise KeyError(key)
KeyError: 'nbpages'
I am using the master branch of nbpages.
Index usage of python libraries by search code cells for "from ... import" and "import ... " expressions. Display as "python_library_index" sorted by library, sublibrary with links to each instance of usage. Add link to the index file in index.html.
The docs directory accumulates unused files after repeated publishing runs. Do a cleanup of
/docs
/docs/figures
/docs/data
before publish. But don't touch _config.yml or any file listed inside _config.yml
(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 92, in main
notebooks.insert_headers()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 365, in insert_headers
self.nbheader.insert(nb)
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 307, in insert
print('- inserting header for {0}'.format(nb.filenapatth_srme))
AttributeError: 'Chapter' object has no attribute 'filenapatth_srme'
I will commit the offending notebooks in https://github.com/ndcbe/cbe67701-uncertainty-quantification in a few minutes.
Should there be a destination directory for /data and /figures which, if specified, causes --publish to copy those, too?
We are using nbpages for this class: https://github.com/ndcbe/cbe67701-uncertainty-quantification/
Students are each going to contribute an example notebook (I gave each one a template) via a pull request. Is there a way I can trigger nbpages to run automatically from GitHub on the master branch after a pull request? This would be convenient but is not critical.
https://help.github.com/en/actions/reference/events-that-trigger-workflows
Perhaps #26 will also fix this. I am opening a separate issue so we remember to test it.
Tags are sorted alphabetically. But the prefix is header section number which needs to be parsed to correct sort. This is a problem if any of the section number components is becomes more than on digit.
This would be the most helpful to a new user.
This will be a bit more complicated
and clean out REAME.rst for notes to developers.
read_config.py was originally used to read the nbpages config. That functionality has been moved into main.py and nbcollection.py. Need to remove file, then test.
Below is output after adopting the changes in PR #24:
(base) esc306856:cbe67701-uncertainty-quantification adowling$ nbpages
- inserting header for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting header for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting header for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 01.00-Introduction-to-Uncertainty-Quantification-and-Predictive-Sciences.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 02.00-Probability-and-Statistics-Preliminaries.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- inserting navbar for 03.00-Input-Parameter-Distributions.ipynb
- writing table of contents file
[NbConvertApp] Converting notebook docs/toc.ipynb to html
[NbConvertApp] Writing 274340 bytes to docs/toc.html
- writing tag index file
[NbConvertApp] Converting notebook docs/tag_index.ipynb to html
[NbConvertApp] Writing 273413 bytes to docs/tag_index.html
- writing index.md
Traceback (most recent call last):
File "/anaconda3/bin/nbpages", line 11, in <module>
load_entry_point('nbpaages', 'console_scripts', 'nbpages')()
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/__main__.py", line 98, in main
notebooks.write_html(NOTEBOOK_DST_DIR, os.path.join("templates", 'nbpages.tpl'))
File "/Users/adowling/DowlingLab/Teaching/nbpages/src/nbpages/nbcollection.py", line 464, in write_html
(body, resources) = html_exporter.from_notebook_node(nb.content)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/html.py", line 96, in from_notebook_node
output, resources = super(HTMLExporter, self).from_notebook_node(nb, resources, **kw)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 315, in from_notebook_node
output = self.template.render(nb=nb_copy, resources=resources)
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 113, in template
self._template_cached = self._load_template()
File "/anaconda3/lib/python3.7/site-packages/nbconvert/exporters/templateexporter.py", line 286, in _load_template
return self.environment.get_template(template_file)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 830, in get_template
return self._load_template(name, self.make_globals(globals))
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 804, in _load_template
template = self.loader.load(self, name, globals)
File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 405, in load
return loader.load(environment, name, globals)
File "/anaconda3/lib/python3.7/site-packages/jinja2/loaders.py", line 125, in load
code = environment.compile(source, name, filename)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 591, in compile
self.handle_exception(exc_info, source_hint=source_hint)
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
reraise(exc_type, exc_value, tb)
File "/anaconda3/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
raise value.with_traceback(tb)
File "/Users/adowling/DowlingLab/Teaching/cbe67701-uncertainty-quantification/templates/nbpages.tpl", line 15, in template
{% endblock any
File "/anaconda3/lib/python3.7/site-packages/jinja2/environment.py", line 497, in _parse
return Parser(self, source, name, encode_filename(filename)).parse()
File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 901, in parse
result = nodes.Template(self.subparse(), lineno=1)
File "/anaconda3/lib/python3.7/site-packages/jinja2/parser.py", line 888, in subparse
self.stream.expect('block_end')
File "/anaconda3/lib/python3.7/site-packages/jinja2/lexer.py", line 384, in expect
self.name, self.filename)
jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got 'any'
Here are the contents of templates/nbpages.tpl
:
<!-- jinja2 template that extendes `full` to add cell tags in the html rendering of notebooks -->
{% extends 'full.tpl'%}
{% block any_cell %}
{% if cell['metadata'].get('tags', []) %}
<div style="background-color:white; border:thin solid grey; margin-left:95px; margin-right:6px">
{% for tag in cell['metadata'].get('tags', []) %}
<a href="https://{github_user_name}.github.io/{github_repo_name}/tag_index.html#{{ tag }}">{{ tag }}</a>
{% endfor %}
</div>
{{ super() }}
{% else %}
{{ super() }}
{% endif %}
{% endblock any
Need testing classes.
Proposal: With a special command-line option, if nbpages
finds a header cell with the tag under-construction
, it will strip out replace all of the subsequent non-header cells that under it with custom text. We can make the default "This notebook section is under construction". Check again soon."
Example with three markdown cells:
## Some time here
blah A blah A blab A
### Subtle point 1
blah B blah B blah B
### Subtle point 2
blah C blah C blah C
If I tagged the first cell with under-construction
, then the contents "blah B blah B, blah B" and "blah C blah C, blah C" would get replaced with "This notebook section is under construction".
Why is this helpful? I want to start migrating old class notebooks but I do not want to make the contents, besides the header structure, public. Some notebooks require a major text or example rewrite. I'd prefer to get everything online and "under construction" then systematically clean notebooks up one section at a time.
This helps find "orphaned" data and figures. It is also a precursor to data and figure TOC.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.