davidleoni / jupman Goto Github PK
View Code? Open in Web Editor NEWA template for online books made with Jupyter notebooks
Home Page: https://jupman.softpython.org
License: Other
A template for online books made with Jupyter notebooks
Home Page: https://jupman.softpython.org
License: Other
Python tutor should work offline and sometimes needs overflowing
Managed to do it this way:
jupman.py
using modulize scriptThere is a new very interesting feature in nbconverter 5.3.0 for filtering cells based on metadata tags I should try when I find the time. You can filter cell input, output and both. More here:
As everything, we really need to make sure it works both in HTML and PDF.
Currently I'm using
> jupyter notebook --version
4.4.1
> nbconvert --version
5.1.1
For jupman projects, I chose to have a directory I named templates
where to store i.e. exam templates.
Looking at conf.py
, I saw a property templates_path=['_templates']
. According to Sphinx docs it should keep html templates, but without understanding it much I had the bad idea to remove the underscore and change it to templates_path=['templates']
. This made notebooks inside templates
automatically excluded during the build :-/
So to avoid confusion we should rename templates
directory to jm-templates
and restore templates_path
to ['_templates']
Note to myself: if I use jm-templates
and link exam tepmlate from Jupman tests notebook, it's fine. But if I include everything by putting jm-templates/**/*
in index.rst
I get this error:
/home/da/Da/prj/jupman/prj/index.rst:24: WARNING: toctree contains reference to document 'jm-templates/exam/theory/README' that doesn't have a title: no link will be generated
/home/da/Da/prj/jupman/prj/jm-templates/exam/exam-yyyy-mm-dd.ipynb:78: WARNING: undefined label: intro.ipynb#commandments (if the link has no caption the label must precede a section header)
/home/da/Da/prj/jupman/prj/past-exams/2000-12-31/exam-2000-12-31.ipynb:78: WARNING: undefined label: intro.ipynb#commandments (if the link has no caption the label must precede a section header)
Process Process-6:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
pipe.send((failed, collector.logs, ret))
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'
Process Process-7:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
pipe.send((failed, collector.logs, ret))
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'
Process Process-8:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/da/.local/lib/python3.5/site-packages/sphinx/util/parallel.py", line 94, in _process
pipe.send((failed, collector.logs, ret))
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'setup.<locals>.<lambda>'
^C
interrupted!
Traceback (most recent call last):
File "./build.py", line 261, in <module>
run_sphinx(manuals, formats)
File "./build.py", line 130, in run_sphinx
res = run(cmd)
File "./build.py", line 99, in run
cwd=cwd
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 695, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.5/subprocess.py", line 1059, in communicate
stdout = self.stdout.read()
KeyboardInterrupt
if first cell contains import jupman but it is not hidden, a warning should be generated
apparently system is not able to resolve correct links from .md files to existing .ipynb files
In particular, nbsphinx stops execution with error:
NotImplementedError: Unknown node: pending_xref
Currently jupman.js and jupman.css are included on ReadTheDocs only when initial cell with jupman.init does not have "nbsphinx": "hidden"
metadata.
To include javascript on RTD we should use standard Sphinx mechanism instead:
http://www.sphinx-doc.org/en/stable/extdev/appapi.html#sphinx.application.Sphinx.add_javascript
I would like ipywidgets to show in generated htmls, tried with softpython project without success.
Apparently requirejs is missing, see this issue:
Apparently parallel builds cause problems:
JupyterLab is certainly the future , but as August 2018 is still v0.33 and NBSphinx does not officially supports it.
Ideally, the processor should recognize solutions from text and transform output accordingly to have hideable panes. Obviously it shouldn't overwrite existing files on user directory.
Currently, there seems to be no standard way to hide cells in Jupyter notebook - with jupyter lab situation seems better.
It would be cool to parse for tests (assert and unittests) and display them in HTML as question "What is this function supposed to output with this input? User should be able to provide an executable answer to check against the expected result. This might also alleviate the technical burden of having to understand asserts in order to solve some exercises
Supposing it is user job on its own computer to run the notebooks before submitting them to readthedocs, when build.py
is launched it could be useful to check if a notebook has unexecuted cells, and it that case fail the build.
Another option would be to automatically run the notebook and save it (we proper messages), but I'm worried this might overwrite files currently open in Jupyter.
Generated PDF is ugly on my laptop, it's also missing the toc for some reason.
Good news is it looks fine on ReadTheDocs
We need to support more patterns and simplify file names.
See also: Restructure folders #33
- stacks.ipynb
- stack_solution.py
- stack_test.py
Generates
- stacks.ipynb
- stack_solution.py
- stack_exercise.py
- stack_test.py
- lists_solution.ipynb
Generates
- lists_solution.ipynb
- lists_exercise.ipynb
(This doesn't actually work, I just use it manually for exams)
- exam_solution.ipynb
- stack_solution.py
- stack_test.py
Generates
- exam_solution.ipynb
- exam_exercise.ipynb
- stack_solution.py
- stack_exercise.py
- stack_test.py
- stacks.ipynb
- stack_solution.py
- stack_test.py
Generates
- stacks.ipynb
- stack.py
- stack_test.py
- stack_sol.py
- lists_sol.ipynb
Generates
- lists_sol.ipynb
- lists.ipynb
- exam_sol.ipynb
- stack_sol.py
- stack_test.py
Generates (pre-publication):
- exam.ipynb
- stack.py
- stack_test.py
Generates (post-publication):
- exam.ipynb
- exam_sol.ipynb
- stack.py
- stack_sol.py
- stack_test.py
... but its pointless
Also, it shouldn't copy generated exercises ...
mmm what a mistery
Relative links in notebooks in zips don't make much sense, as target might be missing from the distribution.
Instead, they should be automatically made to point to the website.
Notebooks are converted to RST, but converted RST files are not saved so it's impossible to trace where errors are, see https://github.com/spatialaudio/nbsphinx/issues/83
Notice this might not be an issue anymore when switch to recommonmark is implemented:
is https://github.com/spatialaudio/nbsphinx/issues/36
should also create a directory like private/2019-02-13-admin/test
for testing exam results
In notebooks they work fine, but in latex I get this error:
! LaTeX Error: Unknown graphics extension: .svg.
See the LaTeX manual or LaTeX Companion for explanation.
Type H <return> for immediate help.
...
l.383 \noindent\sphinxincludegraphics{{cc-by}.svg}
?
! LaTeX Error: Unknown graphics extension: .svg.
See the LaTeX manual or LaTeX Companion for explanation.
Type H <return> for immediate help.
...
l.383 \noindent\sphinxincludegraphics{{cc-by}.svg}
Currently, exercises generated from solutions are stored in a zip, but I would like them to be generated in the same folder as solutions, without write them in source directory (don't want to accidentally overwrite existing files)
Would be nice to allow a container website to embed generated website.
We need to allow some message passing between host and iframe to overcome CORS issues.
Currently using plain pdflatex *.tex
twice, instead, we should use build commands such as readthedocs ones:
python /home/docs/checkouts/readthedocs.org/user_builds/softpython/envs/latest/bin/sphinx-build -b latex -D language=it -d _build/doctrees . _build/latex
cd _builds/latex
latexmk -r latexmkrc -pdf -f -dvi- -ps- -jobname=softpython -interaction=nonstopmode
jupman.py
should be redistributed to users as it is meant to be import in notebooks, but jupman_tools.py
and tests could stay in a proper separate package
It would be appealing to ship student zips with a jupman.py
file containing inside all the required files, If files then need to be physically present on disk, the code could automatically generate them.
See also
After setting a local MathJax (so it doesn't try to fetch it from cdn ), website now display this:
This is infinity: \(\infty\)
In PDF shows correctly.
It's quite easy to forget empty cells, it would be nice to automatically remove them.
Unicode characters such as ✪ don't display in PDFs
To fix it, tried to add this in conf.py
without success:
Taken inspiration from this
#: To fix @ issue, copied from here: https://github.com/sphinx-doc/sphinx/issues/4136#issuecomment-335956163
latex_engine = 'xelatex'
latex_elements = {
'papersize': 'a4paper',
'preamble' : r"""\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
"""
}
I would like to have a sidebar like
- home
- page1
- page2
- Section A
- page3
- page4
- Section B
- page5
- page6
Obviously, it's more complicated then expected.
So far, I used the dirty trick of putting an unholy fake space separator :
.. toctree::
:caption: Index
:maxdepth: 2
:glob:
Home <index.ipynb>
usage.md
jupman-tests.ipynb
|space|
==========
.. |space| unicode:: U+0020 .. space
Part B
============
.. toctree::
:caption: Exercise examples
:maxdepth: 2
:glob:
exercises/python-intro/python-intro.ipynb
Jupyter introduction <exercises/jupyter-intro/jupyter-intro-solution.ipynb>
Part C
============
.. toctree::
:caption: Templates
:maxdepth: 2
:glob:
past-exams.ipynb
exam-project.ipynb
project-ideas.ipynb
jm-templates/project-NAME-SURNAME-ID/markdown.ipynb
jm-templates/project-NAME-SURNAME-ID/requirements.txt
changelog.md
Index
-----------
Now that we have fixed scheme for file names, it would be nice to run tests automatically (in conf.py
)
If code is from exercises without implementation, we should just check the file compiles.
Maybe one day RTD will support OpenGraph, see issue https://github.com/sphinx-doc/sphinx/issues/2645
But if we don't have RTD, we can use this: https://github.com/wpilibsuite/sphinxext-opengraph
By default we use sphinx-rtd-theme, but we can how to customize theme by providing some extra css
Currently we have a huge TOC as home, but I would like a proper page, but it is not so easy (as always...):
eval_rst
index.ipynb
as hometoc.rst
toc-page.rst
as master_doc. Only needed to show the toc in a separate page.Disadvantages:
toc-page.html
, so in toc-page.rst
I put a redirect to index.html
: fixed in 3.3 with jupman.js
#71Index
writing shows at the bottom of the toc instead of the top: fixed in 3.3 with this obscene hackDiscarded, didn't like the redirect
Currently we have this folder structure which gets replicated into zips:
jupman.py
img
- common.png
exercises
- lists
- img
- list-chart.png
- lists_exercise.py
- lists_solution.py
exams
- 2019-10-14
- stack_solution.py
A better alternative would be:
- etc
- jupman.py
- img
- common.png
- lists
- img
- list-chart.png
- lists.py
- lists_sol.py
- lists_test.py
- exams
- 2019-10-14-exam
- stack_sol.py
When zipped, I would like to bring into exercise folders common files, like so:
- lists
- etc
- jupman.py
- img
- common.png
- img
- list-chart.png
- lists_exercise.py
- lists_solution.py
This would require to fix paths in :
![]()
and <img src
format)<a href
)This is weird, files linked from a notebook are not copied when I run sphinx locally, but they copied indeed on ReadTheDocs!
See also nbsphinx documentation
Workaround: put stuff in overlay/_static
like overlay/_static/trial.pdf
, and refer to them like [My trial](_static/trial.pdf)
although this will cause warnings during building, like:
/home/da/Da/prj/jupman/prj/jupman-tests.ipynb: WARNING: file not found: '_static/trial.pdf'
Current custom logging works but is embarassing
On Ubuntu 16.04, jupyterlab-2.1.4 jupyterlab-server-1.1.5 attempting to run Python Tutor I get:
See also Switch to JupyterLab #13
It would be nice to have toggable solution divs for exercises.
NOTE: there are various options, will try to collect some in jupman-tests.ipynb
When there are multiple sections, breadcrumbs always show only the first one, making them useless
As a temporary workaround, we can disable them
Docs » FIRST SECTION» PAGE FROM OTHER SECTION
i18n topic is a bit complex, will start with minimal changes
Default Jupyter PDF export is ugly, adds lots of useless numberings.
There are better Latex templates around, we could use them
Expand icons in the menu in non-current sections are not visible. To expand entries you are forced to click on them which triggers loading of TOC page. This is cumbersome.
Instead, we should always visualize + signs and clicking should just expand the section entry without going to TOC
RTD already provides it, but if we build it on GitHub Action elsewhere we need an extra Sphinx plugin and pass it as a secret
There is an extension sphinxcontrib-googleanalytics but as of July 2020 but doesn't work, see https://github.com/sphinx-contrib/googleanalytics/issues/2
. Resolved by copy-pasting code.
Can't have notebooks in subfolders, as to import jupman module
a. add stuff like sys.path.append('../')
to all files
b. using symlinks to link parent algolab
c. add some new default kernel that includes the paths I want
d. use Jupyter magic like %run ../../lib.py
all solutions which I don't quite like
I tried putting this:
%run -i ../../jupman
jupman_init()
Passing parameters like
--toc
and
--root=../../
Some thoughts:
%run
executes the script, to prevent namespace pollution all functions start with 'jupman_'%run
, last command output is not shown in Jupyter, for that you have to explictly call jupman_init()
function in Jupyter.ipynb
, but discoverd it is not possible to pass parametersFinally adopted this, it's less weird and just works
import sys
sys.path.append('../)
import jupman
jupman.init()
It would be cool to automatically generate exercises from solutions when building .To do so, we need to distinguish two types of exercises.
There can be two kinds of exercises, exercises in python files and exercises in jupyter files.
Since Jupman 0.8, it is possible to automatically generate an exercise from a solution file by stripping text marked with special tags. It is possible to inspect generated files in _build/jupman/
directory
Note: in the zips for the students containing solutions, the tag comments are automatically removed from the solution files as well.
See example: exercises/python-intro/python-intro.ipynb
In this type of exercises, typically you have a jupyter file (like python-intro.ipynb
) that describes the exercise and then the actual exercises are in python files.
If there is a solution file eding in _solution.py
but no file ending in _exercise.py
like this:
python_intro1_solution.py
python_intro1_test.py
then Jupman will try to generate one from a _solution.py
file. To do so, it will look for tags to strip inside the solution file.
If there is already an exercise file like this:
python_intro2_exercise.py
python_intro2_solution.py
python_intro2_test.py
Jupman will just copy the existing file.
See example: exercises/jupyter-intro/jupyter-intro-solution.ipynb
This type of exercises stay in a jupyter notebook itself. In this case, the following applies:
If there is a notebook ending in -solution.ipynb
, the following applies (WARNING: for ipynb
files we use dash -
, not the underscore _
):
the notebook must contain a title as markdown like # bla bla solution
Note text must contain solution
text, which can be customized in conf.py
(you might need to translate it)
the notebook must contain tags to strip
Start tags begin with a #
while end tags begin with a #\
Replaces code inside with an Exception (text is customizable in conf.py
). Be careful to position the comment exactly with the indentation yuoi want the raise to appear. For example:
def add(x,y):
#jupman-raise
return x + y
#/jupman-raise
becomes
def add(x,y):
raise Exception('TODO IMPLEMENT ME !')
Just strips code inside
def f(x):
print(x)
#jupman-strip
def help_func(x,y):
return x - y
#/jupman-strip
def g(y):
return y
becomes
def f(x):
print(x)
def g(y):
return y
This special tag for python code erases whatever is found afterwards the # write solution here
comment
conf.py
w = 5
# write solution here
x = 5 + w
y = 2 + x
becomes
w = 5
# write solution here
Would like to reduce dependency on ReadTheDocs by using Github Actions
Still, to get a full ReadTheDocs-like experience, you can optionally add 'readthedocs_ext.readthedocs'
in conf.py
extensions, see https://github.com/readthedocs/readthedocs-sphinx-ext/issues/27#issuecomment-451181542
Just to remind myself of this oddity
I had to artificially add an extra line at the beginning of shown code in Python tutor otherwise with small input (1 or two lines) was giving strange errors in the browser.
Example exam 2000-12-31 should be created during ReadTheDocs build. Note now it is gitignored.
To do so, it would probably be better convert the bash script create_example_exam.sh
into a Python script, this way it would also become more portable
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.