doconce / doconce Goto Github PK

Slides rendered using deck.js currently have problems that weren't there before.

To reproduce, use https://github.com/lexnederbragt/doconce_demo/blob/master/slide1 and convert slide1.do.txt to slides using

doconce format html slide1.do.txt --html_output=slide1.deck --pygments_html_style=autumn \
	--keep_pygments_html_bg SLIDE_TYPE=reveal SLIDE_THEME=swiss \
	--skip_inline_comments --execute \
	&& doconce slides_html slide1.deck deck --html_slide_theme=swiss

Compare complied slides using the latest version of DocOnce with the rendered versions slide1.deck.pdf and/orslide1.deck.html in https://github.com/lexnederbragt/doconce_demo/blob/master/slide1, which are OK.

• title slide is misrendered
• slides have text under the code, figures, formulas or tables instead of above
• slide 2 has the text separated from the code blocks

(the In [2]: difference mentioned in #117 (comment) is not part of this issue)

git bisect told me that 0f04d50 was the first commit that broke this. I suspect that changing the headers caused the problem.

Travis CI has changed its pricing model. The free tier is now a one-off 10000 credits, which is good for 1000 minutes of builds. The doconce GitHub organisation is currently half way through its free credits see here.

GitHub Actions is a better alternative, as UiO is en enterprise member and has 50000 monthly pre-paid minutes that can be shared amongst employees. Here's the quickstart

This is a copy of ofenerci's issue in the old repository hplgit/doconce #98

Hi,

I wonder there is a syntax file for vim. I saw that there is already one for Emacs ( http://hplgit.github.io/doconce/doc/pub/quickref/quickref.html ). If somebody uses vim to type doconce, could she post vimrc file here?

Best Regards,
Ozhan

Python 3.7 introduced some changes with the re module for regex and some DocOnce commands fail.

re.sub: Changed in version 3.7: Unknown escapes in repl consisting of '' and an ASCII letter now are errors.

For example, an error is thrown when a regular expression is applied to the references in test/papers.pub because the text contains: url: \url{http://some.where.org}

python -V
    Python 3.8.0
cd test
publish import refs1.bib
publish import refs2.bib
publish import refs3.bib
doconce format sphinx testdoc --examples_as_exercises
   ...
   regex._regex_core.error: incomplete escape \u at position 874 (line 26, column 8)

This implies for example that '\u' should either be replaced by '\\u', or the re module should be replaced by regex with import regex as re.

Even when using regex I am still seeing some problems with '{' followed by ascii letters

python -V
    Python 3.8.0
cd test
doconce format sphinx testdoc --examples_as_exercises
   ...
   ... incomplete escape \u at position ..

# This line in the tests fails because { and } should be scaped:
doconce subst '(newtheorem{example}.*)'  '\g<1>\n\\newtheorem{theorem}{Theorem}[section]' testdoc_bigex.p.tex
# Now it works (import regex as re)
doconce subst '(newtheorem\{example\}.*)' '\g<1>\n\\newtheorem{theorem}{Theorem}[section]' testdoc_bigex.p.tex

This is a feature request: currently, the markdown produced by doconce jupyterbook does not allow the use of citations. As per the documentation of Jupyterbook, the formatting used for markdown output should be adjusted. There seem to be only two changes needed:

• adjust how references are cited

• adjust how the bibliography is built

I did some quick tests, and if these changes are implemented citations, in DocOnce files seem to work properly with doconce jupyterbook, both for do.txt files that result in .md as well as those that result in .ipynb.

@MartinHeroux noticed in discussion #62 that the configuration for Travis travis.yml is outdated. For example, Python 2.7 is probably not meant to be tested anymore.

As @@@CODE directives result in the imported code being placed inside a !bc - !ec block, when --execute is used, these code blocks are also executed, at least in .ipynb and .html outputs. This may be a desired result.

However, I sometimes use @@@CODE to show the content of a .csv file, and then 'code execution' is undesired (it often leads to a Python error, in my case).

Regardless of my use case, perhaps @@@CODE could have a -t equivalent?

When generating a bibliography at the end of a Jupyter Notebook, the placement of the tag <div id="authorYear"></div> messes up the formatting of the references.

To reproduce, use https://github.com/lexnederbragt/doconce_demo/tree/master/reference1, and run

doconce format ipynb reference1.do.txt --ipynb_cite=latex-plain

Current result:

Desired result:

Admonitions (notice, summary, warning, question, block) add a dot after their title when compiled in html format. This is not necessary.

Compile the following with 
doconce format html <filename>
Notice that when compiled to html format, the admonition below will read: 
`Example.
Some text
`
That is, a dot will be added to the title.

!bnotice Example
Some text
!enotice

Test file https://github.com/lexnederbragt/doconce_demo/blob/master/exercise1/exercise1.do.txt has a pycod-t block, which should not be executed with --execute.

Converting to html and ipynb gives correct output (no output if the code block is added).

Converting to pdf seems to ignore the -t and output is added also.

Provide a clear explanation for installation. This includes updating the dependencies, moving packages to PyPI, etc. Also refer to discussion #62.

The MOVIE and FIGURE seem to fail when compiling a document with doconce jupyterbook.

For MOVIE: this can be caused by the use of a relative path to the resource and the use of the --dest option because the jupyter-book is created in a different directory. Maybe this can be programmatically fixed or at least a warning should be shown.

For FIGURE: the figure's src seems to be misformed (no starting slash in the path).

## mybook.do.txt
TITLE: Input file for doconce jupyterbook
AUTHOR: Alessandro Marin
AUTHOR: Alessandro Marin again
TOC: on
DATE: today

======= Chapter 1 =======
Movie with relative path can fail with :
MOVIE: [../../doc/src/manual/mov/wave.webm, width=700 height=400] 1D wave in WebM.

Figure with relative or absolute path fails:
FIGURE: [../../doc/src/manual/fig/wave1D, width=500] label{Ch1:figure}

======= Chapter 2 =======
BTW this link ref{Ch1:figure} works.

Commands:

cd test
jupyter-book create mybook
cd mybook
mkdir content

doconce jupyterbook mybook.do.txt --sep=section --dest=content 
jupyter-book build .

Complete the migration from the original hplgit/doconce repository to this one (doconce/doconce). The reason is that, as I am maintaining and developing DocOnce, I need push access to hplgit/doconce. Unfortunately, that is impossible to set up on hplgit's repository.

There are still links in DocOnce (e.g. in documentation, README) that refer to hplgit. Replace them with doconce/doconce

Eventually replace most of the documentation in hplgit/doconce with a reference/links to doconce/doconce

Keep the issues in hplgit/doconce, but do not allow new ones there

Archive hplgit, [about-archiving-repositories](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-archiving-repositories)

I created the current repository (doconce/doconce) from a fork (doconce/doconce_fork) to make this repository "independent" from hplgit/doconce. Close the issues on doconce/doconce_fork. At some point close the whole doconce/doconce_fork repository.

If a table in a DocOnce file appears above a header, bold or italics formatted text, or a !bquote - !equote block, those elements are misformatted the resulting ipynb. It looks like, in the markdown cell of the ipynb file, there should be an empty line inserted between the </table> tag and the next line.

Minimal workday example: convert https://github.com/lexnederbragt/doconce_demo/blob/master/text1/text1.do.txt to ipynb, check for the ## Figure below the table. In the markdown cell, insert an empty line after the </table> line to see desired output.

Modify text1.do.txt to see the effect for text in bold, italics etc following a table.

Note that this is an ipynb specific issue, it seems not to be a problem for other output formats.

Running tests on a previous version revealed that the urllib library was not reliable. Due to this library the tests were occasionally failing, whereas with the requests library the errors were gone. The requests library in fact is known to perform better, at least in Python 3.
Therefore I recommend to substitute all calls to urllib with analogous calls to requests.

Push signed tags so that they become visible in GitHub's tags page. The aim is to be able to clone the repository using tags.

I have been pushing annotated tags as follows, but they do not show up there, e.g.:

git tag -a 1.5.8 94b6c4d4 -m 'signed tag for version 1.5.8'
git push origin 1.5.8

Unfortunately I cannot create signed tags yet:

gpg --gen-key #generate GPG key
gpg --list-secret-keys --keyid-format LONG
    pub ../<some key>  ..
    uid ..
    ssb ..
git config --global user.signingkey <some key>
git tag -s 1.5.9 -m 'signed tag for version 1.5.9'
    error: gpg failed to sign the data
    error: unable to sign the tag

Hi!

It seems that citations in ipython notebooks does not work when they are inside figure captions.

A minimal working example:

===== Figure =====

See figure ref{PythonDict} for an illustration that explains the python dictionary concept.

FIGURE: [PythonDict.jpg] Data structure concept of a dictionary in python. From cite{PythonDict}. label{PythonDict}

Let us cite the image here as well: from cite{PythonDict}.

===== References =====

BIBFILE: papers.pub

And compiling it with doconce format ipynb reference1.do.txt we get the following notebook:

Which has the wrong referencing in the image caption. If this is already a known issue, just disregard this issue. Thanks for your time!

Attachments:
pub file(renamed to .txt as github does not support .pub extension): papers.txt
Image:

See test/execute.do.txt. When formatting to latex (or to html in an upcoming pull request) with doconce format latex execute.do.txt --execute, any code block gets executed using Python/IPython. Therefore, code such as the following:

Compile this document with

!bc sh-t
doconce format ipynb execute.do.txt --execute
!ec

results in an error:

Compile this document with

\bsh
doconce format ipynb execute.do.txt --execute
\esh
\bpyout
  File "<ipython-input-2-909ae4915a71>", line 1
    doconce format ipynb execute.do.txt --execute
                 ^
SyntaxError: invalid syntax
\epyout

According to the manual in the 'Hidden code blocks' paragraph, the !bc *hid environment (e.g. !bc pyhid) can be used to execute code (e.g. import sys) in a hidden cell. Subsequent cells can use the variables or imports in that cell.

Example:

Code block with pyhid (executed but hidden):
!bc pyhid
import sys
test=1
!ec


Code block using pycod:
!bc pycod
print(test)
!ec

Code block using pycod:
!bc pycod
print(sys.version)
!ec

Issue 1: when compiling the example with doconce format ipynb the first pyhid cell does not get hidden. I am not sure if the cell should take precedence on --execute and still be hidden.

Issue 2: when compiling the example with doconce format ipynb --execute, the 2nd and 3rd blocks fail because the variable and the import are not found

With this in test.do.txt:

===== Testing =====
Here is a line of text.
Here is another line that should appear right after.

This line has an empty line above it and should thus start a new paragraph.

and these commands

doconce format pdflatex test.do.txt --latex_code_style=vrb
pdflatex test.tex

I get this as PDF output:

The empty line does not seem to be respected. In html and ipynb there appears an empty line in the right place. I noticed this because for the book I am working on empty lines are no longer respected by the latest DocOnce, while they were before. However, that older version also does not add a new line on the test example above, so there may be something else going on?

Previously, I could run doconce jupyterbook src/file.do.txt. Now that gives a file not found error :

path/to/doconce.py", line 4263, in write_file
    f = open(out_filename, 'w')
FileNotFoundError: [Errno 2] No such file or directory: '../../$DEST/01_file.md'

A fix is to do

cd src
doconce jupyterbook file.do.txt

Is this intended behaviour or the unintended result of a recent change?

Implemented in the cse branch of https://github.com/CINPLA/doconce is the --execute option for DocOnce format ipynb. A demo can be found by installing doconce as per https://github.com/CINPLA/cse_book#readme and cloning the https://github.com/lexnederbragt/doconce_demo repo.

Example command:

 doconce format ipynb text1.do.txt --execute

Expected result: each notebook cell is executed and results are added as output cells.

It would be great if this functionality could be added to the official doconce repo. Note that this was developed by @dragly and/or @simetenn.

doconce jupyterbook writes a setting in the _toc.yml file that has been deprecated: expand_sections. jupyter-book build complains about this but runs anyway. Remove this setting.

Formatting to html shows many orphaned tags such as <p>, or mistakes such as <h1 id="table_of_contents">Table of contents</h2> (notice the use of h1 and h2). The html renders the page ok because it tolerates many mistakes, but it is not a great practice to leave them.

This is a feature request.

DocOnce sets up its table of contents with links to internal sections using #___sec1, #___sec2 etc. Direct links to these then become https://doconce.github.io/doconce/doc/pub/manual/manual.html#___sec1, with <h2 id="___sec1" class="anchor">Demos and Documentation </h2> in the html.

However, the last years another convention is to use the header text as identifier - in lower case and replacing whitespace with dashes. Example: in https://github.com/doconce/doconce/blob/master/README.md, the header ### How to cite becomes an anchor so it can be linked to directly with https://github.com/doconce/doconce/blob/master/README.md#how-to-cite. The html would then have <h3 id="how-to-cite">How to cite</h3> (GitHub uses a more complex version of this).

An advantage of this method of internal links is that changes in the headers (adding headers in the middle or changing their order) does not change these anchors, while in DocOnce, such changes would result in new ___sec numbers. In other words, the links using header text as anchor names are more stable.

The request is, if this is feasible, to add a flag to DocOnce format html that uses the header text as anchor names for internal links (regardless of whether the do.txt file has TOC: on) and in TOCs.

Requested behaviour is then, with that flag, that

https://doconce.github.io/doconce/doc/pub/manual/manual.html#___sec1

becomes

https://doconce.github.io/doconce/doc/pub/manual/manual.html#demos-and-documentation

<h2 id="___sec1" class="anchor">Demos and Documentation </h2>

becomes

<h2 id="demos-and-documentation" class="anchor">Demos and Documentation </h2>

I do not know how duplicate headers are normally treated but I suspect each next one gets a running number.

Copy of issue 190 in hplgit/doconce.
quickref.html shows under the "Section Types" paragraph that exercises can be surrounded by 7 = characters. When I try this I get a compilation error (not shown).

The documentation is probably wrong/obsolete on this point. In fact, according to manual.html Exercises, Problems, Projects, and Examples are subsections and should be surrounded by 5 = characters. Also book.pdf defines exercises as a subsections.

doconce jupyterbook automatically creates .ipynb and .md files that constitute chapters or sections of a jupyter-book. Links in jupyter-book do not work when their destination is in a different jupyter-book chapter or section which constitute a different html document of their own. The reason is that doconce2format does not "automatically" account for links to a different html document.

## mybook.do.txt
======= Chapter 1 =======

Refer to a label in another chapter: ref{Langtangen_2002}

======= Chapter 2 =======

label here: 
label ref{Langtangen_2002}

Commands:

jupyter-book create mybook
cd mybook
mkdir content

doconce jupyterbook mybook.do.txt --sep=section --dest=mybook/content --dest_toc=mybook --allow_refs_to_external_docs
jupyter-book build .

The commands above create two files: 01_mybook.html and 02_mybook.html. ref{} results in a <a href="#Langtangen_2002"> anchor tag instead of <a href="10_mybook#Langtangen_2002">. A user could probably work around this issue by using generalized references, but they would have to know the output filenames beforehand.

Compiling the code below does not show answers and solutions in the subexercise because the subexercise does not have text, but it shows the subexercise counter "a)". I am not sure subexercises without text would make sense, but the current behavior is confusing. I suggest to show answers and solutions anyway.

===== Exercise: week01 =====
label{ex:week01_ex1}

Some text

!bsubex

!bans
some answer
!eans

!bsol
some solution
!esol
!esubex

Compile with:
$ doconce format html mydoc.do.txt
The html output shows:

Exercise 1: week01

Some text

a) 

doconce format html | latex | ipynb --execute allows execution of code blocks written in python, but other programming languages are not supported. A user expressed their interest in executing code blocks in other programming language, e.g. Julia.

This could be done

1. Using an appropriate Jupyter kernel;
2. By saving the code to a temporary file and running the code as a script, e.g. bash <file_with_code>.sh or Rscript <file_with_code>.R.

Option 1 would allow a user to share the scope between different code blocks (e.g. print out a variable defined in the previous code block). However, these kernels should not be a requirement for running DocOnce, so DocOnce should check if an appropriate kernel is installed. Option 2 could be used as a fallback if no kernel is found.

If it comes out that executing different programming languages creates problems, --execute could be use with a list of programming languages to be executed.

I made revisions to the quiz documentation and was trying to compile the various output documents using the bash.sh file provided. Unfortunately, I was not able to generate the Runestone Interactive book:

~/.../quiz$ python automake_sphinx.py --runestone
copy: fig to sphinx-rs/fig

create RunestoneInteractive directory

running git clone https://github.com/RunestoneInteractive/RunestoneComponents.git
Cloning into 'RunestoneComponents'...
remote: Enumerating objects: 349, done.
remote: Counting objects: 100% (349/349), done.
remote: Compressing objects: 100% (177/177), done.
remote: Total 15845 (delta 229), reused 256 (delta 166), pack-reused 15496
Receiving objects: 100% (15845/15845), 29.44 MiB | 5.34 MiB/s, done.
Resolving deltas: 100% (10831/10831), done.
RunestoneInteractive has recently changed its setup - must abort

The current version of RunestoneInteractive/RunestoneComponents requires Python 3.7, but it is possible to obtain an older version that is Python 3.6 compatible.

The automake_sphinx.py includes the following code, which is run when the --runestone flag is passed:

else:
    # Add directory for RunestoneInteractive book
    use_runestonebooks_style = True  # False: use user-chosen style
    print("""

create RunestoneInteractive directory
""")
    sys.path.insert(0, os.curdir)
    import conf as source_dir_conf  # read data from conf.py

    if not os.path.isdir('RunestoneTools'):
        system('git clone https://github.com/RunestoneInteractive/RunestoneComponents.git')
    os.chdir('RunestoneComponents')
    logging.info('creating RunestoneInteractive directory')

    # Edit conf.py
    # This one does not work anymore: run runestone init instead,
    print('RunestoneInteractive has recently changed its setup - must abort')
    sys.exit(1)
    # it's the file runestone/__main__.py and function init()
    # Need to build a bash script that runs the command and feeds the answers
    # See also https://github.com/RunestoneInteractive/RunestoneComponents
    f = open('conf.py.prototype', 'r');  text = f.read();  f.close()
    text = text.replace('<ENTER YOUR PROJECT NAME HERE>', source_dir_conf.project)
    text = text.replace('<INSERT YOUR PROJECT NAME HERE>', source_dir_conf.project)
    text = text.replace('<ENTER YOUR COPYRIGHT NOTICE HERE>', source_dir_conf.copyright)
    text = text.replace('<INSERT YOUR PROJECT NAME OR OTHER TITLE HERE>', source_dir_conf.project)
    text = text.replace('<INSERT YOUR PROJECT NAME OR OTHER SHORT TITLE HERE>', source_dir_conf.project)
    text = text.replace('html_theme_path = ["_templates"]', 'html_theme_path = ["_templates", "../_themes"]')
    [...]

So the solution appears to be to build a bash script that runs the command and feeds the answers. Unfortunately, I am not clear how to do this. How does one run the init() function found in runestone/__main__.py in a bash script, and how does one feed answers to this function?

At present, there is no conf.py.prototype file to speak of. And from my reading of the init() function of runestone/__main__.py, there is a conf.py file generated, not a conf.py.protoype file.

Copy of issue 189 in hplgit/doconce.
Broken links in documentation:

• In doc/pub/quickref/quickref.html the example document points to a 404 Page.
• In the same section the reference to doc/src/make.sh does not seem to be valid (the file does not exist)
• The link to Mako in the last chapter of main_mako.pdf is broken

Of course these corrections should be applied to the corresponding *do.txt files and can apply to other document formats

In addition I noticed a couple of typos:

• In doc/pub/manual/manual.html the "(e.g.)" should probably be removed: line as MYVAR=mytext (e.g.)
• In the same file Different pieces of Python code in Mako is placed should be .. are placed

A recent change leads to tables being made as wide as the page in ipynb output.

Example from https://github.com/lexnederbragt/doconce_demo/blob/main/pub/text1.ipynb:

Currently, the table in this notebook looks like this:

Before, it looked like this:

The change seems to be caused by this line in the notebook, which currently is this:

<table class="table" border="1">

Before, it was this:

<table border="1">

Can the previous version be restored, at least for

EDIT by @aless80 : The original title was: "Runtime error in pdf and html for pycod -t block"

Converting https://github.com/lexnederbragt/doconce_demo/blob/master/reference1/reference1.do.txt to pdf using

doconce format pdflatex reference1.do.txt --latex_code_style=vrb --execute
pdflatex reference1

gives this in pdf output

Running doconce format html reference1.do.txt --execute gives

The relevant python code in reference1.do.txt is in a pycod -t block. If the code were to be executed (which it shouldn't), it should indeed give a NameError as the variable total is undefined.

The same commands on https://github.com/lexnederbragt/doconce_demo/blob/master/text1/text1.do.txt do not have this problem, probably because the variable total is defined earlier.

Expected behaviour: no error messages following pycod -t blocks in pdf or html output.

Universal design of web content requires an alternative text for images. Could a feature be added to DocOnce to allow this? Not sure how to do it but maybe

FIGURE: [/path/to/file.png, width=500 alt="Picture of a flower"] Caption text label{figure_label}

Alternative text is added to html like this:

<img src="img_girl.jpg" alt="Girl in a jacket" width="500" height="600">

I copied most of the following from issue #123.

There seems to be a problem associated with paths in the --dest and -dist_toc options when the doconce command is run in a different folder than the do.txt file.

Executed in the root of the doconce repo folder:

1. test/testdoc.do.txt
   When the path to dest is in the same directory as where doconce runs, doconce jupyterbook works:

mkdir book
doconce jupyterbook test/testdoc.do.txt --examples_as_exercises --dest=book

These two error out:

doconce jupyterbook test/testdoc.do.txt --examples_as_exercises --dest=$PWD/book
...
FileNotFoundError: [Errno 2] No such file or directory: '../../book/01_testdoc.md'

mkdir ~/Desktop/book
doconce jupyterbook test/testdoc.do.txt --examples_as_exercises --dest=$HOME/Desktop/book 
# same with the full path to the book folder
...
FileNotFoundError: [Errno 2] No such file or directory: '../../../book/01_testdoc.md'

Everything works fine when cd'ing in the input file's directory"

cd test
mkdir book

doconce jupyterbook testdoc.do.txt --examples_as_exercises --dest=book
doconce jupyterbook testdoc.do.txt --examples_as_exercises --dest=$PWD/book
doconce jupyterbook testdoc.do.txt --examples_as_exercises --dest=$HOME/Desktop/book

Note that jupyterbook changes to the input file directory (as doconce format does), and then it corrects the paths passed to --dest and --dest_toc based on the output of the find_file_with_extensions:

# Check if the file exists, then read it in
    dirname, basename, ext, filename = find_file_with_extensions(sys.argv[1], allowed_extensions=['.do.txt'])
    if not filename:
        errwarn('*** error: file %s does not exist' % globals.filename)
        _abort()
    if dirname:
        # cd into the DocOnce file's directory, then fix dest and dest_toc
        os.chdir(dirname)
        errwarn('*** doconce format now works in directory %s' % dirname)
        # fix dest, dest_roc, and finally dirname
        dest = os.path.relpath(dest, start=dirname) + '/'
        if dest.startswith('./'):
            dest = dest[2:]
        dest_toc = os.path.relpath(dest_toc, start=dirname) + '/'
        if dest_toc.startswith('./'):
            dest_toc = dest_toc[2:]
        dirname = ''

I recently updated DocOnce to the latest commit (ea41df3) and encountered a problem. I did a ‘git bisect’ and it seems that 8b2db74 was the commit that caused this.

To replicate:

git clone https://github.com/lexnederbragt/doconce_demo
cd doconce_demo
doconce format html text1/text1.do.txt

Output:

Traceback (most recent call last):
  File "/anaconda3/envs/doconce_delete_me/bin/doconce", line 223, in <module>
    bg_session = main()
  File "/anaconda3/envs/doconce_delete_me/bin/doconce", line 213, in main
    retval = eval(command + '()')
  File "<string>", line 1, in <module>
  File "/anaconda3/envs/doconce_delete_me/bin/doconce", line 177, in format
    bg_session = doconce.doconce.format_driver()
  File "/anaconda3/envs/doconce_delete_me/lib/python3.9/site-packages/doconce/doconce.py", line 5369, in format_driver
    filename_preprocessed = preprocess(globals.filename, format, preprocessor_options)
  File "/anaconda3/envs/doconce_delete_me/lib/python3.9/site-packages/doconce/doconce.py", line 4872, in preprocess
    f = open(filename_in, 'r'); filestr = f.read(); f.close()
FileNotFoundError: [Errno 2] No such file or directory: 'text1.do.txt'

When I do this, it works:

cd text1
doconce format html text1.do.txt

So there seems to be a problem when doconce is asked to compile a file in another folder.

Currently, code executions through --execute is implemented for latex and ipynb. Could it be added to html also? Lower priority would be pandoc. I don't currently see a need for other formats.

Using doconce 1.5.7.
With the attached source files tst.zip

 doconce format html tst encoding=utf-8 

Generates html that displays nicely in a web browser. However, when validating with the W3C tool https://validator.w3.org/
a number of errors are generated.

Errors (131) 
    Start tag seen without seeing a doctype first. Expected <!DOCTYPE html>.
    CSS: ___: Property ___ doesn't exist. (12) · Hide all · Show all
        CSS: box-orient: Property box-orient doesn't exist. (5)
        CSS: box-align: Property box-align doesn't exist. (5)
        CSS: box-flex: Property box-flex doesn't exist. (2)
    The center element is obsolete. Use CSS instead. (6)
    The ___ attribute on the ___ element is obsolete. Use CSS instead. (111) · Hide all · Show all
        The align attribute on the img element is obsolete. Use CSS instead.
        The border attribute on the table element is obsolete. Use CSS instead. (2)
        The align attribute on the th element is obsolete. Use CSS instead. (3)
        The align attribute on the td element is obsolete. Use CSS instead. (105)
    An img element must have an alt attribute, except under certain conditions. For details, consult guidance on providing text alternatives for images.

It may not matter for direct display, but it is likely to compromise the successful generation of epub.

label{} and caption{} do get converted when formatting to latex, but not to ipynb or html (upcoming pull request). For example (I will soon add this to test/execute.do.txt:

======= Plotting =======

This is a cell that should plot and output:

!bc pycod
from pylab import *
x = linspace(0, 10, 100)
plot(x, x*x)
show()
!ec
caption{Caption for the image above}
label{image}

To improve the image quality of plots (see ref{image}) ...

The caption{} is converted to text. The label{} does get processed, but it is not "attached" to the figure.

One reason for this is that conversion of code to these formats is done in three different ways. In latex.py there is a loop on filestr i.e. the DocOnce file parsed up to that point. The code can look for label{} and caption{} in the two lines immediately after any code block, and process them. On the other hand html.py loops on the lines of code, which do not include the subsequent text with any label{} and caption{} commands. This means that fixing this properly probably requires an extensive reworking of the code in html.py and ipynb.py.

The DocOnce code base contains various imports that were added (or always existed) to support running DoOnce on Python 2 and 3.

Examples of such imports include:

from __future__ import print_function
from future import standard_library
standard_library.install_aliases()
from builtins import str
from builtins import range
from __future__ import print_function
from __future__ import absolute_import
from __future__ import division
from past.builtins import execfile
from future import standard_library
standard_library.install_aliases()
from builtins import zip
from builtins import str
from builtins import range
from past.builtins import basestring
from past.utils import old_div

As only Python 3 is now supported by DocOnce, moving forward it would be good to clean up the code base. This involves:

1. Remove import statements
2. If needed, make changes to existing code to be Python 3 compatible

What I have tried

As I was new to this type of transition (Python 2/3 to Python 3 only), I created a dedicated Python venv and removed these imports and made the required changes.

I have run my revised version of the code on some of the examples from doconce/doc/src as well as the major test that is doconce/test/make.sh. All of these run without error and produce appropriate output.

What is there left to do?

This would represent a major shift, moving to Python 3 completely. Also, it touches many parts of the code. Thus, I was wondering how I would approach testing my changes before I make a pull-request.

As mentioned, I have run several of the examples and the test make.sh file. Is thus sufficient?

Currently, a bc pycod block gives the following when converting to markdown output:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.Python}
# python code
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

There are many ~ symbols, while current practice is only three. Can the markdown output be changed to:

~~~{.Python}
# python code
~~~

Similarly, a bc pyout block gives this after conversion to markdown:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# python output
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Could this also be changed to:

~~~
# python output
~~~

The issue
Issue with idx{}: when backticks are used in subentries but not in primary entries, the subentries get nested one level too much.

To Reproduce
Compile manual.do.txt with the following commands:

doconce format latex manual.do.txt --no_mako --latex_font=helvetica --no_ampersand_quote --cite_doconce  --no_abort 
doconce ptex2tex manual envir=ans:nt
pdflatex -shell-escape manual
pdflatex -shell-escape manual
makeindex manual

Current behaviour
Index
doconce commands
     apply_edit_comments, 1
    cvs2table, 1
    remove_inline_comments, 1
mainentry, 1
    mainentry
        bug_with_backticks, 1
    working subentry, 1

Expected behaviour
An index that appears like (dashes should not appear; only used to allow:

Index
doconce commands
     apply_edit_comments, 1
    cvs2table, 1
    remove_inline_comments, 1
mainentry, 1
    bug_with_backticks, 1
    working subentry, 1

Copied from issue #158 in hplgit/doconce filed by lexnederbragt.

Add the new feature mentioned in #43 into the manual (at least I can't find it and I really like/need it!)

With the text below having exercises with labels

!split
========= Week 1 – Example of exercises =========
label{chapter:chapter1}
Some text

===== Exercise: week01 =====
label{ex:week01_ex1}
An exercise. 
#Placing a reference here to chapter:chapter1 would suppress the warning

!bsubex
Some question

!bans
An answer to sub-exercise
!eans

!bsol
A solution to sub-exercise
!esol

!esubex

========= References =========
References: ref{ex:week01_ex1}, ref{chapter:chapter1}

DocOnce shows a warning:

$ doconce format html book.do.txt
...
*** Exercise: week01
    label{ex:week01_ex1}
    could be Problem (no refs beyond the exercise itself)
    found info about 1 exercises

I do not fully understand what the code is meant to check (a comment says: "no refs to labels outside the Exercise"). Maybe the exercise is meant to refer to some external label. In any case I suggest to suppress this warning.

This is a feature request for ipynb output. It would make much sense if sections, those parts that start with with 3-9 =signs around the section header, would start in a new markdown cell in Jupyter Notebooks.

The issue is related to #145, as I found out that the final subexercise of an exercise may end up being in the same markdown cell as the start of the next exercise.

A working example not related to exercises:

===== My first section =====

Some text

=== My first subsection ===

More text

===== My second section =====

Some more text

=== My second subsection ===

Even more text

All parts starting with === or ===== are requested to start in a new Markdown cell.

Minimal example for exercises:

===== Exercise: My first exercise =====

!bsubex
Question without code answer:
!bans
Not really.
!eans
!esubex

!bsubex
Another question without code answer.
!bans
Nothing to answer...
!eans
!esubex

===== Exercise: My second exercise =====

!bsubex
Question 2 without code answer:
!bans
Not really.
!eans
!esubex

!bsubex
Another question 2 without code answer.
!bans
Nothing to answer...
!eans
!esubex

All subexericses should be in a new markdown cell, and My second exercise should start in a new one too so that students can add a new cell below the final subexercise from exercise 1

Is this feasible?

The following lines in the installation script doc/src/manual/install_doconce.sh fail (cloud_sptheme, sphinxjp.themes.solarized, pygments-ipython-console):

pip_install --exists-action i -e hg+https://bitbucket.org/ecollins/cloud_sptheme#egg=cloud_sptheme

Obtaining cloud_sptheme from hg+https://bitbucket.org/ecollins/cloud_sptheme#egg=cloud_sptheme
  Cloning hg https://bitbucket.org/ecollins/cloud_sptheme to /home/amarin/doconce/venv/src/cloud-sptheme
abort: HTTP Error 404: Not Found
Command "hg clone --noupdate -q https://bitbucket.org/ecollins/cloud_sptheme /home/amarin/doconce/venv/src/cloud-sptheme" failed with error code 255 in None

pip install --upgrade --exists-action i -e hg+https://bitbucket.org/miiton/sphinxjp.themes.solarized#egg=sphinxjp.themes.solarized
Obtaining sphinxjp.themes.solarized from hg+https://bitbucket.org/miiton/sphinxjp.themes.solarized#egg=sphinxjp.themes.solarized
  Cloning hg https://bitbucket.org/miiton/sphinxjp.themes.solarized to /home/amarin/doconce/venv/src/sphinxjp.themes.solarized
abort: HTTP Error 404: Not Found
Command "hg clone --noupdate -q https://bitbucket.org/miiton/sphinxjp.themes.solarized /home/amarin/doconce/venv/src/sphinxjp.themes.solarized" failed with error code 255 in None

pip install --upgrade --exists-action i -e git+https://bitbucket.org/doconce/pygments-ipython-console#egg=pygments-ipython-console
Obtaining pygments-ipython-console from git+https://bitbucket.org/doconce/pygments-ipython-console#egg=pygments-ipython-console
  Cloning https://bitbucket.org/doconce/pygments-ipython-console to /home/amarin/doconce/venv/src/pygments-ipython-console
remote: Repository doconce/pygments-ipython-console not found
fatal: repository 'https://bitbucket.org/doconce/pygments-ipython-console/' not found
Command "git clone -q https://bitbucket.org/doconce/pygments-ipython-console /home/amarin/doconce/venv/src/pygments-ipython-console" failed with error code 128 in None

Similar errors takes place in the other installation file doc/src/manual/install_doconce.py.

The dependencies have probably moved, for example to cloud-sptheme.readthedocs.io, but they should be tested

Coming back to #74:

I've been thinking a bit more on this and I'd like to explore enabling pycod-t for other formats than only ipynb. I am uncertain what the logic is behind only enabling it for ipynb. If one uses pycod-t one wants to exclude some, but not all, code blocks from being executed with --execute. It seems not to make sense to me that it still should be executed in pdf or html outputs. If one does not want code execution in html/pdf one can skip --execute.

Would it be much work to put this functionality in a branch so I can test it out? (so , not merge it in yet)

Jupyterbook: "Jupyter Book is an open source project for building beautiful, publication-quality books and documents from computational material.". This project is interesting because it allows users to visualize a book composed of text (Markdown) and interactive notebooks (Jupyter Notebooks).

This enhancement is about allowing DocOnce user to create the files necessary to compile a Jupyterbook from a DocOnce file (.do.txt). Once installed, a template Jupyterbook can be created and built with:

pip install jupter-book --user
jupyter-book create mybook 
jupyter-book build mybook 

The main parts of the book are

1. the book chapters, which are Markdown and Jupyter Notebook files;
2. the _toc.yml file, which lists the content of the book and refers to chapter files;
3. configuration settings and other files such as _config.yml.

Doconce should output the files in 1. and 2. from a DocOnce file, so that the user can subsequently compile the book. The chapters files in point 1. can be extracted from DocOnce sections/subsections/chapters. The TOC file in 2. should list the files in 1. DocOnce should be able to extract a chapter title for each book chapter in 1. from the corresponding DocOnce section/subsection/chapter header, if present.

Note that Jupyterbook has moved to version 0.8, which has a different file structure than previous versions.

Fix the sphinx documentation, which is not formatted properly (eg manual)

This is a feature request for exercises compiled to jupyter notebooks (ipynb format).

When I compile exercises to notebooks for my students, I make two versions, one with the exercise only (using --without_solutions --without_answers). Later I'll make one with the exercises including the solutions/answers, and give that to the students.

The first version, exercises only, is meant for the students to work in, they will add their solutions directly into the notebook. For this, they need to add new code or markdown cells below the exercise text.

Currently, consecutive subexercises in the --without_solutions --without_answers version may end up in one markdown cell, which prevents adding a new cell in between. Interestingly, if either the solution (!bsol - !esol) or answer (!bans - !eans) block has a !bc - !ec block, the next subexercise is put in a new codecell (despite --without_solutions --without_answers being used).

Minimal demo: https://github.com/lexnederbragt/doconce_demo/blob/main/pub/exercise1.ipynb and https://github.com/lexnederbragt/doconce_demo/blob/main/pub/exercise1-solutions.ipynb, both generated from https://github.com/lexnederbragt/doconce_demo/blob/main/src/exercise1.do.txt via https://github.com/lexnederbragt/doconce_demo/blob/main/src/exercise1.commands.sh (note the new file structure of the demo repo). In exercise1.ipynb, 2c should be in its own markdown cell, and not in the same cell as 2b.

I hope this does not require complete rewrites...