phockett / quantum-metrology-with-photoelectrons-vol3 Goto Github PK

GH actions currently only configured for basic JupyterBook env, so will fail in cases where notebooks need execution.

Options:

• Only run local build & deploy?
  • Added script for ghp-import in e01e26b (HTML only). Method per https://jupyterbook.org/en/stable/publish/gh-pages.html#option-2-automatically-push-your-build-files-with-ghp-import
• Only run .ipynb builds from cached notebooks on GH.
• Setup full GH action (may be an issue for build complexity?)
• Use Docker container for GH builds?

In testing Nov. 2022:

• Citations generally OK in HTML and PDF outputs.
• unsrt style only working for per-document bibs (not recognised in _config.yml case)?
• unsrt per doc
  • Good for HTML output, and includes back-links.
  • For PDF, get final bib only with mis-numbered (with repeats as per-doc case). Need to use other methods for per-chpt bib if required.
• Final bib with all refs & unsrt works in both cases, no backlinks in HTML though.

For general use, set global bib document which works in both cases (although produces extra header in PDF).

For final outputs, may want to add pre-processing script for per-doc bibs for HTML output (see methods in ePSdata). Add as footer to pages:

% Per document case.

:filter: docname in docnames
:style: unsrt

TODO: decide and test best practice.

• Do inline links appear OK in Latex and PDF builds? Yes, at least with JupyterBook builds.
• Inline links with code formatting? Is this confusing?
• Use full URLs in bibliography only, or drop into text or footnotes too?
• Always use code style for package names?

• Run code online
  • Binder/Colab integration: https://jupyterbook.org/en/stable/interactive/launchbuttons.html
  • Thebe: https://jupyterbook.org/en/stable/interactive/thebe.html
  • See also other platforms, e.g. https://lewtun.github.io/dl4phys/intro.html for Colab, Kaggle, Gradient, Studio Lab examples.
  • Platform overview: https://www.dataschool.io/cloud-services-for-jupyter-notebook/
  • Will potentially have issues with some packages (if not pip/conda or in-notebook installable). May be able to use Docker build...?
• Comments and help
  • Via Jupyter Book, https://jupyterbook.org/en/stable/interactive/comments.html
  • Added help doc/pages with additional links, e.g. package GH issues, Open Science forum etc.
• Zotero bibliography to upload and link

To do, probably first.

• HTML vs. latex/PDF build settings (mainly plotters probably). May also want to use exclude on filenames to allow for different versions of content (although ideally will not need to do this).
• Imports for each notebook? General script as per PEMtk config? (https://github.com/phockett/PEMtk/blob/master/demos/fitting/setup_fit_demo.py)
• Global substitutions? https://jupyterbook.org/content/content-blocks.html#define-substitutions-for-your-whole-book
• Cleanest env config probably to use Docker? Easy to ensure env consistency and tear-down/rebuild/test cycles. See https://github.com/phockett/open-photoionization-docker-stacks

General setup & config notes: https://github.com/phockett/Quantum-Metrology-with-Photoelectrons-Vol3/tree/main/notes/jupyter_book

• Fix Plotly rendering issues in HTML output, probably Sphinx theme as per Read The Docs similar issues, see phockett/ePSproc#27 (comment) for solutions. FIXED IN https://jupyterbook.org/en/stable/interactive/interactive.html#plotly
• Plotly issues with Figure captions and glue, see executablebooks/jupyter-book#1815. Seems to work with Holoviews wrapper OK however. For more tests, see doc-source/part1/theory_101122-Render-debug.html (local tests only so far).

May also expect to see Plotly doesn't render after maths bug, although haven't had an issue with this as yet in HTML or PDF outputs.

Currently renders as flat text block - should convert to proper latex table, not sure why it isn't, might be glue() issue again? (But OK in HTML output.)

• For parameters class, can use plain text or HTML output, e.g. https://github.com/lmfit/lmfit-py/blob/53118ec74784ef9350389a8c0297fbeb07ac94d9/lmfit/parameter.py#L324
  • For current builds, glue() for HTML builds seems to use plain text tabulated output (see also #6) and PDF looks like print(params) plain text dump. Try wrapping with display(data.params._repr_html_()) as a quick fix?
  • See also pretty_print functionality, and printfuncs.py, https://github.com/lmfit/lmfit-py/blob/53118ec74784ef9350389a8c0297fbeb07ac94d9/lmfit/printfuncs.py#L327
  • May need to add LaTex wrapper? Or a way to automate this from HTML?

TODO: test with nbconvert directly, and maybe try a custom Pandoc wrapper (see https://pypi.org/project/pandoc/)?

Final pass for most docs, hopefully, on branch reviewJuly2023, https://github.com/phockett/Quantum-Metrology-with-Photoelectrons-Vol3/tree/reviewJuly2023

31/07/23 Released.

• Release on GH: https://github.com/phockett/Quantum-Metrology-with-Photoelectrons-Vol3/releases/tag/v310723pm
• refBuild_310723pm subdir for current HTML and LaTex/PDF builds (no source files, ~80Mb), also included as archive in release above.
• This build now also on GH pages, https://phockett.github.io/Quantum-Metrology-with-Photoelectrons-Vol3

General todo

• Update text. Versions to new date-stamp as revised, and update TOC to reflect.

  • 58f3773 Adds frame defintion section (needs some work) and alignment section (stub).
  • Theory
    • Swap section ordering? Or add note/figure in intro to this effect.
    • Finish observables by type sections.
    • Promote symmetry subsubsection to subsection (may also fix maths rendering issues in "photoionziation dynamics" parent subsection). UPDATE: did this anyway, but rendering issues also fixed with pkg versions in 15/07/23 builds.
    • Clarify LF/AF coupling term - this currently mixes LF and MF definitions for latter case (now fixed in code, see also MF recon manuscript)?
    • Add some notes on alignment? Currently introduced numerically later. (E.g. alignment tensor section). Should include some plots.
      • NOTE: issues with Matplotlib plots in builds 19/07/23, seem to be related to MolPlot routine triggering errors (Arrow3D issues). BUT OK for <v3.6. pip install matplotlib==3.5.3 is working, now set in Dockerfile_qm3_local_install_from_base.
      • This version OK in Jupyter directly in testing, but failing in book build 21/0723 - still some Matplotlib issue.
      • Ah, looks like issue is Arrow3D class, needs patching for Matplotlib >v3.5, see stackoverflow.com/a/74122407 TODO: add to ePSproc with conditional patching depending on MPL version.
      • Current Docker build to MPL 3.7 after fixing other plotting issues (conda install -c conda-forge firefox geckodriver did this, annoyingly), but so far seems OK, although have WARNINGS OFF.
      • Do now see some issues again, reran sudo pip install matplotlib==3.5.3 and pip install --user matplotlib==3.5.3 in container to fix (hopefully), seemed to have some pip/Conda mismatch issues too.
      • Add 'glue' refs to page for builds.
  • Density matrix page needs a tidy. May move QuTip stuff to info content page or appendix (still giving rendering issues).
    • Build 1458_280723 with QuTip section removed for testing. Didn't fix missing figures, maybe other issues?
    • Captions to debug, check logs but maybe missing glue items?
    • Figures OK after Docker Holoviews/Bokeh export fix per Docker section below.
    • NOTE: didn't retest QuTip output in LaTex builds.
  • Info content page also still to do. Updated in f27e2cb
  • Numerics intro section. Updated 24/07/23, passable - content better, but quite a bit of overlap with Chpt. 2; some parts could move to PARTII?

• Check/test individual notebooks in current Docker build.

  • Fix fig object return in epsproc.sphFromBLMPlot for Plotly case.
  • Fix data.padPlot for basic case, already did this...? Currently issues with missing defaults due to (recent?) changes in code, see broken example here. Should now be fixed (in next build), see phockett/ePSproc#27 (comment)
  • Issue in Density Mat notebook with setup script (%run '../scripts/setup_symmetry_basis_tensors.py') in 19/07/23 builds - but rest of notebook seems OK however. UPDATE: removed from script in 59edb77.
  • Info content notebook now returning all zero BLMs... no idea why
    • Test change in default phaseConventions
    • Test change in default sumDims
    • AH - likely due to addition of dipole filtering in calcs? Need to add per phockett.github.io/Quantum-Metrology-with-Photoelectrons-Vol3/part2/sym-fitting-intro_220423.html#mapping-symmetrized-harmonics-to-fit-parameters. Quick test unsuccessful, will need a more careful revisit.
    • Now fied in 9c60eaf, updated setup script with full symmetry routine and tested info content and density matrix chpts.

• Debug HTML and PDF builds.

  • Density matrix chpt. no figures in PDF builds (as of 11/07/23)? May be related to QuTip issues (see #8 ), although those cells are hidden in PDF build. NOTE: was previously working, and do have static renders in pdf-build dir, although multiple versions, so may just need a clean/non-cached build to fix.
    • Full build (no cache/force) 26/07/23 pm didn't fix. Should debug in single page build.
    • NOW FIXED in builds 30/07/23 after container update as below, figure export was silently failing for HV/Bokeh on this page previously.
    • Retesting QuTip section inclusion... OK
  • MFPAD plots (Plotly) cropping after setting height=800 in plotting routine, but NOT FIXED after setting back to 1200px and clean build. Not sure what is going on there... maybe need bigger and had something cached accidentally previously? Should debug in single page build. NOW FIXED in 27/07/23 builds, back to 1200x800. ALSO SET other figures to None for default height.
    • Updated notebooks in c2125e7, sectioned PAD plotting seems more robust.
    • Also testing separate script with independent figure settings for case studies.
    • Generally working well now, all pages only have 1 row or 2 row layout so can get away with config hack.
  • Added get_ipython().magic('matplotlib inline') to general setup script in 77be80e, hopefully only will do good things. SHOULD ALSO CHECK FIG BUILD CHAIN GENERALLY.
  • In latest Docker container getting RuntimeError: Neither firefox and geckodriver nor a variant of chromium browser and chromedriver are available on system PATH. You can install the former with 'conda install -c conda-forge firefox geckodriver'., in builds AND NOTEBOOK IN LAB, even though installed at container build? Conda env issues, and/or user paths?
    • Fixed with sudo apt-get install --yes libgtk-3-0 libdbus-glib-1-2 in current Docker container (and added to Dockerfile), must have fixed this in old build then forgotten? Solution from bokeh/bokeh#10108
  • Part labelling - need to add explicitly for HTML builds, but this messes up PDF builds currently. (Which adds additional "Part" labels.). For linking to Part II in text can use {ref}Part II chpt:extracting-matrix-elements-overview``.
  • Outstanding issues with data.params (lmfit object) data return in basis sets page.
    • OK in HTML for raw cell output, but no glue() success so far, not sure how to return PD object? Tested a few variants (text and HTML reprs), but no joy. See notebook for more comments, also #6.
    • AH, see also lmfit/lmfit-py#827. NOW added to setup script, should be OK on next build.

• Wrap up case studies.

  • 27/07/23 consolidating general setup script. Note this is from mid-July Docker builds (runs on Fock), and still includes phase convention fix (which may be incorrect...). See phockett/ePSproc#26 (comment)
  • 28/07/23 N2 case study in place. Code a little messy, but otherwise OK.
  • Case study data files maybe >100Mb, breaks Github file size limit. UPDATE: OK as archive, 0d0dd3f
  • TODO: post to Zenodo.
    • Repo linked to Zenodo, https://zenodo.org/record/8284304,
    • Docker image to separate repo, https://zenodo.org/record/8286020,
  • 29/07/23 OCS and C2H4 in place. Code as N2. All OK, except MFPADs for C2H4 case, need to fix remapping/downsampling.
    • Build 29/07/23 includes these for testing/debug.
    • Odd - now get C2H4 MFPADs OK, but not OCS case, maybe plotting bugs?
    • Also N2 case not picking up dataset? No idea why, just grabs an old one since deleted? Issues with buildTmp source dir TBC...
    • Ref runs now working, and saved, for both cases. Manual export to HTML OK. Also note automated figure export to static seems buggy.

• General Docker builds for readers, and push to Docker hub. (see also build chain notes below)

  • Current working version with Jupyter Stacks builds circa May 2023, issues with Jupyter Lab builds in new images (?).
    • Now running tests with v2023-05-15 - but note this is Python 3.10, not Python 3.9.7 as previously!
    • Now running tests with v2023-05-19 - Matplotlib issues in Alignment plotting (v3.7 vs. 3.4 previously).
  • Build 2023-07-22 now deployed to Github. Fixed to Sect. 3.6. NEW ISSUES with Plotly figure rendering however. Due to changes to setup script? Thought this had been undone, but maybe stuck in cached files.
  • Build 2023-07-30 pm now deployed to GH. Still a bit of debugging to do, but generally looking pretty good at a glance.
  • Images to DockerHub (22/08/23 rebuilds), https://hub.docker.com/r/epsproc/quantum-met-vol3

• Tidy/remove/archive drafts

  • Build 2023-07-29_00-19-57_preDraftsCleanUp archived with all drafts.
  • TODO: remove duplicate/unnecessary files from archives, e.g. .ipynb_checkpoints etc. Lazy.
  • Part1 drafts to notes/_drafts/part1
  • Part 2 drafts to notes/_drafts/part2
  • Fitting sections still to finish and tidy.

• Tidy scripts

• Tidy build chain and finalise Docker build files and notes.

  • Note issues with clean PDF builds in current case, seeing data persisting in buildTmp/_latest_build/pdf and buildTmp/_latest_build/html. 30/07/23: manual reset of dir structure for testing.
  • Minor Dockerfile tidy and build test (local only, 22/08/23, see local build notes for details)
  • Added Docker build files and notes, 1b880b6
  • Images to DockerHub (22/08/23 rebuilds), https://hub.docker.com/r/epsproc/quantum-met-vol3
  • Script this for local Python installs, and also for full Docker build.

• Installation guide in main readme, and also at end of "Platform intro" chpt. (Old version here.) UPDATED version 07/07/23, OK aside from Docker details to update.

To do:

• General routine for locating setup-notebook.py script? Still OK to use relative paths here? Now added a note to the Installation section on this. May want to add as an appendix?

• General notes on pkg versions, and link to build versions doc (end of book), also to add. Should release current versions and match to Docker build at completion.

• Numerical implementations/examples

  • Add MF case either in outline or in detail, see pemtk.readthedocs.io/en/latest/fitting/PEMtk_fitting_demo_multi-fit_tests_130621-MFtests_120822-tidy-retest.html and pemtk.readthedocs.io/en/latest/fitting/PEMtk_fitting_backends_demo_010922.html. 24/07/23 now mentioned in passing in Sects. 2.5 and 4.1.3/4.2 (in progress).

Some inconsistencies in default builds...

• \bm not rendering in Firefox, Github builds? (e.g. https://phockett.github.io/Quantum-Metrology-with-Photoelectrons-Vol3/part1/theory_101122.html)
• \boldsymbol not rendering in IceDragon, local builds (either default MathJax, or v3).
• Both seem OK in Firefox, with v3, but \boldsymbol not working in default case. UPDATE: NOW NOT WORKING in either case after more test builds... weird...
• Both seem OK in PDF output (pdflatex) in either case.
• Run with v3 for now and hope for the best... (note all display OK in JupyterLab, except \bm).

For forcing version in _config.py, see https://jupyterbook.org/en/stable/content/math.html#math-and-equations

TODO: clean test page, currently testing in main Theory notebook, might be other issues...?

UPDATE: may actually be issue with source and/or Pandoc conversion, currently failing to case with $\boldsymbol{\mathbf{E}}$, but $\mathbf{E}$ or $\bm{E}$ may be equivalent...?

UPDATE2: things seem OK in maths test page for \boldsymbol (even nested), but not for \bm. May have issue related to executablebooks/jupyter-book#1528 with Plotly? (See also https://jupyterbook.org/en/stable/interactive/interactive.html#plotly) Maybe also issue with extensions or macros, https://docs.mathjax.org/en/latest/input/tex/extensions/boldsymbol.html?

Ignore for now and use standard methods - but should fix!

UPDATE3 21/11/22:

• \boldsymbol currently working on some pages, but not all, so likely issue with some page builds.
• For \bm no Mathjax support, but added work-around to _config.py as per mathjax/MathJax#1219, implemented in 3e0c9a5.

Getting build errors for QuTip objects, specifically matrices.

• Render correctly in notebook.
• In HTML build, renders OK but with extra $
• In Tex, produces warnings like below, although fixed output renders correctly in final PDF (but JupyterBook throws final build errors, note this is with --keep-going set, see https://jupyterbook.org/en/stable/reference/cli.html#cmdoption-jupyter-book-build-keep-going).

! Missing } inserted.
<inserted text> 
                }
l.2540 ...& 1.000\\\end{matrix}\right)$\end{split}
                                                  
? 
! Missing { inserted.
<inserted text> 
                {
l.2540 ...& 1.000\\\end{matrix}\right)$\end{split}
                                                  
? 
! Missing $ inserted.
<inserted text> 
                $
l.2540 ...& 1.000\\\end{matrix}\right)$\end{split}
                                                  
? 
! Extra alignment tab has been changed to \cr.
<recently read> \endtemplate 
                             
l.2540 ...& 1.000\\\end{matrix}\right)$\end{split}

Seems like something similar to use of \protect in figure case (mcmtroffaes/sphinxcontrib-bibtex#276 (comment)) may be required here - something is presumably getting stripped incorrectly during preprocessing. May be to do with mixed nature of object, with text and matrix? In notebook this is clear, but in built output the label text is wrapped too?

Current example from .tex output:

\begin{sphinxVerbatimOutput}

\begin{sphinxuseclass}{cell_output}\begin{equation*}
\begin{split}Quantum object: dims = [[25], [25]], shape = (25, 25), type = oper, isherm = True $ \\ \left(\begin{matrix}1.0 & 0.0 & 0.0 & 0.0 & 0.707 & \cdots & 1.0 & 0.0 & 0.707 & 0.0 & 0.707\\0.0 & 1.000 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 1.0 & 0.0 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.0 & 0.0 & 0.0 & 1.000 & 0.0 & \cdots & 0.0 & 0.0 & 0.0 & 0.0 & 0.0\\0.707 & 0.0 & 0.0 & 0.0 & 1.000 & \cdots & 0.707 & 0.0 & 0.0 & 0.0 & 0.0\\\vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots\\1.0 & 0.0 & 0.0 & 0.0 & 0.707 & \cdots & 1.0 & 0.0 & 0.707 & 0.0 & 0.707\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 1.000 & 0.0 & 1.000 & 0.0\\0.707 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.707 & 0.0 & 1.000 & 0.0 & 1.000\\0.0 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.0 & 1.000 & 0.0 & 1.000 & 0.0\\0.707 & 0.0 & 0.0 & 0.0 & 0.0 & \cdots & 0.707 & 0.0 & 1.000 & 0.0 & 1.000\\\end{matrix}\right)$\end{split}
\end{equation*}
\end{sphinxuseclass}\end{sphinxVerbatimOutput}

Looks like Numba issue.

Currently running
numba 0.54.1 py39h56b8d98_0 conda-forge

Should be able to update or deactivate warnings, see numba/numba#5275

Should be fixed in v>0.56? Not sure why this older version is present in current Docker builds... may need to specify in Dockerfile if something is pulling an older version.