autolink_bare_uris and emoji need to be added to the list:

potentially difficult because of working directories, need to better understand limitations and document them

One of the main things that's currently painful in the lesson template is the way it's updated: via pull request.

I've been able to split away the components for updating it into two packages, {sandpaper} for the engine and {varnish} for the styling, but there is still a third, deployment component that are the github actions.

In out current mode of thinking, we are duplicating the actions across the repositories, but that's problematic because it means that they can be modified and we want to avoid doing that because we know that they will need to be updated in the future. This comment outlines the issue quite well: https://github.community/t/being-dry-centralized-workflows/16548

One solution is creating a stand-alone Composite action: https://docs.github.com/en/free-pro-team@latest/actions/creating-actions/creating-a-composite-run-steps-action

The requirement for this is to make sure that these actions do not require an extra secret to be added to the repos (or maybe that's not so bad with the tradeoff of having smaller actions).

The {learnr} package provides interactive lessons with answer-checking mechanisms that run in a shiny environment written as RMarkdown documents rendering to a learner::tutorial document. We might want to consider a method for compiling the episodes into a package serving learnr tutorials from the episodes directory.

Things to consider (based on initial run):

{learnr} is agnostic of {dovetail}, so both will work well together 💯

knitr::knit(tutorial) does not work for learnr documents

knitr::knit("lrrrrn/lrrrrn.Rmd")


processing file: lrrrrn/lrrrrn.Rmd
  |......                                                                                      |   7%
  ordinary text without R code

  |............                                                                                |  13%
label: setup (with options) 
List of 1
 $ include: logi FALSE

Quitting from lines 8-11 (lrrrrn/lrrrrn.Rmd) 
Error: package or namespace load failed for 'learnr':
 .onAttach failed in attachNamespace() for 'learnr', details:
  call: NULL
  error: The shiny_prerendered_chunk function can only be called from within runtime: shiny_prerendered

rmarkdown::render(tutorial) does not work out of the box.

It produces a markdown file that has a yaml header and nothing else. It also inserts a dput_to_string character vector into my global namespace. I am going to assume that there is a mechanism for building this and I just haven't read it (I didn't read the manual very much at all)

At the moment, the folders in the lesson are all organized underneath the episodes folder. This is.... not optimal because it breaks the relationship between the folder structure you see in the repository with the folder structure that you see on the website. Ideally, you would want to have the folder structure reflect the structure of the website menu.

All of this can change, but it would be a good idea to make sure that we can switch between folder structures (heavily related to #19).

At the moment, it would be good to have a similar structure to what we have for the current template with the flexibility and notion that it could change.

I think the idea behind this is that the folders could look like this:

./
|-- CONDUCT.md
|-- SETUP.md
|-- episodes/
|-- extras/
|-- something/ 
|   `-- hidden/
`- LICENSE.md

The issue with this is how you tell {sandpaper} that this is the configuration you want without getting too in the weeds with yaml files. Why, you ask? Because I want to avoid behemoth config files that look like this: https://github.com/grunwaldlab/Population_Genetics_in_R/blob/master/_site.yml

It might be possible to do something like this, but it might be too complex 😬

- episodes
  path: episodes/
  order:
  - introduction.md
  - getting-to-know-vectors.Rmd
- extras
  path: extras/
  order:
  - factors.Rmd
  - classes.Rmd
- hidden
  path: something/hidden/
  order:
  - hidden-extra.md

Add a CRON job that rebuilds the HTML every week with an updated version of https://github.com/carpentries/varnish/

For R-based lessons, the packages will be kept in check via {renv}. To avoid simply pinning the repository to a 2021 version of R for the rest of time, we will have a monthly action that will update the packages used by {renv} and create a pull request with the changes to the {renv} lockfile.

This will trigger the PR mechanism that will give a preview of the markdown diff due to the changes in the packages. If they are acceptable, the maintainer can merge and everything will be updated.

If there are averse changes to the updates, the maintainer can change the content of the RMarkdown sources in that branch so that they are acceptable.

At the moment, we still have a weird case where we have to trick {pkgdown} into building the site by writing markdown files and Rmd files. This is largely fine, except when there are examples of how to write knitr documents, then it tries to render the examples and things go goofy 🙁

I've tried the trick of putting the lessons into the top of the site/ directory, but the problem is that the div tags are screwed up from pandoc somehow 🙁 and truncated early.

Overview

Extract content from the challenge blocks and translate them to code blocks for each episode.

A concept has been applied to Data Carpentry R Ecology lesson: https://github.com/datacarpentry/R-ecology-lesson/blob/main/make_code_handout.R, but it extracts EVERYTHING </Gary Oldman Voice>.

each exercise should be in its own file

Learners have a document that they can fill out as they go along. (see R ecology DC lesson). This also potentially allow new usage as “magic cells” and code slides.

Magic cells

This is a concept in Jupyter notebooks that allows people to load code into a cell using a single command.

Dr. Sarah Brown presented a poster on magic cells during CarpentryCon 2018 she also provided a repository containing examples.

Time of extraction

These should be created automagically on lesson rendering via pandoc lua filters.

At the moment, {sandpaper} may not be clearing out old markdown files unless reset_site() is called. AFAICT, it should be doing this by default

Note: I've removed a test file that does show up on the website, but is not included in the diff for the markdown outputs. I think this is an error in {sandpaper} where invalid outputs are not cleared away.

Originally posted by @ravmakz in zkamvar/testme#6 (comment)

The local workflow for a lesson maintainer or contributor should include:

1. pulling the repository
2. adding content
3. previewing content
4. creating a pull request

At the moment, {sandpaper} automatically creates a schedule in the YAML for the template lesson materials, which creates frustration if you just want to drop in your file-ordered episodes and go.

Things to do:

• use get_source_files() when the schedule is NULL or blank
• don't auto-write the schedule on lesson creation
• add option for create_episode() to add to the schedule and set it to FALSE
• make sure options in the yaml are not over-written by rescheduling (this is an artifact of the way yaml::read_yaml() parses the files)

I'm wondering if build_lesson() should be renamed build_site() so it matches with reset_site()? but I guess it would clash with {pkgdown} and {blogdown} function names? If we call reset_site() reset_lesson() users might be confused about what that would do (reset content of the lesson files?). It seems that have build_* and reset_* use the same suffix would be useful though...

At the moment, it's not quite possible to build pages that are not in the top level of the site because linking to them gets muddled in the template.

Case in point: zkamvar/glowing-chainsaw@bafd045 attempts to build new pages inside of an episode, but runs into problems because:

1. It cannot build to the site because that has not been initialized at the time the page is generated
2. building to a subfolder of episodes does not link the back and forward links properly because it assumes that they are on the same level.

Some formatting issues are to be expected in the early versions. We should have a disclaimer on the rendered version of the site that says we are working on a new version of the template, and include ways to report issues (GitHub + email?)

Especially for people who will not be familiar with R, abstracting common operations by relying on Make commands may make using {sandpaper} easier.

From our discussion in maintainers meeting.

We thought it would be good to have sections for the both data download part of setup and then also the software install part of setup on the setup page.

The PR workflow has some extra features that require some documentation for the maintainers and contributors to understand what to expect:

1. Source of truth is in the main branch of the repository
2. PRs will have an extra comment that shows the output of the code.
3. Describe checks that will be performed on the HTML and the markdown

mostly work I think, but we need to document usage and limitations

At the moment, we are thinking of using {renv} for dependency management, but there is the question of exactly how much we want to be responsible for in terms of supporting languages.

For example, if we are supporting an R lesson, we have the correct solution with {renv}, but if we wanted to support the environment for a python lesson, then it becomes tricker because of the relationship between {renv} and {reticulate} (See rstudio/renv#543 and rstudio/renv#537)

We know that BASH works and we are fairly confident that SQL works, but the question is: how do we ensure that these lessons can have auto-generated output should the lesson maintainer choose to do so?

This is extremely related to #47 and #48.

When I merged #48, there's a bug in there that seems to remove images that don't need to be removed as evidenced here: zkamvar/testme#6 (comment)

Clearly the strategy is fragile and non-specific :(

I introduced set_episodes() and set_instructors() and set_learners() to allow people to configure the order of their episodes in the lesson.... well, apparently, using set_episodes() and then set_learners() forces the yaml list to collapse into a comma separated list 🤦

# Navigation ------------------------------------------------
# The menu bar will be in the following order:
# 
# - Code of Conduct      (CODE_OF_CONDUCT.md)
# - Setup                (Setup.md)
# - Episodes             (episodes/)
# - Learner Resources    (learners/)
# - Instructor Resources (instructors/)
# - Learner Profiles     (profiles/)
# - License              (LICENSE.md)
# 
# Use the following menu items to specify the order of
# individual pages in each dropdown section

# Order of episodes in your lesson 
episodes: 
- 01-starting-with-data.Rmd
... SNIP ...
- 15-supp-loops-in-depth.Rmd


# Information for Learners
learners: setup.md,reference.md

There are several advantages to separate the {pkgdown} templating into a new package:

1. It easier to update the package on users machines
2. It's stand-alone
3. We can come up with an equally silly and relevant name like {varnish} 😜

All the templates available to us are found in: https://github.com/r-lib/pkgdown/tree/master/inst/templates

It looks like they use a combination of pandoc and mustache templating.

This process is facilitated by {pegboard}, but still needs some attention. Once I document this, I will simplify it and include it as a function.

Valid episodes should have the following attributes

• YAML header:
  • title: string
  • teaching: number
  • exercises: number
• Blocks:
  • Objectives
  • Exercises

At the moment, the episodes directory is used for the working directory for knitting. This causes issues where rendered materials will drop in there accidentally. To prevent this, these should be rendered in the site/built/assets directory.

The following patterns give odd results for the current lua filters:

:::::::::::::::: objectives ::::::::::::::::::::

- Explain how to use markdown with the new lesson template
- Demonstrate how to include pieces of code, figures, and nested challenge blocks

::::::::::::::::::::::::::::::::::::::::::::::::

:::::::::::::::: questions :::::::::::::::::::::

- HOW do you write a lesson using RMarkdown and `{sandpaper}`?

::::::::::::::::::::::::::::::::::::::::::::::::

Here's a fun paragraph that should not be included with the previous blocks.

:::::::::::::::: objectives ::::::::::::::::::::

## Things to consider

- Explain how to use markdown with the new lesson template
- Demonstrate how to include pieces of code, figures, and nested challenge blocks

::::::::::::::::::::::::::::::::::::::::::::::::

:::::::::::::::: questions :::::::::::::::::::::

## Questions to ponder

- HOW do you write a lesson using RMarkdown and `{sandpaper}`?

::::::::::::::::::::::::::::::::::::::::::::::::

@tobyhodges pointed out that the output to rendered code chunks still has ## prepended to the output:

This may be fixed by setting the options in {callr}

https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-commands-for-github-actions#setting-a-warning-message

I've found it helpful in pull requests when warnings and errors label the lines where an error is found. This is achieved in GitHub Actions via workflow commands that are strings output to stdout that start with ::.

For example, this warning was generated from this workflow command:

::warning file=episodes/01-introduction.Rmd,line=10,col=5::This is a warning

Greg wilson brought up to make sure that we don't force users to have RStudio if they don't want it. This means that the setup should work on Windows with a text editor like VSCode or Emacs.

One thing that was brought up by @tobyhodges is the concept that people like to be able to tweak the template. At the moment, this is not explicitly possible, but it is possible without too much effort by piggybacking on {pkgdown}'s approach to allowing custom templates a la tidytemplate: https://tidytemplate.tidyverse.org/ (which is exactly what we did with {varnish}).

Another aspect is the fact that some instructors like to tweak the lessons by forking the official lessons. We want to make sure that they don't have to go through any changes to the site if they don't want to.

Both of these aspects can be achieved by introducing a mechanism by which users can lock the versions of sandpaper and varnish needed to build the site.

Problem with the example when the site dir hasn't been populated:

usethis::create_from_github("zkamvar/miniature-octo-waffle", tempdir())
#> ✔ Creating '/tmp/RtmpkspjdR/miniature-octo-waffle/'
#> ✔ Cloning repo from '[email protected]:zkamvar/miniature-octo-waffle.git' into '/tmp/RtmpkspjdR/miniature-octo-waffle'
#> ✔ Setting active project to '/tmp/RtmpkspjdR/miniature-octo-waffle'
#> ✔ Setting active project to '<no active project>'
sandpaper::build_lesson(file.path(tempdir(), 'miniature-octo-waffle'))
#> Error: [ENOENT] Failed to search directory '/tmp/RtmpkspjdR/miniature-octo-waffle/site/built': no such file or directory

^{Created on 2020-08-28 by the reprex package (v0.3.0)}

This GitHub action will be the entry point into many of the lessons. The idea is that the workflow would look like this (EDIT to reflect #34 (comment)):

1. a pull request is created
2. the site is built using sandpaper::build_lesson()
3. the site is deployed from the artifact on netlify or aws
4. the diff of the built/ directory is stored as an artifact for the PR (Note: we will need to have a way for maintainer to re-trigger these builds in case the artifacts get too old and die)
5. the link to the site and the diff for the generated markdown documents are added as a comment.

Once the PR is accepted and merged, the staging branch is deleted and the main build (on push) runs.

Because of the way the setup-r action is run, it is possible for any script running in that action to have access to that environment variable. This would be a problem if someone were to include a script in the repository that gets run with the lessons harvesting the token during any of the processing steps and gains access. It's not impossible to imagine that this would happen we effectively use that strategy in {sandpaper} to create branches and push the site. All it would take would be for one maintainer to merge code in without checking to see if there are malicious scripts being run. There may be mitigations against this, but I'm not aware of any.

One of the bottlenecks is converting markdown to HTML and it might be a good idea to do two things:

1. cache the varnish environment: This allows us to only translate the markdown files that needed to be rebuilt because we know the style has not changed.
2. cache the schedule: Currently, we read in every episode anew when we want to build the schedule. This requires a couple of rounds of IO and costs some time (perhaps more so on disk drives). It would be good to store the schedule in either tabular format or HTML format for later passing to the converter function.

This will describe the CRON jobs that are deployed as GitHub Actions which update {varnish} and {renv}, respectively.

The <aside> tag is a neat feature of HTML5 that allows you to write side-bars in HTML without the use of JavaScript.

We can have contributors create instructor notes that can be hidden by using fenced div tags:

This is some markdown text

::::::::::::::::::::: instructor

This will be hidden, but appear on the side if a button is pressed.

:::::::::::::::::::::::::::::::

It's possible to achieve this via lua filters, which are available since pandoc 2.0. The R Markdown Cookbook describes these in detail: https://bookdown.org/yihui/rmarkdown-cookbook/lua-filters.html. One particular inspiration is the {govdown} package, which has this bit of lua that describes how we can replace a div tag with pretty much anything we want

If I manipulate this for our purposes, it would look like this:

-- Deal with fenced divs
Div = function(el)

  -- look for the instructor tag
  v,i = el.classes:find("instructor")
  -- If the index is not missing
  if i ~= nil then
    -- remove this class from the div tag
    el.classes[i] = nil

    -- create a new pandoc element, add raw HTML, 
    -- and fill it with the content of the div block
    local html
    local res = pandoc.List:new{}

    html = '<aside class="instructor">'

    table.insert(res, pandoc.RawBlock('html', html))

    for _, block in ipairs(el.content) do
      table.insert(res, block)
    end
    table.insert(res, pandoc.RawBlock('html', '</aside>'))

    return res
  end
  return el
end

Reference: https://zkamvar.github.io/stunning-barnacle#links

Contributors should be able to link to different parts of The Carpentries' ecosystem easily. We also want to make sure that we enforce canonical URLs to avoid situations where we have some sites that go to site.url/page/, site.url/page, and site.url/page.html

The use cases for linking are:

1. within episodes (e.g. link to challenge)
2. within lessons (e.g. link to references)
3. across lessons (e.g. link to bash novice)
4. across carpentries (e.g. link to glosario)

Note that canonical URLs can also be enforced via JavaScript: https://www.jpap.org/blog/2017/04/canonical-url-redirects-for-static-sites/ (but won't work if JavaScript is not enabled)

One of the pain points of the carpentries lessons is that they don't easily compile to courses. We should be able to accommodate that

It's been used in the past by carpentries members, so it should be relatively straightforward?

it seems that relying on knitr::include_graphics() has some advantages but what do we do for non-R based lessons. Do we ask them to use Rmd just for this feature until we can evaluate other languages?

related to #58

Summary

This comes from lessons learned from https://github.com/datacarpentry/astronomy-python/, where Allen Downey has prepared a script for converting from iPython notebooks to markdown.

Since we are using markdown as an intermediate to HTML it should be a matter of dropping in an engine that renders and converts a single jupyter notebook to markdown. The script can be called from the {processx} package.

Tasks

• confirm a python script can be run consistently from R via processx
• create a python engine for conversion
• create tests for python conversion
• add python engine to code, acting on *.myst notebook files
• add optional config for users to insert their own engine

Edit: I'm changing this from ipynb based to myst notebook/jupytext based operations because ipynb format is not git-friendly even if github will render them.

Yes... Yes... I know.

I know what you're going to say and you don't have to say it.

I should have gone with RMarkdown sites all along in the first place 😞

At the moment {pkgdown} works, but there are changes coming down the pike that may make it less feasible. Luckily, because we have the staging area, switching these should not be that difficult... right?

Because we are porting the styling into the {varnish} package, the users should be able to install any version of it at will:

sandpaper::update_varnish() # latest version
sandpaper::update_varnish(version = "1.2.3") # specific version 

The varnish version should also be recorded in the config file.

Originally posted by @zkamvar in #1 (comment)

There are different levels for translation for a website:

1. Translation of the menus and the messages (e.g. 404)
2. Translation of the prose.

The former is relatively easy, the latter is quite complicated and involves tradeoffs.

David Pérez-Suárez gave a really good talk about this in his CarpentryCon talk: https://youtu.be/IzRCuk7XX18

His solution was to have a centralized hub to hold translations and work with git submodules to control when those translations would be updated. It's not a trivial issue because there's a balance of effort on the maintainers and the contributors. David had mentioned that the updates would only flag translation files, but the result would remain unchanged until a translator comes along and translates the changes.

To address the translation of the prose within the document, they modified https://github.com/carpentries-i18n/po4gitbook to work with the kramdown tags. Looking elsewhere, it doesn't seem that there's really any other good solution to translating markdown to po and back again.

I believe some of this process can be improved by implementing translation to XML on the backend since that provides a clear structure for lists, paragraphs, etc. The challenge is to match that to a clear grammar for how the po files are to be structured.

There's a good breakdown of the tasks necessary for i18n: https://wiki.mageia.org/en/What_is_i18n,_what_is_l10n#I18N

I looked into how the Hugo Learn theme does i18n (https://learn.netlify.app/en/cont/i18n/), but it appears that they have a basic structure for the message translation for the menus and messages, but the prose is expected to live in separate files in separate repositories.