Test examples and solutions in CI about purescript-book HOT 1 CLOSED

Thinking more about requirements and weighing some trade-offs of the original solutions-as-branch plan versus alternatives. Nothing seems to meet the requirements in a nice way.

Requirements

1. No duplicated code
2. PRs to master trigger test of all solutions
3. Enforceable no-merge policy for PRs with CI failures, even if changes are made to tests/solutions. This means that tests and solutions need to be coupled together in a PR.

Solutions as branch:

• Tracks all solutions-related changes in a separate branch that is easy to compare against master.
• Incompatible with requirement 3, since a PR to master will omit solutions.
  • Best we can do is report whether CI for solutions branch is failing after master is merged, and then make a PR to fix the solutions. Maybe this is good enough.
• CI is more complex with automatic merging across branches.
• Annoying choices to make around branch management strategy, such as whether to go with:
  • Merge (or cherry-pick) new content from master onto solutions.
    • "Messy", as solutions changes are interleaved with book changes.
  • Rebase solutions onto master.
    • "Cleaner", but breaks golden rule of no rebasing public branches.
    • May squash all solutions commits to avoid having to perform conflict resolution for multiple commits, but this further tramples on history.

Solutions as hidden directory:
Create a .solutions/ directory that matches the structure of exercises/. This only contains the files that were edited for creating solutions:

.solutions
├── chapter3
│   └── test
│       ├── Main.purs
│       └── Solutions.purs
├── chapter_
│   ├── src
│   │   └── Foo
│   │       └── SomeOtherFile.purs -- exercises require editing this file
│   └── test
│       ├── Main.purs
│       └── Solutions.purs
...

Contents of exercises/ for reference:

exercises
├── chapter3
│   ├── packages.dhall
│   ├── spago.dhall
│   ├── src
│   │   ├── Data
│   │   │   └── AddressBook.purs
│   │   └── Main.purs
│   └── test
│       ├── Main.purs
│       └── Solutions.purs
├── chapter_
│   ├── packages.dhall
│   ├── spago.dhall
│   ├── src
│   │   ├── Foo
│   │   │   └── SomeOtherFile.purs
│   │   └── Main.purs
│   └── test
│       ├── Main.purs
│       └── Solutions.purs
...

Then use rsync to merge the two directories for testing during CI.
This cuts-down on some file duplication, but there's still lots of duplicated code. There's also no good way to protect against PRs that add new tests, but neglect to update the previously working old tests and solutions tracked in .solutions.

Solutions as tracked patch file
This is similar to the previous .solutions directory option, but it tracks all of the solutions changes as a .patch file to be applied by CI during testing. This is a lot more robust than the directory solution in that it should definitely catch incompatible modifications. It seems perverse to track a patchfile within git, but I believe this meets all requirements.
Difficulties are the following:

• Generating the correct patchfile when updating solutions. Good instructions are required.
• Using the patchfile to view solutions. Maybe it's good for these to be slightly obfuscated. Should provide instructions on how to apply this patch to the reader's local answer-key branch.
• Viewing solutions change history. Will appear as a diff of a diff.

No need to settle on a particular strategy now, and we can pivot from one strategy to another as we learn more about the ergonomics of each.

from purescript-book.