At least RNP rnpgp/rnpgp.github.io#8 is an example. Probably flawed logic of fetching & reading software docs.

Switching to submodules (#1) would address this stuff as a side effect, but even though I’m leaning towards that feature we want to address this issue now rather than wait a day for submodule refactor.

We want to automatically release the gem when a new version is tagged.

The Rubygems API token is provided at the secret named RIBOSE_RUBYGEMS_API_KEY.

Originally posted by @ronaldtse in #21 (comment)

Filtering software packages or specs by tag is currently not working as intended.

Hi

Please, I receiving the following error when I tried to use open-project theme

/Users/name/gems/gems/jekyll-theme-open-project-helpers-2.1.0/lib/jekyll-theme-open-project-helpers/filterable_index.rb:137:in get_all_items': Collection does not exist: software (RuntimeError)`

Thanks

Description

When trying to build the open.ribose.com project, an error would occur.

From the source, it seems that project['site'] is returning nil.

Steps to reproduce / Actual results

Here's the trace:

# bundle exec jekyll build --trace
Configuration file: /opt/ribose/src/open.ribose.com/_config.yml
 Theme Config file: /opt/ribose/bundle/gems/jekyll-theme-open-project-1.0.10/_config.yml
            Source: /opt/ribose/src/open.ribose.com
       Destination: /opt/ribose/src/open.ribose.com/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
           Warning: Excerpt modified in _projects/Metanorma/_samples/draft-camelot-holy-grenade.md!
                    Found a Liquid block containing separator ' ' and has been modified with the appropriate closing tag.
                    Feel free to define a custom excerpt or excerpt_separator in the document's Front Matter if the generated excerpt is unsatisfactory.
Invalid theme folder: _data
bundler: failed to load command: jekyll (/opt/ribose/bundle/bin/jekyll)
NoMethodError: undefined method `[]' for nil:NilClass
  /opt/ribose/bundle/gems/jekyll-theme-open-project-helpers-1.0.3/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:83:in `block in fetch_and_read_projects'
  /opt/ribose/bundle/gems/jekyll-theme-open-project-helpers-1.0.3/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:78:in `each'
  /opt/ribose/bundle/gems/jekyll-theme-open-project-helpers-1.0.3/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:78:in `fetch_and_read_projects'
  /opt/ribose/bundle/gems/jekyll-theme-open-project-helpers-1.0.3/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:54:in `read'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/site.rb:164:in `read'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/site.rb:69:in `process'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/command.rb:28:in `process_site'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/commands/build.rb:65:in `build'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/commands/build.rb:36:in `process'
  /opt/ribose/bundle/gems/jekyll-3.8.3/lib/jekyll/commands/build.rb:18:in `block (2 levels) in init_with_program'
  /opt/ribose/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `block in execute'
  /opt/ribose/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `each'
  /opt/ribose/bundle/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `execute'
  /opt/ribose/bundle/gems/mercenary-0.3.6/lib/mercenary/program.rb:42:in `go'
  /opt/ribose/bundle/gems/mercenary-0.3.6/lib/mercenary.rb:19:in `program'
  /opt/ribose/bundle/gems/jekyll-3.8.3/exe/jekyll:15:in `<top (required)>'
  /opt/ribose/bundle/bin/jekyll:29:in `load'
  /opt/ribose/bundle/bin/jekyll:29:in `<top (required)>'

FYR, I bundle update-ed jekyll-theme-open-project-helpers from 1.0.2 to 1.0.3, but the same error occurred.

Expected results

It would build the site without errors.

Site build may fail in strange ways if project data fetched by Jekyll from other repositories is found in an unexpected state.

It’s a bit of an edge case, as it’s never an issue in clean CI environment or during routine development. However, it adds unwelcome confusion during initial bootstrapping and troubleshooting of other errors (such as #5).

The code in question is around https://github.com/riboseinc/jekyll-theme-open-project-helpers/blob/master/lib/jekyll-theme-open-project-helpers/project_data_reader.rb#L136, where it expects either “a clean slate” or everything already set up properly, and may get confused by a third possibility.

Goal

The idea is a trivial addition to/around git_sparse_checkout which would check for existing data consistency & perform a cleanup and re-initialization if needed.

As of now build configuration for software docs resides in <docs root>/navigation.adoc (being currently only the navigation structure, since doc pages are converted from adoc/md into HTML using standard Jekyll means), and build configuration for specs resides in <project site>/_specs/<spec>.adoc’s frontmatter (being the navigation and the engine used to generate HTML).

To reduce mental overhead and implementation complexity, a consistent approach is desirable. I’m tentatively reversing my position and considering moving navigation and (for specs) build configuration from item’s frontmatter in project’s Jekyll site source into item’s repo or subtree in that repo (perhaps a dotfile a-la Travis); having Jekyll build discover that dotfile and take it from there.

This is not of dire priority as it doesn’t affect CI builds and doesn’t really hamper dev flow, but it may be a manifestation of a bug somewhere. If the issue is reproducible, we’ll want to deal with it at some point.

This is an urgent requirement now due to some documentation repositories having their branches renamed as part of GitHub’s initiative.

e.g.
https://github.com/metanorma/metanorma.org/runs/3279098951?check_suite_focus=true

/opt/hostedtoolcache/Ruby/2.7.4/x64/lib/ruby/gems/2.7.0/gems/jekyll-theme-open-project-helpers-2.1.7/lib/jekyll-theme-open-project-helpers/spec_builders/png_diagrams.rb:72:in `block in build_spec_pages': Navigation item matching StandardDoc_Form.png (specs/metanorma-model-standoc) was not found (RuntimeError)
26

This problem has consumed numerous (at least a dozen) hours of @opoudjis and my time in tweaking the source index. We need to ensure:

1. If there is a specified image but it doesn't exist, raise an error but DO NOT CRASH.
2. If there is an image that was not specified but it exists, DO NOT RAISE ANY ERROR. This is expected to not fail.

During development with Jekyll serve, each time a minor change is made after a site was already built once, Jekyll pulls all project data anew, which slows down the build and may impair development workflow.

Skipping data re-fetching completely in case project data is already present should be enough, since this is a problem only during development and CI build happens in pristine environment anyway.

This is used for more than just diagrams, e.g. https://github.com/riboseinc/metanorma-model-gb shows an illustration of document’s cover page.

Scenario

Project uses develop as default github branch, and merges into release for releases.

Problem

jekyll 3.8.6 | Error:  git '--git-dir=<path-to-repo>/deltarcm.org/_projects/DeltaMetrics/.git' '--work-tree=<path-to-repo>/deltarcm.org/_projects/DeltaMetrics' checkout '--force' 'origin/master'  2>&1:error: pathspec 'origin/master' did not match any file(s) known to git.

The repo default branch is hard-coded as "master" (Line 9). As best as I can tell, this variable (DEFAULT_REPO_BRANCH) is not touched until the call in git_shallow_checkout.

Solution

I have addressed this for local build by adding to the projects index.md frontmatter, a git_repo_branch, e.g.

site:
  git_repo_url: "https://github.com/DeltaRCM/DeltaMetrics"
  git_repo_branch: "develop"

which is filled with the default values set on Line 9, if it is not provided, with

unless project['site']['git_repo_branch']
  project['site']['git_repo_branch'] = DEFAULT_REPO_BRANCH
end

Finally, I've modified git_shallow_checkout to take the project['site']['git_repo_branch'] as a 3rd argument, and then use it in repo.xxxx calls.

Thoughts

• I'm not a ruby developer, so my solution may be a bit hacky. I'm happy to submit a PR if you'd like, for feedback, or allow you to implement a fix. let me know 👍
• do you lay out defaults somewhere else? An alternative approach to the project_data_reader module might be to lay out a Hash with default values at the top of the module, and then .merge! this with the hash configured by the project frontmatter for use throughout the OpenProjectReader class. This would be a larger refactoring though, I think.
• related to #6
• love the theme, hope to be able to use it!

Currently, hub site pulls project data, and project site pulls software package (and possibly eventually spec) data by checking out their respective repositories during site build. Keeping project data in submodules could reduce build time and remove some of the build logic.

However, it would mean that extra commits would be necessary to “pull” the necessary data into Jekyll site each time.

1. For hub sites, whenever a child project or a “grandchild” software package changes

2. For project sites, whenever a child software package changes

In addition to the process becoming less automated, I believe it’s not possible to add a subdirectory of a repo as a submodule of another repo, which would be important if we want to preserve the sanity of data layout.

See the original riboseinc/open.ribose.com#11.

Currently when software docs get fetched, the gem looks at the presence of “navigation” (.adoc or .md) and uses it to build the navigation across documentation pages.

What could be used instead is just a regular doc like “index” (entry doc filename can be specified in software YAML in open project site contents), with special frontmatter entry.

This causes missing “Last update” markers on software items on Open Project sites:

(Then we’ll be able to reuse it in the ISO TC theme.)

bundler: failed to load command: jekyll (/Users/tony/.rvm/gems/ruby-2.4.4/bin/jekyll)
NameError: undefined local variable or method `subtrees' for #<Jekyll::OpenProjectHelpers::OpenProjectReader:0x00007fd2a535cc60>                                                                           
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:180:in `block in git_shallow_checkout'                        
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:179:in `open'                                                 
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:179:in `git_shallow_checkout'                                 
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:70:in `fetch_hub_logo'                                        
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:62:in `read'                                                  
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-3.8.3/lib/jekyll/site.rb:164:in `read'

Possibly related: metanorma/metanorma.org#23 (comment).

Known to have caused metanorma/metanorma.org#23 (comment).

bundler: failed to load command: jekyll (/Users/tony/.rvm/gems/ruby-2.4.4/bin/jekyll)
Git::GitExecuteError: git '--git-dir=/Volumes/Ribose/metanorma.com/parent-hub/.git' '--work-tree=/Volumes/Ribose/metanorma.com/parent-hub' checkout '--force' 'origin/master'  2>&1:error: Sparse checkout leaves no entry on working directory
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/git-1.5.0/lib/git/lib.rb:967:in `command'
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/git-1.5.0/lib/git/lib.rb:652:in `checkout'
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/git-1.5.0/lib/git/base.rb:306:in `checkout'
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:191:in `git_shallow_checkout'                                 
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:70:in `fetch_hub_logo'                                        
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-theme-open-project-helpers-1.0.6/lib/jekyll-theme-open-project-helpers/project_data_reader.rb:62:in `read'                                                  
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-3.8.3/lib/jekyll/site.rb:164:in `read'
  /Users/tony/.rvm/gems/ruby-2.4.4/gems/jekyll-3.8.3/lib/jekyll/site.rb:69:in `process'

With new customisation docs for metanorma-iso, causes build error on https://github.com/riboseinc/open.ribose.com/:

…was not properly terminated with regexp: /\}\}/ in /Volumes/Ribose/open.ribose.com/_projects/Metanorma/_software/metanorma-iso/docs/customisation.adoc              
bundler: failed to load command: jekyll (/Users/tony/.rvm/gems/ruby-2.4.4/bin/jekyll)       
Liquid::SyntaxError: Liquid syntax error (line 507): Variable '{{`, `{%`) unless you know what you're doing with them: they are how the pages are populated with metadata.

By invoking Metanorma gem function or executable, we want to build source specs, and down the line let the Jekyll theme show spec contents on open project sites.