mnyrop / pagemaster Goto Github PK

View Code? Open in Web Editor NEW

67.0 5.0 17.0 258 KB

Jekyll plugin for generating Markdown collection pages from CSV/YAML/JSON records 🧙📖

License: MIT License

Ruby 100.00%

page-generator markdown jekyll-plugin digital-humanities csv json yaml jekyll-collections

pagemaster's Introduction

pagemaster

jekyll plugin for generating markdown collection pages from .csv, .json, or .yml records

How?

pagemaster takes a specified .yml, .csv, or .json file from your _data folder and 'splits' each item into the front matter of an individually generated markdown page.

Why?

If you have a data set for a Jekyll collection (e.g. a CSV of page titles, image links, dates, tags, and so on), you can completely automate the generation of collection pages by running this plugin with it. And if each page in the collection uses the same custom layout, you can specify that layout in your _config.yml file and generate the look of the pages programatically.

Isn't this what Jekyll collections are for?

Kind of! pagemaster actually uses Jekyll collections, but gives you a lot more control and generates .md pages to root instead of .html pages to _site/.

Will it work with GitHub Pages?

Kind of! Because the pages are generated to root as markdown, you only need to run the plugin locally once. From there GH pages will do what it normally does with Jekyll sites: compile vanilla yaml and markdown to html. So GitHub won't run it, but it shouldn't need to.

To use

Add pagemaster to the :jekyll_plugins group of your Gemfile like so:
```
group :jekyll_plugins do
  gem 'pagemaster'
end
```
Run $ bundle install.
Add pagemaster as a plugin in _config.yml like so:

plugins:
  - pagemaster

Set-up your collection(s) in _config.yml and add pagemaster variables. For example:

collections:
  writers:
    output: true
    source: writer-list.csv
    id_key: id
    layout: writer-profile-page
  scientists:
    output: true
    source: scientist-survey.json
    id_key: orcid
    layout: scientist-profile-page

Run $bundle exec jekyll pagemaster [collection-name], e.g. $bundle exec jekyll pagemaster writers scientists.

Results

For the writers example above, pagemaster will:

look for writer-list.csv in the _data directory,
make a new directory called _writers, and
generate a markdown page for each item in writer-list.csv, named after its id value and using the writer-profile-page.html layout.

For the scientists example above, pagemaster will:

look for scientist-survey.json in the _data directory,
make a new directory _scientists, and
generate a markdown page for each item in scientist-survey.json, named after its orcid value and using the scientist-profile-page.html layout.

+-- _config.yml
+-- _data
|   +-- writer-list.csv
+-- _writers
|   +-- 00001.md
|   +-- 00002.md
|   +-- 00003.md
|   +-- 00004.md
|   +-- 00005.md
|   +-- ...
+-- _layouts
|   +-- default.html
|   +-- writer-profile-page.html
|   +-- scientist-profile-page.html
+-- _scientists
|   +-- 0000-0002-1825-0097.md
|   +-- 0000-0002-1825-0098.md
|   +-- 0000-0002-1825-0099.md
|   +-- 0000-0002-1825-0100.md
|   +-- 0000-0002-1825-0101.md
|   +-- ..

pagemaster's People

Contributors

Stargazers

Watchers

Forkers

cryptowampum sekmet deveah j2l giupea web-work-tools kubicek deepfriedbrain fer-rum whaleen rowe-morehouse jekyllresources cbtelemann ajinkyaghadge jekyllgo yorkei04

pagemaster's Issues

If chinese character can be supported as file name

Dears,

how can I put the csv value into md content part? currently it's all in front matter. could you please advice on it?

Is there a permalink option

Is there a way to control the permalink that gets added to each page? In my use case, I just don't want them to end in html.

use as jekyll command

I like the functionality of this plugin,
just a thought: could be simpler to use implemented as a jekyll command plugin rather than fiddling around with the pm_generate value. Then, every time you want to build/rebuild the collection of .md, you could just say jekyll pagemaster or what ever to run the plugin one time.

thank you.

enable more granular/custom configuration

permalink setting ignored

If I add a permalink field to the settings, it's ignored. Ex)
permalink
blog/title/date

config
blog:
output: true
id_key: title
permalink: permalink_field

output
blog/title.html

Content section

Hi, I trying to use this plugin with examples from sample_site folder.
It works (Jekyll 4.0.0 isn't), I receive this in generated page:

---
pid: dir_imgs_item
gambrel: 'null'
indescribable: Pie. Whoever invented the pie? Here was a great person.
blasphemous: Shunned lurk hideous antiquarian foetid furtive antediluvian. Unmentionable
  accursed gibbous ululate cyclopean. Fainted antiquarian hideous eldritch immemorial
  unnamable dank singular. Ululate daemoniac madness loathsome cat non-euclidean foetid
  swarthy. Noisome manuscript cat nameless gibbous.
furtive: Mathias Buckridge
permalink: "/csv_collection/dir_imgs_item.html"
layout: default
---

How can I fill content section? Of course, I can add content in layout, but I feel its wrong way.

<div class="wrapper">{{ page.blasphemous }}</div>

'jekyll pagemaster' could not be found.

I have the same issue as #17
Running bundle exec jekyll pagemaster monsters results in:

fatal: 'jekyll pagemaster' could not be found. You may need to install the jekyll-pagemaster gem or a related gem to be able to use this subcommand.

my Gemfile:

source "https://rubygems.org"
group :jekyll_plugins do
    gem 'github-pages'
    gem 'jekyll-admin'
    gem 'pagemaster'
end

My _config.yml:

staticman:
  branch: gh-pages
permalink: /:slug-:month:day:short_year
include: ['_pages']

title: Jekyll-db
description:
google_analytics:
show_downloads: true
plugins:
  - pagemaster

collections:
    monsters:
    output: true
    source: monsters.csv
    id_key: id
    layout: entry

I have:

* pagemaster (2.0.2)
	Summary: jekyll pagemaster plugin
	Homepage: https://github.com/mnyrop/pagemaster
	Path: /home/yochai/gems/gems/pagemaster-2.0.2

Bundle info:

Using concurrent-ruby 1.1.7
Using i18n 0.9.5
Using minitest 5.14.1
Using thread_safe 0.3.6
Using tzinfo 1.2.7
Using zeitwerk 2.4.0
Using activesupport 6.0.3.2
Using public_suffix 3.1.1
Using addressable 2.7.0
Using backports 3.18.2
Using bundler 2.1.4
Using coffee-script-source 1.11.1
Using execjs 2.7.0
Using coffee-script 2.4.1
Using colorator 1.1.0
Using ruby-enum 0.8.0
Using commonmarker 0.17.13
Using unf_ext 0.0.7.7
Using unf 0.1.4
Using simpleidn 0.1.1
Using dnsruby 1.61.4
Using eventmachine 1.2.7
Using http_parser.rb 0.6.0
Using em-websocket 0.5.1
Using ffi 1.13.1
Using ethon 0.12.0
Using multipart-post 2.1.1
Using faraday 1.0.1
Using forwardable-extended 2.6.0
Using gemoji 3.0.1
Using sawyer 0.8.2
Using octokit 4.18.0
Using typhoeus 1.4.0
Using github-pages-health-check 1.16.1
Using rb-fsevent 0.10.4
Using rb-inotify 0.10.1
Using sass-listen 4.0.0
Using sass 3.7.4
Using jekyll-sass-converter 1.5.2
Using listen 3.2.1
Using jekyll-watch 2.2.1
Using rexml 3.2.4
Using kramdown 2.3.0
Using liquid 4.0.3
Using mercenary 0.3.6
Using pathutil 0.16.2
Using rouge 3.19.0
Using safe_yaml 1.0.5
Using jekyll 3.9.0
Using jekyll-avatar 0.7.0
Using jekyll-coffeescript 1.1.1
Using jekyll-commonmark 1.3.1
Using jekyll-commonmark-ghpages 0.1.6
Using jekyll-default-layout 0.1.4
Using jekyll-feed 0.13.0
Using jekyll-gist 1.5.0
Using jekyll-github-metadata 2.13.0
Using mini_portile2 2.4.0
Using nokogiri 1.10.10
Using html-pipeline 2.14.0
Using jekyll-mentions 1.5.1
Using jekyll-optional-front-matter 0.3.2
Using jekyll-paginate 1.1.0
Using jekyll-readme-index 0.3.0
Using jekyll-redirect-from 0.15.0
Using jekyll-relative-links 0.6.1
Using rubyzip 2.3.0
Using jekyll-remote-theme 0.4.1
Using jekyll-seo-tag 2.6.1
Using jekyll-sitemap 1.4.0
Using jekyll-swiss 1.0.0
Using jekyll-theme-architect 0.1.1
Using jekyll-theme-cayman 0.1.1
Using jekyll-theme-dinky 0.1.1
Using jekyll-theme-hacker 0.1.1
Using jekyll-theme-leap-day 0.1.1
Using jekyll-theme-merlot 0.1.1
Using jekyll-theme-midnight 0.1.1
Using jekyll-theme-minimal 0.1.1
Using jekyll-theme-modernist 0.1.1
Using jekyll-theme-primer 0.5.4
Using jekyll-theme-slate 0.1.1
Using jekyll-theme-tactile 0.1.1
Using jekyll-theme-time-machine 0.1.1
Using jekyll-titles-from-headings 0.5.3
Using jemoji 0.11.1
Using kramdown-parser-gfm 1.1.0
Using minima 2.5.1
Using unicode-display_width 1.7.0
Using terminal-table 1.8.0
Using github-pages 207
Using rack 1.6.13
Using rack-protection 1.5.5
Using tilt 2.0.10
Using sinatra 1.4.8
Using multi_json 1.15.0
Using rack-test 1.1.0
Using sinatra-contrib 1.4.7
Using jekyll-admin 0.10.2
Using pagemaster 2.0.2

Jekyll does not build after install pagemaster

I installed pagemaster per the instructions in the README and now I cannot build or serve jekyll.
After running bundle install and then bundle exec jekyll serve I get this error:

LoadError: cannot load such file -- /Users/me/Development/project/vendor/bundle/ruby/2.6.0/gems/pagemaster-2.1.0/lib/pagemaster/collection
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/gems/pagemaster-2.1.0/lib/pagemaster.rb:14:in `require_relative'
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/gems/pagemaster-2.1.0/lib/pagemaster.rb:14:in `<top (required)>'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `require'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:74:in `block (2 levels) in require'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `each'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:69:in `block in require'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `each'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler/runtime.rb:58:in `require'
  /Users/me/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0/gems/bundler-2.1.4/lib/bundler.rb:174:in `require'
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/gems/jekyll-4.0.0/lib/jekyll/plugin_manager.rb:53:in `require_from_bundler'
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/gems/jekyll-4.0.0/exe/jekyll:11:in `<top (required)>'
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/bin/jekyll:23:in `load'
  /Users/me/Development/project/vendor/bundle/ruby/2.6.0/bin/jekyll:23:in `<top (required)>'

Here's my gemfile:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

# gem "rails"

gem "jekyll", "~> 4.0"

group :jekyll_plugins do
  gem 'pagemaster'
end

I've followed a few troubleshooting steps related to this error and they all come back to something wrong with pagemaster.

consolidate input and source

merge pm_input into pm_source variable, i.e. pm_source: filename.csv.

Choose output directory

Hello :)

First thanks for making this plugin. I just discovering it after having used datapage_gen and would like to check if it's better, but there are a few things which prevent me from switching to your plugin, so I'm opening this issue (and another one)

I'd like to be able to specify the output directory, because my site has 2 languages and I need to generate files for both english and french layouts, and out put in both the "en" and "fr" directories of my website structure.

As I understand it from the README the output is obviously the same if I use the same data file.

Thanks in advance.

Test + upgrade to use Jekyll 4.0

Conditions for parsing a collection

Could it be possible to apply a condition when parsing a collection ? I have a big data set and would like to avoid splitting it up depending on category.

For example my "tags" collection has a "tagtype" field. I'd like to thus make two sets of files, one for "singers" and "songwriters" for example, but both are in the "tags" collection (only their tagtype differs)

So if I want to output only singers, I'd like to specify that I only want the items which meet the condition (tagtype == singer).

Command bundle exec jekyll pagemaster [collection] not working

I run the following command as stated in the README.
bundle exec jekyll pagemaster cities

And I get the following error.

fatal: 'jekyll pagemaster' could not be found. You may need to install the jekyll-pagemaster gem or a related gem to be able to use this subcommand.

Any thoughts?

Cheers

Question about ID

I see the ID is what's used to determine how to call the .md files created.

Does it work with simple names? For example "John Doe" and such? Some of my data in my collection don't have static uuids which would make them real unique, but their uniqueness is their name. Is the url slugified or something if I provide a string-like id field ? Like "John Doe" becomes "john-doe.md" ?

The reason I'm asking is because I want to be able to create links to the pages generated from other pages, so I would need a "safe" method (slugification for example) to know which URL should I point to from my page to display the "John Doe" page for instance.

Thanks for your answers.

License

I would like to use this for a project but the repo is missing a license? Could you update the repo to contain the MIT license that is mentioned on https://rubygems.org/gems/pagemaster/versions/1.0.0?

add content/body option

jekyll 4.0.0 | Error: Cannot load ...check for typos and rebuild.

Hello,

I have the following in my gemfile and config.yml respectively:

group :jekyll_plugins do
gem 'pagemaster'
end

plugins:

pagemaster

collections_dir: my_collections
collections:
books:
output: true
source: books.yml
id_key: id
layout: page

I have the following file: _data/books.yml

id: 1
title: The Hobbit

Ran $ bundle install

Ran bundle exec jekyll pagemaster books

Then got the following error:

jekyll 4.0.0 | Error: Cannot load books.yml check for typos and rebuild.

I verified that the books.yml was valid yml.

To test further, I used the sample data located here:
pagemaster/spec/sample_site/_data/valid.yml

and replaced what I had with the sample within the file: _data/books.yml

however, I did get the same error:

jekyll 4.0.0 | Error: Cannot load books.yml check for typos and rebuild.

I also tried a separate books.csv and books.json file and got the same error.

I am
Using safe_yaml 1.0.5
Using rainbow 3.0.0
Using pagemaster 2.1.2

Thank you very much for your assistance.

JSON error

This is a weird issue maybe not related to pagemaster itself, but I'm not getting errors when using other plugins or even Jekyll himself to process it.

When I try to run pagemaster on a sample JSON I made, I get an invalid JSON error :

$ bundle exec jekyll pagemaster karas_fr
Processing _data/karas_fr.json....
jekyll 3.8.3 | Error:  Cannot load _data/karas_fr.json. check for typos and rebuild.

Visual Studio Code reports no issues with my JSON file, I'm attaching it to this issue. (renamed to .txt so GitHub accepts it)

Here's my config file :

plugins:
  - pagemaster

collections:
  karas_fr:
    output: true
    source: karas_fr.json
    id_key: kid
    layout: kara_fr

karas_fr.json.txt

I use this file as a collection already (it's in the _data directory) and Jekyll generates a single page as an index with every record in there without issue.

My only lead is that the JSON is a "karas" array and pagemaster won't process it because it expects an array without a name, but I'm not even sure this is valid JSON... And doing so would keep Jekyll from generating an index page. I can access the collection through site.data.karas_fr.karas for example...