minicomp / wax_tasks Goto Github PK

it's currently hard to read/write tests, because rspec best practices are being followed; need to commit time to fixing

Is your feature request related to a problem? Please describe.
This happened twice to me already in completely different contexts, so I'm guessing it will be a relatively common problem. Terminal spits out a nil value error. We might be able to be more specific about the nature of the error so producers can quickly address.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

currently: warns that there is no record found (which is correct behavior) but then throws

NoMethodError: undefined method pid' for nil:NilClass` when trying to update metadata

Learned the hard and painful way

Is your feature request related to a problem? Please describe.
Putting the layout in the front matter of a wax page is unnecessary since it is required to put in the _config.yml file and Jekyll will auto populate the layout from the _config.yml file. If you change it in the _config.yml file you have to reprocess all your pages file, which is unnecessary extra work for the user.

Describe the solution you'd like
Remove the piece of code that puts the layout in the front matter of every wax page.

Describe alternatives you've considered
There could be a setting that will put it in the front matter, but this should not be the default.

Additional context
Line 15
https://github.com/minicomp/wax/blame/e4dcaa10a28b427cd502ac960b8943952805ed7d/_qatar/obj10.md

Currently Wax works with .tiff but not .tif

^ copied from minicomp/wax#55

BUG: wax:iiif includes source dir specified in _config.yml in path for image resources (e.g., thumbnail path, manifest path, etc.)

MUST FIX and write tests before release

... so that you can stop & start wax:derivatives task and wax will keep track of progress made.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

This would require a decent amount of work to refactor but would be worth it.

Current behavior:

Skips writing derivative (in memory) to disk if target file already exists

Desired behavior:

Should skip creating derivative at all if target file already exists.
This means collections need to know all* of their target derivative paths in advance.
If successful, derivative tasks would take the same amount of time the first time they are run, but would be exponentially quicker when rerun.

Is your feature request related to a problem? Please describe.

When debugging an issue in a site, it's important to be able to start over from a clean slate. 'wax:clobber' generally allows this, but as @mnyrop notes in the issue creating it, #46, it doesn't eliminate columns that have been added to _data/{collection}.csv. So if you want to actually do a clean install, you need to maintain a backup copy of your csv and start copying it into _data.

Describe the solution you'd like
That wax overwrites a user-created file with an altered version makes me squeamish. I don't know Jekyll, but every static system I've used maintains a bright line between user generated files and program-generated ones. Like, you would never script something that adds new lines to _config.yml as part of a build process? I can't see a clearcut case where this would be disastrous, but it's nice, for instance, to have the modification date on the data csv be the date that the user actually edited it, not the last time they ran a wax build that updated it.

So just have a file at like _{collection}/{collection}.csv that wax creates which is the user csv from _data/{collection}.csv but with the columns order,layout,collection,thumbnail,full,manifest, and whatever else you're creating added.

Describe alternatives you've considered

Leave it as is? I don't know what you're thinking of to allow this to get clobbered.
I've occasionally used a solution of having something called catalog_derived.json in the same folder, but using the collection folder seems more robust to me.

Additional context
Add any other context or screenshots about the feature request here.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Describe the bug
When using source: my-src-dir configuration option in Jekyll (see: https://jekyllrb.com/docs/configuration/options/) the derivative tasks will mistakenly include that value as part of the URL path.

Expected behavior
Wax_tasks should take note of the source: directory set and drop it from the image resource paths & URIs

• use openseadradon for simple derivatives (see: minicomp/wax#56)
• derivatives:simple should create an array of images to add to collection page front matter IF the item is paged
• derivatives:simple should use the first image for main thumbnail and full banner 🐛

Edit: These changes in wax_tasks would be best written into a v2.0 that makes use of tmp and state-tracking json files so that the array of paged derivatives & their variants can be captured without writing that info back into a cannonical CSV.

Time to rip the band-aid off! 🩹⏲️

• should remove generated pages (e.g., _qatar) and images (e.g., img/derivatives), and search index (e.g., search/index.json)
• should leave the data (e.g., _data/raw_images and _data/qatar.csv)
• BUT should remove generated columns/fields from metadata file (e.g., manifest from _data/qatar.csv)

See: https://travis-ci.org/minicomp/wax_tasks/jobs/577971694#L621

• Track item types e.g.,
  • existence/list/md5 of source image asset(s)
  • existence/list/md5 of IIIF derivatives
  • existence/list/md5 of image variants
• Keep wax-generated fields namespaced & separate from source metadata file
• Do not .gitignore by default??

Copied from minicomp/wax#54

spec.required_ruby_version = '>= 2.4'

copied from wax #93

Describe the bug

When running the rake wax:search qatar task, rake exists with an error:

rake aborted!
WaxTasks::Error::InvalidConfig:
...

To Reproduce
Steps to reproduce the behavior:

1. Clone the repository
2. Run bundle install
3. Run bundle exec rake wax:search qatar
4. See error

Expected behavior
The /search/index.json file to be generated.

Error Trace Output

$bundle exec rake wax:search qatar --trace
rake aborted!
WaxTasks::Error::InvalidConfig:
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/wax_tasks-1.0.2/lib/wax_tasks/config.rb:54:in `search'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/wax_tasks-1.0.2/lib/wax_tasks/site.rb:40:in `generate_static_search'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/wax_tasks-1.0.2/lib/tasks/search.rake:12:in `block (3 levels) in <top (required)>'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/wax_tasks-1.0.2/lib/tasks/search.rake:12:in `each'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/wax_tasks-1.0.2/lib/tasks/search.rake:12:in `block (2 levels) in <top (required)>'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/Users/<user>/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/Users/<user>/.rvm/gems/ruby-2.6.3/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/Users/<user>/.rvm/gems/ruby-2.6.3/bin/rake:23:in `load'
/Users/<user>/.rvm/gems/ruby-2.6.3/bin/rake:23:in `<main>'
/Users/<user>/.rvm/gems/ruby-2.6.3/bin/ruby_executable_hooks:24:in `eval'
/Users/<user>/.rvm/gems/ruby-2.6.3/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => wax:search

Is your feature request related to a problem? Please describe.
I am trying to hook WAX up with GitHub actions so you can create derivatives via GitHub and don't have to download the repo. The problem I am running into is splitting PDFs doesn't work on GitHub actions using an Ubuntu machine and if you use a Mac to create the derivatives it takes twice as long to create the derivatives. What I would like to do is split the images on a Mac and then create the derivatives on an Ubuntu machine. I thought I would get away with running a ruby script that imported the wax_tasks library and run the script but it looks like all the methods are made private or something because they are not showing up as a method I can run using the library.

Describe the solution you'd like
A rake task to create the image derivatives.

Describe alternatives you've considered
Making the methods public to use when importing the library to ruby.

Additional context
n/a

In the "Requirements" section of the Metadata file description, it notes that the pid:

should follow “snake case”, where letters are lowercase, special characters are removed, and spaces are replaced by underscores

Is the lowercase requirement actually a requirement? Out PIDs and corresponding image files include uppercase characters. We could rename the files... but that turns a seamless production workflow into a slightly more complicated / error-prone one.

Reference:

Wax has constraints on metadata that users might not immediately notice--e.g., that ids cannot include a period or uppercase letters, and that certain fields have special meanings.

If these rules are violated, a site will build but break in unpredictable ways. Would you consider instead throwing an error that explains what needs to happen differently? I suspect this will save users time by helping them identify where things have gone wrong.

Custom variant image sizes specified in _config.yml for a given collection are not being generated.

To Reproduce

• In out-of-the-box wax, add custom variant sizes to a collection in _config.yml, following the example in the sample site
• render the iiif derivatives with bundle exec rake wax:derivatives:iiif

Expected behavior
An image such as obj12_00 ought to have the following full-region images in img/derivatives/iiif/images/obj12_00/full/: 50, 250, 1400, full. Instead, only the default sizes are found: 250, full.

Desktop (please complete the following information):

• OS: Ubuntu 19.10
• Browser: n/a
• Version: n/a

Additional context
This is an important feature to make Wax IIIF images fully reusable, since Universal Viewer always requests 90, thumbnails. An item without 90, thumbnails always looks broken in UV.

Describe the bug
There are are additional slashes between the hostname and the path for IIIF resources in manifests, etc.

To Reproduce
Steps to reproduce the behavior:

1. Check out the repo (a minimal wax exhibition): https://github.com/anarchivist/chandlery.git
2. Build using the following:

bundle exec rake wax:derivatives:iiif
bundle exec rake wax:pagemaster
bundle exec jekyll build

3. Inspect the output

Expected behavior
Something likehttps://example.com/img/derivatives/iiif/MKP_392_0032/manifest.json should appear instead of https://example.com//img/derivatives/iiif/MKP_392_0032/manifest.json

this will make it easier to check, e.g., {% if page.manifest %}, since checking for empty string values is more tedious in liquid than in ruby.

Describe the bug
While generating, the message should indicate that the process is happening, rather than wait for it to finish. This is important for big images.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Is your feature request related to a problem? Please describe.

This is probably quite a difficult thing to do. But it is quite hard to continuously update a wax site right now. If a CSV or exhibit is edited, it's hard to know if those changes are getting pushed through to the final version, so it often involves:

1. Shutting down the jekyll server
2. Running wax:clobber (maybe--because it's not clear from the user end whether, e.g., minor changes to images will be passed through into the manifests or not) which requires a rebuild of all the derivative files.
3. Running wax:pages
4. Running wax:search (although I skip this in iterative development)
5. Restarting Jekyll with a clean rebuild.

This takes 30-45 minutes on our collection.

Describe the solution you'd like

In an ideal world, I want to be able to run bundle exec wax serve which wraps jekyll serve with continuous updating of changed files

Describe alternatives you've considered

1. Clear guarantees in the documentation about which rebuild steps are actually necessary when which files are edited. (Like--under what circumstances is it safe to keep the derivatives from a previous build?)
2. Making my own Makefile to handle the rebuilding, since I'm not a ruby dev and don't really know how to plumb the interior of the rakefiles
3. Fine-tuning the clobber command so that you can delete everything except the IIIF manifests. This alone would be a big help, especially in combination with 1.
4. Porting the Wax themes to Svelte-kit instead of Jekyll to get instant hot-module-reloading based on JSON description in dev, with pretty-much identical static HTML produced only at the time of a publication build and optionally rebuilt client-side in-browser. (I have indeed seriously considered this--not saying that you should, but you asked!)

Additional context
This flurry of issues is being caused by my having borked up my open sea dragon displays somehow, and it being extremely slow to trace the flow of changes in the upstream metadata to the user-facing HTML. Nothing in there is Wax's fault, I'm pretty confident, but it would still be nice for the platform to be debuggable.