For future optimization it would be interesting to measure the amount of time used on the different part of the processing chain. Time spend on generating tiles, time spend analyzing ALTO etc.

The histogram function in ImageMagic is extremely memory hungry (~1GB of RAM for a 30MP image). Unfortunately it seems that GraphicsMagic is equally needy, but it should be possible to find a generic, fast and low-mem way of generating histograms from greyscale images.

When looking for outliers, the histograms helps a lot. They could be displayed somewhere on the folder overview page when a mouse-over is performed on one of the thumbnails.

Absolute holes (greyscale intensities with count 0) are strong indicators of post-processing. They are easy to find and counting those would make it easy to detect post-processing.

ImageMagic's histogram generator uses an insane amount of memory: 2GB for a 30MP image. A helper bash script has been implemented (analyze.sh) for experimentation but although it uses a lot less RAM, it is very slow.

The sed command called from quack.sh does not work in OS X 10.9 and older versions too I suspect - as OS X sed handles the -i argument (and other arguments) differently to the Linux version.

A quick work around is to use gnu-sed on OS X.

The long term solution is to fix the script to run on both Unix variants.

It seems that the dark percentage is off when more than just grey(0) is counted. This is possibly triggered by generating all parts, then changing the dark fuzzy-factor and re-generating.

The histogram can optionally be calculated based on a sub part of the image (typically one would want that when there are black borders which will otherwise shew the numbers). The printed statistics should also be based on the crop.

With Graphics Magick 1.3.18 2013-03-10 Q8 (default install under Ubuntu 14.04), the overlay files for blown blacks are not made transparent. Running quack.sh results in errors like

gm convert: Invalid background palette index (/home/te/projects/quack/t1/sub/ACE-17950616-0019A.black.png).

Never touch the source folder!

There seems to be a timing issue when an image page is opened in a browser. With Firefox at least, opening a page with a large image and moving the mouse over the image before the tiles are fully loaded in OpenSeadragon, sometimes stops the overlay setup.

Holes in histograms, spike height, burnout percent and OCR recognition rate are candidates for stats where an overview would help a lot.

The greyscale intensities and their counts are shown at the bottom of the pages. Showing and marking missing values makes it easier to judge whether they are a problem.

User @kaarefc reported the error message "gm convert: Invalid
background palette index (/home/kfc/public_html/kfc/PDF_Batch01-3/A-1999-11-08-P-0482.white.png)" when processing images. A quick investigation showed that the transparent pixels of the generated overlay were not transparent but white.

Specifying a crop percentage is essential for generating usable histograms and histogram statistics, when the source image has been rotated. However, this cropping is not shown anywhere, which raises the question as to what part of the image the histogram really represents.

A simple box (maybe shown only on mouse-over on the histogram?) showing the crop area would be one solution.

At any given level of the folder tree, it would be interesting to know the top X darkest and top Y brightest images.

The numbers for the individual images are already there as .grey-files, they just needs to be collected.

Alternative images-links can be derived from the image filename and does not require an ALTO-file. Currently the alternatives resolver is hidden behind a check for ALTO-file existence.

The grep for histogram data is faulty and adds some colormap data if there are less than 256 greyscale values.

This should be fairly trivial to fix with a check for colormap and a subsequent length cut with head.

On the image page, a mouse-over on the histogram shows the (optional) crop box for histogram data. This box is not shown if TextBlocks are disabled.

When an existing Quack-structure is updated, spotting changes would be a lot easier if the last-changed date was displayed along with the sub folder name.

As the presentation images imposes quite an extra load on the browser, takes up storage space and are not always needed or even wanted, they should be optional.

The first time Quack is executed, it exits with an error asking the user to download the OpenSeadragon JavaScript. This should be fetched automatically, if possible,

The generated HTML does correct links to the openseadragon.min.js but the images used by OpenSeadragon for navigation overlays are not similarly adjusted.

Add some sort of statistics for the estimated OCR quality from the ALTO files to the image pages.

Biggest problems are text iteration with processing and HTML generation with templating through bash. This really isn't bash's forte.

In the ALTO format, hyphenated words, broken over two lines, can be stated by explicit duplication in the form of the parts + the whole. The current text extractor extracts everything, resulting in confusing OCR display for the duplicate case. This should be collapsed to just the full word, including the hyphenation-char.

It should be possible to have custom HTML-snippets placed in image-folders inserted into the generated pages. This would make it possible to have descriptive meta-data stored with the sources.

A single page with all images from all sub folders can be easier to handle than a deep directory structure. This page could be generated for all projects and accessed with a link from the standard pages.

One problem is that the Previous and Next links on the image pages would have little relation to the image order on the overview page, but that problem is already present when sorting the image list on a folder page.

It is messy to have everything in the root folder. A simple refactoring of file location would be in order.

All the values are calculated anyway so inserting the raw histogram data is trivial.

It helps a lot to have a date-time when backtracking just which batch some images were from.

When new ALTO-image-pairs has arrived and Quack is re-executed, progress is tracked against all potential images instead of the ones that has not been processed yet. Quack should start by estimating how many images has already been processed.

Switching between thumbnails & histograms in the folder view and toggling overlays in image view should be accessible by keyboard shortcuts for faster inspection.

Probably solvable by quoting excessively or enforcing a World-wide ban on spaces in filenames.

Currently all histograms are vertically scaled to the maximum count. Comparing histograms between images is hard with such scaling. One possibility would be to set a fixed height for the histogram, in terms of percentage of the total pixels.

In addition to the fixed histogram, the standard non-fixed could also be generated, but only be shown on mouse-over.

When the user points to a segment, the next segment in the group is marked with a surrounding box. This is highly confusing as the expectation is that something happens to the element pointed to. @kaarefc suggests marking both the current and the following box.

For a quick overview, each article (group of linked segments) should have an unique color.

If a Quack-execution is aborted, it sometimes leaves zero-length files. Checks for file existence is currently with bash '-f', which simply checks for existence. The check '-s' checks for existence of a file of length > 0 and would be better in this situation.

Comparison of different presentation copies is easier if the file size is stated.

Concatenation of path and file is done with $PATH/$FILE. At the root level, $PATH is empty so the result is /$PATH. This should be $PATH.

Some of the variable substitutions are single-value and does not work well when a newline is added to the inserted value. GREY_PERCENT_FIRST is an example of this.

It should be possible to avoid the newline.

The logic for locating alternate versions of the same image (same name but different location) does not seem to handle sub-folders properly.

Together with source image file size, the dimensions (in pixels) of the image are usable for judging the material.

The biggest spikes are not always the darkest or the brightest intensity. A better statistics display would be to locate the spikes no matter where they occur.

With burnout, both dark and light, the spike will very often be the same as one of the burnouts. Spike search should skip the outer edges of the histogram, in order to produce a third interesting measurement.

When processing folders or files, some sort of progress indicator would be preferable. Maybe the current image count vs. the total image count?

This does not help locally to a folder, so maybe quack should start by counting all images? Remember that image conversion is threaded, so beware updating the counter.

The table with links to image pages in the folder view supports sorting by any column. This should be linked to the thumbnails so that their order matches the table at all time.

Idea for implementation: Add a hook for clicks on table header, step through the list. For each entry, remove its thumbnail-div from the thumbnail outer box and insert it again as the last element. Do the same for thumbnails.

It's a simple indicator for OCR coverage.