We have apparent 'gaps' under OutbackCDX + pywb, as records for

https://www.webarchive.org.uk/wayback/archive/*/http://www.webarchive.org.uk/index.html

are separate from records for

https://www.webarchive.org.uk/wayback/archive/*/http://www.webarchive.org.uk/

Our users expect to see these together. This seems to be a URL canonicalisation issue with OutbackCDX, but I'm recording the issue here for now.

Add an index source to support loading from the xml interface as provided by outbackcdx (tinecdxserver)

Add support in pywb to allow 'single session access' to every top-level page, but not embedded resources.

Current workflow to be supported:

• A user visits, and is given a session that is stored using a browser cookie.
• Each timestamp+URL combination the user visits is ‘locked’ to that session. If another user requests the same URL and the same timestamp, they should get a suitable error page.
• There should be ‘logoff’ page that expires the cookie and releases the locks.
• There should be a page for administrators that lists all the session, and the items locked by each session, and provides a link where a session can be logged off (the cookie expired and all locks released).
• It is sufficient that the locks are held in RAM. e.g. using a HashMap. They do not need to be stored permanently.
• All locks should expire overnight. This can be done within pywb or the service could provide a ‘release all locks’ URL that can be initiated from outside.

EDIT Some clarifications:

• Not all UKWA deployments need this feature, so it should be configuration per deployment.
• If it helps, the 'reference implementation' is here
• The above was implemented as a servlet filter, and uses a series of checks as to whether to lock specific resources, e.g. embeds/transclusions are not locked

The calendar layout would look better if it switched to a long layout for small screens.

The Google Analytics hook is not installed in the main playback banner, only the calendar page.

The Robot Framework test for concurrent usage would be cleaner and more robust if it used Switch Browser.

Initial support based on the proposal explained in: mementoweb/rfc-extensions#6

Add themed UI for all templates, banner for replay

Working on the video support, we found some issues.

The necessary metadata for a URL like http://this.page/ is stored under a URL like metadata://this.page/. However, when using OutbackCDX, these records are not canonicalised in the same way, and so do not get stored under the same key. Hence, when pywb performs the vi_/http://this.page/ lookup, it doesn't find the record.

I can hack my way around this by telling OutbackCDX that the metadata URL is an alias for the http one. This means the pywb playback finds the metadata and playback work fine! However, it also shows up in the calendar as if it was another instance of the original http URL.

I'm not sure how best to proceed.

My preference is that we change the JavaScript so that it explicitly looks for a record like vi_/metadata://this.page, but I'm not sure if this is breaking the semantics of the vi_ hook?

Alternatively, could the vi_/http://this.page/' hook actually convert this into a lookup for metadata://this.page/` in the back-end?

We have problems with some URLs on a particular site, where some requests appear to work and others hang.

This hangs:

#  curl -v -x ''  http://access.n45.wa.bl.uk:8280/archive/20191024125648mp_/http://www2.gov.scot/Topics/marine/seamanagement
* About to connect() to access.n45.wa.bl.uk port 8280 (#0)
*   Trying 192.168.45.25...
* Connected to access.n45.wa.bl.uk (192.168.45.25) port 8280 (#0)
> GET /archive/20191024125648mp_/http://www2.gov.scot/Topics/marine/seamanagement HTTP/1.1
> User-Agent: curl/7.29.0
> Host: access.n45.wa.bl.uk:8280
> Accept: */*
>

But http://access.n45.wa.bl.uk:8280/archive/20191024125646mp_/http://www2.gov.scot/Topics/marine/seamanagement works.

Looking at the CDX:

...
<result>
<compressedoffset>173790774</compressedoffset>
<mimetype>text/html</mimetype>
<file>
/heritrix/output/frequent-npld/20191010210458/warcs/BL-NPLD-20191024103625885-00622-76~npld-heritrix3-worker-1~8443.warc.gz
</file>
<redirecturl>-</redirecturl>
<urlkey>scot,gov)/topics/marine/seamanagement</urlkey>
<digest>6AZTIB2Q5GWAB47AIIHXHFVTZKOD2UBU</digest>
<httpresponsecode>200</httpresponsecode>
<robotflags>-</robotflags>
<url>https://www2.gov.scot/Topics/marine/seamanagement</url>
<capturedate>20191024125646</capturedate>
</result>
<result>
<compressedoffset>188646081</compressedoffset>
<mimetype>warc/revisit</mimetype>
<file>
/heritrix/output/frequent-npld/20191010210458/warcs/BL-NPLD-20191024102939401-00620-76~npld-heritrix3-worker-1~8443.warc.gz
</file>
<redirecturl>-</redirecturl>
<urlkey>scot,gov)/topics/marine/seamanagement</urlkey>
<digest>3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ</digest>
<httpresponsecode>0</httpresponsecode>
<robotflags>-</robotflags>
<url>http://www2.gov.scot/Topics/marine/seamanagement</url>
<capturedate>20191024125648</capturedate>
</result>
</results>
</wayback>

So, it looks like our revisit records are causing some kind of problem?

If JavaScript is disabled in the user's browser, you currently get no playback at all. It should be possible to at least get a basic re-written version back, even if most of the dynamic stuff will be broken (but that's the normal experience for browsing without JavaScript).

As noted in #45, this would also mean the main page could respond with a 404 for items that are not in the archive, which would be less surprising.

On our production APIs, I visit:

http://192.168.45.25:8081/qa-access/20180228215703/http://www.bbc.co.uk/news

I end up in a loop of requests:

GET /qa-access/20180228215703mp_/http://www.bbc.co.uk/news HTTP/1.1
Host: 192.168.45.25:8081
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: http://192.168.45.25:8081/qa-access/20180228215703/http://www.bbc.co.uk/news
Connection: keep-alive
Upgrade-Insecure-Requests: 1

But the response is:

HTTP/1.1 301 Moved Permanently
X-Cache-Action: PASS (no-cache-control)
X-Archive-Orig-Vary: Accept-Encoding
X-Cache-Age: 0
Content-Type: text/html;charset=utf-8
Date: Fri, 26 Jan 2018 12:10:26 GMT
Location: http://192.168.45.25:8081/qa-access/20180228215703mp_/http://www.bbc.co.uk/news
X-Archive-Orig-X-XSS-Protection: 1; mode=block
X-Mozart-Location: eu-west-1
X-Content-Type-Options: nosniff
X-Archive-Orig-X-Frame-Options: SAMEORIGIN
Content-Length: 0
X-Archive-Orig-Connection: close
Memento-Datetime: Wed, 28 Feb 2018 21:57:03 GMT
Link: <https://www.bbc.co.uk/news>; rel="original", <http://192.168.45.25:8081/qa-access/mp_/https://www.bbc.co.uk/news>; rel="timegate", <http://192.168.45.25:8081/qa-access/timemap/link/https://www.bbc.co.uk/news>; rel="timemap"; type="application/link-format", <http://192.168.45.25:8081/qa-access/20180228215703mp_/https://www.bbc.co.uk/news>; rel="memento"; datetime="Wed, 28 Feb 2018 21:57:03 GMT"; collection="qa-access"
Preference-Applied: rewritten
Vary: Prefer
Content-Security-Policy: default-src 'unsafe-eval' 'unsafe-inline' 'self' data: blob: mediastream: ws: wss: ; form-action 'self'

This appears to happen for revisit records, because if I go to the datestamps for other records playback works.

In this case, we have this record in the CDX server:

<result>
	<compressedoffset>235449396</compressedoffset>
	<mimetype>warc/revisit</mimetype>
	<file>/heritrix/output/warcs/daily/20180228120038/BL-20180228180040873-00015-62~ukwa-h3-pulse-daily~8443.warc.gz</file>
	<redirecturl>-</redirecturl>
	<urlkey>uk,co,bbc)/news</urlkey>
	<digest>3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ</digest>
	<httpresponsecode>0</httpresponsecode>
	<robotflags>-</robotflags>
	<url>https://www.bbc.co.uk/news</url>
	<capturedate>20180228215703</capturedate>
</result>

and scrolling back quite a lot, the corresponding response record, based on digest:

<result>
	<compressedoffset>325775660</compressedoffset>
	<mimetype>text/html</mimetype>
	<file>/heritrix/output/warcs/daily/20180126120026/BL-20180126120519587-00001-62~ukwa-h3-pulse-daily~8443.warc.gz</file>
	<redirecturl>-</redirecturl>
	<urlkey>uk,co,bbc)/news</urlkey>
	<digest>3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ</digest>
	<httpresponsecode>301</httpresponsecode>
	<robotflags>-</robotflags>
	<url>http://www.bbc.co.uk/news/</url>
	<capturedate>20180126121046</capturedate>
</result>

I'm guessing that pywb is going back far enough to find the record, so maybe this is a problem with the way we populated our CDX index?

Ensure docker run -it -p 8080:8080 ukwa/ukwa-pywb starts up to a server pointing to test data.
Install test data under /webarchive mount, to allow overriding via a volume there.

Support a per-url exclusion system for pywb, including the following modes:

• A SURT-prefix- based white list of allowed content, and return a HTTP 451
  message if content is outside the white list.
• A SURT-based blacklist of disallowed content that should return a HTTP 404.

Some CDX queries timeout, some don't. e.g. this doesn't work but this one works fine

[pid: 18|app: 0|req: 295807/6259025] 193.61.220.3 () {50 vars in 1267 bytes} [Wed Feb  6 12:10:25 2019] GET /wayback/en/archive/cdx?url=http%3A%2F%2Fwww.bardsey.org%2F&output=json&allowFuzzy=false => generated 37973 bytes in 236 msecs (HTTP/1.1 200) 1 headers in 48 bytes (5 switches on core 359)
[pid: 16|app: 0|req: 436657/6259124] 194.66.232.92 () {48 vars in 1398 bytes} [Wed Feb  6 11:59:59 2019] GET /wayback/en/archive/cdx?url=http%3A%2F%2Fwww.jeremycorbyn.org.uk%2F&output=json&allowFuzzy=false => generated 81575 bytes in 634587 msecs (HTTP/1.1 200) 1 headers in 48 bytes (7 switches on core 366)

i.e. these queries are taking 10 mins!

Support running locally and in Docker from same deployment.
Add env var support to config.yaml to allow setting index and archive paths via env vars.

OpenWayback permits a prefix search mode, where it lists the URLs under a given prefix. Can pywb do the same?

We have a specific Twitter URL recorded as a 304 Not Modified (via a browser rendering cache checking event).

https://www.webarchive.org.uk/wayback/archive/20190312214115/https://ton.twimg.com/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js

it seems to be played back as a 304 rather than skipped. The CDX records look like

...
com,twimg,ton)/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js 20190312213613 https://ton.twimg.com/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js application/javascript 200 NMGUH2FASUEU67TMO7EM7Y7FJ7W5ZUNI 0
com,twimg,ton)/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js 20190312213614 https://ton.twimg.com/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js application/http 304 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ 0
com,twimg,ton)/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js 20190312213615 https://ton.twimg.com/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js application/http 304 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ 0
com,twimg,ton)/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js 20190312213616 https://ton.twimg.com/tfw/js/native_bundle_v1_2c6dc837d4dedb22fff5faf9e125064ba2cbeb8a.js application/http 304 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ 0
...

For some pages, we hit OutbackCDX's default limit=10000 for numbers of results, e.g. https://beta.webarchive.org.uk/wayback/archive/*/https://www.theguardian.com/uk which stops in 2018.

How can we change pywb to use a larger limit?

Add README, documentation on deployment and new features.

Add support for per-site JS injection, probably by SURT prefix.

Can be specified in a format similar to the access system:
com,example)/ - {"insert": "custom_file.js"}

There are already extensive server-side rewriting options in rules.yaml, perhaps combine with those in some way?

Ensure Prefer header also works with proxy mode.

Use of Prefer should result in no redirect and serve either the raw content or standard proxy-mode rewritten content and include Preference-Applied.

Probably should have a different Preference-Applied for proxy-mode rewriting, which only injects the banner, instead of full rewriting. Perhaps Preference-Applied: banner-injected

There may be little we can do about this during this development cycle, but we've noticed that the banner can stay blank for a long time while the page loads. Would it be possible for it to load something quickly, so people can tell the page is working? e.g. just the blue background and logo?

We have an oddity, in that some URLs, like this one:

http://3.bp.blogspot.com/-W8IWj9tFz-I/UTCS2D5Pt-I/AAAAAAAAAI4/8BCbTLsJ3tI/s320/African+women4.png

Playback fine in OpenWayback: https://www.webarchive.org.uk/wayback/archive/20130307094428im_/http://3.bp.blogspot.com/-W8IWj9tFz-I/UTCS2D5Pt-I/AAAAAAAAAI4/8BCbTLsJ3tI/s320/African+women4.png

But do not play back in pywb: https://alpha.webarchive.org.uk/wayback/en/archive/20130307094428im_/http://3.bp.blogspot.com/-W8IWj9tFz-I/UTCS2D5Pt-I/AAAAAAAAAI4/8BCbTLsJ3tI/s320/African+women4.png

In the back-end OutbackCDX service (which is old, which might matter here), the thing checks out:

> http://bigcdx.n45.wa.bl.uk:8080/data-heritrix?url=http%3A%2F%2F3.bp.blogspot.com%2F-W8IWj9tFz-I%2FUTCS2D5Pt-I%2FAAAAAAAAAI4%2F8BCbTLsJ3tI%2Fs320%2FAfrican%2Bwomen4.png
< com,blogspot,bp,3)/-w8iwj9tfz-i/utcs2d5pt-i/aaaaaaaaai4/8bcbtlsj3ti/s320/african+women4.png 20130307094428 http://3.bp.blogspot.com/-W8IWj9tFz-I/UTCS2D5Pt-I/AAAAAAAAAI4/8BCbTLsJ3tI/s320/African+women4.png image/png 200 GRDKSKQHAA72NSYKH6UUAOAELGHBKGPW 0

From the pywb logs we see:

127.0.0.1 - - [2018-10-19 10:12:54] "POST /archive/resource/postreq?url=http%3A%2F%2F3.bp.blogspot.com%2F-W8IWj9tFz-I%2FUTCS2D5Pt-I%2FAAAAAAAAAI4%2F8BCbTLsJ3tI%2Fs320%2FAfrican%2Bwomen4.png&matchType=exact&closest=20130307094428 HTTP/1.1" 404 536 0.060883

So perhaps the issue is the + getting escaped?

Looking at https://timetravel.mementoweb.org/guide/api/#timemap shows a nice, paged TimeMap is an option for the JSON-format one. The various Wayback implementations seem inconsistent on this point, but I think I'd prefer the paged form shown on the Memento page. Apart from anything else, it might help us cope with how many hits we have for some URLs.

As of right now, running the Docker ukwa/ukwa-pywb image against our production APIs, the calendar doesn't work. It partially loads and there's this in the console:

Syntax error, unrecognized expression: #month_2010_December }}

Arising at line 111 of query.js:

if (! $('#month_' + currentYear + '_' + currentMonth).length){

Which looks like a missed ).

Looks good otherwise tho!

Following #38 we would still like to pursue further optimisations, at least for TLD/SLD lookups, to avoid linear searches if we can.

Odd error from https://www.webarchive.org.uk/act/wayback/archive/20190502112435/https://www.wekissthescreens.com/book/1?author=1

{'args': {'coll': 'archive', 'type': 'replay', 'archive_paths': 'webhdfs://192.168.45.13:14000', 'index_paths': 'xmlquery+http://192.168.45.21:8080/data-heritrix'}, 'error': '{"message": "Internal Error: object of type \'NoneType\' has no len()"}'}

The {{ url }} template value does not exist, so the links are coming out blank. We can use err_msg which contains the URL in this case.

UKWA need to make suitable APIs available for WebHDFS and our old version of OutbackCDX.

Add webhdfs:// as a registered BlockLoader scheme for loading via WebHDFS

To avoid showing status for old page while new one is loading, reset banner text to Loading...

For example, https://webarchive.org.uk/wayback/archive/2019/www.coinbase.com
In the wa-www nginx log, this request is recorded as a status code 200 whereas the on-screen message is "Url Not Found
The url http://www.coinbase.com/ could not be found in this collection. ".

As per this thread, we have a report that the Memento TimeGate is not where it is expected according to the documentation.

Specifically it seems the URI of the UKWA TimeGate has changed from:

https://www.webarchive.org.uk/wayback/archive/

to:

https://www.webarchive.org.uk/wayback/archive/mp_/

This seems likely to be a bug, probably introduced by our theming/overlaid changes. We should probably add an integration test to check the TG is there.

If we go to:

https://beta.webarchive.org.uk/wayback/archive/*/https://www.gov.uk/search/all?organisations[]=foreign-commonwealth-office&order=updated-newest

We get

80 captures of https://www.gov.uk/search/all?organisations[]=foreign-commonwealth-office&order=updated-newest

but the results point to: https://www.gov.uk/search/all?organisations[]=foreign-commonwealth-office

i.e. the additional parameter has been dropped somewhere. Doesn't seem to happen on older pywb

https://www.webarchive.org.uk/wayback/archive/*/https://www.gov.uk/search/all?organisations[]=foreign-commonwealth-office&order=updated-newest

If I go direct to the container, it also loses the second query parameter.

Playback URLs seem to work,

https://beta.webarchive.org.uk/wayback/archive/20191018085010/https://www.gov.uk/search/all?organisations[]=foreign-commonwealth-office&order=updated-newest

... so I think this is a problem with the new calendar page:

Update Docker container to run in uWSGI instead of default gevent WSGI container.
uWSGI will run on http port 8080, and uwsgi socket on 8081 in Docker container.
Set up 10 processes of 400 workers each by default.

• Setup integration tests with HTTP/S proxy mode access
• setup custom CA or ignore certs for HTTPS proxy

We have this URL:

https://alpha.webarchive.org.uk/wayback/en/archive/20180701041928im_/https://www.bl.uk/britishlibrary/resources/global/images/bl_logo_100.gif

But in the CDX we have:

uk,bl)/britishlibrary/resources/global/images/bl_logo_100.gif 20180701021331 https://www.bl.uk/britishlibrary/resources/global/images/bl_logo_100.gif image/gif 200 IXISNGDKQAOS4A4TXCKMMYUZVJCEOOUW 0
uk,bl)/britishlibrary/resources/global/images/bl_logo_100.gif 20180701041928 https://www.bl.uk/britishlibrary/resources/global/images/bl_logo_100.gif application/http 429 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ 0
uk,bl)/britishlibrary/resources/global/images/bl_logo_100.gif 20180701140344 https://www.bl.uk/britishlibrary/resources/global/images/bl_logo_100.gif application/http 429 3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ 0
uk,bl)/britishlibrary/resources/global/images/bl_logo_100.gif 20180801140650 https://www.bl.uk/britishlibrary/resources/global/images/bl_logo_100.gif image/gif 200 IXISNGDKQAOS4A4TXCKMMYUZVJCEOOUW 0

It's hitting the 429, because we got throttled by our own site! We'll talk to the team about removing the throttle, but maybe we should avoid playing these back? Or should we avoid indexing them?

Hm, playback not working at present, throwing Internal Error: '_Q' is undefined.

Rolling back to 2.4.0-beta-2 resolves this.

(thanks for noticing this promptly @GilHoggarth !)

When attempting to replay in a browser that's too old, such as IE10, redirect to home page and/or show warning message.

It would be preferable if the internal configuration details were not accessible in this way.

After attempting to run the service directly against our APIs internally, I found a gap that #1 missed. Our production APIs require the user to add a user.name=<USERNAME> field as per this documentation. Soon, our newer cluster may also require a delegation=<TOKEN> field. Would it be possible to add these to the configuration easily enough?

I'm seeing

Ymweld â safle byw a chyfredol
Edrych os oes copïau o'r Url hwn gyda archifoedd y we eraill

But the source unicode looks about right:

So somewhere along the way the locale is breaking things.

Set query page to expand years by default.
Add improved error messages for collection not found and static file not found.

As per ukwa/w3act#614

Things like /index.html should really get flattened down to / I think?

Maybe not. Looking at the OpenWayback documentation, I don't think it's usual to canonicalise the /index.html in this way https://github.com/iipc/openwayback/wiki/Technical-overview#canonicalization-scheme

Can we gracefully reload the whitelists/blacklists?

On a few pages the page title/frame still says ‘Loading...’ even though it’s stopped.

One example is a PDF https://www.webarchive.org.uk/wayback/archive/20180523082513/https:/assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/705269/Post_16_education_highest_level_of_achievement_by_age_25.pdf

Support localized UI text for collections to support multiple language deployment.

We are moving to a new, simpler implementation of the NPLD limitations. Rather than going through a remote desktop, clients will access Wayback directly. This means we need to do a few things:

1. Modify the single-concurrent-use locking
2. Limit how much text can be copied at once
3. Use cache control headers to limit how much content gets stored on the client
4. Prevent download of non-web content

Single-Concurrent-Use

The will be no login/logout hooks, so a simple alternative locking mechanism is proposed.

The default behaviour is that all 'top-level' URLs will be lock to a user's cookie session, set to time-out at midnight later that day. As before, transcluded items should not be locked. These locks are managed server-side.

To enable the lock to be released earlier, the lock can be polled and repeatedly updated from the Wayback JavaScript client, with a time-out set to a few minutes in the future. While a page is being viewed, it will still be locked to the current user, but once they move on it should time out in a few minutes as the lock is no longer being updated.

This means files that get downloaded will be locked for the whole day, but most pages should be released promptly.

Limit cut-and-paste

The client-side JavaScript should intervene during cut/copy events and limit the text to a configurable amount.

Limit local caching

The server should add headers to limit local caching, as per https://stackoverflow.com/questions/9884513/avoid-caching-of-the-http-responses -- this may be better done via NGINX?

Prevent downloads of non-web content

We need to try to prevent content being downloaded to local machines, and use a secondary service for rendering some formats to HTML.

First step is to intercept direct downloads of content other than HTML. These will then either be blocked (probably with a custom 451 error) or passed to an external service for rendering.

We will need some lookup table that maps Content Types to URL templates, e.g.

application/msword, http://service.things.com/url={url}

Or similar. When we hit a non-web type, we should open up the block page, and if there's a mapping, offer to redirect the user to that URL for access. For all types, we should ensure the Content-Disposition header is blocked so downloads can't be forced that way.

i.e. this is similar to the old Interject idea (source code & tech docs here).