Revisit ukwa-backstage integration as part of W3ACT with files and archived resources and collections and targets

And also integrate W3ACT users via https://github.com/duke-libraries/devise-remote-user

We currently use Trifecta to check on Kafka queues, but this is not that widely used. We can switch to akhq which seems to be more widely used and well supported.

version: '3.7'
services:
  akhq:
    image: tchiotludo/akhq
    ports:
      - "58080:8080"
    environment:
      AKHQ_CONFIGURATION: |
        akhq:
          connections:
            fc:
              properties:
                bootstrap.servers: "192.168.45.34:9094"
          security:
            default-group: reader

• Review current implementation, test it's working.
• Migrate older records to DB.
• ukwa/ukwa-manage#84

The stacks hardcode the www/beta/dev domains in a few places, whereas it should be possible to pick these up from Host headers/context.

Current plan is to siphon crawl events into a large database, for recently crawled FC material. We will use Solr at first because we know how to run it at scale, reconsidering CockroachDB later if we need e.g. proper SQL or ACID transactions etc.

Start with a simple Solr indexed version of the standard crawl log, so we can:

• Find/filter the crawl log (see crawl-log-viewer) but much quicker than using Kafka.
• Find crawl launch outcomes.

See ukwa/crawl-db#1

Need to find a way to tidy up the H3 log parsing code and related code that is spread around:

• crawl-log-viewer e.g. parser
• crawl-analysis should be notebooks I think.
• crawl-streams
• crawl-db
• Deployment bits currently in ukwa-services.
• Whether Hadoop jobs should be in ukwa-hadoop-tasks or in the relevant repository e.g. crawl-db (given MrJob is a pretty lightweight dependency compared to Luigi, that might be fine?)

Update the access-time webrenderer to 2.2.x, and set RUN_BEHAVIOURS=false, to speed up rendering of cards etc.

The repeated tests in browse.robot might lend themselves to being simplified/deduplicated using a custom keyword.

To resolve some complex playback issues (Twitter, HuffPo) we need to be able to play back POST requests.

This requires some coordination with Ilya as he's been changing how he does it.

Once the indexing scheme is stable, we need to use a version of OutbackCDX that supports it, and re-index the CDX data (at least the last couple of years).

Updating the Java stack is quite involved: ukwa/webarchive-discovery#244

Might be time to switch to Python for this MR Job. Use PyWB indexer and POST them to OutbackCDX.

Also need OutbackCDX 0.8.0 to handle the lookups properly.

Some other examples of similar code:

• mr_cdx_job.py
• xperimental_tasks_python_warc.py
• warctasks.py

MrJob

• https://mrjob.readthedocs.io/en/latest/guides/writing-mrjobs.html#passing-entire-files-to-the-mapper
• https://mrjob.readthedocs.io/en/latest/guides/setup-cookbook.html?highlight=virtualenv#using-a-virtualenv

Using mapper_raw means MrJob arranges for a copy of each WARC to be placed where we can get to it:
(This breaks data locality, but streaming through large files is not performant because they get read into memory)
(A FileInputFormat that could reliably split block GZip files would be the only workable fix)
(But TBH this is pretty fast as it is)

Play with a WARC processor with https://pypi.org/project/boilerpy3/ and e.g. Spacy

• Verify with @ikreymer when the approach has been finalised, and what version of OutbackCDX it works with.
• CDXJ Indexer indexes metadata records, which is what we want for video metadata etc. Are those application/warc-fields fields from metadata records from Heritrix3 okay in the CDX?
• Index recent material into a fresh Outback (>= 0.8.0) index and check playback.
• Convert metadata URIs to embed URNs.
• Drop 451/429?

See https://github.com/ukwa/ukwa-hadoop-tasks/tree/master/warc_indexing

The website is currently deployed on the access server. It should be running from the PROD Swarm.

• Include PyWB 2.6.3 in deployment.
• Deploy to BETA Swarm.
• Run tests against beta.webarchive.org.uk to verify all is working as expected
• Deploy to PROD Swarm.
• Run tests against prod1.n45.wa.bl.uk to verify all is working as expected.
• Set up API proxywebsite.api.wa.bl.uk:80, pointing to prod1:80
• Run tests against website.n45.wa.bl.uk to verify all is working as expected.

The final deployment steps will need to be done together, to make sure everything is consistent. This should also be done early or late in the day to minimise disruption.

• Update public-facing NGINX proxy configuration to match BETA, using website.api.wa.bl.uk:80 as the back-end.
• Run tests against www.webarchive.org.uk to make sure all is well.

The archivist role can add the problematic URL to W3ACT already, under a Black List field.

Then, we need to pick up white_list,black_list URLs from targets.csv and include them in the crawl feeds. Should be combined with the in-scope and nevercrawl lists (respectively).

After that, we need to check the crawler will pick up changes to the scope and block files, and add a w3act_export service to the FC stack that pulls and updates them. This does mean the block list might lag behind the launches a little, so we probably want to update them more often than daily.

(clearly, we should consider wildcard/regex support, but that's more difficult to use. Maybe use plain URLs for URL blocks, but allow #-delimited lines for RegEx?

https://www.bl.uk/?mobile=on
#twitter\.com/.*?lang=#

Hmm.

Also, take ukwa/ukwa-heritrix#85 into account

This is to note any outcomes from ukwa/w3act#662

• Carlos is hitting examples where the inner iframe of playback says login, and this ends up logging into Wayback within the frame. The login redirect should target the full page frame. Possible a new frame and then recomment a reload?
• The /wayback/*/ to /wayback/archive/*/ redirect is not working, ended up at http://prod1/act/wayback/archive//https://www.scottishpower.co.uk/
• Determine if the JSESSIONID Set-Cookie headers are what's tripping up everything else - see e.g. this page.
• Copy the W3ACT cookie into every response as a Set-Cookie header, when viewing Wayback.
• Deal with the SameSite warning (see below, and ukwa/w3act#663).

Cookie “PLAY_SESSION” will be soon rejected because it has the “SameSite” attribute set to “None” or an invalid value, without the “secure” attribute. To know more about the “SameSite“ attribute, read https://developer.mozilla.org/docs/Web/HTTP/Headers/Set-Cookie/SameSite

e.g add a script to pull this from GitLab, run that from install and from Airflow?

It seems the 2018 and 2019 domain crawls may not have been CDX indexed. We need to design a suitable Airflow DAG that will be able to perform these backfill tasks.

The idomatic Airflow version would be a proper backfilling task, with a start date in e.g. 2010, using the last-modified date of the files on HDFS, and where each chunk loops through the total available to be indexed. e.g. an @monthly task, that lists all WARCs corresponding to that previous month, and then indexes them in chunks of e.g. 2000 WARCs.

This would mean changing the windex utility to (a) be able to filter on a date block instead of X years back, and (b) able to loop over all matching WARCs rather than just running one batch.

Building on #83, improve crawl management:

• Make crawl launches 'back-fillable` so we can re-run launches if the don't happen:
  • Needs date-stamped crawl feed files.
  • Needs a separate task that is dependent on the data export, or the current w3act_export needs to be made back-fillable.
• Blocks, seeds and scope files in use by the crawlers need to be updated:
  • Blocks and scope managed via Watched Files, less clear if/how seeds should be blanket-updated.
  • Not clear how best to do that. Probably push rather than pull, as this means Airflow is always in charge of things. But then, a shared volume updated directly by Airflow? Files made available and remote task or service prompted to pull them down?
• Launch metrics need to be posted to Prometheus.

Rather than continuing to roll our own search and visualization tool, we should consider adopting SolrWayback and collaborating with the NAS team on it. We could start by making it available as an internal tool, within the W3ACT stack. For this, we need to:

• Dockerise it in a way that allows it to work to our Solr indexes.
• Cope with content_type being either single or multiValued.
• Ensure required configuration options to be overridden from environment variables.
• Tomcat should log the response from the server when calls to Solr fail: add alternative logback config for Docker so logs go to the console not a file.
• Implement HTTP and/or WebHDFS WARC record retrieval back-ends so it can work properly with our WARC store.
• SolrWayback relies on url_norm for many queries, but our older indexes do not contain it. Can we work around this somehow?
• To facilitate automated CI testing, switch to a test WARC as well as test Solr documents, i.e. use consistent records so we can search and replay from the test system.
• Allow WAR name to be overridden on launch, so I can use act#solrwayback and hence get it to deploy in the right place.
• Allow alternative playback engine to be overridden. Added ALT_PLAYBACK_PREFIX env var.
• Also override regex for path as our Solr has name only not full path.
• ukwa/ukwa-warc-server#12
• Adjust SolrWayback so it can be deployed at an alternative path (e.g. /act/solrwayback).

It's possible we can't resolve some of these issues without re-indexing. In which case, this will have to wait.

Notes on public use moved to #73

See the old Ingest Stack ideas: https://github.com/anjackson/ukwa-services/tree/intranet-2020/manage/intranet

These should be part of the core W3ACT stack, using the new auth mechanism, etc.

• Use Metabase as main report system, rather than a faceted browser. Fall back on Voila notebooks if necessary.
• Use Grafana as the main report system, rather than a faceted browser. Fall back on Voila notebooks if necessary.
• #70
• #38
• API and recent screenshots
• [ ] TrackDB browser ???
• [ ] Airflow Dashboard ?

• Updating https://wiki.bl.uk:8443/display/WAG/UKWA+Technical+Architecture and deleting old stuff.
• Moving useful stuff into https://github.com/ukwa/ukwa-documentation
• Roadmap in the style of https://www.gov.uk/roadmap

The new host_no_auth env var was introduced to get around errors raised in robot runs by the inline authentication for the dev service, but it looks a bit extraneous/messy. Might be useful to revisit.

The procedure for launching robot tests might need some clarification or refactoring, especially wrt to environment vars.

eg. The .env files need "export"'s removing before normal docker-compose usage, but have been retained as they are used in that format elsewhere.

The website scripts (as example) are here:
https://github.com/ukwa/ukwa-services/blob/master/access/website/deploy-access-website.sh

Avoid https://www.bl.uk/?mobile=on to prevent accidentally crawling the mobile version of www.bl.uk.

Just patch the regex block list bean. Document how it's done to the live crawler, ideally wrap as a h3cc script.

But see also #36

To be broken down into issues and milestones...

• Push metrics from w3act_export to Prometheus.
• Resolve webrecorder/pywb#591 (this is currently worked-around by dropping revisits)
• Resolve ukwa/ukwa-pywb#61 (Fixing Twitter requires indexing POST requests etc.)
• Find any and all places where webrender-puppeteer is used with a proxy and ensure there is no trailing slash because this make puppeteer go crazy. ukwa/webrender-puppeteer#13
• Consider hooking Flask app into Sentry
• Make recently-crawled screenshots (and content!) available. To do so, we have to finish off the warc-server idea:
  • Update warc-server to maintain filename-HDFS mapping?
  • Then proxy across crawler warc-server and HDFS warc-server?
  • Or have a separate redirect service that bounces HDFS filenames to WebHDFS, and to the crawler warc-server otherwise?
  • Also, update warc-server to update shared file list in background thread. Could be separate services, using NGINX proxy_next_upstream to manage them.

See also ukwa/ukwa-access-api#1 and the other issues https://github.com/ukwa/ukwa-access-api/issues

The image server container we use is not being kept up to date: https://hub.docker.com/r/lyrasis/cantaloupe

There are some others on Docker Hub to investigate, but it's probably easier to make a new ukwa/cantaloupe one and make sure we're up to date.

Can we leverage the CDX or ELK indexes to support alerts corresponding to the Targets owned by a curator and/or organization? Also relates to the idea of using CDX to provide RSS feeds of changes to URLs as an API service.

This covers the deployment files and documentation for running the PyWB reading room services as a Docker Swarm service stack. The documentation starts at: https://github.com/ukwa/npld-access-stack#readme

Note that, at this time, PDF and ePubs are not handled properly. PDFs will be rendered in the browser directly, for example. This will remain the case until ukwa/ukwa-pywb#74 is complete.

• Confirm required network location. Do we need to be on the Access VLAN?
• Ensure staff access can be separated out. May require separate IP address.
• Understand various redundancies/backup services needed.
• Verify assumption that all failover redirection, SSL encryption, authentication, token validation or user identification are handled upstream.
• Understand reporting needs and whether this is all handled upstream.
• Configure logging as appropriate.
• Consider training options, e.g. this
• If offline installation needed, @anjackson to document how that can be done (following this example or this one that does multiple images).
• @anjackson Add NGINX rules to map expected URLs to PyWB URLs.
• @anjackson Add LOCKS_AUTH=admin:password
• @anjackson Add in known test cases for manual testing below.
• @anjackson Allow access to the multi-cluster WARC Server as warc-server.api.wa.bl.uk
• @anjackson ukwa/ukwa-pywb#77
• @anjackson ukwa/ukwa-pywb#78
• ukwa/docker-robot-framework#6

There are problems with the QA Wayback proxy in W3ACT - it struggles already, but will definately struggle with POST requests that PyWB can now support. We can check user cookies at the NGINX level via a cached sub-request.

See:

• https://developer.okta.com/blog/2018/08/28/nginx-auth-request
• https://serverfault.com/questions/897625/caching-nginx-auth-request

The website and W3ACT tests (see e.g. #42) should should be extended to read the blocked-URLs list, and check those records are not available via CDX or Solr.

Following ukwa/ukwa-pywb#71, upgrade W3ACT's Wayback and verify with @crarugal that ukwa/ukwa-pywb#70 is resolved.

Update to latest version, inc. small patch to rotate workers and hopefully stop the occasional outages we're seeing.

This ticket summarises the overall status of this project, also known as the 'Ericom Replacement Project'. In short, we need to be able to access NPLD content from the UK LDL in a way that meets accessibility needs while also maintaining sufficient security. The current approach uses a remote desktop that is accessed via a HTML canvas, and as such this does not provide an route that meets accessibility legislation. The proposed solution makes the content more accessible, while carefully managing the security issues and NPLD contraints (e.g. the single-concurrent-use lock).

The solution works by extending our PyWB service to access PDFs and ePubs, and ensuring that the resulting web service is only access via authorised web browsers that prevent copies of items being taken away. For some reading rooms, this requires a dedicated NPLD Player.

There are two main work streams:

• UKWA Team helping App Support deploy the PyWB services and integrate them into LDL Reading Rooms.
  • #69
  • #86
  • Supporting all expected URL forms, including IDs with no prefix, e.g. vdc_100031420983.0x000001 rather than ark:/81055/vdc_100031420983.0x000001, and including /welcome.html?ID....
• Webrecorder creating or extending the tools to make this possible:
  • ukwa/ukwa-pywb#74
  • ukwa/npld-player#3

n.b. Some tasks need to be run on other services, but AirFlow can make the SSH connection dependencies nice and explicit (see example), and the remote task can still just run a Docker run command so we can manage the software distribution (note we might need a docker pull ukwa/ukwa-manage:latest as part of the remote script if were running latest).

Should also rationalise what's in:

• https://github.com/ukwa/ukwa-manage making sure to cover any fixes from the 2019-prod branch
• https://github.com/ukwa/ukwa-tasks
• https://github.com/ukwa/ukwa-hadoop-tasks

[x] Add https://github.com/epoch8/airflow-exporter so we can integrate with Prometheus.

Ingest

• w3act_backup daily):
  • Back-up W3ACT PostgreSQL database to HDFS as CSV and SQL.
  • Clean out older backups??? (optional)
• w3act_export (hourly):
  • Update blocks/allows.txt/aclj and annotations.json.
  • Update the Collection Solr.
  • Update crawl feed/job specifications.
  • python-w3act, w3act run_w3act-qa-check prod (currently, a weekly cron of 0 8 * * Mon ssh access@prod2 'sh /home/access/github/python-w3act/run_w3act-qa-check prod' >> /var/log/w3act-qa-check.log 2>&1)
  • GenerateW3ACTTitleExport (optional)
• Analyse Crawl Logs/Run Document Harvester (hourly) see here
• Update Intranet Reports (GenerateHDFSReports, dead seeds, etc.) (optional) currently some in ukwa-manage here but also GenerateHDFSReports and ukwa-reports.

Crawl

• crawl_launch (hourly):
  • Launch crawls, based on crawl job specifications (hourly) (from here to crawlstreams).
• crawl_warc_tidy (hourly): (to be part of ukwa-manage? OR h3cc?)
  • Close old open WARCs. (optional)
  • Move WEBRENDER WARCs see here.
• Move WARCs and crawl logs to HDFS (hourly) ??? (optional) (new store command)
• Refillers, e.g. (optional)
  • post-process Kafka log or CDX and re-queue URLs that match certain criteria.
  • Run with CrawlCache and do screenshots/device emulation.

Management

• Update TrackDB from HDFS:
  • Update all HDFS daily, update WARCs locations hourly.
  • Metrics (generate metrics from TrackDB and push to the Prometheus push gateway) ??? OR stats_pusher.
• Back up TrackDB to HDFS. (optional)
• HDFS file hash job to TrackDB. (optional)
• #102

Access

• Update CDX Index with latest WARCs on HDFS, based on TrackDB (hourly) website/scripts/run-cdx-indexer.sh
• #63
• Back-up the Shine PostgreSQL database (daily)
• Run the test suite (daily after the above updates?) and raise an alert if the website is misbehaving (optional)

I think part of the reason warcprox is pulling in so much data is that it does not do any deduplication. We should enable deduplication.

Need to check it's the 'right kind' of deduplication, something we can cope with at playback time.

We need to make sure all important Docker images are scanned for security issues as part of the GitHub Actions process, before the images are pushed to Docker Hub.

To do this, we can reuse GitHub Actions workflows across repositories, to ensure we build, scan and upload Docker Images consistently.

This is an example of a container that uses the shared workflow: https://github.com/ukwa/ukwa-warc-server/blob/master/.github/workflows/push-to-docker-hub.yml

The task here is to go through the stacks in this repository and update every referenced container build to re-use this shared workflow. Every change should be proposed as a PR on each repository, and linked here for @anjackson to review.

Beyond internal access, to use SolrWayback as a public service (replacing the faceted search part of ukwa-ui), we need to consider more complex issues:

• Add localization support: netarchivesuite/solrwayback#23
• Implement as an accessible, responsive design, e.g. follow Warclight Bootstrap for basic facet design and UI. Consider TailwindCSS for layout with Shoelaces for components, to be consistent with Webrecorder's work.
• Pages like the Toolbox etc. should also be routed and bookmarkable.
• Display an error message in the UI when calls the the back-end fail, rather than appearing to still be busy.
• Security review - make sure all APIs can safely be made public.

As well as some minor changes:

• Make sure the indexer config default (and in SolrWayback Docker Compose) uses url_norm.
• Add support for common deployment context headers (X-Forwarded-Proto, X-Forwarded-Host etc.) to SolrWayback, so hardcoding the base URL is no longer necessary.
• Possible optimisation: Change ArcHTTPResolver to check response for Content-Bytes rather than make two requests per request (where the first probes for Accept-Ranges: bytes (Or at least only check the first time?)

This needs to be weighed up against the difficulties in adapting the off-the-shelf options.

Currently, one file contains three workflows, because they share code for dumping the W3ACT DB, and each runs their own dump in case of conflicts due to workflows running simultaniously. To me a bit more canonical-Airflow in style and a bit easier to manage, the workflows could be changed as follows:

• For w3act_export
  • This runs hourly, to keep access services up to date. As it's the most frequent, this is the one that should export W3ACT data.
  • Rather than maintaining a single folder and replacing, we should use a shared per-run folder, like /var/tmp/w3act_export_2021-12-10T09:00:00Z/ so that each run get's it's own output folder.
  • These will get cleaned up automatically every 30 days by the OS.
• For w3act_backup and w3act_report
  • These would use Airflows ExternalTaskSensor to await the completion of the w3act_export workflow for the hour at which they run, e.g. 2021-12-10T00:00:00Z. They would then refer to the corresponding W3ACT DB dump and use that instead of a separate dump.

This would make it easier to keep them in separate files, which is also more canonical for Airflow, and makes things a bit easier to understand.

It would be very useful to be able to browse and filter Targets in Collections based on whether they are OA or not. e.g.

• In this collection, show only OA items.
• Show all OA items relating to this subject.

If we wrap IIIF around the page screenshotter, we get a lot of the features we'll need, like easy specification of sizes etc, for different purposes.

To make this work, given the format of IIIF URIs, we could use PWID's and Base64 encode them. e.g.

urn:pwid:webarchive.org.uk:2008-11-29T00:41:42Z:page:http://www.jisc.ac.uk/whatwedo/programmes/programme_preservation/2008sigprops.aspx

Becomes...

dXJuOnB3aWQ6d2ViYXJjaGl2ZS5vcmcudWs6MjAwOC0xMS0yOVQwMDo0MTo0Mlo6cGFnZTpodHRwOi8vd3d3Lmppc2MuYWMudWsvd2hhdHdlZG8vcHJvZ3JhbW1lcy9wcm9ncmFtbWVfcHJlc2VydmF0aW9uLzIwMDhzaWdwcm9wcy5hc3B4

Which we use as the identifier in the IIIF {scheme}://{server}{/prefix}/{identifier}/{region}/{size}/{rotation}/{quality}.{format} URLs, like this:

/iiif/2/dXJuOnB3aWQ6d2ViYXJjaGl2ZS5vcmcudWs6MjAwOC0xMS0yOVQwMDo0MTo0Mlo6cGFnZTpodHRwOi8vd3d3Lmppc2MuYWMudWsvd2hhdHdlZG8vcHJvZ3JhbW1lcy9wcm9ncmFtbWVfcHJlc2VydmF0aW9uLzIwMDhzaWdwcm9wcy5hc3B4/0,0,200,200/full/0/default.jpg

This uses the page level precision-spec, as this is what makes sense in this context. The prefix of the URL would have to be used to distinguish between the archived and crawl-time images.

/render/archive/iiif/2/dXJuOnB3aWQ6d2ViYXJjaGl2ZS5vcmcudWs6MjAwOC0xMS0yOVQwMDo0MTo0Mlo6cGFnZTpodHRwOi8vd3d3Lmppc2MuYWMudWsvd2hhdHdlZG8vcHJvZ3JhbW1lcy9wcm9ncmFtbWVfcHJlc2VydmF0aW9uLzIwMDhzaWdwcm9wcy5hc3B4/0,0,200,200/full/0/default.jpg
/render/capture/iiif/2/dXJuOnB3aWQ6d2ViYXJjaGl2ZS5vcmcudWs6MjAwOC0xMS0yOVQwMDo0MTo0Mlo6cGFnZTpodHRwOi8vd3d3Lmppc2MuYWMudWsvd2hhdHdlZG8vcHJvZ3JhbW1lcy9wcm9ncmFtbWVfcHJlc2VydmF0aW9uLzIwMDhzaWdwcm9wcy5hc3B4/0,0,200,200/full/0/default.jpg

This could be done by running a Cantaloupe IIIF image server, which wraps plain image servers nicely, is used by our partners, and has lots of nice features like handling caching. This would pass the Base64 PWID on to a modified webrender-puppeteer which would decode the pwid64 and render the page at full size and ideally at high resolution. Cantaloupe would then cache this output and handle generating all necessary derivatives.

Cantaloupe can also overlay e.g. the UKWA logo which might work quite nicely.

(We could also add http://labs.mementoweb.org/aggregator_config/archivelist.xml and use that to determine the right web archive endpoint for other archives.)

The script that runs w3act exports should also post metrics to Prometheus. Proposal is to shift to being powered by ukwa-manage rather than just python-w3act and define the task script there, and add code to post-process the python-w3act output and post metrics to Prometheus.

Some screenshot problems need fixing by updating the webrender-api service to use the latest webrender-puppeteer release. However, even then, there will be problems getting the timings right because the machine is so heavily loaded when running screenshots in the morning. We need to spread things out a bit more, so we much reduce the number of workers for webrender-api.

When one crawler froze up, and we switched to another, this caused problems because both were running the same networked Gluster filesystem (used for Kafka, Prometheus) whereas the crawl state (frontier and caches) were locally held. This caused problems with Kafka and Prometheus on startup.

This ticket is to consider how to handle this:

• Only have crawl output on Gluster?
• Move Kafka/Prometheus/etc. onto local disk?
• Make Kafka an distinct, fully distributed service? (Similar to how the crawl-time CDX is a separate service).
• And improve documentation to cover crawler failover.

Add an Airflow DAG, based on the rclone/rclone Docker image, running e.g.

rclone copy --hdfs-namenode h3nn.wa.bl.uk:54310 --hdfs-username ingest  --max-age 24h --no-traverse /mnt/gluster/fc/heritrix/output :hdfs:/heritrix/output --include "*.warc.gz" --include "crawl.log.cp*"


rclone copy --max-age 48h --no-traverse /mnt/gluster/fc/heritrix/output/frequent-npld hadoop3:/heritrix/output/frequent-npld --include "*.warc.gz" --include "crawl.log.cp*"

The newer W3ACT stack improves the authenication method to access e.g. QA Wayback, but it also makes it a bit easier to accidentally remove the access limit. To look our for this, we need some additional automated tests to check logins are required etc.

To check that the following areas are only accessible if logged into W3ACT:

• /act/wayback/
• /act/nbapps/
• /act/logs/

The test system is at https://github.com/ukwa/ukwa-services/tree/master/ingest/ingest_tests and is a set of Robot Framework tests that perform some simple live-system tests. i.e. all tests added to here should be safe to run on live/production services.

So we can upgrade/migrate as needed, we need to be able to run across two Hadoop clusters.

• Airflow #50
• ukwa/ukwa-warc-server#8
• ukwa/ukwa-manage#80
• ukwa/ukwa-manage#82
• Add Airflow DAGs to run the above tools on both Hadoop services

NLW report that these websites should all have instances continually from 2004 onwards:
http://www.bloc.org.uk/
http://www.enlli.org/
http://www.morfablog.com/
http://www.cymruarywe.org/
http://www.waleswatch.welshnet.co.uk/
http://academi.org/
http://www.eglwysfair.org/
http://www.eisteddfod.org.uk/
https://mennaelfyn.co.uk/
http://www.fortunecity.com/business/pencil/1572/
http://www.grahamedavies.com/
https://www.iwa.wales/
http://www.ewrop.com/
http://gwleidydd.blogspot.com/

There are instances from 2008 onwards in QA Wayback but the earlier instances are missing.

While investigating showing the Collection Areas, @min2ha found the data in the Collections Solr didn't match up with what was in W3ACT. Following team discussion it seems that a significant change of data model has happened.

Specifically, in the schema, the collectionAreaId is not multiValued. This means each Collection can only belong to one Collection Area (which it seems I had assumed was part of the intention, to make the list manageable, but it seems there are many Collections in multiple Collection Areas).

So, to fix this, we need to:

• Change the ukwa-ui-collections-solr schema so it allows multiple collection areas, and deploy that to DEV.
• Change the python-w3act scripts to make use of the multiple values for the collection areas.
• Keep working on ukwa-ui to take advantage of this data.

Rather than depending on processes running on Ingest, the crawls should be launched from Airflow. First, we'll just port the current mechanism over. (see #??? for planned improvements).

• First use the bypm crawl launch as a test case, check it all works fine.
• Check those crawls launched properly and down-stream tasks are working as expected.
• Then add the npld crawl launch task, working in the same way.

There are a few things we could look at to simplify and secure how we deploy under Docker Swarm:

• Docker Swarm Configs
• Docker Swarm Secrets
• Stop running anything as root, and isolate containers with a user namespace

We're not able to shift off Google Analytics just yet, but can at least switch data sharing features off.

This likely means UKWA-UI and ukwa-pywb, perhaps other stuff too.

After changing this, I'm hoping this report will be clear of issues: https://themarkup.org/blacklight?url=www.webarchive.org.uk