Giter VIP home page Giter VIP logo

ukwa-pywb's Introduction

UKWA PYWB Access System

This repository represents a customized extension of pywb for use with the UK Web Archive. It includes custom features:

...and configuration files for standard PyWB features:

This repository builds an ukwa-pywb container image, which extends the offical pywb container image release.

In the past, additional PyWB functionality has been developed on the ukwa/pywb fork of pywb, but this has now all been merged upstream. This allows us to depend on PyWB directly rather than our own release.

Upgrading PyWB

This project can be upgraded by simply the version of the PyWB docker container referred to in the Dockerfile. In general, upgrades should usually be smooth, but as in this case, sometimes changes have been made to PyWB that affects the things we've modified or extended, like the banner template handling.

To check this, the new image can be built and tested using the integration test system, as outlined below.

Once tested, the version can be tagged, using the PyWB version as a base. i.e. the version of ukwa/ukwa-pywb based on webrecorder/pywb:2.6.2 should be 2.6.2. If any further releases are required to resolve unexpected problems, while sticking to the same version of PyWB, a point suffix can be added, e.g. 2.6.2.1, 2.6.2.2 and so on.

Once the image has been built, it should be rolled out across the relevant ukwa-services. This includes the website, w3act and reading room service stacks.

Development Setup

To build and run from a checked-out repository, you can set up a virtualenv (or similar) using Python 3.7 or later. Install the dependencies e.g.

python setup.py install

Then run

uwsgi uwsgi.ini

And ukwa-pywb should start using the configuration in uwsgi.ini, i.e. available on port 8080, and using the CDX and WARC files in ./integration-test/test-data/. If the port is in use, create a copy of the uwsgi.ini file configuring alternative port(s) and use that instead.

To perform more complex testing that includes additional services (like Redis), you can use the integration test setup. First

cd integration-test/

Then you can use this to re-build the containerized version locally:

docker-compose build pywb

And run it using:

docker-compose up populate pywb

The first time you do this, the populate container will make the test data available and populate the system with it. You will need to do this again if you fully remove the integration testing containers. However, if you're just re-building pywb for testings, you can just do this:

docker-compose up pywb

When running locally, the service should be available at: http://localhost:8081 and should contain:

See Integration Tests for more details.

Deployment and Configuration

The project can be deployed locally or in Docker.

  • See Deployment for more information about deploying this repository and running tests.

  • See Configuration for more information about the config.yaml file and its options.

  • See Integration Tests on how to run the included integration test suite.

ukwa-pywb's People

Contributors

anjackson avatar ikreymer avatar tw4l avatar m4rk3r avatar n0tan3rd avatar ldbiz avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.