This repository is the database and API underlying the EDGI Web Monitoring Project.

It’s a Rails app that:

• Acts as a database of monitored pages and revisions that have been made to them
• Allows other services to add new tracked pages/versions (we are currently focused on Versionista, but this database will soon host data from other sources, such as the Internet Archive)
• Provides an API to get that version data and allow analysts or other automated tools to annotate those versions with metadata

1. Ensure you have Ruby 2.6.3+.

   You can use rbenv to manage multiple Ruby versions

2. Ensure you have PostgreSQL 9.5+. If you are on MacOS, we recommend Postgres.app. It makes running multiple versions of PostgreSQL much simpler and gives you easy access to start and stop your databases.

3. Ensure you have Redis

   On MacOS:

   $ brew install redis

   On Debian Linux:

   $ apt-get install redis

4. Ensure you have a JavaScript Runtime

   On MacOS:

   You do not need to do anything. Apple JavaScriptCore fulfills this dependency.

   On Debian Linux:

   $ apt-get install nodejs

   If you wish to use another runtime you can use one listed here.

5. Clone this repo

6. If you don’t have the bundler Ruby gem, install it:

   $ gem install bundler

7. Wherever you cloned the repo, go to that directory and install dependencies:

   $ bundle install --without production

8. Copy the .env.example file to .env - this allows for easy configuration locally.

   $ cp .env.example .env

   Take a moment to look through the variables here and change any that make sense for your local environment. If you need set variables differently when running tests, make a .env.test file that has your test-specific variables.

9. Set up your database.

   • If your Postgres install trusts local users and you have a superuser (this is the normal situation with Postgres.app), run:

     $ bundle exec rake db:setup

     That will create a database, set up all the tables, create an admin user, and add some sample data. Make note of the admin user e-mail and password that are shown; you’ll need them to log in and create more users, import more data, or make annotations.

     If you’d like to do the setup manually or don’t want sample data, see manual postgres setup below.

   • If your Postgres install has a superuser, but doesn't trust local connections, you'll need to configure database credentials in .env. Find the line for DATABASE_URL in your .env file, uncomment it, and fill it in with your username and password. Make another file named .env.test and copy that line, but change the database line at the end to configure your test database. Then run the same command as above:

     $ bundle exec rake db:setup

     If you’d like to do the setup manually or don’t want sample data, see manual postgres setup below.

   • If you’d like to configure your Postgres DB to use a specific user, you’ll need to do a little more work:

     1. Log into psql and create a new user for your databases. Change the username and password to whatever you’d like:

        CREATE USER wm_dev_user WITH SUPERUSER PASSWORD 'wm_dev_password';

        Unfortunately, Rails' test fixtures require nothing less than superuser privileges in PostgreSQL.

     2. (Still in psql) Create a development and a test database:

        -- Development database
        $ CREATE DATABASE web_monitoring_dev ENCODING 'utf-8' OWNER wm_dev_user;
        $ \c web_monitoring_dev
        $ CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
        $ CREATE EXTENSION IF NOT EXISTS "pgcrypto";
        $ CREATE EXTENSION IF NOT EXISTS "plpgsql";
        $ CREATE EXTENSION IF NOT EXISTS "citext";
        -- Repeat for test database
        $ CREATE DATABASE web_monitoring_test ENCODING 'utf-8' OWNER wm_dev_user;
        $ \c web_monitoring_test
        $ CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
        $ CREATE EXTENSION IF NOT EXISTS "pgcrypto";
        $ CREATE EXTENSION IF NOT EXISTS "plpgsql";
        $ CREATE EXTENSION IF NOT EXISTS "citext";

     3. Exit the psql console and open your .env file. Find the line for DATABASE_URL in your .env file, uncomment it, and fill it in with your credentials and database name from above:

        DATABASE_URL=postgres://wm_dev_user:wm_dev_password@localhost:5432/web_monitoring_dev

        Make a .env.test file and set the same value there, but with the name of your test database:

        DATABASE_URL=postgres://wm_dev_user:wm_dev_password@localhost:5432/web_monitoring_test

     4. Set up all the tables and test data in your DB by running:

        # Set up tables, indexes, and general database schema:
        $ bundle exec rake db:schema:load
        # Add sample data and an admin user:
        $ bundle exec rake db:seed

        For more on this last step, see manual postgres setup below.

10. Start the server!

    $ bundle exec rails server

    You should now have a server running and can visit it at http://localhost:3000/. Open that up in a browser and go to town!

11. Bulk importing, automated analysis, and e-mail invitations all run as asynchronous jobs, managed by a Redis queue. If you plan to use any of these features, you must also start a Redis server and worker.

    Start redis:

    $ redis-server

    Start a worker:

    $ QUEUE=* VERBOSE=1 bundle exec rake environment resque:work

    If you only want to run particular type of job, you can set a list of queue names in the QUEUES environment variable:

    $ QUEUES=mailers,import,analysis VERBOSE=1 bundle exec rake environment resque:work

    Each job type runs on a different queue:

    • mailers: Sending e-mails. (There's no job associated with this queue because it is automatically processed by ActionMailer, a built-in component of Rails.)
    • import: Bulk version imports (processing data sent to the /api/v0/imports endpoint).
    • analysis: Auto-analyze changes between versions and create annotations with the results.

If you don’t want to populate your DB with seed data, want to manage creation of the database yourself, or otherwise manually do database setup, run any of the following commands as desired instead of rake db:setup:

$ bundle exec rake db:create       # Connects to Postgres and creates a new database
$ bundle exec rake db:schema:load  # Populates the database with the current schema
$ bundle exec rake db:seed         # Adds an admin user and sample data

If you skip rake db:seed, you’ll still need to create an Admin user. You should not do this through the database since the password will need to be properly encrypted. Instead, open the rails console with rails console and run the following:

User.create(
  email: '[your email address]',
  password: '[the password you want]',
  admin: true,
  confirmed_at: Time.now
)

The Dockerfile runs the rails server on port 3000 in the container. To build and run:

docker build --target rails-server -t envirodgi/db-rails-server .
docker build --target import-worker -t envirodgi/db-import-worker .
docker run -p 3000:3000 envirodgi/db-rails-server -e <ENVIRONMENT VARIABLES> .
docker run -p 6379:6379 envirodgi/db-import-worker -e <ENVIRONMENT VARIABLES> .

Point your browser or curl at http://localhost:3000.

This repository falls under EDGI's Code of Conduct.

This project wouldn’t exist without a lot of amazing people’s help. Thanks to the following for all their contributions! See our contributing guidelines to find out how you can help.

(For a key to the contribution emoji or more info on this format, check out “All Contributors.”)

Copyright (C) 2017 Environmental Data and Governance Initiative (EDGI)

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.0.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See the LICENSE file for details.