alphagov / govuk-docker Goto Github PK

View Code? Open in Web Editor NEW

72.0 41.0 19.0 2.15 MB

GOV.UK development environment using Docker 🐳

License: MIT License

Makefile 19.60% Dockerfile 7.28% Ruby 45.22% Shell 27.90%

govuk

govuk-docker's Introduction

GOV.UK Docker

GOV.UK development environment using Docker.

The GOV.UK website uses a microservice architecture. Developing in this ecosystem is a challenge, due to the range of environments to maintain, both for the app being developed and its dependencies.

The aim of govuk-docker is to make it easy to develop any GOV.UK app. It achieves this by providing a variety of environments or stacks for each app, in which you can run commands, and the app itself.

RFC 106: Use Docker for local development describes the background for choosing Docker. See the list of projects which work with govuk-docker.

Installation

First make sure you have the following dependencies.

brew
git
A govuk directory in your home directory

Next, add the following line to your bash/zsh config.

# in ~/.bashrc or ~/.zshrc
export PATH=$PATH:${HOME}/govuk/govuk-docker/exe

Run echo $SHELL if you're not sure which shell you use. After saving, you will need to run source ~/.bashrc or source ~/.zshrc to apply this change to your current terminal session.

Now in ~/govuk , run the following setup commands.

git clone [email protected]:alphagov/govuk-docker.git
cd govuk-docker
bin/setup

👉 Check the troubleshooting guide if you have a problem.

Then make sure you give Docker enough resources.

Running GOV.UK applications can be resource intensive. To give Docker more resources on Mac, click the Docker whale icon in the macOS menu bar, select 'Preferences'. We suggest the following minimum resources:

6 CPUs
12 GB RAM
64GB+ Disk

Check out the how-to guide to customise your setup.

Usage

Do this the first time you work on a project:

make [app-name]

...where app-name is the name of the app you want to build. For example, if you want to make collections-publisher you would run:

make collections-publisher

👉 Check the troubleshooting guide if you have a problem.

Each project provides a number of 'stacks' for different use cases. You can see the stacks for a project in its config file. To provide consistency, all projects should follow these conventions for stacks:

The `lite` stack

This stack provides only the minimum number of dependencies to run the project code. This is useful for running the tests, or a Rails console, for example.

Do this to run the tests for a project:

govuk-docker run [app-name]-lite bundle exec rake

👉 Check the troubleshooting guide if you have a problem.

The `app` stack

This stack provides the dependencies necessary to run an app e.g. in a browser. If the app is a web app, you will then be able to visit it in your browser at app-name.dev.gov.uk.

Do this to start a GOV.UK web app:

govuk-docker up [app-name]-app

👉 Replicate data locally (or use the app-live stack).

👉 Check the troubleshooting guide if you have a problem.

The `app-*` stacks

Variations on the app stack are allowed where necessary such as:

app-draft: used for testing the authenticating-proxy against a draft version of the router app.
app-live: used to test a read-only frontend app against live GOV.UK APIs (avoids having to replicate data locally).

Some app stacks also depend on a worker stack, to run asynchronous tasks [example].

Resources

Contributing

Check out the CONTRIBUTING guide.

Licence

MIT License

govuk-docker's People

Contributors

Stargazers

Watchers

Forkers

benthorner nogtini alphagov-mirror mac-manu denomas marcelovani tubbz-alt neilha uk-gov-mirror claudiosfpessoa dilwoarh bhupsal1 benbhall rrbanda rarikus cl-gds maidamoh peterhippocarter kejau

govuk-docker's Issues

Bouncer will not build

Steps to reproduce

make bouncher

Stack trace

Some kind of issue in the DB setup perhaps...

dropdb --if-exists transition_test
dropdb: could not connect to database template1: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
rake aborted!
Command failed with status (1): [dropdb --if-exists transition_test...]
/govuk/bouncer/Rakefile:11:in `block (2 levels) in <top (required)>'
/home/build/.rbenv/versions/2.6.6/bin/bundle:23:in `load'
/home/build/.rbenv/versions/2.6.6/bin/bundle:23:in `<main>'
Tasks: TOP => db:reset
(See full trace by running task with --trace)
make: *** [/home/huw/govuk/govuk-docker/projects/bouncer/Makefile:2: bouncer] Error 1

It appears to be this line that is causing the issue:
govuk-docker run bouncer-lite sh -c 'bundle exec rake db:reset'

Would be good if someone else can confirm this is reproducable on their machine and not some wierd artifact of some database containers I might not have cleared out?

Initial installation assumes rbenv but not setup

We have an issue that the installation instructions expects a bundle install step before bin/setup, on a new machine this will use system Ruby and on GDS current dev machines this won't work due to the bundle lockfile version. Example error:

/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems.rb:283:in `find_spec_for_exe’: Could not find ‘bundler’ (2.1.4) required by your /Users/matt.hill/govuk/govuk-docker/Gemfile.lock. (Gem::GemNotFoundException)
To update to the latest version installed on your system, run `bundle update --bundler`.
To install the missing version, run `gem install bundler:2.1.4`
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems.rb:302:in `activate_bin_path’
	from /usr/bin/bundle:23:in `<main>'

We install rbenv as part of bin/setup however rbenv is not configured nor is a Ruby version installed so this doesn't really help anyone progress with a Ruby installation.

Some options I could see to proceed with this:

Drop bundle install from initial install step, you don't actually need it until you run govuk-doctor or the tests. This seems a bit funky as you'll have parts of the repo that might not work and need extra instructions to set them up
Remove rbenv from Brewfile and suggest a dev sets that up first with a link to rbenv instructions - this is probably the lowest effort approach.
Set-up rbenv as part of bin/setup - this is most consistent with current approach - this would mean us having to modify a ~/.zshrc file run source on it and then rbenv install and bundle install. This one is a bit of a pain for being invasive (changing the zshrc file feels like a confirm action) or assuming someone uses zsh

I'm somewhat of the view that 2 is the best option where 3 just seems like more things for us to maintain and to break, but I feel like I'm a little out of step with how far we want bin/setup to go (it already seems to go pretty far)

Govspeak will not build

Steps to reproduce

Join our plucky cast as they attempt to run:
make govspeak

Stack trace

And encounter the mysterious case of the missing Argument

/home/huw/govuk/govuk-docker/exe/govuk-docker run govspeak-lite rbenv install -s || (/home/huw/govuk/govuk-docker/exe/govuk-docker build --no-cache govspeak-lite; /home/huw/govuk/govuk-docker/exe/govuk-docker run govspeak-lite rbenv install -s)
docker-compose -f [...] run govspeak-lite rbenv install -s
/home/huw/govuk/govuk-docker/exe/govuk-docker run govspeak-lite sh -c 'gem install --conservative --no-document bundler -v $(grep -A1 "BUNDLED WITH" Gemfile.lock | tail -1)'
docker-compose -f [...] run govspeak-lite sh -c gem install --conservative --no-document bundler -v $(grep -A1 "BUNDLED WITH" Gemfile.lock | tail -1)
grep: Gemfile.lock: No such file or directory
ERROR:  While executing gem ... (OptionParser::MissingArgument)
    missing argument: -v
make: *** [Makefile:41: bundle-govspeak] Error 1

Seems related to:
#389

Implies #396
May not be a complete fix.

`govuk-docker doctor` says "Dnsmasq not found"

Running govuk-docker doctor:

❌ Dnsmasq not found.
You should install it with `brew install dnsmasq`.
For a manual installation, visit http://www.thekelleys.org.uk/dnsmasq/doc.html

But running brew install dnsmasq:

Warning: dnsmasq 2.80 is already installed and up-to-date
To reinstall 2.80, run `brew reinstall dnsmasq`

Dnsmasq is installed on my machine but not being found in the doctor check.

content-store no longer builds after switching from MongoDB to PostgreSQL

The content-store repository recently switched from MongoDB to PostgreSQL. The problem is that it cannot connect to PostgreSQL, because of the missing environment variable specifying that connection.

Problem replicating publishing-api to local machine

I'm working on something that relies on publishing-api to provide a content object from a content and attachment id, so I need to replicate the publishing-api database locally. If I run the replicate command:

gds aws govuk-integration-readonly --art 6h ./bin/replicate-postgresql.sh publishing-api

(--art 6h to allow the 20Gb file to download)

...it gets the file, then runs through a huge list of docker containers trying to stop them, spins up the PG 13.1 container but then fails to do the import with the error:

Error response from daemon: OCI runtime start failed: cannot start a container that has stopped: unknown

(More details below, from the start of step 2 after all the containers have been stopped/removed)

docker-compose -f [...] up -d postgres-13
[+] Running 2/2
 ⠿ Network govuk-docker_default          Created                                                                                                                                           0.0s
 ⠿ Container govuk-docker-postgres-13-1  Started                                                                                                                                           0.3s
waiting for postgres...
docker-compose -f [...] config
docker-compose -f [...] run postgres-13 /usr/bin/psql -h postgres-13 -U postgres -c DROP DATABASE IF EXISTS "publishing-api"
DROP DATABASE
docker-compose -f [...] run postgres-13 /usr/bin/createdb -h postgres-13 -U postgres publishing-api
docker-compose -f [...] run postgres-13 /usr/bin/pg_restore -h postgres-13 -U postgres -d publishing-api --no-owner --no-privileges
^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347372^@
^@^@^@access_limits^@^E^@^@^@TABLE^@^B^@^@^@^@%^A^@^@CREATE TABLE public.access_limits (
    id integer NOT NULL,
    created_at timestamp without time zone NOT NULL,
    updated_at timestamp without time zone NOT NULL,
    edition_id integer,
    users jsonb DEFAULT '[]'::jsonb NOT NULL,
    organisations jsonb DEFAULT '[]'::jsonb NOT NULL
);
^@!^@^@^@DROP TABLE public.access_limits;
^A^A^@^@^@^@^F^@^@^@public^@^@^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347380^@^T^@^@^@access_limits_id_seq^@^H^@^@^@SEQUENCE^@^B^@^@^@^@}^@^@^@CREATE SEQUENCE public.access_limits_id_seq
    START WITH 1
    INCREME^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347412^@^P^@^@^@documents_id_seq^@^H^@^@^@SEQUENCE^@^B^@^@^@^@y^@^@^@CREATE SEQUENCE public.documents_id_seq
    START WITH 1
    INCREMENT BY 1
   ^C^@^@^@^@^@^@^@^@^@?^O^@^@^@^@^@^@^@^@^A^@^@^@0^@^A^@^@^@0^^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347414^@^H^@^@^@editions^@^E^@^@^@TABLE^@^B^@^@^@^@^R
^@^@^@1259^@	^@^@^@217347414^@^H^@^@^@editions^@^E^@^@^@TABLE^@^B^@^@^@^@^E^@^@CREATE TABLE public.editions (
    id integer NOT NULL,
    title text,
    publi^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^^@^@^@expanded_links_id_seq^@^H^@^@^@SEQUENCE^@^B^@^@^@^@~^@^@^@CREATE SEQUENCE public.expa^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@^@^@^@217347468^@^L^@^@^@links_id_seq^@^H^@^@^@SEQUENCE^@^B^@^@^@^@u^@^@^@CREATE SEQUENCE public.links_id_seq
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;^C^@^@^@^@^@^@^@^@^@?^O^@^@^@^@^@^@^@^@^A^@^@^@0^@^A^@^@^@0^@^L^@^@^@links_id_seq^@^C^@^@^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347476^@^X^@^@^@path_reservations_id_seq^@^H^@^@^@SEQUENCE^@^B^@^@^@^@?^@^@^@CREATE SEQUENCE public.path_reservations_id_seq
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347478^@^@^@^@schema_migrations^@^E^@^@^@TABLE^@^B^@^@^@^@R^@^@^@CREATE TABLE public.schema_migrations (
    version character varying NOT NULL
^C^@^@^@^@^@^@^@^@^@?^O^@^@^@^@^@^@^@^@^A^@^@^@0^@^A^@^@^@0^@^T^@^@^@unpublishings_id_seq^@^C^@^@^@^@^@^@^@^@^@?^@^@^@^@^@^@^@^@^@^@^@^@1259^@	^@^@^@217347492^@^E^@^@^@users^@^E^@^@^@TABLE^@^B^@^@^@^@?^A^@^@CREATE TABLE public.users (
    id integer NOT NULL,
    name character varying,
    email character varying,
    uid character varying,
    organisation_slug character varying,
    organisation_content_id character varying,
    app_name character varying,
    permissions text,
    remotely_signed_out boolean DEFAULT false,
    disabled boolean DEFAULT false,
    crea^C^@^@^@^@^@^@^@^@^@?^O^@^@^@^@^@^@^@^@^A^@^@^@0^@^A^@^@^@0^@^L^@^@^@users_id_seq^@^C^@^@^@^@^@^@^@^@^@?^N^@^@^@^@^@^@^@^@^@^@^@2604^@	^@^@^@217347503^@
^@^@^@actions id^@^G^@^@^@DEFAULT^@^B^@^@^@^@h^@^@^@ALTER TABLE ONLY public.actions ALTER COLUMN id SET DEFAULT nextval('public.actions_id_seq'::regclass);
^@9^@^@^@ALTER TABLE p^C^@^@^@^@^@^@^@^@^@?^N^@^@^@^@^@^@^@^@^@^@^@2604^@	^@^@^@217347506^@^K^@^@^@editions id^@^G^@^@^@DEFAULT^@^B^@^@^@^@j^@^@^@ALTER TABLE ONLY public.editions ALTER COLUMN id SET DEFAULT nextval('public.editions_id_seq'::regclass);
^@:^@^@^@ALTER TABLE^C^@^@^@^@^@^@^@^@^@?^N^@^@^@^@^@^@^@^@^@^@^@2604^@	^@^@^@217347508^@^@^@^@expanded_links id^@^G^@^@^@DEFAULT^@^B^@^@^@^@v^@^@^@ALTER TABLE ONLY public.expan^C^@^@^@^@^@^@^@^@^@?^O^@^@^@^@^@^@^@^@^A^@^@^@0^@^A^@^@^@0^@^P^@^@^@documents_id_seq^@^L^@^@^@SEQU^^C^@^@^@^@<^@^@^@SELECT pg_catalog.setval('pub^C^@^@^@^@^@^@^@^@^@^O^@^@^@^@^@^@^@^@^@^@^@2606^@	^@^@^@222875031^@^X^@^@^@documents documents_pkey^@
^@^@^@CONSTRAINT^@^@^@^@^@V^@^@^@ALTER TABLE ONLY public.documents
    ADD CONSTRAINT documents_pkey PRIMARY KEY (id);
^@B^@^@^@^C^@^@^@^@^@^@^@^@^@^N^O^@^@^@^@^@^@^@^@^@^@^@^\^@^@^@index_editions_on_updated_at^@^E^@^@^@INDEX^@^@^@^@^@W^@^@^@CREATE INDEX index_editions_on_updated_at ON public.??^He[?TX{Uy?? ????L??   ??^O^F?6M???!????2???@j???k?#??i?֭^T^\<F8^A?-<?@^C?|^T<KA?m?^R
?|^T<KA?m???qe^P(ğ^T<??????^Z?j^Z?9_?^C??^Z?^H?h??6^G)@?A
??O?m?V?,bd|?R^Z_?R?v????Ԥ?^B?Q%?г?
Ӌ?^Y?U??[o???tܽМ_?R>?I?G?_????!6<٠B?^@^@^P^@^@??^G????U??y?X]?AE^@?b^_??_j?^A????^^!??(;?A?^\?B]27bd??&?Cp!??@^B??<?
jH^R
jH?^R
jH???'y?&?^[cl?7<:?01$!'??^N?*?w??~(^E^[?Ryc?΅(U??P?W??D?_䧦\?^X]b?AW?qx^L?^K?WP?????ʡ?^R
?[??7?B%?H?/^Z?^Y)dǶ%?!Ц^\?XR?UR?GZ*??Z??^TN?p?Ϛέ?^\U??"?.+?II??                                                                                                                              Zٔ ????G?D|p/?-^P?׿?^[????^R
????G?D|p/?-^P?׿?^[?????0??eؠ?$?^[?@
??                                                      
j?Q?/?^F.G?^N???a?
jR?x0?X??U??(^KW??I?6?I?^]E??2T?K'O?<dA"???t^Z?4???b?=^X?Q?????N*??%4h?o_??	j?<^F?*?|Mr????^]?^O?????D????Zu2^LB??\??????D?B?K?r5???;?A]?;^A?רXy!?"^^? ?OError response from daemon: OCI runtime start failed: cannot start a container that has stopped: unknown
 128KiB 0:00:00 [ 182KiB/s] [>                                                                                                                                                 ]  0%            
docker-compose -f [...] stop postgres-13
[+] Running 5/5
 ⠿ Container govuk-docker-postgres-13-1                 Stopped                                                                                                                            0.2s
 ⠿ Container govuk-docker_postgres-13_run_b2cf0b1c4b99  Stopped                                                                                                                            0.0s
 ⠿ Container govuk-docker_postgres-13_run_4bd59df1a7a6  Stopped                                                                                                                            0.0s
 ⠿ Container govuk-docker_postgres-13_run_1764a2258737  Stopped                                                                                                                            0.0s
 ⠿ Container govuk-docker_postgres-13_run_1f71fb33b0c6  Stopped

MySQL 8 container crashing

Nothing obvious in why this happens, and I'm running Whitehall against it but it'll work ok for a bit and then just shut. Logs don't show anything useful:

Attaching to govuk-docker_mysql-8_1
mysql-8_1                               | 2022-01-26 10:12:43+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26 10:12:43+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mysql-8_1                               | 2022-01-26 10:12:43+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26T10:12:44.716721Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27) starting as process 1
mysql-8_1                               | 2022-01-26T10:12:44.824462Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-8_1                               | 2022-01-26T10:12:45.588340Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql-8_1                               | 2022-01-26T10:12:46.176760Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
mysql-8_1                               | 2022-01-26T10:12:46.190993Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
mysql-8_1                               | 2022-01-26T10:12:46.254257Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T10:12:46.254325Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T10:12:46.262567Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql-8_1                               | 2022-01-26T10:12:46.262727Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql-8_1                               | 2022-01-26T10:12:46.267732Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mysql-8_1                               | 2022-01-26T10:12:46.346044Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
mysql-8_1                               | 2022-01-26T10:12:46.346280Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | 2022-01-26 15:34:05+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26 15:34:06+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mysql-8_1                               | 2022-01-26 15:34:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26T15:34:06.646950Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27) starting as process 1
mysql-8_1                               | 2022-01-26T15:34:06.667793Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-8_1                               | 2022-01-26T15:34:07.482439Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql-8_1                               | 2022-01-26T15:34:08.066329Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
mysql-8_1                               | 2022-01-26T15:34:08.076439Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
mysql-8_1                               | 2022-01-26T15:34:08.170263Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T15:34:08.170324Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T15:34:08.175524Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql-8_1                               | 2022-01-26T15:34:08.175774Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql-8_1                               | 2022-01-26T15:34:08.179023Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mysql-8_1                               | 2022-01-26T15:34:08.263484Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
mysql-8_1                               | 2022-01-26T15:34:08.263560Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | mbind: Operation not permitted
mysql-8_1                               | 2022-01-26 16:05:22+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26 16:05:23+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mysql-8_1                               | 2022-01-26 16:05:23+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26T16:05:23.837576Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27) starting as process 1
mysql-8_1                               | 2022-01-26T16:05:23.995042Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-8_1                               | 2022-01-26T16:05:24.789074Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql-8_1                               | 2022-01-26 16:11:07+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26 16:11:07+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mysql-8_1                               | 2022-01-26 16:11:07+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26T16:11:08.091119Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27) starting as process 1
mysql-8_1                               | 2022-01-26T16:11:08.159346Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-8_1                               | 2022-01-26T16:11:08.730307Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql-8_1                               | 2022-01-26T16:11:10.451006Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
mysql-8_1                               | 2022-01-26T16:11:10.486955Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
mysql-8_1                               | 2022-01-26T16:11:21.669591Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T16:11:21.671668Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T16:11:22.089113Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql-8_1                               | 2022-01-26T16:11:22.202066Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql-8_1                               | 2022-01-26T16:11:22.743651Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mysql-8_1                               | 2022-01-26 16:31:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26 16:31:53+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
mysql-8_1                               | 2022-01-26 16:31:53+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.27-1debian10 started.
mysql-8_1                               | 2022-01-26T16:31:53.941218Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.27) starting as process 1
mysql-8_1                               | 2022-01-26T16:31:54.028684Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql-8_1                               | 2022-01-26T16:31:54.650976Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql-8_1                               | 2022-01-26T16:31:56.197887Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
mysql-8_1                               | 2022-01-26T16:31:56.221086Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
mysql-8_1                               | 2022-01-26T16:32:04.087143Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T16:32:04.105629Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
mysql-8_1                               | 2022-01-26T16:32:04.440798Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql-8_1                               | 2022-01-26T16:32:04.479445Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql-8_1                               | 2022-01-26T16:32:04.688937Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
mysql-8_1                               | 2022-01-26T16:32:19.932435Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
mysql-8_1                               | 2022-01-26T16:32:19.956338Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.27'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
govuk-docker_mysql-8_1 exited with code 137

Foreign key constraint failure in collections-publisher-lite db:setup

On a first-ever run of make collections-publisher, I get this error:

docker-compose -f [...] run collections-publisher-lite bin/rake db:setup
Creating govuk-docker_collections-publisher-lite_run ... done
Created database 'collections_publisher_development'
Created database 'collections_publisher_test'
/home/al/govuk/govuk-docker/exe/govuk-docker run collections-publisher-lite env RAILS_ENV=test bin/rake db:setup
docker-compose -f [...] run collections-publisher-lite env RAILS_ENV=test bin/rake db:setup
Creating govuk-docker_collections-publisher-lite_run ... done
Database 'collections_publisher_test' already exists
rake aborted!
ActiveRecord::InvalidForeignKey: Mysql2::Error: Cannot delete or update a parent row: a foreign key constraint fails
/govuk/collections-publisher/db/schema.rb:25:in `block in <top (required)>'
/govuk/collections-publisher/db/schema.rb:13:in `<top (required)>'
Caused by:
Mysql2::Error: Cannot delete or update a parent row: a foreign key constraint fails
/govuk/collections-publisher/db/schema.rb:25:in `block in <top (required)>'
/govuk/collections-publisher/db/schema.rb:13:in `<top (required)>'
Tasks: TOP => db:schema:load
(See full trace by running task with --trace)

I've worked around it using @kevindew 's suggestion in this Slack thread -

run govuk-docker run collections-publisher-lite bin/rails db:drop
comment out the second db:setup invocation for the test environment on line 3 of the collections-publisher Makefile
re-run make collections-publisher

but it would be good to fix the issue for the next new starter

`govuk-docker doctor` dnsmasq check fails on comments in /etc/resolver/...

If you have something like this in /etc/resolver/dev.gov.uk:

# some comment
nameserver 127.0.0.1

...then govuk-docker doctor fails with:

❌ Something else (possibly Vagrant-dns) is configured to resolve DNS requests for dev.gov.uk.
          The /etc/resolver/dev.gov.uk file needs to be updated to only contain
          nameserver 127.0.0.1

It should ignore comments, as we can save new joiners a few moments of confusion by not erroring on comments. Comments were in /etc/resolver/dev.gov.uk by default on mine and @AgaDufrat's machines, so I suspect a lot of devs come across this.

Running tests in frontend and collections does not work

Two unrelated issues prevent the tests (e.g. govuk-docker run frontend-lite bundle exec rake) from being run.

Issue 1

Branch names were changed from master to main. Because of that pact tests fail.

I had to add an environment variable -e PACT_CONSUMER_VERSION=branch-main when running govuk docker for tests.

Issue 2

Recent updates to selenium prevent running tests which require it. Selenium now automatically downloads the browsers that it uses, which generates an error related to version mismatch.

I had to remove the downloaded browser govuk-docker run --rm collections-lite rm -Rf /root/.cache/selenium and then use an environment variable -e SE_AVOID_BROWSER_DOWNLOAD=true when running govuk-docker for tests so it is not downloaded again.

Compatibility documentation incorrect for some apps

The compatiblity doc should be corrected:

Clarify what support there is for "frontend".
Clarify what support there is for "whitehall".
Clarify what support there is for "calculators".

DNS resolution strategy incompatible with Rails 6

Arises from #174.

Applications running Rails 6 have "Host Authorization" middleware to prevent against DNS rebinding attacks. This means that only requests from 0.0.0.0, ::, and localhost are allowed by default.

To get the custom alias domains (such as http://collections-publisher.dev.gov.uk/) to work, applications will either need to:

Explicitly add the local domain to its config.hosts (as has been done in link-checker-api), or
Clear the entire whitelist (config.hosts.clear), preferably on local dev only. This lets through requests for all hostnames.

Test DB not created when running "rake db:setup"

This isn't specific to GOV.UK Docker, but seems to affect all repos running Rails 6.1+. A workaround for this is to run:

govuk-docker-run env RAILS_ENV=test bundle exec rake db:setup

While it's something of a regression, it appears to be intentional and related to this Rails issue. Next steps:

Find the change in Rails that lead to this change in behaviour. Hopefully this will lead to documentation about the recommended approach, or we could refer to the mainstream docs.
Agree the fix and apply this to each affected app. For example, we may agree to add the above workaround to each Makefile. Ideally we should preserve the old behaviour, as that was simpler.

Documentation error in README.md?

https://github.com/alphagov/govuk-docker/blob/main/README.md?plain=1#L93

appears to not be how the cli works now.

It says that you need to put in the name of the app when using the up command

govuk-docker-up imminence-app

...but for the apps I've tried so far (imminence, feedback) appear to automatically supply the app name, so doing it that way fails (it tries to bring up imminence-imminence-app). What works is:

govuk-docker-up app

Make Pull doesn't work for generic-ruby-libraries or main branches

Describe the issue

To reproduce run:
make pull

For generic-ruby-library you now get:
generic-ruby-library directory not found

Also for any repository using main over master it will fail.

Expected behaviour

Ideally it would follow up with generic-ruby-libaries and clone all those separately.
It's still important those codebases are up to date i think.

GDS has committed to switching from master to main` in all our repositories, we should prep for that change or allow both.

Fixture file based tests fail due to Linux Kernel issue

People have experienced problems running the test suites of Content Publisher and Whitehall (I'm not aware of other apps but there could be more) using govuk-docker. It transpires this issue is a problem with the Linux kernel, in particular the version that Docker-for-mac is using currently (on my machine it's 5.10.47-linuxkit) but I'm recording this here so the GOV.UK issue is documented and can be tracked to a degree.

The evidence of the problem is tests that fail like so:

     MiniMagick::Invalid:
       `identify /tmp/mini_magick20211206-1-7orhba.jpg` failed with error:
       identify-im6.q16: insufficient image data in file `/tmp/mini_magick20211206-1-7orhba.jpg' @ error/jpeg.c/ReadJPEGImage_/1166.

Error:
ImageUploaderTest#test_should_store_all_the_versions_of_a_bitmap_image_in_asset_manager:
CarrierWave::ProcessingError: Failed to manipulate with MiniMagick, maybe it is not an image? Original Error: `convert /govuk/whitehall/tmp/1321009871-957757668381030-0168-8098/s960/minister-of-funk.960x640.jpg -auto-orient -resize 960x640^ -gravity Center -background rgba(255,255,255,0.0) -extent 960x640 /tmp/image_processing20111111-428-11fn1kn.jpg` failed with error:
convert-im6.q16: insufficient image data in file `/govuk/whitehall/tmp/1321009871-957757668381030-0168-8098/s960/minister-of-funk.960x640.jpg' @ error/jpeg.c/ReadJPEGImage_/1166.
convert-im6.q16: no images defined `/tmp/image_processing20111111-428-11fn1kn.jpg' @ error/convert.c/ConvertImageCommand/3229.

The source of the issue is a relatively obscure set of circumstances: a file is stored on a docker mount (i.e. a test fixture), it is then copied to be a temporary file (i.e. a test faking the upload of a file) and then is copied again (i.e copying the uploaded file for processing).

The issue is discussed (at length) in: docker/for-linux#1015. It looks like versions of Linux Kernel > 5.10 are fixed and it appears that a backported fix for 5.10 is imminent, however it's completely unclear how long it will take for any fixes to make their way into a Docker-for-mac releases.

The two options I see ahead of us for this are:

Wait it out, expect that a future release of Docker Desktop for mac resolves it
Refactor test code that is affected by this issue to no longer be affected by the circumstances - there seems to be some examples based off the issue

By default, we're effectively doing 1) now - I imagine if this issue continues for multiple months we may switch to taking 2)

Build steps can fail when services aren't ready

Just jotting this down while it's fresh in my mind.

It's quite common for an application build to fail when dependencies haven't started fast enough. A couple of examples of this are: build elasticsearch indexes - elasticsearch seems to be a slow service to start, and publishing routes - Publishing API is often not ready.

A common approach to resolve this problem in the docker compose world are using a wait script, Docker itself doesn't provide a solution for reasons which don't really apply to a setup. A couple of examples of the scripts are wait-for-it and dockerize. In the publishing-e2e-tests this is achieved with compose healthchecks and ruby loops.

Router development is not currently possible in govuk-docker

The main branch of govuk-docker is now M1-compatible, and part of that is removing the mongo-2.6 container and pointing router to mongo-3.6. This will work for general use, but the router-3.6 container in govuk-docker is not set up to use a replica set, which docker relies on to detect updates to the data. This will causes failures in the following situations:

developers actively working on router will not be able to run the test suite
developers using the stack in govuk-docker who publish new content and expect it to appear in their local dev environment through router (rather than directly accessed) will not see that new content.

This issue is related in part to this one: alphagov/router#210

Developers needing a workaround should try this govuk-docker branch which sets mongo-3.6 to use the replica set:
https://github.com/alphagov/govuk-docker/tree/m1-compatibility-keith

Alternatively, work in this router branch will render this replica-set requirement moot when complete:
https://github.com/alphagov/router/tree/change-stream

(This issue repeated in router since it's not 100% clear where it should be, since the fix is in both places: alphagov/router#267)

Packages removed from apt-get repos preventing builds

It appears that dependencies in three files were removed from the apt-get repositories. After extensive testing locally it appears that these are no longer needed at all, and a pull request removing them has been made here. However this issue is posted for discussion and logging.

The error in question occurs in Asset Manager and Whitehall, and results in this failure:

------                                                                          
> [ 2/12] RUN apt-get update -qq && apt-get install -y libxss1 libappindicator1 libindicator7:                                                                 
#5 10.39 Reading package lists...                                               
#5 10.78 Building dependency tree...                                            
#5 10.89 Reading state information...
#5 10.96 E: Unable to locate package libappindicator1
#5 10.96 E: Unable to locate package libindicator7
------
executor failed running [/bin/sh -c apt-get update -qq && apt-get install -y libxss1 libappindicator1 libindicator7]: exit code: 100
ERROR: Service 'whitehall-lite' failed to build : Build failed

Removing the associated requirements for libxss1 libappindicator1 libindicator7 from three files; Dockerfile.govuk-base projects/asset-manager/Dockerfile and projects/whitehall/Dockerfile fixes the issue. That is what the pull request does in turn.

People who have these repos cached will not experience this issue, however if someone was to build an app they had previously not cached, or reset their docker environment, they would be unable to continue.

To clear the cache and test the problem (and the fix) for yourself, run docker builder prune in the govuk-docker folder.

Unable to select option in make, seeing ^M when hitting enter

Running make support, I'd get:

/Users/christopher.ashton/govuk/govuk-docker/bin/branch_checks.sh support
Fetching recent updates for support
You are not currently on the main branch for support (your branch: dependabot/bundler/redis-5.0.5)
Choose an option
1) Checkout main
2) Ignore
3) Quit
#? 2^M

It just wouldn't take my chosen input of '2'.

I tried a simple select option in "First" "Second" "Third"; do echo "You chose $REPLY"; done (a simplified derivative of the code that was failing for me in govuk-docker) directly in my terminal and that worked, so I thought this was either an issue with the branch_checks.sh file, or it not being compatible with my ZSH setup.

Eventually overwrote the entire file with that minimal example, and realised that was failing, i.e. the bash select functionality worked directly on the CLI but always failed when run from a script.

Did a lot of Googling and eventually found the answer here: https://unix.stackexchange.com/a/580621

TLDR: running stty sane fixed the issue for me.

Documenting here in case it comes in handy for someone in the future.

Cannot replicate collection-publisher's MySQL database

Actual

~/govuk $ cd govuk-docker
~/govuk/govuk-docker $ grep adapter ../collections-publisher/config/database.yml
  adapter: mysql2
~/govuk/govuk-docker $ gds aws govuk-integration-poweruser ./bin/replicate-mysql.sh collections-publisher
~/govuk/govuk-docker $ gds aws govuk-integration-poweruser ./bin/replicate-mysql.sh collections-publisher
Enter MFA code for <REMOVED>: <REMOVED>
Replicating mysql for collections-publisher
couldn't figure out backup filename in S3 - if you're sure the app uses MySQL, file an issue in alphagov/govuk-docker.

Expected

Mysql database populated.

Comments

I am a new developer for GDS, so I might now know that it is not supposed to work, so my assumption it should might be wrong for some reason, please let me know. Replication works for other databases I use.

Router API cannot connect to Mongo 2.6

We've encountered a chaining problem when looking into how publishing-api tries to put things onto the rabbitMQ, which we've traced:
Publishing API -> Content Store -> Router API -> Router

The problem seems to be that router-api cannot find a server:

To replicate:

➜  router-api git:(main) govuk-docker-run bundle exec rails c
docker-compose -f [...] run router-api-lite bundle exec rails c
Creating govuk-docker_router-api-lite_run ... done
Loading development environment (Rails 6.0.3.7)
irb(main):001:0> Route.count
Traceback (most recent call last):
        1: from (irb):1
Mongo::Error::NoServerAvailable (No primary server is available in cluster: #<Cluster topology=Unknown[mongo-2.6:27017] servers=[#<Server address=mongo-2.6:27017 GHOST>]> with timeout=30, LT=0.015)

The mongo container does run, and you can watch logs though it is in a big loop of opening and closing connections punctuated by the following failry suspect message:

2021-12-03T16:06:57.809+0000 [rsStart] warning: getaddrinfo("48703775aaf0") failed: Name or service not known
2021-12-03T16:06:57.846+0000 [rsStart] getaddrinfo("48703775aaf0") failed: Name or service not known
2021-12-03T16:06:57.846+0000 [rsStart] replSet info Couldn't load config yet. Sleeping 20sec and will try again.

@kevindew spotted that if we comment out this line things start working again.

That seems to have been introduced during work to to resolve differences in how rs.status responds between mongo v.2.6 (which router runs in prod) and more modern versions.

#499

This may have been an attempt to resolve this issue: alphagov/router#210

Question to answer: what was L46 trying to resolve? Does it still serve that purpose? Can we replace it with something doesn't block local dev, or remove it altogether?

Should we have some automated build testing?

Was chatting with @injms and thinking perhaps, once a week we pull this repo and a bot / github action tries to run make all-apps and then reports back in some way if any build fails?

Strikes me that there's a few images in here that are lesser used, but might be returned to.
Especially whilst we're in a world of pausing projects to jump onto pressing national concerns, it would be good if there was a way to monitor which of the images have rotted away whilst we weren't looking?

Is this something we could do with a github action?

"bin/doctor" requires Ruby and Gems to be installed

Background

We refer to this utility in our troubleshooting guidance, but using it requires someone to install Rbenv, Ruby and Gems. This can take a long time if someone doesn't already have the required version of Ruby installed on their system. Having to do all these steps also makes it likely the user will need support with the support utility 😆.

Suggestions

Since most of the checks are just simple shell commands, we could rewrite the script in bash to avoid the need for Ruby / Gems. Alternatively, we could raise a PR, or conduct an informal poll, to remove the utility: we should prioritise fixing problems at source, and empowering people to fix harder problems themselves, with guidance.

M1 Laptops cannot run all features of govuk-docker

An issue to track findings on running govuk-docker on M1 Laptops.

Current issues:

When initially running a 'make ' the make fails when trying to install google-chrome-stable. This can be replaced with chromium to get past the initial step, but tests runners later may rely on google-chrome-stable and cause failures in test suites.
MySQL versions below 8 are not available in arm64 architectures (possibly replace with MariaDB?)
MongoDB versions below 3.6 are not available in arm64 architectures (so router and router-api cannot be run. 3.6 is probably a safe replacement, since we're going to try to upgrade the production database to that anyway)

MySQL 8.0 changes default auth plugin from `mysql_native_password` to `caching_sha2_password`

#550 documents a workaround by altering the root users config to set it back to mysql_native_password which lets things build again.

We should try integrate a solution into govuk-docker itself, either by placing this command into the makefile.
Or seeing if we can meet the config requirements for mysql_native_password.

The Plugin docs are here

GDS API Adapters fails to build

In this episode of Will it build:

Steps to reproduce

Join our plucky cast as they attempt to run:
make gds-api-adapters

Stack trace

And encounter the mysterious case of the missing Argument

/home/huw/govuk/govuk-docker/exe/govuk-docker run gds-api-adapters-lite rbenv install -s || (/home/huw/govuk/govuk-docker/exe/govuk-docker build --no-cache gds-api-adapters-lite; /home/huw/govuk/govuk-docker/exe/govuk-docker run gds-api-adapters-lite rbenv install -s)
docker-compose -f [...] run gds-api-adapters-lite rbenv install -s
/home/huw/govuk/govuk-docker/exe/govuk-docker run gds-api-adapters-lite sh -c 'gem install --conservative --no-document bundler -v $(grep -A1 "BUNDLED WITH" Gemfile.lock | tail -1)'
docker-compose -f [...] run gds-api-adapters-lite sh -c gem install --conservative --no-document bundler -v $(grep -A1 "BUNDLED WITH" Gemfile.lock | tail -1)
grep: Gemfile.lock: No such file or directory
ERROR:  While executing gem ... (OptionParser::MissingArgument)
    missing argument: -v
make: *** [Makefile:41: bundle-gds-api-adapters] Error 1

Also note

So... I suspect this is because the central Makefile has an expectation there'll be a Gemfile.lock coming with each package.
But in this case gds-api-adapters appears to be a gem itself.

So i guess in this case something more custom in the makefile, (or perhaps another task that focuses on the steps for Gems?) is the route?

Branch Checks fail if a branch does not have a remote

If you've created a new branch and haven't yet pushed it, the branch check will allow you to ignore that it isn't main, but will then immediately crash when trying to compare it to the origin version.

(To test: go to any app, create a new branch for it, then make the app in govuk-docker and select "ignore" when it tells you that you're not making main).

It should just skip over the comparison in this case.

Errno::ECONNREFUSED when the publishing-api Makefile runs the setup_exchange Rake task

I'm trying to get the whitehall app set up for the first time on an M1 Mac, but I ran into the following error:

~/govuk/govuk-docker/exe/govuk-docker run publishing-api-lite bin/rake setup_exchange
docker-compose -f [...] run publishing-api-lite bin/rake setup_exchange
[+] Running 3/0
 ⠿ Container govuk-docker-postgres-13-1  Running                                                                               0.0s
 ⠿ Container govuk-docker-rabbitmq-1     Running                                                                               0.0s
 ⠿ Container govuk-docker-redis-1        Running                                                                               0.0s
W, [2023-03-20T14:44:17.712951 #1]  WARN -- #<Bunny::Session:0x1f798 guest@rabbitmq:5672, vhost=/, addresses=[rabbitmq:5672]>: Could not establish TCP connection to rabbitmq:5672: Connection refused - connect(2) for 172.18.0.6:5672
rake aborted!
Bunny::TCPConnectionFailedForAllHosts: Could not establish TCP connection to any of the configured hosts
/govuk/publishing-api/lib/tasks/setup_exchange.rake:6:in `block in <main>'

Caused by:
Bunny::HostListDepleted: No more hosts to try in the supplied list of hosts
/govuk/publishing-api/lib/tasks/setup_exchange.rake:6:in `block in <main>'

Caused by:
Bunny::TCPConnectionFailed: Could not establish TCP connection to rabbitmq:5672: Connection refused - connect(2) for 172.18.0.6:5672
/govuk/publishing-api/lib/tasks/setup_exchange.rake:6:in `block in <main>'

Caused by:
Errno::ECONNREFUSED: Connection refused - connect(2) for 172.18.0.6:5672
/govuk/publishing-api/lib/tasks/setup_exchange.rake:6:in `block in <main>'
Tasks: TOP => setup_exchange
(See full trace by running task with --trace)
make: *** [publishing-api] Error 1

The error seems to be triggered by this line:

https://github.com/alphagov/publishing-api/blob/30dfc8355e957dd546c755af35322ca1ba4defb0/lib/tasks/setup_exchange.rake#L6

Which in turn was triggered by this line:

govuk-docker/projects/publishing-api/Makefile

Line 5 in 6c45979

$(GOVUK_DOCKER) run $@-lite bin/rake setup_exchange

Which in turn was ultimately triggered by me running the make whitehall.

I can't see anything relevant in the troubleshooting guide and, although there are some mentions of problems with M1 Macs in a couple of issues, none of them seem immediately relevant.

Can anyone help?

search-api fails to build

Steps to reproduce

make search-api

Stack trace

I, [2020-08-06T09:39:59.460022 #1]  INFO -- sentry: ** [Raven] Raven 3.0.0 configured not to capture errors: DSN not set
rake aborted!
Gem::ConflictError: Unable to activate sidekiq-5.2.9, because redis-4.2.1 conflicts with redis (>= 3.3.5, < 4.2)
/govuk/search-api/lib/rummager.rb:27:in `<top (required)>'
/govuk/search-api/Rakefile:8:in `<top (required)>'

Caused by:
Gem::ConflictError: Unable to activate sidekiq-5.2.9, because redis-4.2.1 conflicts with redis (>= 3.3.5, < 4.2)
/govuk/search-api/lib/rummager.rb:27:in `<top (required)>'
/govuk/search-api/Rakefile:8:in `<top (required)>'

Caused by:
LoadError: cannot load such file -- sidekiq
/govuk/search-api/lib/rummager.rb:27:in `<top (required)>'
/govuk/search-api/Rakefile:8:in `<top (required)>'
(See full trace by running task with --trace)
make: *** [/home/huw/govuk/govuk-docker/projects/search-api/Makefile:4: search-api] Error 1

Also note

As search API is a dependency of finder-frontend that also cannot fully build until this is resolved.

Thoughts

This is a head scratchers for me...
The Gemfile.lock in search-api demands:

   sidekiq (5.2.9)
      connection_pool (~> 2.2, >= 2.2.2)
      rack (~> 2.0)
      rack-protection (>= 1.5.0)
      redis (>= 3.3.5, < 4.2)

Does <4.2 also exclude any minor versions other than .0?
I've tried hacking around in the Gemfile.lock to see if I can force sidekiq to take a compatable version... I also can't see anywhere in either our docker compose files or ruby gems where we're specifying a version of Redis...

Has this one cropped up elsewhere too?

Why was I running this?

¯\_(ツ)_/¯ wanted to see what would happen if I ran make all-apps, this was one of the things that happened

contacts-admin fails to build

Steps to reproduce

make contacts-admin

Stack trace

/home/huw/govuk/govuk-docker/exe/govuk-docker run contacts-admin-lite sh -c 'bin/rake db:migrate 2>/dev/null || bin/rake db:create db:migrate'
docker-compose -f [...] run contacts-admin-lite sh -c bin/rake db:migrate 2>/dev/null || bin/rake db:create db:migrate
Starting govuk-docker_mysql_1 ... done
Starting govuk-docker_redis_1 ... done
Database 'contacts_development' already exists
Database 'contacts_test' already exists
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Directly inheriting from ActiveRecord::Migration is not supported. Please specify the Rails release the migration was written for:

  class CreateContacts < ActiveRecord::Migration[4.2]
/govuk/contacts-admin/db/migrate/20130814072057_create_contacts.rb:1:in `<main>'

Caused by:
StandardError: Directly inheriting from ActiveRecord::Migration is not supported. Please specify the Rails release the migration was written for:

  class CreateContacts < ActiveRecord::Migration[4.2]
/govuk/contacts-admin/db/migrate/20130814072057_create_contacts.rb:1:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
make: *** [/home/huw/govuk/govuk-docker/projects/contacts-admin/Makefile:2: contacts-admin] Error 1

Thoughts

Interesting, I would have thought this would have come up in the contacts app by now if this was failing, also seems that in whitehall we have classes that inherit from ActiveRecord::Migration without a version, and i don't think this error is occuring there... a red herring perhaps?

Can anyone else give this a go and verify this is happening for them too?

Notes

Why was I running this?
¯\_(ツ)_/¯ wanted to see what would happen if I ran make all-apps, this was one of the things that happened

alphagov/static fails to start when starting alphagov/publisher

When trying to start alphagov/publisher using the command govuk-docker-up, the start-up process proceeds until recreating alphagov/static. It fails with the following output:

Recreating govuk-docker_static-app_1 ... error

ERROR: for govuk-docker_static-app_1  Cannot start service static-app: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"bin/rails\": stat bin/rails: no such file or directory": unknown

ERROR: for static-app  Cannot start service static-app: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"bin/rails\": stat bin/rails: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.

Any idea where to start with resolving this?

Unable to specify stacks that don't have `app-` prefix

E.g. govuk-content-schemas:

$govuk-docker startup
Unknown stack: app.

Available stacks:

lite


$govuk-docker startup lite

Unknown stack: app-lite.

Available stacks:

lite

Have tried passing --stack=lite which used to work but has now been removed from govuk-docker.

Link-Checker-API not working inside Collections Publisher

Inside collections-publisher:

govuk-docker startup e2e

Then navigate to http://collections-publisher.dev.gov.uk/step-by-step-pages and create or view a step by step and then click "Check for broken links"; we get a 503 error.

~~This happens even if you run govuk-docker startup inside link-checker-api in another tab.~~ As of alphagov/link-checker-api#284, you can now have link-checker-api running in another tab and this is available to collections-publisher, fixing the issue. But we shouldn't have to do this - the e2e stack should account for the link-checker-api dependency and resolve it automatically.

Generic Ruby Library will not build (and shouldn't try to!)

Steps to reproduce

make all-apps

Stack trace

Issue

We've talked about removing all-apps in the past.
It currently tries to build the following list:

asset-manager authenticating-proxy cache-clearing-service calculators collections collections-publisher contacts-admin content-data-admin content-data-api content-publisher content-store content-tagger datagovuk_publish email-alert-api email-alert-frontend email-alert-service feedback finder-frontend frontend generic-ruby-library government-frontend govuk_crawler_worker govuk-developer-docs govuk_publishing_components hmrc-manuals-api imminence info-frontend licence-finder link-checker-api local-links-manager manuals-frontend manuals-publisher mapit maslow miller-columns-element publisher publishing-api release router-api router search-admin search-api service-manual-frontend service-manual-publisher short-url-manager signon smart-answers specialist-publisher special-route-publisher static support-api support transition travel-advice-publisher whitehall

So it will try and build generic-ruby-library.

However running make generic-ruby-library fails because the makefile there has a set of more specifc rules.

Solution

We should remove all-apps or ensure that it does not include generic-ruby-library or provide a separate make step to cover everything in that makefile.

NB: This also means that the five generic ruby files do not appear in ALL_APPS, if we're trying to test builds that way.

finder-frontend fails to build without publishing-api running

Impact: Low
Workaround: Run publishing-api and static docker containers before building and running finder-frontend

After the implementation of the dynamic list work, finder-frontend will error when using govuk-docker build unless publishing-api is already built and running prior:

/Users/jonathonshire/govuk/govuk-docker/bin/govuk-docker compose run search-api-setup rake publishing_api:publish_eu_exit_business_finder
docker-compose -f [...] run search-api-setup rake publishing_api:publish_eu_exit_business_finder
Starting govuk-docker_postgres_1 ... done
Starting govuk-docker_nginx-proxy-app_1 ... done
Starting govuk-docker_mongo_1           ... done
Starting govuk-docker_rabbitmq_1        ... done
Starting govuk-docker_redis_1           ... done
Starting govuk-docker_router-app-draft_1 ... done
Starting govuk-docker_router-api-app-draft_1 ... done
Creating govuk-docker_router-app_1              ... done
Creating govuk-docker_content-store-app-draft_1 ... done
Creating govuk-docker_router-api-app_1          ... done
Creating govuk-docker_content-store-app_1       ... done
Creating govuk-docker_publishing-api-worker_1   ... done
Creating govuk-docker_publishing-api-app-e2e_1  ... done
I, [2019-10-23T12:00:59.342136 #1]  INFO -- sentry: ** [Raven] Raven 2.11.3 configured not to capture errors: DSN not set
I, [2019-10-23T12:01:01.635643 #1]  INFO -- sentry: ** [Raven] Raven 2.11.3 configured not to capture errors: DSN not set
I, [2019-10-23T12:01:03.719706 #1]  INFO -- : Publishing Find Brexit guidance for your business (a finder_email_signup content item)
rake aborted!
GdsApi::HTTPBadGateway: URL: http://publishing-api.dev.gov.uk/v2/content/2818d67a-029a-4899-a438-a543d5c6a20d
Response body:
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.17.3</center>
</body>
</html>

It appears we're not running the app via the finder-frontend call at this point, which is resulting in the 502. Manually starting publishing-api before attempting to build and run finder-frontend solves the problem.

It's likely just a small fix involving what starts up in the build process, I just don't have time to look at it right now.

alphagov / govuk-docker Goto Github PK

govuk-docker's Introduction

GOV.UK Docker

Installation

Usage

The lite stack

The app stack

The app-* stacks

Resources

Contributing

Licence

govuk-docker's People

Contributors

Stargazers

Watchers

Forkers

govuk-docker's Issues

Steps to reproduce

Stack trace

Steps to reproduce

Stack trace

Issue 1

Issue 2

Describe the issue

Expected behaviour

Actual

Expected

Comments

Background

Suggestions

Steps to reproduce

Stack trace

Also note

Steps to reproduce

Stack trace

Also note

Thoughts

Why was I running this?

Steps to reproduce

Stack trace

Thoughts

Notes

Steps to reproduce

Stack trace

Issue

Solution

Recommend Projects

Recommend Topics

Recommend Org

The `lite` stack

The `app` stack

The `app-*` stacks