circlesubi / circles-docker Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 2.0 349 KB

Infrastructure provisioning for Circles development

License: GNU Affero General Public License v3.0

Makefile 36.53% Shell 48.68% Dockerfile 3.54% JavaScript 11.26%

development-tools docker provisioning

circles-docker's People

Contributors

Stargazers

Watchers

Forkers

circlesland carlfairclough

circles-docker's Issues

Check the pathfinder2 docker image used in tests

Jon fixed the docker builds after we moved the repository from chris. There are now current images available again: https://hub.docker.com/r/circlesubi/pathfinder2/tags

circles-core tests passing

Integrate pathfinder2 services in docker setup

Create these services: blockchain-indexer, database, pathfinder-updater, pathfinder2 from https://github.com/CirclesUBI/circles-platform/blob/main/ansible/roles/apps/templates/docker-compose.yaml.j2 in the docker setup.

Align pathfinder versions to what is production

For the new services that we integrated and are needed for the pathfinder service, we are using the dev version. We need to change it to reflect the specific versions that are used in prod.

Right now we use dev bc we assume this is what it's in prod. Ref: https://github.com/CirclesUBI/land-local/blob/main/modes/from-image/docker-compose.yml

But we'll use version ids when we integrate the services in our IAC.

Optional docker-compose configuration for small production setups

For other communities to run their own Circles platforms with a simple docker-compose setup.

Requirements:

LetsEncrypt certificates
Change containers to run with production env vars

This already existed in an older version of this repository: https://github.com/CirclesUBI/circles-docker/tree/b539d24508736537d1db74da1f038aaac251752a

In logs: "Cannot store the access token because dependencies are missing"

Describe the bug
Logs from make subgraph:

> [email protected] deploy
> ./scripts/deploy.sh

graph deploy CirclesUBI/circles-subgraph --ipfs http://localhost:5001 --node http://localhost:8020 /.../circles-docker/.tmp/subgraph/.subgraph.tmp.yaml
Could not get access token from libsecret (usually gnome-keyring or ksecretservice): 

Cannot store the access token because dependencies are missing. If you
are on Linux, try installing 'libsecret-1-dev' (Debian, Ubuntu etc.) or
'libsecret-devel' (RedHat, Fedora etc.) and reinstalling Graph CLI
afterwards.

The original error was: Cannot find module 'keytar'
Require stack:
- /.../circles-docker/.tmp/subgraph/node_modules/@graphprotocol/graph-cli/src/command-helpers/auth.js
- /.../circles-docker/.tmp/subgraph/node_modules/@graphprotocol/graph-cli/src/commands/auth.js
- /.../circles-docker/.tmp/subgraph/node_modules/gluegun/build/loaders/module-loader.js
- /.../circles-docker/.tmp/subgraph/node_modules/gluegun/build/loaders/command-loader.js
- /.../circles-docker/.tmp/subgraph/node_modules/gluegun/build/runtime/runtime.js
- /.../circles-docker/.tmp/subgraph/node_modules/gluegun/build/domain/builder.js
- /.../circles-docker/.tmp/subgraph/node_modules/gluegun/build/index.js
- /.../circles-docker/.tmp/subgraph/node_modules/@graphprotocol/graph-cli/src/cli.js
- /.../circles-docker/.tmp/subgraph/node_modules/@graphprotocol/graph-cli/bin/graph
      
Continuing without an access token

To Reproduce
Steps to reproduce the behavior:

deploy docker containers:

make down
git pull
make clean
make up EXPOSE_PORTS=1
make subgraph

See logs of the last command

Expected behavior
I expected no warnings.

Desktop-Computer:

OS: Debian 10

Substitute ganache-cli by ganache

https://github.com/trufflesuite/ganache

Update node version for all repositories, and test the integration

This issue will be solved by several PRs:

Integration of the subgraph updated to node 16 and deployed as a docker image: #77
circles-contracts v3.3.2: https://github.com/CirclesUBI/circles-contracts/releases/tag/v3.3.2
circles-core v4.5.0: https://github.com/CirclesUBI/circles-core/releases/tag/v4.5.0
circles-api v1.9.0: https://github.com/CirclesUBI/circles-api/releases/tag/v1.9.0
db migration needed? Not in docker

Do not expose Relayer static pages on production

Graph Node crashes when using Version >=0.15.1

See graphprotocol/graph-node#1378

Consider updating postgres:13-alpine

Or at leat the same versin we have in prod

Graph Node version 0.31.0 seems to build incorrectly

Updated version of graph seems to generate problems with building.
Using node v16.20.1
Previous version of graph works fine

Compress backups

Use caddy to remove the etc/hosts entries

Reference: https://hub.docker.com/_/caddy

Suggested by Land team

Error when deploying local subgraph

When deploying the subgraph in local the next error appears:

Cloning into '.tmp/subgraph'...
HEAD is now at 7e20c22 1.4.0
Installing npm dependencies ..

> [email protected] codegen /home/runner/work/circles-core/circles-core/circles-docker/.tmp/subgraph
> ./scripts/codegen.sh

graph codegen --output-dir ./src/types /home/runner/work/circles-core/circles-core/circles-docker/.tmp/subgraph/.subgraph.tmp.yaml
./scripts/common.sh: line 80: ./node_modules/.bin/graph: No such file or directory

Add backup system

Daily moving the current postgres data to an external (S3) instance ala Borg.

Revisit staging environment

Update graph-node

When updating to graph-node v.0.24.2, erros in the logs:

thread '<unnamed>' panicked at 'failed to connect notification listener to Postgres: Error { kind: ConfigParse, cause: Some(InvalidValue("port")) }', store/postgres/src/notification_listener.rs:131:22
stack backtrace:
   0:     0x55ff5bebee20 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he1640d5f0d93f618
   1:     0x55ff5bee32dc - core::fmt::write::h88012e1f01caeebf
   2:     0x55ff5beb7bd5 - std::io::Write::write_fmt::h360fa85b30182555
   3:     0x55ff5bec104b - std::panicking::default_hook::{{closure}}::h84fe124cd0864662
   4:     0x55ff5bec0b21 - std::panicking::default_hook::h5a8e74a76ce290a7
   5:     0x55ff5bec1714 - std::panicking::rust_panic_with_hook::h67c812a4fe9d4c91
   6:     0x55ff5bec11f7 - std::panicking::begin_panic_handler::{{closure}}::h33f9c1b96af300d7
   7:     0x55ff5bebf31c - std::sys_common::backtrace::__rust_end_short_backtrace::h51bae64be5921f0e
   8:     0x55ff5bec1159 - rust_begin_unwind
   9:     0x55ff5a911091 - core::panicking::panic_fmt::h12a3a3c256485fca
  10:     0x55ff5a911183 - core::result::unwrap_failed::h2d8d0952e3250de9
  11:     0x55ff5ad3e487 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::he8094a7b91ab0aab
  12:     0x55ff5adf46ad - std::sys_common::backtrace::__rust_begin_short_backtrace::hea80286588b72259
  13:     0x55ff5ad92ffe - core::ops::function::FnOnce::call_once{{vtable.shim}}::ha3813d852ee359a1
  14:     0x55ff5bec55f7 - std::sys::unix::thread::Thread::new::thread_start::hbba5bc368baac205
  15:     0x7f272cb5cfa3 - start_thread
  16:     0x7f272c9034cf - clone
  17:                0x0 - <unknown>

update graph-node version in dockers

Acceptance Criteria:
Core test passing

Ipfs container exits immediately

Logs from the container show:

Changing user to ipfs
ipfs version 0.4.23
Found IPFS fs-repo at /data/ipfs
Initializing daemon...
go-ipfs version: 0.4.23-6ce9a35
Repo version: 7
System version: amd64/linux
Golang version: go1.12.16

Error: no version file found, please run 0-to-1 migration tool.
See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.

It's fixed by deleting all containers (docker system prune -a and docker volume prune) but reappears again eventually on a future make up

Populate news activities in database

insert into news (message_en, date, "iconId", "isActive") values ('News 1', '2022-01-01', 12131232, true);
      insert into news (message_en, date, "iconId", "isActive") values ('News 2', '2022-02-02', 12131232, false);
      insert into news (message_en, date, "iconId", "isActive") values ('News 3', '2022-03-03', 12131232, true);
      insert into news (message_en, date, "iconId", "isActive") values ('News 4', '2022-04-10', 12131232, false);
      insert into news (message_en, date, "iconId", "isActive") values ('News 5', '2022-05-11', 12131232, true);
      insert into news (message_en, date, "iconId", "isActive") values ('News 6', '2022-06-12', 12131232, false);
      insert into news (message_en, date, "iconId", "isActive") values ('News 7', '2022-07-20', 12131232, true);
      insert into news (message_en, date, "iconId", "isActive") values ('News 8', '2022-08-21', 12131232, false);
      insert into news (message_en, date, "iconId", "isActive") values ('News 9', '2022-09-22', 12131232, true);
      insert into news (message_en, date, "iconId", "isActive") values ('News 10', '2022-09-30', 12131232, true);
      ```

Revisit CORS settings

Requests to servers should only be accepted from trusted origins

Improve docker build times

Some ideas:

Move all docker images from Digital Ocean to a public repository (we have to change all the tagbuild.yaml github actions and change the image urls in the circles-iac Helm charts)
Use these images in our circles-docker repo instead, they are already built and faster to use
Change the Makefile to support a different API, something like:

# This will build your local versions of the circles-api and safe-relay-service
make build api relayer

# This will use all pre-built images from the public repository
make build