Graphql Apollo service failure under Ubuntu 20.04 LTS about server HOT 2 CLOSED

@fdlind Yeah I don't expect the apollo service to start up without the rest of the stack running (it's the top layer that requires the graphql service which requires postgres etc.) When you run prefect server start do you see the migrations being applied to the Postgres database? Wondering if somehow they're being missed or are unable to run. There might also be a port issue where some service is not able to be created due to something using the default port. You could try setting different ports for the prefect server start command and that may resolve it if there's a conflict:

--postgres-port     TEXT    Port used to serve Postgres, defaults to '5432'
--hasura-port       TEXT    Port used to serve Hasura, defaults to '3001'
--graphql-port      TEXT    Port used to serve the GraphQL API, defaults to '4001'
--ui-port           TEXT    Port used to serve the UI, defaults to '8080'
--server-port       TEXT    Port used to serve the Core server, defaults to '4200'

from server.

Hi,

Make a clean conda install per standard instructions. Tried different port combinations from the command line as above. Doesn't seem to actually change the server ports. Also edited the toml config file manually and that will change ports but it doesn't fix the issue.

The trace associated with the clean conda install and server startup with an alternate graphql port is :

(av_prefect2) av_developer@ubuntu:~$ prefect server start --graphql-port 3100
Pulling postgres ... done
Pulling hasura ... done
Pulling graphql ... done
Pulling apollo ... done
Pulling scheduler ... done
Pulling ui ... done
Creating network "prefect-server" with the default driver
Creating tmp_postgres_1 ... done
Creating tmp_hasura_1 ... done
Creating tmp_graphql_1 ... done
Creating tmp_scheduler_1 ... done
Creating tmp_apollo_1 ... done
Creating tmp_ui_1 ... done
Attaching to tmp_postgres_1, tmp_hasura_1, tmp_graphql_1, tmp_scheduler_1, tmp_apollo_1, tmp_ui_1
apollo_1 |
apollo_1 | > @ serve /apollo
apollo_1 | > node dist/index.js
apollo_1 |
graphql_1 |
graphql_1 | Running Alembic migrations...
graphql_1 | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
graphql_1 | INFO [alembic.runtime.migration] Will assume transactional DDL.
graphql_1 | INFO [alembic.runtime.migration] Running upgrade -> 723da8052cbd, create extensions
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 723da8052cbd -> 2f249ccdcba7, set utc timezone
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 2f249ccdcba7 -> 7ff37edbf446, Initial migration
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 7ff37edbf446 -> 6c48e68b5e97, Add task run trigger
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6c48e68b5e97 -> aa594a91210c, Add run timing details trigger
graphql_1 | INFO [alembic.runtime.migration] Running upgrade aa594a91210c -> 8666a0ceb70e, Add trigger to update runs when states are inserted
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 8666a0ceb70e -> 6b0389a57a0a, Add updated trigger to all tables
graphql_1 | INFO [alembic.runtime.migration] Running upgrade 6b0389a57a0a -> 7e65dadba625, fix log default timestamp
graphql_1 | Error: HTTPConnectionPool(host='hasura', port=3000): Max retries exceeded with url: /v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5989db8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
graphql_1 |
graphql_1 | Could not upgrade the database!
hasura_1 | {"type":"startup","timestamp":"2020-07-15T21:58:58.629+0000","level":"info","detail":{"kind":"server_configuration","info":{"live_query_options":{"batch_size":100,"refetch_delay":1},"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","plan_cache_options":{"plan_cache_size":100},"enabled_log_types":["http-log","websocket-log","startup","webhook-log"],"server_host":"HostAny","enable_allowlist":false,"log_level":"info","auth_hook_mode":null,"use_prepared_statements":true,"unauth_role":null,"stringify_numeric_types":false,"enabled_apis":["metadata","graphql","config","pgdump"],"enable_telemetry":true,"enable_console":true,"auth_hook":null,"jwt_secret":null,"cors_config":{"allowed_origins":"","disabled":false,"ws_read_cookie":null},"console_assets_dir":null,"admin_secret_set":false,"port":3000}}}
hasura_1 | {"internal":"could not connect to server: Connection refused\n\tIs the server running on host "postgres" (172.31.0.2) and accepting\n\tTCP/IP connections on port 5432?\n","path":"$","error":"connection error","code":"postgres-error"}
postgres_1 | The files belonging to this database system will be owned by user "postgres".
postgres_1 | This user must also own the server process.
postgres_1 |
postgres_1 | The database cluster will be initialized with locale "en_US.utf8".
postgres_1 | The default database encoding has accordingly been set to "UTF8".
postgres_1 | The default text search configuration will be set to "english".
postgres_1 |
postgres_1 | Data page checksums are disabled.
postgres_1 |
postgres_1 | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1 | creating subdirectories ... ok
postgres_1 | selecting default max_connections ... 100
postgres_1 | selecting default shared_buffers ... 128MB
postgres_1 | selecting default timezone ... Etc/UTC
postgres_1 | selecting dynamic shared memory implementation ... posix
postgres_1 | creating configuration files ... ok
postgres_1 | running bootstrap script ... ok
postgres_1 | performing post-bootstrap initialization ... ok
postgres_1 | syncing data to disk ... ok
postgres_1 |
postgres_1 | WARNING: enabling "trust" authentication for local connections
postgres_1 | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1 | --auth-local and --auth-host, the next time you run initdb.
postgres_1 |
postgres_1 | Success. You can now start the database server using:
postgres_1 |
postgres_1 | pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1 |
postgres_1 | waiting for server to start....2020-07-15 21:58:58.417 UTC [45] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-07-15 21:58:58.442 UTC [46] LOG: database system was shut down at 2020-07-15 21:58:58 UTC
postgres_1 | 2020-07-15 21:58:58.450 UTC [45] LOG: database system is ready to accept connections
postgres_1 | done
postgres_1 | server started
postgres_1 | CREATE DATABASE
postgres_1 |
postgres_1 |
postgres_1 | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/
postgres_1 |
postgres_1 | waiting for server to shut down...2020-07-15 21:58:58.832 UTC [45] LOG: received fast shutdown request
postgres_1 | .2020-07-15 21:58:58.834 UTC [45] LOG: aborting any active transactions
postgres_1 | 2020-07-15 21:58:58.835 UTC [45] LOG: background worker "logical replication launcher" (PID 52) exited with exit code 1
postgres_1 | 2020-07-15 21:58:58.836 UTC [47] LOG: shutting down
scheduler_1 | [2020-07-15 21:59:01,520] INFO - prefect-server.Scheduler | Scheduler will start after an initial delay of 264 seconds...
postgres_1 | 2020-07-15 21:58:58.848 UTC [45] LOG: database system is shut down
postgres_1 | done
postgres_1 | server stopped
postgres_1 |
postgres_1 | PostgreSQL init process complete; ready for start up.
postgres_1 |
postgres_1 | 2020-07-15 21:58:58.953 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres_1 | 2020-07-15 21:58:58.953 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres_1 | 2020-07-15 21:58:58.965 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1 | 2020-07-15 21:58:58.981 UTC [63] LOG: database system was shut down at 2020-07-15 21:58:58 UTC
postgres_1 | 2020-07-15 21:58:58.996 UTC [1] LOG: database system is ready to accept connections
postgres_1 | 2020-07-15 21:59:00.483 UTC [71] ERROR: duplicate key value violates unique constraint "pg_extension_name_index"
postgres_1 | 2020-07-15 21:59:00.483 UTC [71] DETAIL: Key (extname)=(pgcrypto) already exists.
postgres_1 | 2020-07-15 21:59:00.483 UTC [71] STATEMENT: CREATE EXTENSION IF NOT EXISTS pgcrypto SCHEMA public
ui_1 | 👾👾👾 UI running at localhost:8080 👾👾👾
apollo_1 | 2020-07-15T21:59:02.760Z Error fetching GraphQL health: FetchError: request to http://graphql:4201/health failed, reason: connect ECONNREFUSED 172.31.0.4:4201
apollo_1 | 2020-07-15T21:59:02.761Z Error: Could not safely build a schema!
apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11)
apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7) Could not safely build a schema! Error: Could not safely build a schema!
apollo_1 | at safelyBuildSchema (/apollo/dist/index.js:129:11)
apollo_1 | at process._tickCallback (internal/process/next_tick.js:68:7)
apollo_1 | 2020-07-15T21:59:02.762Z
apollo_1 | Trying again in 3 seconds...
apollo_1 |
apollo_1 | 2020-07-15T21:59:05.772Z Building schema...
apollo_1 | 2020-07-15T21:59:06.464Z Building schema complete!
apollo_1 | Server ready at http://0.0.0.0:4200 🚀
apollo_1 | 2020-07-15T21:59:06.577Z Sending telemetry to Prefect Technologies, Inc.: {"source":"prefect_server","type":"startup","payload":{"id":"009531e9-baa7-42f6-8ba0-266a47db7dd1"}}

from server.