Giter VIP home page Giter VIP logo

memphis-k8s's Introduction

Github (2)

Discord Code Of Conduct GitHub release (latest by date)

CNCF Silver Member CNCF Silver Member

Memphis.dev is a highly scalable, painless, and effortless data streaming platform.
Made to enable developers and data teams to collaborate and build
real-time and streaming apps fast.

Memphis Kubernetes Deployment

Helm is a k8s package manager that allows users to deploy apps in a single, configurable command.
More information about Helm can be found here.

Memphis is cloud-native and cloud-agnostic to any Kubernetes on any cloud.

Requirements

Minimum Requirements (Without high availability)

ResourceQuantity
Minimum Kubernetes version1.20 and above
K8S Nodes1
CPU2 CPU
Memory4GB RAM
Storage12GB PVC

Recommended Requirements (With high availability)

Resource Minimum Quantity
Minimum Kubernetes version 1.20 and above
K8S Nodes 3
CPU 4 CPU
Memory 8GB RAM
Storage 12GB PVC Per node

Installation

Production

Production-grade Memphis with three memphis brokers configured in cluster-mode

helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && helm install memphis memphis/memphis --set global.cluster.enabled="true" --create-namespace --namespace memphis --wait
Dev

Standard installation of Memphis with a single broker

helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && 
helm install memphis memphis/memphis --create-namespace --namespace memphis --wait

* Optional * Helm deployment options

Option Description Default Value Example
global.cluster.enabled Cluster mode for HA and Performance "false" "false"
exporter.enabled Prometheus exporter "false" "false"
exporter.serviceExposed.enbaled Expose metrics port with memphis service "false" "true"
cluster.enabled Enables Memphis cluster deployment. For fully HA configuration use global.cluster.enabled "false" "true"
cluster.replicas Memphis broker replicas "3" "5"
memphis.image Memphis image name "memphisos/memphis:x.x.x-stable" "memphisos/memphis:latest"
memphis.ui.port Dashboard's (GUI) port 9000 9000
memphis.hosts.uiHostName Which URL should be seen as the "UI hostname" "" "https://memphis.example.com"
memphis.hosts.restgwHostName Which URL should be seen as the "REST Gateway hostname" "" "https://restgw.memphis.example.com"
memphis.hosts.brokerHostName Which URL should be seen as the "broker hostname" "" "memphis.example.com"
memphis.configFile.logsRetentionInDays Amount of days to retain system logs 3 3
memphis.configFile.gcProducerConsumerRetentionInHours Amount of hours to retain producer/consumer in system 3 3
memphis.configFile.tieredStorageUploadIntervalSeconds Interval in seconds between uploads to tiered storage 8 8
memphis.configFile.dlsRetentionHours Amount of hours to retain messages in DLS 3 3
memphis.configFile.userPassBasedAuth Authentication method selector.
true = User + pass
false = User + connection token		 "true" "true"
memphis.creds.rootPwd Root password for the dashboard. Randomly generated. "" "superpass"
memphis.creds.connectionToken Token for connecting an app to the Memphis Message Queue. Auto generated.Randomly generated. "" "connectionToken
memphis.creds.jwtSecret For internal traffic. Randomly generated. "" "<JWT_TOKEN>"
memphis.creds.refreshJwtSecret For internal traffic. Randomly generated. "" "<JWT_TOKEN>"
memphis.creds.encryptionSecretKey Encryption secret key for internal encryption. Randomly generated. "" ""
memphis.creds.secretConfig.name *Optional*
Name of the secret		 "memphis-creds" "memphis-creds"
memphis.creds.secretConfig.existingSecret *Optional*
Is this secret an existing secret		 "false" "false"
memphis.creds.secretConfig.rootPwd_key *Optional*
Name of the key in secret		 "ROOT_PASSWORD" "rootPwd"
memphis.creds.secretConfig.connectionToken_key *Optional*
Name of the key in secret		 "CONNECTION_TOKEN" "connectionToken"
memphis.creds.secretConfig.jwtSecret_key *Optional*
Name of the key in secret		 "JWT_SECRET" "jwtSecret"
memphis.creds.secretConfig.refreshJwtSecret_key *Optional*
Name of the key in secret		 "REFRESH_JWT_SECRET" "refreshJwtSecret"
memphis.creds.secretConfig.encryptionSecretKey_key *Optional*
Name of the key in secret		 "ENCRYPTION_SECRET_KEY" "encryptionSecretKey"
memphis.creds.secretConfig.refreshJwtSecretRestGW_key *Optional*
Name of the key in secret		 "REFRESH_JWT_SECRET_REST_GW" "refreshJwtSecretRestGW"
memphis.creds.secretConfig.jwtSecretRestGW_key *Optional*
Name of the key in secret		 "JWT_SECRET_REST_GW" "jwtSecretRestGW"
memphis.extraEnvironmentVars.enabled *Optional*
List of additional environment variables for memphis.		 "" vars:
- name: KEY
- valye: value
memphis.tls.verify *Optional*
For encrypted client-memphis communication. Verification for the CA autority. SSL.		 "" "true"
memphis.tls.secret.name *Optional*
For encrypted client-memphis communication.
K8S secret name that holds the certs. SSL.		 "" "memphis-client-tls-secret"
memphis.tls.cert *Optional*
For encrypted client-memphis communication.
.pem file to use. SSL.		 "" "memphis_client.pem"
memphis.tls.key *Optional*
For encrypted client-memphis communication.
Private key file to use. SSL.		 "" "memphis-key_client.pem"
memphis.tls.ca *Optional*
For encrypted client-memphis communication.
CA file to use. SSL.		 "" "rootCA.pem"
websocket.enabled *Optional* Memphis GUI using websockets for live rendering. "" "false"
websocket.port Memphis GUI using websockets for live rendering. The port can be configured "7770" "77777"
websocket.host Websocket host can be handled on separate LB/DNS. "localhost" "ws.example.com"
websocket.noTLS Websocket can be configured with tls, default is noTLS. "true" "false"
websocket.tls.secret.name *Optional* Memphis GUI using websockets for live rendering.
K8S secret name for the certs		 "" "memphis-ws-tls-secret"
websocket.tls.cert *Optional*
Memphis GUI using websockets for live rendering.
.pem file to use		 "" "memphis_local.pem"
websocket.tls.key *Optional*
Memphis GUI using websockets for live rendering.
key file		 "" "memphis-key_local.pem"
metadata.postgresql.username *Optional*
Username for postgres db		 "postgres" "postgres"
metadata.postgresql.existingSecret *Optional*
An ability to provide predefined secret for metadata PostgreSQL credentials		 "" "metadata-creds.yaml"
metadata.pgpool.existingSecret *Optional*
An ability to provide predefined secret for metadata PG credentials		 "" "metadata-creds.yaml"
metadata.pgpool.tls.enabled *Optional*
Enabling TLS-based communication with PG		 "false" "false"
metadata.pgpool.tls.certificatesSecret *Optional*
PG TLS cert secret to be used		 "" "tls-secret"
metadata.pgpool.tls.certFilename *Optional*
PG TLS cert file to be used		 "" "tls.crt"
metadata.pgpool.tls.certKeyFilename *Optional*
PG TLS key to be used		 "" "tls.key"
metadata.pgpool.tls.certCAFilename *Optional*
PG TLS cert CA to be used		 "" "ca.crt"
metadata.external.enabled *Optional*
For using external PG instead of deploying dedicated one for Memphis		 "false" "true"
metadata.external.dbTlsMutual *Optional*
External PG TLS-basec communication		 "true" "true"
metadata.external.dbName *Optional*
External PG db name		 "" "memphis"
metadata.external.dbHost *Optional*
External PG db hostname		 "" "metadata.example.url"
metadata.external.dbPort *Optional*
External PG db port		 "" 5432
metadata.external.dbUser *Optional*
External PG db user		 "" "postgres"
metadata.external.dbPass *Optional*
External PG db password		 "" "12345678"
metadata.external.secret.enabled *Optional*
Enable an option to use secret for password store		 "false" "true"
metadata.external.secret.name *Optional*
Secret name		 "" "metadata-secret"
metadata.external.secret.dbPass_key *Optional*
Name of the key in the secret		 "" "dbPass"
restGateway.enabled *Optional*
Memphis Rest Gateway can be disabled if not in use		 "true" "false"
restGateway.jwtSecret *Optional*
Manual Jwt Token configurtion		 "" ""
restGateway.refreshJwtSecret *Optional*
Manual Refresh Jwt Token configurtion		 "" ""
auth.enabled *Optional*
Enable using predefined parameters		 "false" "true"
auth.enabled.mgmt *Optional*
Management users that will be created at first deployment		 "" ""
auth.enabled.client *Optional*
Client users that will be created at first deployment		 "" ""
Here is how to run an installation command with additional options - 
helm install memphis --set cluster.replicas=3,memphis.creds.rootPwd=rootpassword" memphis/memphis --create-namespace --namespace memphis

Deployed pods

  • memphis. Memphis broker.
  • memphis-rest-gateway. Memphis REST Gateway.
  • memphis-metadata. Metadata store.
  • memphis-metadata-coordinator. Metadata coordinator

For more information on each component, please head to the architecture section.

Deploy Memphis with TLS (encrypted communication via SSL)

0. Optional: Create self-signed certificates

a) Generate a self-signed certificate using mkcert

$ mkcert -client \
-cert-file memphis_client.pem \
-key-file memphis-key_client.pem  \
"127.0.0.1" "localhost" "*.memphis.dev" ::1 \
email@localhost [email protected]

b) Find the rootCA

$ mkcert -CAROOT

c) Create self-signed certificates for client

$ mkcert -client -cert-file client.pem -key-file key-client.pem  localhost ::1

1. Create namespace + secret for the TLS certs

a) Create a dedicated namespace for memphis

kubectl create namespace memphis

b) Create a k8s secret with the required certs

kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
tls:
  secret:
    name: memphis-client-tls-secret
  ca: "rootCA.pem"
  cert: "memphis_client.pem"
  key: "memphis-key_client.pem"

2. Deploy Memphis with the generated certificate

helm install memphis memphis \
--create-namespace --namespace memphis --wait \
--set \
global.cluster.enabled="true",\
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="memphis-client-tls-secret",\
memphis.tls.ca="rootCA.pem"

Upgrade existing deployment

For adding TLS support

  1. Create a k8s secret with the provided TLS certs
kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
  1. Upgrade Memphis to use the TLS certs
helm upgrade memphis memphis -n memphis --reuse-values \
--set \
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="tls-client-secret",\
memphis.tls.ca="rootCA.pem"

Deployment diagram

Memphis Architecture (1)

memphis-k8s's People

Contributors

avitaltrifsik avatar bjwschaap avatar bkochauri-memphis avatar elchinmemphis avatar idanasulin2706 avatar lp-francois avatar petemac88 avatar saarryan avatar shohamroditimemphis avatar valerabr avatar yanivbh1 avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

saarryan kfirfer vishalsingh17 gastromatic petemac88 iqqbot valerabr winn-solutions madestreel fredericve lp-francois bjwschaap

memphis-k8s's Issues

ServiceMonitor resource endpoint spec is using port instead of targetPort

According to the endpoint spec of a ServiceMonitor resource either of port: or targetPort: need to be used. In the case of port: it should reference the name of a port defined in the Service. In the case of targetPort: it should reference a port of a pod behind the service.

Since the metrics port is not defined in the Service it leads to a prometheus operator not finding what to monitor. In order not to expose the exporter port in the service I suggest to use targetPort: instead.

Helm deployment fails on K8S Cluster

Please let me know what went wrong.

command: helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && helm install memphis memphis/memphis --create-namespace --namespace memphis --wait

  • kubectl get all -n memphis
NAME                                                READY   STATUS             RESTARTS          AGE
pod/memphis-0                                       0/2     Pending            0                 12h
pod/memphis-metadata-0                              0/1     Pending            0                 12h
pod/memphis-metadata-coordinator-6f5b7c9b59-qhwd7   0/1     CrashLoopBackOff   197 (3m34s ago)   12h
pod/memphis-rest-gateway-649774875b-bfmdl           0/1     Init:0/1           0                 12h

NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                               AGE
service/memphis                        ClusterIP   None             <none>        9000/TCP,7770/TCP,6666/TCP,8222/TCP   12h
service/memphis-metadata               ClusterIP   10.107.188.205   <none>        5432/TCP                              12h
service/memphis-metadata-coordinator   ClusterIP   10.108.3.182     <none>        5432/TCP                              12h
service/memphis-metadata-headless      ClusterIP   None             <none>        5432/TCP                              12h
service/memphis-rest-gateway           ClusterIP   10.106.37.203    <none>        4444/TCP                              12h

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/memphis-metadata-coordinator   0/1     1            0           12h
deployment.apps/memphis-rest-gateway           0/1     1            0           12h

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/memphis-metadata-coordinator-6f5b7c9b59   1         1         0       12h
replicaset.apps/memphis-rest-gateway-649774875b           1         1         0       12h

NAME                                READY   AGE
statefulset.apps/memphis            0/1     12h
statefulset.apps/memphis-metadata   0/1     12h
  • kubectl -n memphis logs deployment.apps/memphis-metadata-coordinator
pgpool 23:00:59.79
pgpool 23:00:59.79 Welcome to the Bitnami pgpool container
pgpool 23:00:59.80 Subscribe to project updates by watching https://github.com/bitnami/containers
pgpool 23:00:59.80 Submit issues and feature requests at https://github.com/bitnami/containers/issues
pgpool 23:00:59.80
pgpool 23:00:59.80 INFO  ==> ** Starting Pgpool-II setup **
pgpool 23:00:59.83 INFO  ==> Validating settings in PGPOOL_* env vars...
pgpool 23:00:59.85 INFO  ==> Initializing Pgpool-II...
pgpool 23:00:59.86 INFO  ==> Generating pg_hba.conf file...
pgpool 23:00:59.86 INFO  ==> Generating pgpool.conf file...
pgpool 23:01:00.03 INFO  ==> Generating password file for local authentication...
pgpool 23:01:00.04 INFO  ==> Generating password file for pgpool admin user...

pgpool 23:01:00.05 INFO  ==> ** Pgpool-II setup finished! **
pgpool 23:01:00.08 INFO  ==> ** Starting Pgpool-II **
2024-01-23 23:01:00.094: main pid 1: LOG:  Backend status file /opt/bitnami/pgpool/logs/pgpool_status does not exist
2024-01-23 23:01:00.094: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.094: main pid 1: LOG:  memory cache initialized
2024-01-23 23:01:00.094: main pid 1: DETAIL:  memcache blocks :64
2024-01-23 23:01:00.094: main pid 1: LOG:  allocating (144190784) bytes of shared memory segment
2024-01-23 23:01:00.094: main pid 1: LOG:  allocating shared memory segment of size: 144190784
2024-01-23 23:01:00.225: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.225: main pid 1: LOG:  health_check_stats_shared_memory_size: requested size: 12288
2024-01-23 23:01:00.226: main pid 1: LOG:  memory cache initialized
2024-01-23 23:01:00.226: main pid 1: DETAIL:  memcache blocks :64
2024-01-23 23:01:00.228: main pid 1: LOG:  pool_discard_oid_maps: discarded memqcache oid maps
2024-01-23 23:01:00.234: main pid 1: LOG:  unix_socket_directories[0]: /opt/bitnami/pgpool/tmp/.s.PGSQL.5432
2024-01-23 23:01:00.234: main pid 1: LOG:  listen address[0]: *
2024-01-23 23:01:00.234: main pid 1: LOG:  Setting up socket for 0.0.0.0:5432
2024-01-23 23:01:00.235: main pid 1: LOG:  Setting up socket for :::5432
2024-01-23 23:01:00.241: main pid 1: LOG:  find_primary_node_repeatedly: waiting for finding a primary node
2024-01-23 23:01:00.244: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.244: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:00.246: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.246: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:00.248: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:00.248: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:01.251: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.251: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:01.253: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.253: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:01.258: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:01.258: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:02.261: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.261: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:02.263: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.263: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:02.264: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:02.264: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:03.267: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:03.268: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:06.821: child pid 151: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:06.821: child pid 151: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:06.821: child pid 151: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:08.271: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:08.271: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:08.273: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:08.273: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:08.273: main pid 1: LOG:  reaper handler
2024-01-23 23:01:08.274: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:09.276: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.276: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:09.278: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.278: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:09.279: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:09.280: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:10.282: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.282: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:10.283: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.283: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:10.285: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:10.285: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:11.288: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.288: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:11.290: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.290: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:11.291: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.291: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:11.814: child pid 141: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:11.814: child pid 141: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:11.814: child pid 141: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:11.815: main pid 1: LOG:  reaper handler
2024-01-23 23:01:11.816: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:12.294: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.294: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:12.296: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.296: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:12.297: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:12.297: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:13.300: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.300: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:13.301: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.301: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:13.307: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:13.307: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:14.310: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.310: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:14.311: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.311: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:14.313: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:14.313: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:15.316: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.316: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:15.317: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.317: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:15.319: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:15.319: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:16.322: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.322: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:16.324: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.324: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:16.325: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:16.325: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:17.328: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.328: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:17.329: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.329: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:17.331: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:17.331: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:18.334: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.334: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:18.335: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.335: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:18.337: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:18.337: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:19.339: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.339: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:19.341: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.341: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:19.342: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:19.342: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:20.345: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.345: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:20.346: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.346: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:20.348: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:20.348: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:21.351: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.351: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:21.353: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.353: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:21.355: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.355: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:21.820: child pid 148: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.820: child pid 148: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:21.820: child pid 148: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:21.821: main pid 1: LOG:  reaper handler
2024-01-23 23:01:21.828: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:21.878: child pid 164: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:21.878: child pid 164: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:21.878: child pid 164: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:21.886: main pid 1: LOG:  reaper handler
2024-01-23 23:01:21.887: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:22.357: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.357: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:22.358: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.358: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:22.360: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:22.360: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:23.362: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.362: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:23.363: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.363: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:23.365: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:23.365: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:24.367: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.367: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:24.369: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.369: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:24.371: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:24.371: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:25.373: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.374: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:25.375: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.375: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:25.377: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:25.377: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:26.380: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.380: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:26.381: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.381: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:26.383: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.383: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:26.824: child pid 197: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:26.824: child pid 197: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:26.824: child pid 197: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:26.825: main pid 1: LOG:  reaper handler
2024-01-23 23:01:26.825: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:27.385: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.385: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:27.386: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.386: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:27.387: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:27.387: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:28.391: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.391: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:28.394: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.394: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:28.395: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:28.395: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:29.398: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.398: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:29.400: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.400: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:29.403: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:29.403: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:30.408: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.408: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:30.410: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.410: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:30.415: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:30.415: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:31.419: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.419: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:31.421: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.421: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:31.423: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.423: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:31.821: child pid 153: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:31.821: child pid 153: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:31.821: child pid 153: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:31.822: main pid 1: LOG:  reaper handler
2024-01-23 23:01:31.823: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:32.426: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.426: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:32.427: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.428: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:32.429: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:32.429: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:33.433: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.433: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:33.434: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.434: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:33.436: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:33.436: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:34.440: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.440: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:34.442: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.442: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:34.444: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:34.444: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:35.447: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.447: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:35.448: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.448: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:35.450: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:35.450: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:36.452: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.452: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:36.454: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.454: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:36.455: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.455: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:36.844: child pid 160: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.845: child pid 160: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:36.845: child pid 160: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:36.846: child pid 158: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:36.846: child pid 158: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:36.846: child pid 158: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:36.846: main pid 1: LOG:  reaper handler
2024-01-23 23:01:36.847: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:36.849: main pid 1: LOG:  reaper handler
2024-01-23 23:01:36.849: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:37.457: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.458: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:37.459: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.459: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:37.460: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:37.460: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:38.463: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.463: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:38.464: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.464: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:38.466: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:38.466: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:39.469: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.469: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:39.470: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.470: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:39.472: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:39.472: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:40.475: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.475: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:40.476: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.476: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:40.478: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:40.478: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:41.481: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.481: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:41.483: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.483: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:41.485: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.485: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:41.821: child pid 140: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:41.821: child pid 140: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:41.821: child pid 140: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:41.822: main pid 1: LOG:  reaper handler
2024-01-23 23:01:41.823: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:42.487: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.487: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:42.488: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.488: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:42.490: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:42.490: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:43.493: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.493: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:43.495: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.495: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:43.498: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:43.498: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:44.501: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.501: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:44.503: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.503: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:44.504: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:44.504: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:45.507: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.507: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:45.509: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.509: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:45.511: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:45.511: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:46.514: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.514: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:46.515: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.515: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:46.516: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.517: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:46.820: child pid 154: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.820: child pid 154: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:46.820: child pid 154: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:46.824: main pid 1: LOG:  reaper handler
2024-01-23 23:01:46.824: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:46.865: child pid 240: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:46.865: child pid 240: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:46.865: child pid 240: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:46.866: main pid 1: LOG:  reaper handler
2024-01-23 23:01:46.867: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:47.519: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.519: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:47.521: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.521: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:47.522: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:47.522: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:48.526: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.526: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:48.527: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.527: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:48.529: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:48.529: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:49.532: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.532: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:49.533: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.533: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:49.535: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:49.535: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:50.538: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.538: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:50.539: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.539: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:50.540: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:50.540: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:51.543: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.543: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:51.545: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.545: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:51.546: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.546: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:51.815: child pid 273: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:51.815: child pid 273: FATAL:  failed to create a backend 0 connection
2024-01-23 23:01:51.815: child pid 273: DETAIL:  not executing failover because failover_on_backend_error is off
2024-01-23 23:01:51.816: main pid 1: LOG:  reaper handler
2024-01-23 23:01:51.817: main pid 1: LOG:  reaper handler: exiting normally
2024-01-23 23:01:52.549: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.549: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:52.551: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.551: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:52.553: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:52.554: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
2024-01-23 23:01:53.556: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.556: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 0
2024-01-23 23:01:53.557: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.557: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 1
2024-01-23 23:01:53.559: main pid 1: WARNING:  failed to connect to PostgreSQL server, getaddrinfo() failed with error "Name or service not known"
2024-01-23 23:01:53.559: main pid 1: LOG:  find_primary_node: make_persistent_db_connection_noerror failed on node 2
  • kubectl -n memphis logs deployment.apps/memphis-rest-gateway
Defaulted container "memphis-rest-gateway" out of: memphis-rest-gateway, k8s-busybox-waits4broker (init)
Error from server (BadRequest): container "memphis-rest-gateway" in pod "memphis-rest-gateway-649774875b-bfmdl" is waiting to start: PodInitializing
  • kubectl -n memphis logs statefulset.apps/memphis
Defaulted container "memphis" out of: memphis, reloader, memphis-metadata-readiness (init)
  • kubectl -n memphis logs statefulset.apps/memphis-metadata
NONE

Check Memphis deployment on OCP

Test Memphis deployment on top of the Red Hat Openshift platform. Need to be tested as a Standalone installation and as a cluster as well.

REST Gateway Pods missing tolerations

The REST Gateway pods do include the Helm chart's pod (anti-)affinity settings, but ignore the tolerations.

This is problematic because I have a k8s node group that is tainted to specifically only allow memphis pods to land on them. So I need nodeSelector, podAntiAffinity and tolerations to work on all pods to schedule my memphis pods on a specific node group, and also spread them over my AZs.

Create k8s operator

Memphis K8S operator will be based on Golang.
The operator needs to cover all day 1 and day 2 automation phases:

  • Basic Install
  • Seamless upgrades
  • Full lifecycle
  • Deep insights
  • Autopilot

Can't install using flux due to duplicate key

Hi,
I'm trying to install this using flux but ran into the following error:

Helm install failed: error while running post render on files: map[string]interface {}(nil): yaml: unmarshal errors: line 18: mapping key "JWT_SECRET_REST_GW" already defined at line 16

This seems to have occurred with other charts as well and I think the cause is this duplication of the env vars. Is it possible to remove it?

Thanks

Edit: After looking at this open PR, I think the refactoring there will already solve this issue 🥳

Move user configuration to a secret

Hey,

Thanks for adding the user provisioning through a configuration file! The only issue here is that passwords need to be in plain text in our configuration file (we could also auto generate them in a template file but this would change their value at each apply) our we need to save them to a secret, read from it and write that value to the configuration. This is not very practical...

Furthermore, I think passwords should be in secrets anyway...

Would it be possible to change this so that the user configuration would be fetched from a custom / pre-defined secret instead of being part of the default configuration ConfigMap.

I suppose this would require some changes on the server side to be able to read the configuration from another directory since it is not possible to mount two ConfigMap / Secret at the same place.

and I would then have a volume that would be mounted like this

volumes:
    - name: initial-user-configuration
      secret:
        secretName: {{ .Values.auth.existingSecret }}

And in the container definition

volumeMounts:
  - name: initial-user-configuration
    readOnly: true
    mountPath: "/etc/nats-config/conf.d/user-configuration"

Finally, in the memphis.conf I would include everything that is in /etc/nats-config/conf.d/**/*.conf and on the memphis server read the user configuration directly from the main configuration file.

That way configuration is make more dynamic and gives the possibility to user to mount extra configuration volumes to /etc/nats-config/conf.d like they which.

Of course this is just a proposition, if you prefer this can also be a ConfigMap.

External mongodb

Is it possible to connect Memphis with externally hosted MongoDB? I think the chart does not have any setting to change the connection string and does not self-host the MongoDB in the k8s cluster.

Bug: Helm deployment fails on K3S Cluster

Describe the bug

The helm deployment fails on a k3s cluster. k8s-busybox-waits4broker runs but the memphis-http-proxy is always stuck waiting and that is as far as the deployment gets.

The logs from the busybox container show:

waiting for broker
nslookup: can't resolve 'memphis-cluster.memphis.svc.cluster.local'
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

DNS resolution in the cluster is working. The cluster is using the default k3s core-dns container.

I do get the successful deployment message printed out with the deployment details.

Let me know if there is any other output that i can provide to help debug

Steps to reproduce

Run the helm deploy commands on a k3s based Kubernetes cluster.

Affected services

Broker

Platforms

Kubernetes

If UI - Browsers

No response

Environment

Production, Development, Testing

Additional context

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct

Contributing Docs

  • I agree to follow this project's Contribution Docs

Cant install two memphis deployments with the same name on the same cluster

helm install my-memphis memphis/memphis --create-namespace --namespace memphis2

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "my-memphis-memphis-busybox-role" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "memphis2": current value is "memphis"

Unable to specify a secret as external PostgreSQL credential source

The chart in its current form is unable to use a secret as source for external PostgreSQL configuration.

https://github.com/memphisdev/memphis-k8s/blob/9a91a11ed74519fbb66a0a37fe275d992c416698/memphis/templates/memphis_statefulset.yaml#L405-L418

Instead of configuring credentials like so:

    metadata:
      external:
        enabled: true
        dbUser: "foo"
        dbPass: "bar"

I'd like to set:

    metadata:
      external:
        enabled: true
        dbUser:
          value: "foo"
        dbPass:
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password

The change can be made backwards compatible by checking whether the field is set to a string instead of a dictionary.

I'd also suggest you deprecate hard-coding the credentials like that, immediately after implementing the new syntax. It's simply a very bad practice.

P.S. Is a PostgreSQL connection URL (postgesql://...) theoretically supported by Memphis?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.