orange-opensource / galera-operator Goto Github PK

Galera Operator automates tasks for managing a Galera cluster in Kubernetes

License: Apache License 2.0

Dockerfile 1.42% Go 91.95% Makefile 0.95% Shell 5.69%

galera-operator's Introduction

Galera Operator

Project overview

Galera Operator makes it easy to manage highly-available galera clusters deployed to Kubernetes and automates tasks related to operating a galera cluster.

Galera is a popular, free, open-source, multi-master system for mySQL, MariaDB and Percona XtraDB databases. Galera Operator is tested and build for MariaDB.

Today, Galera Operator can:

Create and Destroy
Resize cluster size (number of nodes in a galera cluster)
Resize Containers and Persistent Volumes (Vertical Scalability)
Failover
Rolling upgrade
Backup and Restore

Please read the full documentation in ./doc/user to view all features.

30 000 ft view

Requirements

Kubernetes 1.12+
MariaDB 10.1.23 for Backup and Restore, only mariabackup is currently implemented
S3 storage for Backup and Restore, S3 is currently the only object storage implemented

Note: Operator image provided for example purpose are build for Kubernetes 1.15+. If you want to use it on an older Kubernetes you need to rebuild the image

Features

Deploy Galera Operator

First, we need to deploy new kind of resources used to describe galera cluster and galera backup, note upgrade-config and upgrade-rule is a forthcoming feature used to validate if a cluster can be upgraded:

$ kubectl apply -f ./example-manifests/galera-crd/galera-crd.yaml
$ kubectl apply -f ./example-manifests/galera-crd/galerabackup-crd.yaml
$ kubectl apply -f ./example-manifests/galera-crd/upgrade-config-crd.yaml
$ kubectl apply -f ./example-manifests/galera-crd/upgrade-rule-crd.yaml

Now we can create a Kubernetes namespace in order to host Galera Operator.

$ kubectl apply -f ./example-manifests/galera-operator/00-namespace.yaml

Galera Operator needs to have some rights on the Kubernetes cluster if RBAC (recommended) is set:

$ kubectl apply -f  ./example-manifests/galera-operator/10-operator-sa.yaml
$ kubectl apply -f  ./example-manifests/galera-operator/20-role.yaml
$ kubectl apply -f  ./example-manifests/galera-operator/30-role-binding.yaml
$ kubectl apply -f  ./example-manifests/galera-operator/40-cluster-role.yaml
$ kubectl apply -f  ./example-manifests/galera-operator/60-cluster-role-binding.yaml

Deploy the operator using a deployment:

$ kubectl apply -f  ./example-manifests/galera-operator/70-operator-deployment.yaml

NOTE: do not deploy ./example-manifests/galera-operator/50-cluster-role-clusterwide.yaml, it is only used if you want to deploy the operator to manage galera objects clusterwide, see full documentation for that.

Galera Operator also have some options configured by flags (have a look by starting Galerator Operator with help flag)

Finally if Prometheus Operator is deployed, you can use it to collect Galera Operator metrics

$ kubectl apply -f  ./example-manifests/galera-monitoring/operator-monitor.yaml

Managed Galera Cluster

A galera cluster is made of several nodes, each galera node is mapped on a kubernetes pod (be carefull between galera nodes and kubernetes nodes, it is not the same thing). All galera nodes are not the same, labels are set to specify which pod is the galera node for read, write of be the backup.

We can find on each pod an initcontainer called bootstrap and a container called galera. Metric container is optional and is used to have metrics collected by external system (as prometheus). On each pod we find a persistant volume claim mapping /var/lib/mysql.

The backup pod is not the same as other pods, because there is an extra container (backup) and an extra persistant volume claim. Backup container is designed to expose an API to manage restore and backup operation from the operator. This API is missing today as all operations are done through cli.

Special pod is used for adding large amount of data on existing cluster (like adding tables). It is designed to be used for a limited time, adding or removing this pod does not change the controller revision used to follow change during the galera cluster lifecycle.

For a deeper undestanding of galera cluster object, please read ./doc/design/design_overview.md

Create Galera Cluster

Once Galera Operator is deployed, Galera can be deployed. You need to deploy a ConfigMap to specify the my.cnf configuration. Operator will overwrite some parameters or add it if not present. A Secret is used to provide access to Galera Operator to interact with the managed galera cluster.

$ kubectl apply -f ./example-manifests/galera-cluster/10-priority-class.yaml
$ kubectl apply -f ./example-manifests/galera-cluster/20-galera-config.yaml
$ kubectl apply -f ./example-manifests/galera-cluster/30-galera-secret.yaml
$ kubectl apply -f ./example-manifests/galera-cluster/40-galera.yaml

If Prometheus Operator is deployed and if a metric image is provided, you can collect metrics:

$ kubectl apply -f  ./example-manifests/galera-monitoring/galera-monitor.yaml

Resize Galera Cluster

Galera cluster can be resized in different ways:

1 by changing the number of replicas
2 by specifing new resource requirements
3 by changing volume claim spec, using different storage classes or a new specifying storage request, for example by changing request resource size

Note that each time a modification on the Galera object is made (except if it is the number of replicas), a controllerRevisions is created.

Rolling upgrade

Simply specify a new container image, today the rolling upgrade is done if the new version is greater that the current on deployed. Image must follow the semver format, for example "10.4.5"

TODO : new CRDs provided to the operator will be used to implement plugins and check if upgrade can be done with the provided my.cnf.

Backup Galera Cluster

Galera clusters can be backuped:

$ kubectl apply -f ./example-manifests/galera-backup/10-backup-secret.yaml
$ kubectl apply -f ./example-manifests/galera-backup/20-galera-backup.yaml

NOTE: change $YOURKEY, $YOURSECRET and $YOURENDPOINT to the credentials and url of your S3 solution.

Restore Galera Cluster

Galera clusters can be restored (you need to specify the name of the gzip file) :

$ kubectl -n galera get gl
NAME            TIMESTARTED   TIMECOMPLETED   LOCATION                                  METHOD        PROVIDER
galera-backup                 14m             gal-galera-backup.20200128172501.sql.gz   mariabackup   S3
$ kubectl apply -f ./example-manifests/galera-restore/restore-galera.yaml

Collect Metrics

If Prometheus Operator is deployed, collect the metrics by deploying the service monitor:

$ kubectl apply -f ./example-manifests/galera-monitoring/galera-monitor.yaml

Building the operator

Kubernetes version: 1.13+

This project uses Go modules to manage its dependencies, so feel free to work from outside of your GOPATH. However, if you'd like to continue to work from within your GOPATH, please export GO111MODULE=on.

k8s.io/kubernetes is not primarily intended to be consumed as a module. Only the published subcomponents are (and go get works properly with those). We need to add require directives for matching versions of all of the subcomponents, rather than using go get. Please read carefully Installing client-go and have a look to go.mod. To help you find the matchind dependency you can also use ./hack/dependencies.sh

This operator use the kubernetes code-generator for:

clientset: used to manipulate objects defined in the CRD (GaleraCluster)
informers: cache for registering to events on objects defined in the CRD
listers
deep copy

To get code-generator, you need to explicitly git clone the branch matching the kubernetes version to ./vendor/k8s.io/code-generator

The clientset can be generated using the ./hack/update-codegen.sh script or using the makefile codegen.

The update-codegen script will automatically generate the following files and directories:

pkg/apis/apigalera/v1beta2/zz_generated.deepcopy.go
pkg/client/

The following code-generators are used:

deepcopy-gen - creates a method func (t* T) DeepCopy() *T for each type T
client-gen - creates typed clientsets for CustomResource APIGroups
informer-gen - creates informers for CustomResources which offer an event based interface to react on changes of CustomResources on the server
lister-gen - creates listers for CustomResources which offer a read-only caching layer for GET and LIST requests.

Changes should not be made to these files manually, and when creating your own controller based off of this implementation you should not copy these files and instead run the update-codegen script to generate your own.

galera-operator's People

Contributors

Stargazers

Watchers

Forkers

sebs42 dav1dde shurubelnita xiaomaojames maxshilov gmemcc moussdia heartshare isabella232 alexandertsybin pinkbluersglobal letshavefun wayneburlingame rentbarkerproject emilianobaum naderelabed

galera-operator's Issues

Passing parameters to the docker prom/mysql-exporter

According to the documentation of the prom/mysqld-exporter docker, it is possible to pass parameters to the mysqld-exporter program to be able to collect or delete certain metrics. For this, the collect flags must be passed under the args tag

Example deployment file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysqld-exporter
  labels:
    app: mysqld-exporter
spec:
  selector:
    matchLabels:
      app: mysqld-exporter
  replicas: 1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9104"
      labels:
        app: mysqld-exporter
    spec:
      containers:
      - name: mysqld-exporter
        image: prom/mysqld-exporter
        args: [--collect.auto_increment.columns, --no-collect.slave_status]
        env:
        - name: DATA_SOURCE_NAME
          value: root:password@(mysql:3306)/

Currently, the Galera operator does not interpret the args tag at the metric container

Use the new image prom/mysql-exporter image: v0.12.1

With version v0.11.0 we detect in the metric logs many connection problems to the MariaDB instance. This results in a non-rise of the metrics at Promotheus.
We no longer have these problems with image v0.12.1

publish on operatorhub.io

Hi,
I did not find info about the maturity of this operator, is it safe to run in production environments?
Have you consider to publish it in operatorhub.io?, thanks!
Cheers,
Jacq

Restoring a Galera Cluster : 1 node is enough

When operatoir restore a Galera Cluster, ir restore MariaBackup backup on all nodes of the cluster.

This is of no interest, because after the bootstrap of the 1st node, the other nodes are not able to provide the last number of trasanction galera realize.
There is automatically a synchronization of the node via SST.

So just restore on first node (bootstrap)

Cluster crashing

When I try to run sysbench tests against cluster created with example-manifests MySQL is crashing

Console with Sysbench output:

sysbench --db-debug=on --db-driver=mysql --mysql-user=root --mysql_password=test --mysql-db=test_db --mysql-host=10.109.107.2 ./sysbench/src/lua/oltp_write_only.lua run
sysbench 1.1.0-bbee5d5 (using bundled LuaJIT 2.1.0-beta3)

Running the test with following options:
Number of threads: 1
Initializing random number generator from current time


Initializing worker threads...

Threads started!

FATAL: mysql_drv_query() returned error 2013 (Lost connection to MySQL server during query) for query 'BEGIN'
FATAL: `thread_run' function failed: /usr/local/share/sysbench/oltp_common.lua:412: SQL error, errno = 2013, state = 'HY000': Lost connection to MySQL server during query

logs gathered with kubectl logs -l galera.v1beta2.sql.databases/galera-namespace=galera -f -c galera -n galera:

2020-10-20 19:03:49 3 [Note] WSREP:     id: 4a7546ae-1303-11eb-b229-7a289af995ab name: gal-g8jhx
2020-10-20 19:03:49 3 [Note] WSREP:     id: 6475a4d4-1303-11eb-907d-dfe54094ec8f name: gal-kjkxd
2020-10-20 19:03:49 3 [Note] WSREP: =================================================
2020-10-20 19:03:49 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-10-20 19:03:50 0 [Note] WSREP: Member 0.0 (gal-zsfp6) requested state transfer from '*any*'. Selected 1.0 (gal-g8jhx)(SYNCED) as donor.
2020-10-20 19:03:51 0 [Note] WSREP: (6475a4d4, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-10-20 19:04:02 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 30)
2020-10-20 19:04:02 1 [Note] WSREP: Server gal-g8jhx synced with group
2020-10-20 19:04:02 1 [Note] WSREP: Server status change joined -> synced
2020-10-20 19:04:02 1 [Note] WSREP: Synchronized with group, ready for connections
2020-10-20 19:04:02 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
WSREP_SST: [INFO] Total time on donor: 0 seconds (20201020 19:04:02.883)
WSREP_SST: [INFO] Cleaning up temporary directories (20201020 19:04:02.887)
2020-10-20 19:04:04 0 [Note] WSREP: async IST sender served
2020-10-20 19:04:04 0 [Note] WSREP: 0.0 (gal-zsfp6): State transfer from 1.0 (gal-g8jhx) complete.
2020-10-20 19:04:04 0 [Note] WSREP: Member 0.0 (gal-zsfp6) synced with group.
2020-10-20 19:04:02 0 [Note] WSREP: 1.0 (gal-g8jhx): State transfer to 0.0 (gal-zsfp6) complete.
2020-10-20 19:04:02 0 [Note] WSREP: Member 1.0 (gal-g8jhx) synced with group.
2020-10-20 19:04:04 0 [Note] WSREP: 0.0 (gal-zsfp6): State transfer from 1.0 (gal-g8jhx) complete.
2020-10-20 19:04:04 0 [Note] WSREP: Member 0.0 (gal-zsfp6) synced with group.
2020-10-20 19:04:04 0 [Note] WSREP: Receiving IST...100.0% (5/5 events) complete.
2020-10-20 19:04:04 3 [Note] WSREP: IST received: 364f2641-1303-11eb-b68f-c32d75c48920:30
2020-10-20 19:04:04 0 [Note] WSREP: 0.0 (gal-zsfp6): State transfer from 1.0 (gal-g8jhx) complete.
2020-10-20 19:04:04 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 30)
2020-10-20 19:04:04 0 [Note] WSREP: Member 0.0 (gal-zsfp6) synced with group.
2020-10-20 19:04:04 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 30)
2020-10-20 19:04:04 3 [Note] WSREP: Server gal-zsfp6 synced with group
2020-10-20 19:04:04 3 [Note] WSREP: Server status change joined -> synced
2020-10-20 19:04:04 3 [Note] WSREP: Synchronized with group, ready for connections
2020-10-20 19:04:04 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.



mysqld: /home/buildbot/buildbot/build/mariadb-10.4.2/wsrep-lib/src/client_state.cpp:108: int wsrep::client_state::before_command(): Assertion `transaction_.active() == false || (transaction_.state() == wsrep::transaction::s_executing || transaction_.state() == wsrep::transaction::s_aborted || (transaction_.state() == wsrep::transaction::s_must_abort && server_state_.rollback_mode() == wsrep::server_state::rm_async))' failed.
201020 19:15:05 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.4.2-MariaDB-1:10.4.2+maria~bionic
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=1
max_threads=402
thread_count=10
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1015316 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fe24c00ae78
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fe268305dd8 thread_stack 0x49000
mysqld(my_print_stacktrace+0x2e)[0x55af6850161e]
mysqld(handle_fatal_signal+0x5a5)[0x55af67f8a015]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fe2899db890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fe288cebe97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fe288ced801]
/lib/x86_64-linux-gnu/libc.so.6(+0x3039a)[0x7fe288cdd39a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30412)[0x7fe288cdd412]
mysqld(_ZN5wsrep12client_state14before_commandEv+0x503)[0x55af6856ed63]
mysqld(_Z10do_commandP3THD+0x1da)[0x55af67d99ada]
mysqld(_Z24do_handle_one_connectionP7CONNECT+0x2c2)[0x55af67e6a2a2]
mysqld(handle_one_connection+0x3d)[0x55af67e6a36d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fe2899d06db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe288dce88f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 557
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Fatal signal 11 while backtracing
2020-10-20 19:15:05 0 [Note] WSREP: (6475a4d4, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.17.0.5:4567
2020-10-20 19:15:05 0 [Note] WSREP: (3c45d8de, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://172.17.0.5:4567
2020-10-20 19:15:06 0 [Note] WSREP: (3c45d8de, 'tcp://0.0.0.0:4567') reconnecting to 4a7546ae (tcp://172.17.0.5:4567), attempt 0
2020-10-20 19:15:07 0 [Note] WSREP: (6475a4d4, 'tcp://0.0.0.0:4567') reconnecting to 4a7546ae (tcp://172.17.0.5:4567), attempt 0
2020-10-20 19:15:10 0 [Note] WSREP: evs::proto(3c45d8de, OPERATIONAL, view_id(REG,3c45d8de,5)) suspecting node: 4a7546ae
2020-10-20 19:15:10 0 [Note] WSREP: evs::proto(3c45d8de, OPERATIONAL, view_id(REG,3c45d8de,5)) suspected node without join message, declaring inactive
2020-10-20 19:15:11 0 [Note] WSREP: (3c45d8de, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://172.17.0.5:4567 timed out, no messages seen in PT3S
2020-10-20 19:15:11 0 [Note] WSREP: evs::proto(6475a4d4, GATHER, view_id(REG,3c45d8de,5)) suspecting node: 4a7546ae
2020-10-20 19:15:11 0 [Note] WSREP: evs::proto(6475a4d4, GATHER, view_id(REG,3c45d8de,5)) suspected node without join message, declaring inactive
2020-10-20 19:15:11 0 [Note] WSREP: (6475a4d4, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://172.17.0.5:4567 timed out, no messages seen in PT3S
2020-10-20 19:15:11 0 [Note] WSREP: declaring 3c45d8de at tcp://172.17.0.3:4567 stable
2020-10-20 19:15:11 0 [Note] WSREP: declaring 6475a4d4 at tcp://172.17.0.6:4567 stable
2020-10-20 19:15:11 0 [Note] WSREP: Node 3c45d8de state prim
2020-10-20 19:15:11 0 [Note] WSREP: Node 3c45d8de state prim
2020-10-20 19:15:11 0 [Note] WSREP: view(view_id(PRIM,3c45d8de,6) memb {
        3c45d8de,0
        6475a4d4,0
} joined {
} left {
} partitioned {
        4a7546ae,0
})
2020-10-20 19:15:11 0 [Note] WSREP: save pc into disk
2020-10-20 19:15:11 0 [Note] WSREP: view(view_id(PRIM,3c45d8de,6) memb {
        3c45d8de,0
        6475a4d4,0
} joined {
} left {
} partitioned {
        4a7546ae,0
})
2020-10-20 19:15:11 0 [Note] WSREP: save pc into disk
2020-10-20 19:15:11 0 [Note] WSREP: forgetting 4a7546ae (tcp://172.17.0.5:4567)
2020-10-20 19:15:11 0 [Note] WSREP: (3c45d8de, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-10-20 19:15:11 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2020-10-20 19:15:11 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 9370b8ea-1308-11eb-8fc3-7a2570d6c9f7
2020-10-20 19:15:11 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 9370b8ea-1308-11eb-8fc3-7a2570d6c9f7
2020-10-20 19:15:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9370b8ea-1308-11eb-8fc3-7a2570d6c9f7 from 0 (gal-zsfp6)
2020-10-20 19:15:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9370b8ea-1308-11eb-8fc3-7a2570d6c9f7 from 1 (gal-kjkxd)
2020-10-20 19:15:11 0 [Note] WSREP: Quorum results:
        version    = 5,
        component  = PRIMARY,
        conf_id    = 5,
        members    = 2/2 (joined/total),
        act_id     = 31,
        last_appl. = 0,
        protocols  = 1/10/4 (gcs/repl/appl),
        vote policy= 0,
        group UUID = 364f2641-1303-11eb-b68f-c32d75c48920
2020-10-20 19:15:11 0 [Note] WSREP: Writing down CC checksum: d311c5ac d07ea68d 0b0c95ae e22db27a at offset 184
2020-10-20 19:15:11 0 [Note] WSREP: Flow-control interval: [23, 23]
2020-10-20 19:15:11 0 [Note] WSREP: Trying to continue unpaused monitor
2020-10-20 19:15:11 3 [Note] WSREP: ####### processing CC 32, local, ordered
2020-10-20 19:15:11 3 [Note] WSREP: ####### drain monitors upto 31
2020-10-20 19:15:11 3 [Note] WSREP: REPL Protocols: 10 (5, 3)
2020-10-20 19:15:11 3 [Note] WSREP: ####### My UUID: 3c45d8de-1303-11eb-b4c2-d27bc643c6ac
2020-10-20 19:15:11 3 [Note] WSREP: ####### ST not required
2020-10-20 19:15:11 3 [Note] WSREP: Skipping cert index reset`

Wrong annotation to manager a cluster by clusterwide operator

In the example deployment file 40-galera.yaml , it is indicated

  ## Adding this annotation make this cluster managed by clusterwide operators
  ## namespaced operators ignore it
   annotations:
    sql.databases/scope: clusterwide

But this annotation is incorrect for it to work you have to put

  annotations:
    galera.v1beta2sql.databases/scope: clusterwide

Is this project still active?

I would like to use this project for managing my Galera MariaDB clusters at scale. But it looks like this project has gone quiet.

I have deployed the operator on v1.21.5+rke2r1, but it looks like the GO library is out-of-date; just go panics.

fatal error: mlock failed

runtime stack:
runtime.throw(0x144afb9, 0xc)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc000622d80)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc0004a8a80)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc0004a8a80)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc000068000, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:1390 +0x14e
runtime.newm(0x0, 0xc000068000)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0x100000000)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:1869 +0x12a
runtime.injectglist(0xc000357e98)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2439 +0x13f
runtime.findrunnable(0xc00004a000, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2128 +0xc9c
runtime.schedule()
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2520 +0x2fc
runtime.park_m(0xc000622c00)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:318 +0x5b

goroutine 1 [chan receive]:
main.main()
	/Users/seb/dev/galera/galera-operator/cmd/galera-operator/main.go:190 +0xf9b

goroutine 6 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x1fd20e0)
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/klog/klog.go:990 +0x8b
created by k8s.io/klog.init.0
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/klog/klog.go:404 +0x6c

goroutine 14 [chan receive]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc0001f5500, 0xc000246000)
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/client-go/tools/cache/shared_informer.go:478 +0x53
k8s.io/apimachinery/pkg/util/wait.(*Group).StartWithChannel.func1()
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:54 +0x2e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc000046490, 0xc0002625a0)
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/Users/seb/dev/galera/galera-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

goroutine 52 [IO wait]:
internal/poll.runtime_pollWait(0x7fcc0b01af18, 0x72, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc000048098, 0x72, 0x0, 0x0, 0x1446fe8)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000048080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/internal/poll/fd_unix.go:384 +0x1d4
net.(*netFD).accept(0xc000048080, 0xc00001cda8, 0xc000100000, 0x7fcc31ce7108)
	/usr/local/Cellar/go/1.14/libexec/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0002624a0, 0xc00001cde8, 0x40c858, 0x30)
	/usr/local/Cellar/go/1.14/libexec/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0002624a0, 0x1389420, 0xc000232fc0, 0x1293a00, 0x1fbd900)
	/usr/local/Cellar/go/1.14/libexec/src/net/tcpsock.go:261 +0x64
net/http.(*Server).Serve(0xc0001d01c0, 0x164efe0, 0xc0002624a0, 0x0, 0x0)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/server.go:2901 +0x25d
net/http.(*Server).ListenAndServe(0xc0001d01c0, 0xc0001d01c0, 0xc000075fb8)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/server.go:2830 +0xb7
net/http.ListenAndServe(0x1445e64, 0x5, 0x0, 0x0, 0x1ffba78, 0x203000)
	/usr/local/Cellar/go/1.14/libexec/src/net/http/server.go:3086 +0x74
created by main.main
	/Users/seb/dev/galera/galera-operator/cmd/galera-operator/main.go:99 +0x47b

[question] Exposing cluster to outside world.

Could you please provide an example of how I can expose cluster for the outside world for read/write ops?

Is it right that service/gal-writer-bkp-svc and service/gal-writer-svc are not supposed to receive read ops?

user mysql exporter is not created in image: sebs42/mariadb:10.4.12-bionic

In image 10.4.12, user export creation is missing in docker-entrypoint.sh

We need to create
CREATE USER 'exporter'@'localhost' IDENTIFIED BY '${EXPORTER_PASSWORD}' WITH MAX_USER_CONNECTIONS 3; GRANT PROCESS,REPLICATION CLIENT,SELECT ON *.* TO 'exporter'@'localhost';
and update documentation https://github.com/Orange-OpenSource/galera-operator/blob/master/doc/design/galera_images.md

How to get root password

how to get root password ??

wrong result with operator galera metric

4 of 6 operator galera metric do not give the right values :

galera_operator_controller_clusters is the same value of galera_operator_controller_clusters_created, not number of clusters managed by the controller
galera_operator_controller_clusters_deleted, galera_operator_controller_clusters_failed and galera_operator_controller_clusters_modified always return 0

replicas >= 5, re-creates too many pods after deleting writer pod

When using more than 3 replicas (5, 7, 9) and deleting the writer pod on a fully populated cluster, the operator creates replicas+1 pods.

Situation:

pod-1: role=writer
pod-2: reader=true
pod-3: reader=true
pod-4: reader=true
pod-5: reader=true

Action: delete pod-1

Situation:

pod-1: TERMINATING
pod-2: reader=false
pod-3: reader=true
pod-4: reader=true
pod-5: reader=true

After a while:

pod-1: reader=true
pod-2: reader=false
pod-3: reader=false; writer=true
pod-4: reader=true
pod-5: reader=true

After some more time:

pod-1: reader=true
pod-2: reader=false
pod-3: reader=false; writer=true
pod-4: reader=true
pod-5: reader=true
pod-6: reader=true

Now there are 6 pods, pod-2 is not recognized as a member of the cluster anymore because its label is reader=false without any role. Caused by the logic here: https://github.com/Orange-OpenSource/galera-operator/blob/master/pkg/controllers/cluster/galera_control.go#L626

				if addPod == nil {
					if reader, exist := pod.Labels[apigalera.GaleraReaderLabel]; exist && reader == "true" {
						addPod = pod
					}
				}

The problem why this happens seems to be in setLabels: https://github.com/Orange-OpenSource/galera-operator/blob/master/pkg/controllers/cluster/galera_control.go#L959

	default:
		if status.Members.Writer == "" {
			var candidate *corev1.Pod
                        // ... candidate selection here

                        // new writer gets selected and the labels for the new writer are updated:
                        // labels are now `reader=false; role=writer`
                        // Writer Status is updated: status.Members.Writer  
			status.Members.Writer = candidate.Name
			if err := gc.podControl.PatchPodLabels(galera, candidate.DeepCopy(), RemoveReader(), AddWriter()); err != nil {
				return err
			}
		}

		if status.Members.BackupWriter == "" {
                    // unrelated logic
		}

                // here is where the problem starts - `status.Members.Writer` is updated and the writer node has the correct labels
                // but the operation was done on a `DeepCopy` and this overwrites the labels for the writer node with the old labels
		for _, pod := range readyPods {
			if pod.Name == status.Members.Writer {
				if pod.DeepCopy().Labels[apigalera.GaleraReaderLabel] == "true" {
					return gc.podControl.PatchPodLabels(galera, pod.DeepCopy(), RemoveReader())
				}
			}
		}

See my comments in the code, but basically what happens is, the newly selected writer nodes get's the role label added and right after there is another update for the new writer node which resets the labels to the old labels again and additionally sets the reader label to false, leaving the one node with only reader=false and no role.

This patch seems to fix the issue:

--- a/pkg/controllers/cluster/galera_control.go
+++ b/pkg/controllers/cluster/galera_control.go
@@ -978,38 +987,38 @@ func (gc *defaultGaleraControl) setLabels(galera *apigalera.Galera, readyPods []
 			if err := gc.podControl.PatchPodLabels(galera, candidate.DeepCopy(), RemoveReader(), AddWriter()); err != nil {
 				return err
 			}
-		}
-
-		if status.Members.BackupWriter == "" {
-			var candidate *corev1.Pod
-			for _, pod := range readyPods {
-				if _, exist := pod.Labels[apigalera.GaleraBackupLabel]; exist {
-					continue
-				}
-				if _, exist := pod.Labels[apigalera.GaleraRoleLabel]; exist {
-					continue
-				}
-				// needed because readyPods is not update by previous patching for the Writer
-				if status.Members.Writer != pod.Name {
-					candidate = pod
-					break
+		} else {
+			if status.Members.BackupWriter == "" {
+				var candidate *corev1.Pod
+				for _, pod := range readyPods {
+					if _, exist := pod.Labels[apigalera.GaleraBackupLabel]; exist {
+						continue
+					}
+					if _, exist := pod.Labels[apigalera.GaleraRoleLabel]; exist {
+						continue
+					}
+					// needed because readyPods is not update by previous patching for the Writer
+					if status.Members.Writer != pod.Name {
+						candidate = pod
+						break
+					}
 				}
-			}

-			if candidate == nil {
-				return errors.New(fmt.Sprintf("no BackupWriter Role candidate for galera %s/%s ", galera.Namespace, galera.Name))
-			}
+				if candidate == nil {
+					return errors.New(fmt.Sprintf("no BackupWriter Role candidate for galera %s/%s ", galera.Namespace, galera.Name))
+				}

-			status.Members.Writer = candidate.Name
-			if err := gc.podControl.PatchPodLabels(galera, candidate.DeepCopy(), AddReader(), AddBackupWriter()); err != nil {
-				return err
+				status.Members.Writer = candidate.Name
+				if err := gc.podControl.PatchPodLabels(galera, candidate.DeepCopy(), AddReader(), AddBackupWriter()); err != nil {
+					return err
+				}
 			}
-		}

-		for _, pod := range readyPods {
-			if pod.Name == status.Members.Writer {
-				if pod.DeepCopy().Labels[apigalera.GaleraReaderLabel] == "true" {
-					return gc.podControl.PatchPodLabels(galera, pod.DeepCopy(), RemoveReader())
+			for _, pod := range readyPods {
+				if pod.Name == status.Members.Writer {
+					if pod.DeepCopy().Labels[apigalera.GaleraReaderLabel] == "true" {
+						return gc.podControl.PatchPodLabels(galera, pod.DeepCopy(), RemoveReader(), AddWriter())
+					}
 				}
 			}
 		}

I am not sure if I am causing other sideeffects with this patch though.

Clone with restoring

If you want to restore galera cluster backup on a new galera cluster, you need to rename backup on S3 before deploy.
It's not possible to use an existing backup, no error in log

<old name of galera cluster>-galera-backup.YYYYMMDDHHMMSS.sql.gz

<new name of galera cluster>-galera-backup.YYYYMMDDHHMMSS.sql.gz

apiVersion: sql.databases/v1beta2
kind: Galera
metadata:
  name: <new name of galera cluster>
...
spec:
  replicas: 3
  restore:
    name: <new name of galera cluster>-galera-backup.YYYYMMDDHHMMSS.sql.gz

other points, if you make a mistake on the s3 address or on the bucket name, there is no error in the operator logs

User exporter's password is not valued in DATA_SOURCE_NAME

In the example file 40-GALERA.yaml, it is proposed to reconstitute the variable DATA_SOURCE_NAME by passing the password of the user EXPORTER via an environment variable.

- name: DATA_SOURCE_NAME
  value: exporter:${EXPORTER_PASSWORD}@(localhost:3306)/

With this method, the password is not valued when the mysqld-exporter program uses the variable DATA_SOURCE_NAME. It would be a shame to be forced to supply the variable DATA_SOURCE_NAME with the password in clear when it is encrypted in the "secret" file.

annotation support

adding annotations to the
sql.databases/v1beta2 Galera template
which could be added to the Galera pods created by the operator would be very useful feature. (maybe labels also?)

Deployment stuck at "Desired 3" - "Current 0"

Followed the instructions at the examples-manifests, using "Three member cluster" example, galera-operator is deployed using the kustomize found in #12

Then manually running kubectl create -f <galera-cluster>.yaml deployment get stuck at Desired 3 - Current 0

This is what I could get my hands on:

galera-operator check:

kubectl get deployments.apps --namespace galera
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
galera-operator   1/1     1            1           62m

crd's check:

$ kubectl get crd
NAME                           CREATED AT
addons.k3s.cattle.io           2020-09-30T00:37:13Z
helmcharts.helm.cattle.io      2020-09-30T00:37:13Z
galeras.sql.databases          2020-09-30T00:38:01Z
galerabackups.sql.databases    2020-09-30T00:39:26Z
upgradeconfigs.sql.databases   2020-09-30T00:39:27Z
upgraderules.sql.databases     2020-09-30T00:40:01Z

Config map is embebed with yaml below, using the default provided in the spec example, nothing changed,

Full yaml (skipping configmap data):

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: galeraconf
  namespace: galera
data:
...

---
apiVersion: sql.databases/v1beta2
kind: Galera
metadata:
  name: zalo1
  namespace: galera
  ## Adding this annotation make this cluster managed by clusterwide operators
  ## namespaced operators ignore it
  #annotations:
  #  sql.databases/scope: clusterwide
  labels:
    stage: dev
    owner: sebs42
spec:
  replicas: 3
  pod:
    credentialsSecret:
      name: galera-secret
    image: sebs42/mariadb:10.4.2-bionic
    mycnfConfigMap:
      name: galeraconf
    env:
      - name: MYSQL_ROOT_PASSWORD
        value: $ROOTPASSWORD
  persistentVolumeClaimSpec:
    accessModes: [ "ReadWriteOnce" ]
    storageClassName: $STORAGECLASS
    resources:
      requests:
        storage: 5Gi

Outputs:

NAMESPACE   NAME    DESIRED   CURRENT   AGE   PHASE   IMAGE                          METRICIMAGE   SPECIAL
galera      zalo1   3                   23m           sebs42/mariadb:10.4.2-bionic

Description of cluster deployment:

$ kubectl describe galeras.sql.databases/zalo1 --namespace galera
Name:         zalo1
Namespace:    galera
Labels:       owner=sebs42
              stage=dev
Annotations:  <none>
API Version:  sql.databases/v1beta2
Kind:         Galera
Metadata:
  Creation Timestamp:  2020-09-30T00:48:21Z
  Generation:          1
  Managed Fields:
    API Version:  sql.databases/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:owner:
          f:stage:
      f:spec:
        .:
        f:persistentVolumeClaimSpec:
          .:
          f:accessModes:
          f:resources:
            .:
            f:requests:
              .:
              f:storage:
          f:storageClassName:
        f:pod:
          .:
          f:credentialsSecret:
            .:
            f:name:
          f:env:
          f:image:
          f:mycnfConfigMap:
            .:
            f:name:
        f:replicas:
    Manager:      kubectl-create
    Operation:    Update
    Time:         2020-09-30T00:48:21Z
    API Version:  sql.databases/v1beta2
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2020-09-30T00:57:57Z
  Resource Version:  1794
  Self Link:         /apis/sql.databases/v1beta2/namespaces/galera/galeras/zalo1
  UID:               84b75575-9646-41b6-81c5-6ffdcfd7098f
Spec:
  Persistent Volume Claim Spec:
    Access Modes:
      ReadWriteOnce
    Resources:
      Requests:
        Storage:         5Gi
    Storage Class Name:  $STORAGECLASS
  Pod:
    Credentials Secret:
      Name:  galera-secret
    Env:
      Name:   MYSQL_ROOT_PASSWORD
      Value:  $ROOTPASSWORD
    Image:    sebs42/mariadb:10.4.2-bionic
    Mycnf Config Map:
      Name:  galeraconf
  Replicas:  3
Events:      <none>

The only error message found was with:

$ kubectl logs galeras.sql.databases/zalo1 --namespace galera
error: no kind "Galera" is registered for version "sql.databases/v1beta2" in scheme "k8s.io/kubectl/pkg/scheme/scheme.go:28"

Is it possible to change readiness and liveness parameters?

I would like to change the readiness and liveness parameters of the galera instances. At this moment, the sst is trying to recover the database data, but the liveness healthcheck is killing the container, because it is not ready.

add to https://operatorhub.io/

hi,
you've done a great job!!
why not add this operator to https://operatorhub.io/

thanks