Report

$ kubectl apply -f deploy/cr.yaml
Warning: metadata.finalizers: "delete-pxc-pods-in-order": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
perconaxtradbcluster.pxc.percona.com/cluster1 created

More about the problem

See above

Steps to reproduce

1. git checkout 1.14
2. Deploy

Versions

1. Kubernetes EKS 1.29
2. Operator 1.14

Anything else?

No response

Proposal

Currently there is only one configuration file that is used for both mysql datadir initialization as well as mysql startup. It will be convenient to have a way to inject mysql configuration post data dir initialization as some plugins are not available when using "mysql --initialize" causing the datadir to not be initialized.

Use-Case

When trying to configure a cluster to write audit_logs by using percona audit_log plugin the cluster fails to initialize because of unknown variable errors if the audit_log configuration is specified. Currently there is no way to launch a cluster with a complete configuration of this plugin.

This causes a lot of headaches when trying to following GitOps principles as new Databases need to be deployed in a two step approach. First, deploy the cluster without audit_log configuration to allow it to initialize. Second, you need to update your configuration and add the missing audit_log variables, restart the cluster and have those configured.

By allowing a hypothetical spec.pxc.configuration_post_init configuration of plugins could be placed there and only loaded by mysql only after post-init. This would allow for a single configuration to bot hinitialize and manage clusters.

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

Use the following snippet if you want to test out the error:

spec:
  pxc:
    configuration: |
      [mysqld]
      # Enable audit_log
      audit_log_file=audit.log
      audit_log_strategy=PERFORMANCE
      audit_log_format=JSON
      audit_log_policy=LOGINS
      audit_log_exclude_accounts=xtrabackup@%,replication@%,proxyadmin@%,operator@%,monitor@%,clustercheck@%

Make sure you are launching on a uninitialized PVC. The following errors will show in /var/lib/mysql/mysqld-error.log:

2024-02-13T20:31:52.517092Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library 'none'
2024-02-13T20:31:52.523257Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-02-13T20:31:53.596678Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-13T20:31:53.596678Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-02-13T20:31:54.110877Z 0 [Warning] [MY-013501] [Server] Ignoring --plugin-load[_add] list as the server is running with --initialize(-insecure).
2024-02-13T20:31:54.704477Z 0 [ERROR] [MY-000067] [Server] unknown variable 'audit_log_file=audit.log'.
2024-02-13T20:31:54.704564Z 0 [ERROR] [MY-013455] [Server] The newly created data directory /var/lib/mysql//sst-xb-tmpdir/ by --initialize is unusable. You can remove it.
2024-02-13T20:31:54.705448Z 0 [ERROR] [MY-010119] [Server] Aborting

Report

When not providing credentialsSecret in the s3 section of the deploy/cr.yaml backups can't be taken.

More about the problem

Documentation states:

Using AWS EC2 instances for backups makes it possible to automate access to AWS S3 buckets based on IAM roles for Service Accounts with no need to specify the S3 credentials explicitly.
Following steps are needed to turn this feature on:

• Create the IAM instance profile * and the permission policy within where you specify the access level that grants the access to S3 buckets.
• Attach the IAM profile to an EC2 instance.
• Configure an S3 storage bucket and verify the connection from the EC2 instance to it.
• Do not provide s3.credentialsSecret for the storage in deploy/cr.yaml.

When trying to drop s3.credentialsSecret the operator logs show the following errors:

2024-04-04T08:46:20.926Z	INFO	Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes reqeueuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler	{"controller": "pxcbackup-controller", "namespace": "data", "name": "backup1", "reconcileID": "e1e6a9aa-8aff-4997-abdd-bdcdcbc75730"}
2024-04-04T08:46:20.926Z	ERROR	Reconciler error	{"controller": "pxcbackup-controller", "namespace": "data", "name": "backup1", "reconcileID": "e1e6a9aa-8aff-4997-abdd-bdcdcbc75730", "error": "create backup job: Job.batch \"xb-backup1\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]", "errorVerbose": "Job.batch \"xb-backup1\" is invalid: [spec.template.spec.containers[0].env[4].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), spec.template.spec.containers[0].env[5].valueFrom.secretKeyRef.name: Invalid value: \"\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')]\ncreate backup job\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup.(*ReconcilePerconaXtraDBClusterBackup).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxcbackup/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227

cr.yaml:

...
backup:
    image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
    schedule:
      - keep: 3
        name: daily
        schedule: 0 0 * * *
        storageName: s3
    serviceAccountName: test-mysql-percona-s3
    storages:
      s3:
        s3:
          bucket: test-mysql-percona-s3
          region: eu-west-1
        type: s3
...

Steps to reproduce

1. Configure s3 backup according to https://docs.percona.com/percona-operator-for-mysql/pxc/backups-storage.html
2. Create an on-demand backup
3. watch the operator logs

Versions

1. Kubernetes v1.27.9-eks
2. Operator 1.14.0
3. Database perconalab/percona-xtradb-cluster-operator:main-pxc8.0

Anything else?

No response

Report

I can successfully create pxc cluster when I do not add sidecars mysqld_exporter Adding sidecars mysqld_exporter cannot create pxc pods cluster

More about the problem

kubectl get all -n pxc

NAME                                                   READY   STATUS    RESTARTS      AGE
pod/cluster1-haproxy-0                                 1/2     Running   2 (76s ago)   7m17s
pod/percona-xtradb-cluster-operator-7d6b49c48c-n9cpr   1/1     Running   0             8m52s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                 AGE
service/cluster1-haproxy                  ClusterIP   100.64.198.221   <none>        3306/TCP,3309/TCP,33062/TCP,33060/TCP   7m17s
service/cluster1-haproxy-replicas         ClusterIP   100.64.73.9      <none>        3306/TCP                                7m17s
service/cluster1-pxc                      ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP            7m17s
service/cluster1-pxc-unready              ClusterIP   None             <none>        3306/TCP,33062/TCP,33060/TCP            7m17s
service/percona-xtradb-cluster-operator   ClusterIP   100.64.244.152   <none>        443/TCP                                 8m42s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/percona-xtradb-cluster-operator   1/1     1            1           8m52s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/percona-xtradb-cluster-operator-7d6b49c48c   1         1         1       8m52s

NAME                                READY   AGE
statefulset.apps/cluster1-haproxy   0/3     7m17s
statefulset.apps/cluster1-pxc       0/3     7m17s

percona-xtradb-cluster-operator logs:
2024-03-05T10:27:52.211Z INFO Waiting for HAProxy to be ready before smart update {"controller": "pxc-controller", "namespace": "pxc", "name": "cluster1", "reconcileID": "29d4bc1a-0928-4ea7-8408-bc6da726381f"}

Steps to reproduce

deploy/cr.yaml sidecars config:

  pxc:
    size: 3
    image: percona-xtradb-cluster:8.0.32-24.2
    autoRecovery: true
    sidecars: 
    - image: mysqld-exporter:v0.15.1
      name: mysqld_exporter
      env: 
        - name: EXPORTER_PASS
          valueFrom:
            secretKeyRef:
              name: cluster1-secrets 
              key: monitor      
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP      
        - name: DATA_SOURCE_NAME
          value: "monitor:$(EXPORTER_PASS)@tcp(127.0.0.1:3306)/"
      args: ["--web.listen-address=$(POD_IP):9104"]
      resources:
        requests:
          memory: 100M
          cpu: 100m
        limits:
          memory: 200M
          cpu: 200m      

Versions

1. Kubernetes v1.28.3+rke2r2
2. Operator 1.13.0
3. Database percona-xtradb-cluster:8.0.32-24.2

Anything else?

No response

change Kind to "PerconaXtraDBCluster"
https://github.com/Percona-Lab/percona-xtradb-cluster-operator/blob/31a80d6b99eab82b4a61a330d5e0413239dd79d4/config/config.yaml#L2

Report

Cannot scale up pxc and proxysql instances storage from the CR definition of the pxc cluster

More about the problem

If we try to increase the size of the pxc instances storage defined in the spec.pxc.volumeSpec.persistentVolumeClaim.resources.requests.storage CR attribute, the cluster switch to error state because the operator try to update the related attribute in the pxc statefulset but this operation is prevented as we can see from the operator logs :

"Forbidden: updates to statefulset spec for fields other than …"

If we try to apply the same operation for the proxysql instances storage defined in the spec.proxysql.volumeSpec.persistentVolumeClaim.resources.requests.storage, nothing happens.

As far as I know, the only way to scale up the storage size of these instances is to directly update the related pvc definition, assuming that these pvc are provided by storageclasses allowing the volume expansion of course. But this is not very convenient as the CR definition and related statefulsets are not updated behind.

It would be a great improvement to allow volume expansion from the CR definition by creating external pvc attached to the pod using Claims As Volumes method as implemented with the PostgreSQL operator instead of the dynamic pvc creation from statefulset definition with the Volume Claim Templates method.

Steps to reproduce

1 - Create a pxc cluster with proxysql section enabled and define a specific storage request size for both pxc and proxysql section
2 - Wait for the cluster to be ready and see that pvc were created with the desired storage sizes
3 - Edit the pxc CR definition by increasing spec.pxc.volumeSpec.persistentVolumeClaim.resources.requests.storage attribute
4 - See the cluster switching to error state and the related log errors from the operator pod. Pvc are not scaled up
5 - Revert the change in the CR definition
6 - The cluster is back to ready state
7 - Edit the pxc CR definition by increasing spec.proxysql.volumeSpec.persistentVolumeClaim.resources.requests.storage attribute
8 - See that nothing happen, operator do not take any action and pvc are not scaled up

Versions

1 - Kubernetes - v1.27.6
2 - Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0
3 - Database - MySQL XtraDB 8.0

Anything else?

1 - Kubernetes - v1.27.6
2 - Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0
3 - Database - MySQL XtraDB 8.0

We use the pxc operator password auto generation feature for pxc accounts like root, xtrabackup ...

Unfortunately, pxc operator can't auto generate pmmserverkey.
So we have to add the pmmserverkey manually to the existing k8s secret which already contains the auto generated passwords for root,xtrabackup ...
https://github.com/percona/percona-xtradb-cluster-operator/blob/main/deploy/secrets.yaml#L11
https://docs.percona.com/percona-operator-for-mysql/pxc/monitoring.html#install-pmm-server

This is a manual step which we would like to avoid. We are using argocd for deployment of several pxc clusters.
So, one option would be, to use kustomize to add pmmserverkey to the existing k8s secret. This is not an option for us because we don't store passwords in clear text in git.
We would like to use sealed secrets, but sealed secrets doesn't support addition of a key, value pair to an existing k8s secret.

Would it be possible to store auto generated and manual created credentials in separate k8s secrets?

Hey,

I wanted to try Percona and installed the newest version on Kubernetes (PXC).
So I started to follow this tutorial percona-operator-for-mysql for kubernetes. I also tried to use the Helm Quickstart but in both caseses there was no Pod created. You can see the Deployment is there, but after that nothing happens and there are no logs or I don't know how to find them.

I am running a cluster with 3 nodes (2 worker, 1 controller [no taints]) on Kubernetes v1.24.6 with Ubuntu 22.04.1 LTS and containerd://1.6.8. If more details are needed please tell me what I am missing.

Report

PXC restore jobs pod resources requests values are overwritten by spec.resources.limits attributes from pxc-restore resource definition.

More about the problem

When you create a new pxc-restore resource, you can specify spec.resources.requests and spec.resources.limits attributes in the yaml definition. However, no matter what you put inside the spec.resources.requests attributes, both resources requests and limits in the related jobs pod will take the values specified in spec.resources.limits attributes.

Steps to reproduce

1 - Create a pxc cluster with a backup storage configured
2 - Run a pxc backup of the cluster and wait for its successfull completion
3 - Prepare a pxc restore definition targeting the previous backup with both spec.resources.requests and spec.resources.limits attributes defined with different values
4 - Apply the pxc restore resource on the cluster
5 - See that the related job pod resources requests and limits are only filled by the values specified in the spec.resources.limits attributes of the restore definition. spec.resources.requests attributes are just ignored.

Versions

1 - Kubernetes - v1.27.6
2 - Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0
3 - Database - MySQL XtraDB 8.0

Anything else?

No response

Proposal

run a fresh MySQL with data from a xtrabackup

Use-Case

For QA , we run MySQL with provisionned data from xtrabackup

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

As a workaround, we create the PVC from a volume snapshot containing the xtrabackup

Report

PXC backup and restore job pods do not bear the classic app.kubernetes.io/* labels used by other components

More about the problem

When a new pxc cluster is deployed, all the running pod related are automatically created with these usefull labels :

app.kubernetes.io/component=<pxc/haproxy/proxysql/pitr>
app.kubernetes.io/instance=<cr.Name>
app.kubernetes.io/managed-by=percona-xtradb-cluster-operator
app.kubernetes.io/name=percona-xtradb-cluster
app.kubernetes.io/part-of=percona-xtradb-cluster

But it is not the case for the backup and restore jobs pods, even if it's possible to add custom labels for backup in the spec.backup.storages..labels section of the CR definition but nothing for restore as far as I know.
Anyway, it would be a great improvement to automatically add these labels for the backup and restore pods as it already the case with the PostgreSQL operator.

Steps to reproduce

1 - Create a pxc cluster with a backup storage configured without additional label in the definition
2 - Run a pxc backup of the cluster
3 - See that the backup job pod does not bear the classic app.kubernetes.io/* labels
4 - Run a pxc restore on the cluster
5 - See that the restore job pod does not bear the classic app.kubernetes.io/* labels neither

Versions

1 - Kubernetes - v1.27.6
2 - Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0
3 - Database - MySQL XtraDB 8.0

Anything else?

No response

Proposal

I would like to be able to configure and restrict the number of completed PerconaXtraDBClusterBackup's that are retained in Kubernetes, maybe default to 5. Current there seems to be an endless number of these which i have to manually remove to clean up my kubectl get pods display.

Also, Is there a reason CronJobs are not used here instead of Jobs?

Use-Case

No response

Is this a feature you are interested in implementing yourself?

Maybe

Anything else?

No response

Report

Hi, I used helm to install percona/pxc-operator and percona/pxc-db version 1.14, and enabled the built-in proxysql as the proxy for the cluster.

Recently, I noticed that the memory usage of proxysql is increasing, indicating a memory leak issue.
The version of proxysql being used is 2.5.5-percona-1.1, with the original settings in place and query digest turned off.

More about the problem

jemalloc_resident increasing to 583 MB

SELECT Variable_Name, Variable_Value / 1024 / 1024 as "Usage in MB" FROM stats.stats_memory_metrics ORDER BY 2 DESC;
+------------------------------+-------------+
| Variable_Name                | Usage in MB |
+------------------------------+-------------+
| jemalloc_mapped              | 629         |
| jemalloc_resident            | 583         |
| jemalloc_active              | 545         |
| jemalloc_allocated           | 462         |
| SQLite3_memory_bytes         | 428         |
| jemalloc_retained            | 134         |
| stack_memory_admin_threads   | 40          |
| stack_memory_cluster_threads | 30          |
| stack_memory_mysql_threads   | 20          |
| jemalloc_metadata            | 10          |
| Auth_memory                  | 0           |
| query_digest_memory          | 0           |
| mysql_query_rules_memory     | 0           |
| mysql_firewall_users_table   | 0           |
| mysql_firewall_users_config  | 0           |
| mysql_firewall_rules_table   | 0           |
| mysql_firewall_rules_config  | 0           |
+------------------------------+-------------+
17 rows in set (0.01 sec)

Steps to reproduce

1. install helm chat percona/pxc-operator, percona/pxc-db
2. disable query digest
3. wait 2 days

Versions

1. Kubernetes: 1.25.16
2. Operator: 1.14
3. Database: 1.14

Anything else?

No response

We run 2 pxc instances with same name "pxc-mysql" in 2 different namespaces: "a" , "b".
We try to schedule a backup for "a" but this is not working, it seem the backup job is removed all the time:

{"level":"info""msg":"Creating or updating backup job","controller":"pxc-controller","namespace":"a","name":"pxc-mysql","reconcileID":"fa7b0439-....83ee7e","name":"80527-jayjay","schedule":"*/2 * * * *"}
{"level":"info",,"msg":"deleting outdated backup job","controller":"pxc-controller","namespace":"b","name":"pxc-mysql","reconcileID":"1e5fa590....9df3","name":"80527-jayjay"}

dont know "deleting outdated backup job" in namespace "b"

Changing the name of instance in ns "a" did the trick.

Is pxc cluster name must be unique across namespaces?

Report

*-monit containers resources in ProxySQL pods does not take the values specified in spec.proxysql.resources attributes of the pxc cluster definition

More about the problem

It seems not possible to provide resources cutom values for the pxc-monit and proxysql-monit containers that run inside proxysql pods. The configuration specified in the spec.proxysql.resources attributes of the cluster definition only apply to the proxysql container of the pod, but the *-monit containers resources section remains empty. It would be a great feature to add options in the CR definition to specify custom resources values for the *-monit containers of the proxysql pod or use the existing spec.proxysql.resources attributes to fill them.

Steps to reproduce

1 - Create a pxc cluster with proxysql section enabled and defined with custom resources configuration in the spec.proxysql.resources fields
2 - See that only the container named proxysql take the custom resources values in the related pod but the resources section remains empty for pxc-monit and proxysql-monit containers

Versions

1 - Kubernetes - v1.27.6
2 - Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0
3 - Database - MySQL XtraDB 8.0

Anything else?

No response

As the title.

Proposal

Whether arm is supported?

Use-Case

I want to deploy my application on ARM machines, so I need to deploy Percona XtraDB Cluster (PXC) on these machines too.

Is this a feature you are interested in implementing yourself?

No

Anything else?

No response

I'm experiencing an intermittent issue where at times I encounter the error described below, while at other times, everything functions normally.

https://percona-xtradb-cluster-operator.percona.svc:443/validate-percona-xtradbcluster?timeout=10s:](https://percona-xtradb-cluster-operator.percona.svc/validate-percona-xtradbcluster?timeout=10s:) tls: failed to verify certificate: x509: certificate signed by unknown

if I remove the kubernetes admission: percona-xtradbcluster-webhook this error never happens again

Report

Random TLS certificate verification failure when calling the percona xtradb cluster validating webhook

More about the problem

When we deploy the pxc operator in cluster wide mode (watchAllNamespaces=true) and with more than one replica (replicaCount>1), tls certificate verification failure appears at random on validating webhook call.
These errors can be seen when a user try to apply or edit a CR definition of a pxc cluster, or from the operator logs during reconciliation operations. The logs are the following :*

"Internal error occured: failed calling webhook "validationwebhook.pxc.percona.com": failed to call webhook: Post "[https://percona-xtradb-cluster-operator.namespace.svc:443/validate-percona-xtradbcluster?timeout=10s](https://percona-xtradb-cluster-operator.namespace.svc/validate-percona-xtradbcluster?timeout=10s)": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "Root CA")

After some investigations, I noticed that the ca bundle configured in the validating webhook change each time a pxc-operator replica pod take the lead of the operations and only this pod has valid tls certificate.

This can be checked by recovering the ca-bundle from the validating webhook and the tls.crt from the pxc-operator leader pod and verify the signature with openssl :

kubectl get validatingwebhookconfiguration percona-xtradbcluster-webhook -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d > ca-bundle.crt
kubectl exec pxc-operator-6bc5fb656b-2grl7 -- cat /tmp/k8s-webhook-server/serving-certs/tls.crt > leader-tls.crt
openssl verify -CAfile ca-bundle.crt leader-tls.crt
leader-tls.crt: OK

But if we extract the tls.crt from another pxc-operator replica pod, the verification fails :

openssl verify -CAfile ca-bundle.crt replica-tls.crt
error 7 at 0 depth lookup:certificate signature failure
139771764295568:error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:rsa_pk1.c:116:
139771764295568:error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed:rsa_eay.c:761:
139771764295568:error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib:a_verify.c:249:

And if we delete the leader pod, the ca-bundle configured in the validating webhook change to match the certificate of the new leader.
As the percona-xtradb-cluster-operator k8s service point to any of the pxc-operator replica pods, this explains why the error appears at random if the validation webhook call is redirected to a non-leader pxc-operator replica pod.
This was also confirm by the fact that the problem disappears when we scale down the operator to only one replica.

Steps to reproduce

1. Deploy the pxc operator in cluster wide mode with more than one replica (helm values watchAllNamespaces=true and replicaCount>1), the more replica the easier the bug to reproduce.
2. Deploy a pxc cluster with any valid configuration
3. Wait a bit of time a check the operator logs : tls verification failure should appear at random during reconciliation operation. You can also check that the signature is valid for only one of the operator pod certificate.

Versions

1. Kubernetes - v1.27.6
2. Operator - Percona Operator for MySQL based on Percona XtraDB Cluster 1.13.0

Anything else?

No response