altinity / clickhouse-operator Goto Github PK

View Code? Open in Web Editor NEW

1.8K 28.0 437.0 47.84 MB

Altinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse clusters running on Kubernetes

Home Page: https://altinity.com

License: Apache License 2.0

Dockerfile 0.58% Go 52.02% Shell 7.10% Python 40.16% Smarty 0.14%

clickhouse-operator kubernetes clickhouse kubernetes-operator

clickhouse-operator's Introduction

Altinity Kubernetes Operator for ClickHouse

Altinity Kubernetes Operator for ClickHouse creates, configures and manages ClickHouse clusters running on Kubernetes.

Features

Creates ClickHouse clusters defined as custom resources
Customized storage provisioning (VolumeClaim templates)
Customized pod templates
Customized service templates for endpoints
ClickHouse configuration management
ClickHouse users management
ClickHouse cluster scaling including automatic schema propagation
ClickHouse version upgrades
Exporting ClickHouse metrics to Prometheus

Requirements

Kubernetes 1.19+

Documentation

Quick Start Guide

Advanced configuration

Maintenance tasks

Monitoring

How to contribute

Documentation index

License

Altinity Kubernetes Operator for ClickHouse is licensed under the Apache License 2.0.

See LICENSE for more details.

Commercial Support

Altinity is the primary maintainer of the operator. It is the basis of Altinity.Cloud and is also used in self-managed installations. Altinity offers a range of services related to ClickHouse and analytic applications on Kubernetes.

Official website - Get a high level overview of Altinity and our offerings.
Altinity.Cloud - Run ClickHouse in our cloud or yours.
Altinity Support - Get Enterprise-class support for ClickHouse.
Slack - Talk directly with ClickHouse users and Altinity devs.
Contact us - Contact Altinity with your questions or issues.
Free consultation - Get a free consultation with a ClickHouse expert today.

clickhouse-operator's People

Contributors

Stargazers

Watchers

Forkers

sunsingerus hodgesrm alex-zaitsev blinkov keevol shirkevich ikruglov artsilenkov dmitryfill twikio clemrodriguez jessestuart maxulan luizgusta laurettasz sonalihk kcking akoloth et2010 teralype introspection3 jb-abbadie ferpega sudeepknair kubernetes-try tianshangjun b3zn0gim an-excellent-choice shenjiaqi maskshell shurshun linevsd garrettlish achan-ns shinoi2 ilyshav leadcap traff1k kushagra391 guoshuai2016 sallery-x halcyondude boston01 wiktor2200 ayima fredchenbj ziplex vuliad zxshinxz leandroloi adheipsingh fourspaces ivanblack19 slimtom95 yuzhichang sunny19930321 shadowmodder easel npvinhsd arnonrgo shunpeizhang genezhang isgasho yan234280533 clooney2007 mohamedelbahr benjamin555 lookhua keznikl davydkov loganlinn stumitchell lomocadem sunset3000 buleant jangocheng 7hong utkarsh2811 allen-cao lumiseven yibeizj dxm-sec-hawking tamhinsf shadowdy tceason pguedes spongedu movsoftware sufian22 kunaldawn alexerm justhonor xjzrc ben-efiz thryvehealth erikdubbelboer leegkon den-crane xiedeyantu sarosejoshi

clickhouse-operator's Issues

Annotations don't work

Dear Sir or Madam

Could you help me?
I try to configure clickhouse-cluster without External_IP, but can't have this result.
I use this annotation:
service.beta.kubernetes.io/openstack-internal-load-balancer: "true"
but clickhouse-operator doesn't set it for LoadBalancer

How can I solve my problem?

What are all these configuration directories for?

Hey,

I have a question regarding the bunch of volumes/ConfigMaps mounted into the container. What are all these different config directories defined in the Deployment for, why do they need to be mounted by the operator and how is the influence to the actual ClickHouse pods spawned by the operator?

Mount Path	Mount Name	ConfigMap Name
`/etc/clickhouse-operator`	`etc-clickhouse-operator-folder`	`etc-clickhouse-operator-files`
`/etc/clickhouse-operator/conf.d`	`etc-clickhouse-operator-confd-folder`	`etc-clickhouse-operator-confd-files`
`/etc/clickhouse-operator/config.d`	`etc-clickhouse-operator-configd-folder`	`etc-clickhouse-operator-configd-files`
`/etc/clickhouse-operator/templates.d`	`etc-clickhouse-operator-templatesd-folder`	`etc-clickhouse-operator-templatesd-files`
`/etc/clickhouse-operator/users.d`	`etc-clickhouse-operator-usersd-folder`	`etc-clickhouse-operator-usersd-files`

I couldn't find any documentation how these are handled by the operator.

operator-files -- ok, seems to be the basic configuration of the operator
confd-files -- is empty? Why do we need it then?
configd-files - What's this? Is this configuration copied to all clickhouse instances? What is the difference to confd-files?
Seems to contain templates for ClickHouseInstallationTemplate, but what is ClickHouseInstallationTemplate any why do I need that? Is a ClickHouseInstallation resource not enough for starting Clickhouse?
usersd-files -- Why do we need that? Why is it not included in configd-files?

Best regards and thanks in advance
Matthias

PVC not found

https://github.com/Altinity/clickhouse-operator/blob/0.6.0/docs/custom_resource_explained.md

Follow the instruction here and cannot allocate specific PVC, fail to create specific Pod.

Support defining pod templates instead of pod specs

In the Clickhouse installation CRD currently we see that the ChiPodTemplate has a spec which is v1.PodSpec. this prohibits defining custom labels when the resources are created. allowing to define custom labels allows selecting resources of an installation with the same selector. To do so, it needs to be v1.PodTemplateSpec instead.

Could this be considered?

Using ODBC drivers

Hello.

We need an odbc driver in our cluster. How can we do this now?

We can add Unix-packages to the docker image, but how do we add ODBC configuration files via operator?

Thanks.

Using files section in kind: "ClickHouseInstallation" causes panic: assignment to entry in nil map

Hi!
Using the latest version I encountered the following problem, it does not occur in version 0.8.0.

Steps to reproduce:

install clickhouse-operator in version 0.9.0
apply ClickHouseInstallation attached below

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "simple-01"

spec:
  configuration:
    files:
      dict1.xml: |
        <yandex>
            <!-- ref to file /etc/clickhouse-data/config.d/source1.csv -->
        </yandex>

check clickhouse-operator logs

I0218 14:45:43.026586       1 creator.go:57] createServiceChi(clickhouse/clickhouse-simple-01)
I0218 14:45:43.026682       1 creator.go:60] Reconcile Service clickhouse/clickhouse-simple-01
I0218 14:45:43.026751       1 creator.go:64] Update Service clickhouse/clickhouse-simple-01
E0218 14:45:43.045957       1 runtime.go:69] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:522
/usr/local/go/src/runtime/map_faststr.go:204
/go/src/github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_settings.go:92
/go/src/github.com/altinity/clickhouse-operator/pkg/model/ch_config.go:69
/go/src/github.com/altinity/clickhouse-operator/pkg/model/ch_config_sections.go:52
/go/src/github.com/altinity/clickhouse-operator/pkg/model/creator.go:240
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:254
/go/src/github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:397
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:233
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:173
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:125
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:56
/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/controller.go:389
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1337
panic: assignment to entry in nil map [recovered]
	panic: assignment to entry in nil map

goroutine 124 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x1224180, 0x15533e0)
	/usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.Settings.GetStringMap(0xc0003d7d40, 0xc0003e2000)
	/go/src/github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_settings.go:92 +0x120
github.com/altinity/clickhouse-operator/pkg/model.(*ClickHouseConfigGenerator).GetFiles(...)
	/go/src/github.com/altinity/clickhouse-operator/pkg/model/ch_config.go:69
github.com/altinity/clickhouse-operator/pkg/model.(*configSections).CreateConfigsCommon(0xc0007c7a00)
	/go/src/github.com/altinity/clickhouse-operator/pkg/model/ch_config_sections.go:52 +0xcd
github.com/altinity/clickhouse-operator/pkg/model.(*Creator).CreateConfigMapChiCommon(0xc000819a10, 0xc0002521e0)
	/go/src/github.com/altinity/clickhouse-operator/pkg/model/creator.go:240 +0x33
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).reconcileChi(0xc00068cdc0, 0xc00065e000, 0xc000190e00, 0x20)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:254 +0x69
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1.(*ClickHouseInstallation).WalkTillError(0xc00065e000, 0xc000757990, 0xc000757980, 0xc000757970, 0xc000757960, 0xc000190e00, 0x2031e98)
	/go/src/github.com/altinity/clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_chi.go:397 +0x3d
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).reconcile(0xc00068cdc0, 0xc00065e000, 0x0, 0x0)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:233 +0x283
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).updateChi(0xc00068cdc0, 0xc000683c00, 0xc00065e000, 0x6, 0x1390e4d)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:173 +0x6e5
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).addChi(0xc00068cdc0, 0xc000026e00, 0xc00051dd70, 0xc00051dcd0)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:125 +0x1e2
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*worker).processItem(0xc00068cdc0, 0x116f660, 0xc000369ba0, 0xc00037cb00, 0xc0004fce88)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/worker.go:56 +0x8dd
github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).runWorker(0xc0001381e0)
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/controller.go:389 +0x1fb
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000760400)
	/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000760400, 0x3b9aca00, 0x0, 0x46f201, 0xc000427e00)
	/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc000760400, 0x3b9aca00, 0xc000427e00)
	/go/src/github.com/altinity/clickhouse-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/altinity/clickhouse-operator/pkg/controller/chi.(*Controller).Run
	/go/src/github.com/altinity/clickhouse-operator/pkg/controller/chi/controller.go:370 +0x213

Possible to monitor all namespaces or by pattern

I see in the documentation you can specify which namespaces should be monitored by the operator:

watchNamespaces:
- dev

It appears when watchNamespaces is left empty the operator only monitors the namespace it is installed into. Is it possible to have it monitor all namespaces or namespaces by pattern. For example:

watchNamespaces:
- customers-*

We have multiple tenants sharing our K8s cluster which are segregated by namespace. While possible to update the operators config when a new tenant is added or to run the operator for each tenant, this is less than ideal.

Thanks for all the work on the operator. The schema maintenance is really slick!

Resources not cleaned up

I'm having difficulty using the operator. The first thing I notice is that resources are not cleaned up even when I delete the ClickHouseInstallation. The main problem I'm having is setting up a cluster with a specific disk size and a specific docker image.

Here is my resource:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: dev
  namespace: timeseries
spec:
  defaults:
    deployment:
      podTemplate: clickhouse-v19.4.3.11
      volumeClaimTemplate: default
  templates:
    volumeClaimTemplates:
    - name: default
      persistentVolumeClaim:
        metadata:
          name: USE_DEFAULT_NAME
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
    podTemplates:
    - name: clickhouse-v19.4.3.11
      containers:
      - name: clickhouse
        volumeMounts:
        - name: default
          mountPath: /var/lib/clickhouse
        image: yandex/clickhouse-server:19.4.3.11
        resources:
          requests:
            memory: 512Mi
            cpu: 500m
          limits:
            memory: 2Gi
            cpu: 2
  configuration:
    zookeeper:
      nodes:
        - host: ts-zoo0
          port: 2181
        - host: ts-zoo1
          port: 2181
        - host: ts-zoo2
          port: 2181
    clusters:
    - name: ts1
      layout:
        type: Standard
        shardsCount: 1
        replicasCount: 1

The pod fails to start with the error:

create Pod chi-1de443-1579-0-0-0 in StatefulSet chi-1de443-1579-0-0 failed error: Pod "chi-1de443-1579-0-0-0" is invalid: [spec.containers[0].volumeMounts[0].name: Not found: "clickhouse-data-test", spec.containers[0].volumeMounts[4].mountPath: Invalid value: "/var/lib/clickhouse": must be unique]

Checking the statefulset, there are in fact two conflicting mount paths:

Mounts:
      /etc/clickhouse-server/conf.d/ from chi-1de443-deploy-confd-1579-0-0 (rw)
      /etc/clickhouse-server/config.d/ from chi-1de443-common-configd (rw)
      /etc/clickhouse-server/users.d/ from chi-1de443-common-usersd (rw)
      /var/lib/clickhouse from clickhouse-data-test (rw)
      /var/lib/clickhouse from clickhouse-data (rw)

Not sure why there are two mounts there.

Dropping tables on installation deletion

Recently we had an issue with tables being dropped after we accidentally deleted the ClickHouse installation object. Normally, we guard against any accidental deletion of in-cluster objects with PersistentVolumes reclaim policy set to "Retain" and the fact that ClickHouse doesn't delete stuff by itself but rather puts it in detached state.

I found out that the issue we had is caused by this function:

clickhouse-operator/pkg/model/schemer.go

Line 223 in b7c71f8

 func (s *Schemer) HostGetDropTables(host *chop.ChiHost) ([]string, []string, error) { 

What is the use-case for dropping the actual tables from ClickHouse in this case?
If you'd want to release storage, could the same be achieved with just deleting associated PersistentVolumeClaim and leave the underlying data cleaning mechanism to PV (i.e. Retain, Recycle, Delete)?

ReplicateTable are not getting created in the new Replica

Whenever we increase replication, from operator logs i see that it's running below query to find which tables to create in new replica

 SELECT DISTINCT
        database AS name,
        concat('CREATE DATABASE IF NOT EXISTS ', name) AS create_db_query

It often results in a message like

Creating replicated objects: []

Even though there are many replicated tables, it doesn't create any. Manually running the generated query in clickhouse gives the correct result.

Can someone point out what may be the issue here?

Zookeeper authorization problem

Hello.

I think that there is a problem with Zookeeper authorization in clickhouse configured with clickhouse-operator.
I deployed Zookeeper with the latest chart from bitnami https://github.com/bitnami/charts/tree/master/bitnami/zookeeper

Without auth enabled in Zookeeper, everything works fine, but when I tried to enable it, I got an error during table creation

Received exception from server:
Code: 999, e.displayText() = Coordination::Exception: Not authenticated, path: /clickhouse (version 19.16.10.44 (official build))


2020.02.27 12:50:04.027507 [ 35 ] {ebc7c680-14d4-416e-8a9d-83743ed7087e} <Error> executeQuery: Code: 999, e.displayText() = Coordination::Exception: Not authenti │
│ cated, path: /clickhouse (version 19.16.10.44 (official build)) (from 10.2.15.119:63674) (in query: CREATE TABLE events_local on cluster '{cluster}' (     event_ │
│ date  Date,     event_type  Int32,     article_id  Int32,     title       String ) engine=ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard │
│ }/{database}/{table}', '{replica}', event_date, (event_type, article_id), 8192) ), Stack trace:                                                                   │
│ 2020.02.27 12:50:04.027694 [ 35 ] {ebc7c680-14d4-416e-8a9d-83743ed7087e} <Error> HTTPHandler: Code: 999, e.displayText() = Coordination::Exception: Not authentic │
│ ated, path: /clickhouse, Stack trace:

While zkcli is able to connect without any problem

$ brew install zookeeper
$ cat /Users/user/.jaas.conf
Client{
      org.apache.zookeeper.server.auth.DigestLoginModule required
      username="zk_user"
      password="password";
};

JVMFLAGS="-Djava.security.auth.login.config=/Users/user/.jaas.conf" zkcli

Part of ClickHouseInstallation:

kind: 'ClickHouseInstallation'

metadata:
  name: 'demo'

spec:
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper-host
      identity: "zk_user:password"

I tried a couple of times with the same result.
Versions:
k8s: 1.14.9
clickhouse: 19.16.10.44
clickhouse-operator: 0.9.1
zookeeper chart: 5.4.2

Adopt f1yegor's dashboard to operator's metrics naming convention

https://grafana.com/grafana/dashboards/882

modify profile to user "default" will change password to "default" and cause other problems

Hi,
I want to change the permission of default user to forbid ddl, so I have the following config:

profiles:
  not_allow_ddl/allow_ddl: 0
users:
  default/profile: not_allow_ddl
  admin/password: admin
  admin/profile: default

but these codes change the password to "default" like this
https://github.com/Altinity/clickhouse-operator/blob/master/pkg/apis/clickhouse.altinity.com/v1/type_config.go#L267-L269

users.xml: |
<yandex>
    <users>
        <admin>
            <networks>
                <ip>::/0</ip>
            </networks>
            <password>admin</password>
            <profile>default</profile>
            <quota>default</quota>
        </admin>
        <default>
            <networks>
                <ip>::/0</ip>
            </networks>
            <password>default</password>
            <profile>not_allow_ddl</profile>
            <quota>default</quota>
        </default>

I am working on a shard env, then I have found this issue #265 which means I should not change the password, and I do not want to change it either

please help me.

metrics exporter lose monitored ClickHouse instances list and stop publish metrics, after restart metrics-exporter container inside clickhouse-operator pod

Steps to reproduce

install clickhouse-operator
install any kind: ClickhouseInstallation from chi-examples
reboot metrics-exporter via

kubectl exec -n $( kubectl get pods --all-namespaces | grep clickhouse-operator | awk '{print $1 " " $2}') -c metrics-exporter reboot

run port-forward to metrics exporter 8888

kubectl -n $( kubectl get svc --all-namespaces | grep clickhouse-operator-metrics | awk '{print $1}') port-forward service/clickhouse-operator-metrics 8888

open http://localhost:8888/chi

Expected results

json array with current running clickhouse instances

Actual results

empty json

This state also happens in following situations:

OOP Kill clickhouse-operator/metrics-exporter container
reschedule clickhouse-operator/metrics-exporter
whole restart kubernetes node / cluster

Solution proposal

simple add retries and timeout options to on MakeRESTCall in file https://github.com/Altinity/clickhouse-operator/blob/master/pkg/apis/metrics/rest.go
add retries and logging to https://github.com/Altinity/clickhouse-operator/blob/master/pkg/controller/chi/controller.go on updateWatchAsync

Cluster scale up/down - moving data

https://github.com/Altinity/clickhouse-operator/blob/master/docs/schema_migration.md

Is it possible to scale up cluster without losing data?

ClickHouse Performance in Container(Docker).

Hi Altinity team,
Does your team make any performance benchmark over clickhouse on k8s/Docker?
In case, yes, What docker storage driver, CSI plugin, you use?
And what about CNI plugins?

single transient failure stops metrics fetcher

right now a single failure to fetch metrics from any chi pod will stop all fetching of metrics for that installation, until it is updated or the ch-operator pod is rolled.

Instead, some more resilient error handling should be done so that a single transient failure doesnt stop all metric collection

Recreate from backup strategy

I could not find any strategy for reusing a PVC/PV.

Imagine a cloud provider like DigitalOcean or GCE where you have snapshots and you are able to mount a disk from snapshot as PV.

How could I specify in the ClickhouseInstallation resource to reuse an existing data source?
As far as I understand, this is not supported currently, correct?

Trivial example url returns 404

Link https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/docs/examples/01-simple-layout-01-1shard-1repl.yaml returns 404.

Mentioned in"
https://github.com/Altinity/clickhouse-operator/blob/master/docs/quick_start.md#trivial-example

[Feature] expose query_log metrics from system.query_log table

The query log table in clickhouse provides information like successful/errored queries, query duration, and which user and quota were used to execute the query. Would the ch-operator team entertain a PR to expose these per-query statistics?

How to create a external-ip for the loadBalancer

I use the 01-simple-layout-01-1shard-1repl.yaml to create the cluster，and I can
kubectl -n testch exec -it chi-simple-01-cluster-0-0-0 -- clickhouse-client to connect the cluster。

But my service‘s EXTERNAL-IP is pending’
How can I slove this to make the service be visit by the out side？

Some of ZK recommended settings are not set in provided manifest

Compare: https://github.com/Altinity/clickhouse-operator/blob/master/deploy/zookeeper/quick-start-persistent-volume/zookeeper-3-nodes.yaml
And: https://clickhouse.yandex/docs/en/operations/tips/#zookeeper

preAllocSize=131072
snapCount=3000000
leaderServes=yes
standaloneEnabled=false

JVM settings are also not set:

JAVA_OPTS="-Xms{{ cluster.get('xms','128M') }} \
    -Xmx{{ cluster.get('xmx','1G') }} \
    -XX:+UseParNewGC \
    -XX:+UseConcMarkSweepGC \
    -XX:+CMSParallelRemarkEnabled"

User Configuration

Hi, I'm trying out ClicHouse Operator recently and have some problems figuring out the user configuration.

My approach is creating a YAML file like the one below and use kubectl apply -f myfile.yaml --namespace myspace to deploy it.

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "test"

spec:
  defaults:
    templates: 
      dataVolumeClaimTemplate: default
      podTemplate: clickhouse:19.6
  configuration:
    users:
      admin/password_sha256_hex: 85fe7e87fc89408f80fdd936c3967c4cd040c4ad676d2e5ec37017c7f5167345
      default/password_sha256_hex: 7388286b8d0a085d1fb57a8a4a164005d228cdb305d2f04f84090c077d2be8ce
      default/networks/ip:
        - "127.0.0.1"
    settings:
      compression/case/method: zstd
    zookeeper:
      nodes:
      - host: zookeeper.zoons
    clusters:
      - name: test
        layout:
          shardsCount: 3
          replicasCount: 2
    
  templates:
    volumeClaimTemplates:
      - name: default
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 500Gi
    podTemplates:
      - name: clickhouse:19.6
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.6.2.11
              resources:
                requests:
                  memory: 4Gi
                  cpu: 2

I check the user.xml file and it looks alright.

<yandex>
    <users>
        <admin>
            <networks>
                <ip>::/0</ip>
            </networks>
            <password_sha256_hex>85fe7e87fc89408f80fdd936c3967c4cd040c4ad676d2e5ec37017c7f5167345</password_sha256_hex>
            <profile>default</profile>
            <quota>default</quota>
        </admin>
        <default>
            <networks>
                <ip>127.0.0.1</ip>
            </networks>
            <password_sha256_hex>7388286b8d0a085d1fb57a8a4a164005d228cdb305d2f04f84090c077d2be8ce</password_sha256_hex>
            <profile>default</profile>
            <quota>default</quota>
        </default>
        <test>
            <networks>
                <ip>::/0</ip>
            </networks>
            <password>default</password>
            <profile>default</profile>
            <quota>default</quota>
        </test>
    </users>
</yandex>

The problem is the password_sha256_hex settings seem to function incorrectly.
admin and default should have different passwords but I can log in with the same password.

root@chi-smart-medic-smart-medic-2-1-0:/etc/clickhouse-server/users.d# clickhouse-client -u admin --password ZHJ5Y2FyZWVzc2VudGlhbDEyMw==
ClickHouse client version 19.6.2.11 (official build).
Connecting to localhost:9000 as user admin.
Code: 193. DB::Exception: Received from localhost:9000, 127.0.0.1. DB::Exception: Wrong password for user admin. 

root@chi-test-2-1-0:/etc/clickhouse-server/users.d# clickhouse-client -u admin --password ZHJ5Y2FyZWVzc2VudGlhbA==
ClickHouse client version 19.6.2.11 (official build).
Connecting to localhost:9000 as user admin.
Connected to ClickHouse server version 19.6.2 revision 54418.

chi-test-2-1-0.chi-test-2-1.clickhouse.svc.cluster.local :) exit
Happy new year.
root@chi-test-2-1-0:/etc/clickhouse-server/users.d# clickhouse-client --password ZHJ5Y2FyZWVzc2VudGlhbA==
ClickHouse client version 19.6.2.11 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 19.6.2 revision 54418.

chi-test-2-1-0.chi-smart-medic-smart-medic-2-1.clickhouse.svc.cluster.local :) exit
Bye.

Another problem is the test user has generated automatically and I don't want it.

Any suggestions would be great. Thanks.

"latest" tag used in the installation template

Hi!
"latest" tag is used in for clickhouse-operator in default yaml (https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install.yaml)

This could be a problem, because operator version could be updated unexpectedly, and it could break some installations (0.8.0 requires installation name shorter, that 15 symbols).
Should we replace it with the current release versions?

password of default user is not propagated to shards

Hi,

when creating a cluster where the password of default user is set in the manifest, it will not be propagated to remote_servers.xml. This leads to errors using umbrella tables:

From this example: https://github.com/Altinity/clickhouse-operator/blob/master/docs/replication_setup.md

SELECT * FROM events;

Code: 194, e.displayText() = DB::Exception: Received from chi-test-cho-with-shards-simple-1-0:9000. DB::Exception: Password required for user default. (version 20.1.4.14 (official build))

This is my manifest file:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "test-cho-with-shards"
spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: data-volume-template
  configuration:
    users:
      default/password: secret
    zookeeper:
      nodes:
        - host: zookeeper.zoons
    clusters:
      - name: "simple"
        layout:
          shardsCount: 3
  templates:
    volumeClaimTemplates:
      - name: data-volume-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

I think I could insert user/password manually but am applied change in manifest will remove it again.

One-to-one mapping between StatefulSets and Shards

Based on my limited experience with the ClickHouse operator, it seems that it deploys a Kubernetes StatefulSet per each shard and replica. That is, my test cluster which is composed of 2 shards and 2 replicas per shard yields 4 Kubernetes StatefulSet objects.

This is the ClickHouse CRD:

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  labels:
    environment: dev
  name: replicated
  namespace: infra
spec:
  configuration:
    clusters:
    - layout:
        replicasCount: 2
        shardsCount: 2
      name: replicated
...

The problem is that the operator code creates StatefulSets with just one replica, as seen here:

clickhouse-operator/pkg/model/creator.go

Line 229 in a71190f

replicasNum := int32(1)

Would it be possible to change this layout to a layout where each StatefulSet maps to a shard, and has as many replicas as the number of replicas in the shard?

With the current design, and based on my CRD that describes a ClickHouse cluster composed of two shards and two replicas, I get 4 StatefulSets (1 replica each) which is not very manageable neither in Prometheus or by back ups (e.g. Stash).

$ kubectl -n infra get sts
NAME                            READY   AGE
chi-replicated-replicated-0-0   1/1     111m
chi-replicated-replicated-0-1   1/1     110m
chi-replicated-replicated-1-0   1/1     109m
chi-replicated-replicated-1-1   1/1     109m
zookeeper                       3/3     111m

I would expect that only 2 StatefulSets are created. Something like this:

$ kubectl -n infra get sts
NAME                            READY   AGE
chi-replicated-replicated-0     2/2     111m
chi-replicated-replicated-1     2/2     109m
zookeeper                       3/3     111m

In total, still 4 Pods (2 shards x 2 replicas/shard) but distributed in a different way.
Is there a reason why this is not done like this?

Pentaho can not insert/update clickhouse database

failed pentaho transformation to clickhouse,

Why not use pod annotations to scrape metrics?

Hello.

I've found that it's supposed to monitor clickhouse with ServiceMonitor https://github.com/Altinity/clickhouse-operator/blob/master/deploy/prometheus/prometheus-clickhouse-operator-service-monitor.yaml

But why not use pod annotations to scrape metrics? This is the part of clickhouse-operator-install-template.yaml file with two lines added in the end.

---
# Possible Template Parameters:
#
# clickhouse-demo
# altinity/clickhouse-operator:0.9.1
# altinity/metrics-exporter:0.9.1
#
# Setup Deployment for clickhouse-operator
# Deployment would be created in kubectl-specified namespace
kind: Deployment
apiVersion: apps/v1
metadata:
  name: clickhouse-operator
  namespace: clickhouse-demo
  labels:
    app: clickhouse-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: clickhouse-operator
  template:
    metadata:
      annotations:
        prometheus.io/port: '8888'
        prometheus.io/scrape: 'true'

Include deployment of CHproxy?

See https://github.com/Vertamedia/chproxy

Cluster with replication

I have the following cluster setup:

clusters:
    - name: ts1
      layout:
        type: Standard
        shardsCount: 3
        replicasCount: 2

The operator worked great and got all the nodes up and running.

I created the following tables:

CREATE TABLE observation_local ON CLUSTER ts1
(
    ts DateTime,
    env UInt16,
    host UInt16,
    metric String,
    value Float64 CODEC(Delta(1), ZSTD)
)
ENGINE = MergeTree
PARTITION BY (env, toYYYYMMDD(ts), host)
ORDER BY (metric, ts);

CREATE TABLE observation ON CLUSTER ts1
(
    ts DateTime,
    env UInt16,
    host UInt16,
    metric String,
    value Float64 CODEC(Delta(1), ZSTD)
)
ENGINE = Distributed(ts1, default, observation_local, host);

I inserted 60 seconds of data for a single host which landed on a single shard - due to the shard key of host. All that looks good.

When I query the distributed table, sometimes I get 60 rows and sometimes I get 0 rows back.

I tracked this down to the shards are not replicating the data. Here is the config on one pod:

<yandex>
    <remote_servers>
        <ts1>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>chi-1de443-1579-0-0</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>chi-1de443-1579-0-1</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>chi-1de443-1579-1-0</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>chi-1de443-1579-1-1</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <internal_replication>true</internal_replication>
                <replica>
                    <host>chi-1de443-1579-2-0</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>chi-1de443-1579-2-1</host>
                    <port>9000</port>
                </replica>
            </shard>
        </ts1>
    </remote_servers>
</yandex>

Shouldn't internal_replication be set to false as described here https://clickhouse.yandex/docs/en/operations/table_engines/distributed/?

Metrics fetches never timeout

Hello all.

We have found an issue in our cluster where the operator metric fetch loop can get stuck waiting for a DB Ping or Exec that will never time out. I believe this can be solved by using PingContext and ExecContext inside of clickhouse.go, and passing in a context.Context with a reasonable timeout -- maybe something like 10 seconds.

What do you folks think? Thanks for the awesome operator!

Build new Docker images

Could you please build new Docker images, for the Operator and Metrics Exporter from "master"?

Operator drops previous monitoring

When deploying new clickhouse installation information about old chi monitoring is dorpped.

How to reproduce:

Deploy clickhouse-operator to k8s.
Map port for monitoring kubectl port-forward svc/clickhouse-operator-metrics 8888
Visit http://localhost:8888/chi
There should be empty array as expected.
Deploy dev chi to dev namespace.
Visit http://localhost:8888/chi
There will be something like [{"namespace":"dev","name":"dev","hostnames":["chi-dev-0-0.dev.svc.cluster.local","chi-dev-0-1.dev.svc.cluster.local","chi-dev-1-0.dev.svc.cluster.local","chi-dev-1-1.dev.svc.cluster.local"]}]
Deploy prod chi to prod namespace.
Visit http://localhost:8888/chi
I expect to see second element in the array, but instead there are something like [{"namespace":"dev","name":"dev","hostnames":["chi-prod-0-0.prod.svc.cluster.local","chi-prod-0-1.prod.svc.cluster.local","chi-prod-1-0.prod.svc.cluster.local","chi-prod-1-1.prod.svc.cluster.local"]}]

k8s version :

GitVersion:"v1.17.0", 
GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean",
BuildDate:"2019-12-13T11:51:44Z", GoVersion:"go1.13.4", 
Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"16",
GitVersion:"v1.16.3", 
GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean",
BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12",
Compiler:"gc", Platform:"linux/amd64"}

clickhouse-operator version :
altinity/clickhouse-operator:0.7.0

test

Support for ClickHouse exporter

In order to allow for monitoring ClickHouse, it would be nice if the operator could automatically deploy sidecar containers in each StatefulSet that use https://github.com/f1yegor/clickhouse_exporter.

Disabling Loadbalancer using serviceTemplate

Can you please tell me how to disable clickhouse-operater from creating a public load balancer. I have a separate service that creates an internal load balancer (AWS) Currently I am trying to override the serviceTemplate to disable ELB bit it is not working (using v 0.6)

serviceTemplates:
      - name: chi-service-template
        metadata:
          name: clickhouse-service-tdd-{chi}
        spec:
          ports:
            - name: http
              port: 8123
            - name: client
              port: 9000
            - name: mysql
              port: 9001
          type: ClusterIP
          ClusterIP: None

Thanks

Make it possible to watch all namespaces from non kube-system namespace

Currently deploying the operator in another namespace then kube-system implicitly disallows global watching:

clickhouse-operator/pkg/apis/clickhouse.altinity.com/v1/type_config.go

Lines 323 to 329 in aaee541

 namespace := os.Getenv(OPERATOR_POD_NAMESPACE) 

 if namespace == "kube-system" { 

 // Do nothing, we already have len(config.WatchNamespaces) == 0 

 } else { 

 // We have WATCH_NAMESPACE specified 

 config.WatchNamespaces = []string{namespace} 

 }

It would be nice to have a parameter for this behavior, we deploy all operators in the operator namespace and they are cluster global.

I am open to implement this but what is a nice approach to this? A WATCH_ALL_NAMESPACES boolean? When WATCH_NAMESPACES or WATCH_NAMESPACE is defined it will be ignored.

PS why not consolidate WATCH_NAMESPACE and WATCH_NAMESPACES into just WATCH_NAMESPACES? Nothing wrong with a list with one entry or am I missing something? Feels weird as enduser :)

Add affinity and tolerations to ChiPodTemplate

Currently only Name, Containers and Volumes from the PodTemplate are copied to the Pod spec. I'm trying to use dedicated nodes for Clickhouse, so it would be nice if the affinity and tolerations are copied from the template, so I can do something like this:

    podTemplates:
    - name: clickhouse-v19.6.2.11
      tolerations:
        - key: "dedicated"
          operator: "Equal"
          value: "clickhouse"
          effect: "NoSchedule"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: dedicated
                operator: In
                values:
                - clickhouse

Problem with is_local in system.clusters table

I have a cluster of 2 shards / 2 replicas deployed on real machines and another one deployed on Kubernetes with Clickhouse-operator. I noticed a strange difference in the value of is_local in system.clusters table. It is a problem as our application relies on this to determine the shard it is connected to.

Kubernetes:

# kubectl -n test exec -it chi-dev-vscluster-0-0-0 -- clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       chi-dev-vscluster-0-0   10.244.2.84     1
1       2       chi-dev-vscluster-0-1   10.244.0.64     1
2       1       chi-dev-vscluster-1-0   10.244.2.85     0
2       2       chi-dev-vscluster-1-1   10.244.2.86     0
# kubectl -n test exec -it chi-dev-vscluster-0-1-0 -- clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       chi-dev-vscluster-0-0   10.244.2.84     1
1       2       chi-dev-vscluster-0-1   10.244.0.64     1
2       1       chi-dev-vscluster-1-0   10.244.2.85     0
2       2       chi-dev-vscluster-1-1   10.244.2.86     0
# kubectl -n test exec -it chi-dev-vscluster-1-0-0 -- clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       chi-dev-vscluster-0-0   10.244.2.84     0
1       2       chi-dev-vscluster-0-1   10.244.0.64     0
2       1       chi-dev-vscluster-1-0   10.244.2.85     0 <= should be 1!
2       2       chi-dev-vscluster-1-1   10.244.2.86     0 <= should be 1!
# kubectl -n test exec -it chi-dev-vscluster-1-1-0 -- clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       chi-dev-vscluster-0-0   10.244.2.84     0
1       2       chi-dev-vscluster-0-1   10.244.0.64     0
2       1       chi-dev-vscluster-1-0   10.244.2.85     0 <= should be 1!
2       2       chi-dev-vscluster-1-1   10.244.2.86     0 <= should be 1!

Real machines:

# clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       dca-test2.iv.local      10.148.20.42    1
1       2       dca-test4.iv.local      10.148.20.44    1
2       1       dca-test3.iv.local      10.148.20.43    0
2       2       dca-test5.iv.local      10.148.20.49    0
# clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       dca-test2.iv.local      10.148.20.42    0
1       2       dca-test4.iv.local      10.148.20.44    0
2       1       dca-test3.iv.local      10.148.20.43    1
2       2       dca-test5.iv.local      10.148.20.49    1
# clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       dca-test2.iv.local      10.148.20.42    1
1       2       dca-test4.iv.local      10.148.20.44    1
2       1       dca-test3.iv.local      10.148.20.43    0
2       2       dca-test5.iv.local      10.148.20.49    0
# clickhouse-client -q "select shard_num,replica_num,host_name,host_address,is_local from system.clusters where cluster='vscluster'"
1       1       dca-test2.iv.local      10.148.20.42    0
1       2       dca-test4.iv.local      10.148.20.44    0
2       1       dca-test3.iv.local      10.148.20.43    1
2       2       dca-test5.iv.local      10.148.20.49    1

This is the YAML I used to deploy:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "dev"
spec:
  defaults:
    templates:
      volumeClaimTemplate: volumeclaim-template
  configuration:
    zookeeper:
      nodes:
      - host: zookeeper.test
        port: 2181
    clusters:
      - name: "vscluster"
        layout:
          shardsCount: 2
          replicasCount: 2
  templates:
    volumeClaimTemplates:
      - name: volumeclaim-template
#        reclaimPolicy: Retain
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi

Do you have an idea why is_local=0 in some of the containers?

How to deal with DNS caching when Pod IPs can/will change

It looks like you are deploying each "node" as a single instance StatefulSet, and relying on the DNS name of each Pod under a headless Service for replication (since the default interserver_http_host is hostname -f) as well as cluster definitions in <yandex><remote_server>....</remote_servers></yandex>. This works fine on initial deployment. However, in the event that a Pod is restarted (planned or unplanned), the DNS caching on each node creates a problem. Is the intention to trigger something like SYSTEM DROP DNS CACHE on each node after the restarted Pod is ready?

On another note, in our own deployment using a Helm chart, we tried dealing with this in another way. We placed a regular Service in front of each StatefulSet, knowing that it's ClusterIP would be static (unless we remove that Service, which is much less likely). We then explicitly set the <interserver_http_host> to that Service name, and used it in <remote_servers> as well. This seems to work for everything EXCEPT any ON CLUSTER queries. We have no idea WHY those queries seem to be blocked at a network layer -- they're just timing out. We haven't traced that yet, but were wondering if you maybe know why these queries fail.

clickhouse cluster create database error

we create clickhouse cluster with attached yaml file
cluster-latest.txt

status from kubernetes:

We are using apache superset as our clickhouse IDE, we had error to create database on cluster, the database is created but there is error:

We have error when create database on cluster,

but the database was created,

Is this behaviour correct ?
Is our create database command on cluster correct ?
we also see the error from kubectl exec -it , how to resolve this issue ??

ExternalIP for every chi-service

Hello

Help me please. I can't deploy chi-services with external IP.
How can I do it?

Thanks

Is there a Project Roadmap?

Cool project! 🎉

My team is deeply invested in Clickhouse. We are already running it in production and we want to build a new cluster in Kubernetes. This project looks promising because I tried and it is simple. We are planning to use something like this operator in production. What we would like to know if there are any existing issues and maybe future features or enhancements.

how to change clickhouse default password in kubernetes

We have deployed clickhouse on kubernetes, with detail :

and we can access clickhouse from command line:

core@manager-02 ~ $ cat clickhouse.sh
#! /usr/bin/bash
sudo kubectl exec -it $(sudo kubectl get pod -l clickhouse.altinity.com/app=chop -o jsonpath='{.items[0].metadata.name}' ) -- clickhouse-client

but tabix can not access clickhouse due to user default do not configured with any passwod, from documentation we need to allow 0.0.0.0:8123 and give it password from xml file, in kubernetes we can not change xml, because if kubernetes destroy and re-create clickhouse pod we will loose the password again

so how to resolve this issue ??

we want to ingest data to clickhouse kubernetes via pentaho also, but also can not connect:

`Error connecting to database [clickhouse] :org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

Error connecting to database: (using class ru.yandex.clickhouse.ClickHouseDriver)
ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 195, host: 10.9.56.132, port: 8123; Code: 195, e.displayText() = DB::Exception: User default is not allowed to connect from address 10.244.34.0 (version 19.6.2.11 (official build))

org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

at org.pentaho.di.core.database.Database.normalConnect(Database.java:472)
at org.pentaho.di.core.database.Database.connect(Database.java:370)
at org.pentaho.di.core.database.Database.connect(Database.java:341)
at org.pentaho.di.core.database.Database.connect(Database.java:331)
at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:83)
at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestResults(DatabaseFactory.java:112)
at org.pentaho.di.core.database.DatabaseMeta.testConnectionSuccess(DatabaseMeta.java:2811)
at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:621)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:263)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:687)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594)
at org.eclipse.swt.widgets.Display.executeNextEvent(Display.java:1217)
at org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1198)
at org.eclipse.swt.widgets.Display.safeReadAndDispatch(Display.java:1181)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1173)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:869)
at org.eclipse.jface.window.Window.open(Window.java:845)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47)
at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:114)
at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:61)
at org.pentaho.di.ui.spoon.delegates.SpoonDBDelegate.editConnection(SpoonDBDelegate.java:96)
at org.pentaho.di.ui.spoon.Spoon.editConnection(Spoon.java:2795)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem.access$100(JfaceMenuitem.java:43)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem$1.run(JfaceMenuitem.java:106)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:493)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:575)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:492)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:403)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:109)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:687)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:594)
at org.eclipse.swt.widgets.Display.executeNextEvent(Display.java:1217)
at org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1198)
at org.eclipse.swt.widgets.Display.safeReadAndDispatch(Display.java:1181)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1173)
at org.eclipse.rap.rwt.application.AbstractEntryPoint.createUI(AbstractEntryPoint.java:69)
at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle.createUI(RWTLifeCycle.java:177)
at org.eclipse.rap.rwt.internal.lifecycle.RWTLifeCycle$UIThreadController.run(RWTLifeCycle.java:290)
at java.lang.Thread.run(Thread.java:748)
at org.eclipse.rap.rwt.internal.lifecycle.UIThread.run(UIThread.java:107)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error connecting to database: (using class ru.yandex.clickhouse.ClickHouseDriver)
ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 195, host: 10.9.56.132, port: 8123; Code: 195, e.displayText() = DB::Exception: User default is not allowed to connect from address 10.244.34.0 (version 19.6.2.11 (official build))

at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:585)
at org.pentaho.di.core.database.Database.normalConnect(Database.java:456)
... 57 more

Caused by: java.lang.RuntimeException: ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 195, host: 10.9.56.132, port: 8123; Code: 195, e.displayText() = DB::Exception: User default is not allowed to connect from address 10.244.34.0 (version 19.6.2.11 (official build))

at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:78)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:47)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:29)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:567)
... 58 more

Caused by: ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 195, host: 10.9.56.132, port: 8123; Code: 195, e.displayText() = DB::Exception: User default is not allowed to connect from address 10.244.34.0 (version 19.6.2.11 (official build))

at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:58)
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:28)
at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:820)
at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:616)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:127)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:110)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:105)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:89)
... 65 more

Caused by: java.lang.Throwable: Code: 195, e.displayText() = DB::Exception: User default is not allowed to connect from address 10.244.34.0 (version 19.6.2.11 (official build))

at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:53)
... 73 more

Custom URL: jdbc:clickhouse://10.9.56.132:8123/database
Custom Driver Class: ru.yandex.clickhouse.ClickHouseDriver

clickhouse kubernetes service loadbalancer ip 10.9.56.132 , need you help and recommendation how to resolve this,

Thank you

Add shard does not automatically creates schema

Root cause is JOIN logic changed in new version of clickhouse, but operator does not adaptation.

https://clickhouse.yandex/docs/en/query_language/select/#select-join
https://github.com/Altinity/clickhouse-operator/blob/master/pkg/model/schemer.go#L125
clickhouse-operator log:

I1230 03:55:53.121183       1 schemer.go:61] clickhouse-opod.default.svc.cluster.local
I1230 03:55:53.172104       1 clickhouse.go:97] FAILED Query(http://clickhouse_operator:clickhouse_operator_password@c
lickhouse-opod.default.svc.cluster.local:8123/) Code: 48, Message: Old ANY INNER|RIGHT|FULL JOINs are disabled by defa
ult. Their logic would be changed. Old logic is many-to-one for all kinds of ANY JOINs. It's equil to apply distinct f
or right table keys. Default bahaviour is reserved for many-to-one LEFT JOIN, one-to-many RIGHT JOIN and one-to-one IN
NER JOIN. It would be equal to apply distinct for keys to right for SQL: SELECT DISTINCT 
        database AS name, 
        concat('CREATE DATABASE IF NOT EXISTS ', name) AS create_db_query
FROM 
(
        SELECT DISTINCT database
        FROM cluster('opod-cluster', system, tables)
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT DISTINCT extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS shard_database
        FROM cluster('opod-cluster', system, tables) 
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
) 
UNION ALL
SELECT DISTINCT 
        name, 
        replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW)', 'CREATE \\1 IF NOT EXISTS')
FROM 
(
        SELECT 
                database, 
                name,
                2 AS order
        FROM cluster('opod-cluster', system, tables)
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
        UNION ALL
        SELECT 
                extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS shard_database, 
                extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS shard_table,
                1 AS order
        FROM cluster('opod-cluster', system, tables)
        WHERE engine = 'Distributed'
        SETTINGS skip_unavailable_shards = 1
) 
ANY INNER JOIN (select distinct database, name, create_table_query from cluster('opod-cluster', system, tables) SETTIN
GS skip_unavailable_shards = 1) USING (database, name)
ORDER BY order
I1230 03:55:53.172148       1 retry.go:39] FAILED attempt 1 of 10, sleep 5 sec and retry: SELECT DISTINCT

Operator Pod Memory leak

Operator version - 0.6.0

The problem is that after operator pod reach some memory limit, it would be Evicted by kubernetes. We have to manually clear these pods to prevent them from using resources.

PVC volume resize on clickhouse

I tried an option to resize PVC to a larger volume. I am using AWS. I see no changes after deleting the pod for changes to take effect. Is this possible through clickhouse-operator?

Support zookeeper configuration on clickhouse cluster level

Hi Team

Does your team have a plan to support zookeeper configuration on clickhouse cluster level.
For the case, there are thousands tables, and join queries are supported on these tables. We want to create these tables into different clickhouse cluster and each clickhouse cluster use different zookeeper cluster.

Thanks

Apply two yaml files successively using operator. Why not create the next chi cluster if one chi cluster is not created successfully

CR Grafana + service, type: LoadBalancer, kops/AWS service stay in pending external-ip status

I have kops/AWS k8s 1.11.x without installed ingress deployment
I need create kind: Service with type: LoadBalancer for allow access to grafana outside from k8s cluster

I tried grafana-operator:latest and grafana-operator:3.1.0
with following custom Grafana resource

apiVersion: integreatly.org/v1alpha1
kind: Grafana
metadata:
  name: "grafana"
spec:
  ingress:
    enabled: False
  service:
    type: LoadBalancer
  client:
    timeout: 30
    preferService: True
  config:
    log:
      mode: "console"
      level: "warn"
    security:
      admin_user: "admin"
      admin_password: "admin"
    auth:
      disable_login_form: False
      disable_signout_menu: True
    # require for Dashboard Installation - https://github.com/integr8ly/grafana-operator/issues/92
    auth.basic:
      enabled: True
    auth.anonymous:
      enabled: True
  dashboardLabelSelector:
  - matchExpressions:
    - {key: app, operator: In, values: [grafana]}

service grafana-service was created
but
kubectl describe -n my-namespace service/grafana-service

show following error:

Name:                     grafana-service
Namespace:                my-namespace
Labels:                   <none>
Annotations:              <none>
Selector:                 app=grafana
Type:                     LoadBalancer
IP:                       100.66.199.39
Port:                     grafana  3000/TCP
TargetPort:               grafana-http/TCP
NodePort:                 grafana  30120/TCP
Endpoints:                100.96.11.188:3000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                      Age                    From                Message
  ----     ------                      ----                   ----                -------
  Normal   EnsuringLoadBalancer        2m16s (x9 over 2m23s)  service-controller  Ensuring load balancer
  Normal   EnsuredLoadBalancer         2m16s (x8 over 2m21s)  service-controller  Ensured load balancer
  Warning  CreatingLoadBalancerFailed  2m16s (x8 over 2m21s)  service-controller  Error creating load balancer (will retry): not persisting update to service 'altinity-kube/grafana-service' that has been changed since we received it: Operation cannot be fulfilled on services "grafana-service": the object has been modified; please apply your changes to the latest version and try again

what i need add to service section of Grafana Custom resource to solve this problem?

	namespace := os.Getenv(OPERATOR_POD_NAMESPACE)
	if namespace == "kube-system" {
	// Do nothing, we already have len(config.WatchNamespaces) == 0
	} else {
	// We have WATCH_NAMESPACE specified
	config.WatchNamespaces = []string{namespace}
	}

altinity / clickhouse-operator Goto Github PK

clickhouse-operator's Introduction

Altinity Kubernetes Operator for ClickHouse

Features

Requirements

Documentation

License

Commercial Support

clickhouse-operator's People

Contributors

Stargazers

Watchers

Forkers

clickhouse-operator's Issues

Expected results

Actual results

Solution proposal

Recommend Projects

Recommend Topics

Recommend Org