ReplicateTable are not getting created in the new Replica about clickhouse-operator HOT 17 CLOSED

@phatak-dev , that could be an issue with the old ClickHouse version you are using.

We recommend using Altinity stable versions, see releases there: https://www.altinity.com/blog/tag/Releases

We have not tested operator extensively for ClickHouse before 19.11 version. I will test with 19.6.2.11 to see if we can reproduce the problem.

from clickhouse-operator.

@phatak-dev , what is your operator version and ClickHouse version? There was a bug in operator 0.7.0 an earlier, but it should be good now.

from clickhouse-operator.

@alex-zaitsev I am using the latest (0.9) operator version and clickhouse version 19.6.2.11. I still see the issue.
Can you please specify what was the bug fixed so that I can look into is the same bug causing the issue in the latest?

from clickhouse-operator.

@phatak-dev , I could not reproduce it in 19.6.2.11 version. Could you give me an example of Replicated table you are using?

from clickhouse-operator.

@alex-zaitsev Below are the tables. One is replicated and one is distributed

CREATE TABLE events_local on cluster '{cluster}' (
    event_date  Date,
    event_type  Int32,
    article_id  Int32,
    title       String
) engine=ReplicatedMergeTree('/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}', '{replica}')
PARTITION BY toYYYYMM(event_date)
ORDER BY (event_date, intHash32(article_id))
SAMPLE BY intHash32(article_id);

CREATE TABLE events on cluster '{cluster}' AS events_local
ENGINE = Distributed('{cluster}', default, events_local, rand());

It works the first time when I increase from 1 to 2 replicas. But when I reduce to 1 and increase to 2 again the tables will be not created.

from clickhouse-operator.

@alex-zaitsev Tested with 19.16.12.49 also. It has same issue. I am running the operator on a AWS kops cluster.

from clickhouse-operator.

@alex-zaitsev Is the below bug may be causing the issue?
ClickHouse/ClickHouse#7268

from clickhouse-operator.

@alex-zaitsev In the newly created pod, I see below error

2020.02.25 09:53:44.849122 [ 6 ] {} <Debug> DDLWorker: Will not execute task query-0000000000: There is no a local address in host list

from clickhouse-operator.

@phatak-dev , could you share your ClickHouseInstallation resource, please?

from clickhouse-operator.

@alex-zaitsev

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"

metadata:
  name: "repl-05"

spec:
  defaults:
    templates: 
      dataVolumeClaimTemplate: default
      podTemplate: clickhouse:19.16
 configuration:
    zookeeper:
      nodes:
      - host: zookeeper.zoo1ns
    clusters:
      - name: replicated
        layout:
          shardsCount: 1
          replicasCount: 1
  templates:
    volumeClaimTemplates:
      - name: default
        reclaimPolicy: Delete
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
                    storage: 300Mi
    podTemplates:
      - name: clickhouse:19.16
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.16.12.49

from clickhouse-operator.

@alex-zaitsev Once in a while, I get below error when I create the first pod

0225 11:51:15.299328       1 controller.go:497] ERROR update CHI (default/repl-05): "Operation cannot be fulfilled on clickhouseinstallations.clickhouse.altinity.com \"repl-05\": the object has been modified; please apply your changes to the latest version and try again"
I0225 11:51:15.299366       1 worker.go:169] UNABLE to write normalized CHI (default/repl-05). It can trigger update action again. Error: "Operation cannot be fulfilled on clickhouseinstallations.clickhouse.altinity.com \"repl-05\": the object has been modified; please apply your changes to the latest version and try again"

from clickhouse-operator.

@phatak-dev , this is some sort of race condition. It does not affect the operator's functionality.

from clickhouse-operator.

@alex-zaitsev tested the code in minikube. It works perfectly fine.

from clickhouse-operator.

@alex-zaitsev Whenever there is dropped table, I am getting context cancelled error in clickhouse-operator side.

from clickhouse-operator.

@alex-zaitsev @sunsingerus Commenting out defer cancel() in func (c *Conn) Query(sql string) (*sqlmodule.Rows, error) of connection.go file seems to be fixing the issue. As I am not expert in go, not sure what may be this doing and what am I doing correct or not. My current understanding is context getting closed much earlier than actual rows are read.

from clickhouse-operator.

@phatak-dev , it is a recommended practice to call defer cancel() once context is acquired:
https://stackoverflow.com/questions/44393995/what-happens-if-i-dont-cancel-a-context

defer ensures it is called after returning from the surrounding function. So the code looks safe to me.

from clickhouse-operator.

@phatak-dev , we have changed schema creation logic in 0.9.9 quite a lot, it should fix your issue.

from clickhouse-operator.