cluster_secure_connection option ignored while creating Replicated database? about clickhouse HOT 7 CLOSED

How did you configure your cluster? Can you add the openssl part too?

Did you look at this guide to configure SSL-TLS: https://clickhouse.com/docs/en/guides/sre/configuring-ssl ?

from clickhouse.

@melvynator
I use ZooKeeper for cluster coordination (not TLS enforced at the moment)
zookeeper.xml:

<zookeeper>
        <node>
            <host>ip1</host>
            <port>2181</port>
        </node>
        <node>
            <host>ip2</host>
            <port>2181</port>
        </node>
        <node>
            <host>ip3</host>
            <port>2181</port>
        </node>
</zookeeper>

ssl.xml:

<openSSL>
        <server> <!-- Used for https server AND secure tcp port -->
            <certificateFile>/etc/clickhouse-server/ssl/clickhouse.crt</certificateFile>
            <privateKeyFile>/etc/clickhouse-server/ssl/clickhouse.key</privateKeyFile>
            <verificationMode>relaxed</verificationMode>
            <caConfig>/etc/clickhouse-server/ssl/ca.crt</caConfig>
            <cacheSessions>true</cacheSessions>
            <disableProtocols>sslv2,sslv3</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
        </server>

        <client> <!-- Used for connecting to https dictionary source -->
            <loadDefaultCAFile>true</loadDefaultCAFile>
            <caConfig>/etc/clickhouse-server/ssl/ca.crt</caConfig>
            <cacheSessions>true</cacheSessions>
            <disableProtocols>sslv2,sslv3</disableProtocols>
            <preferServerCiphers>true</preferServerCiphers>
            <!-- Use for self-signed: <verificationMode>none</verificationMode> -->
            <verificationMode>relaxed</verificationMode>
            <invalidCertificateHandler>
                <!-- Use for self-signed: <name>AcceptCertificateHandler</name> -->
                <name>RejectCertificateHandler</name>
            </invalidCertificateHandler>
        </client>
</openSSL>

and ports defined like that:

<http_port>8123</http_port>
<https_port>8443</https_port>
<tcp_port>9000</tcp_port>
<tcp_port_secure>9440</tcp_port_secure>

<interserver_http_port remove="true">9004</interserver_http_port>
<interserver_https_port>9010</interserver_https_port>

from clickhouse.

@melvynator I've checked your link and it would be great if I can use remote_servers section to define other nodes' endpoints. But when I create replicated database new cluster entity created and it is not controlled via this config file (remote_severs section).

The system.clusters system table contains a cluster named like the replicated database, which consists of all replicas of the database. This cluster is updated automatically when creating/deleting replicas, and it can be used for Distributed tables.

Using this page as a reference: https://clickhouse.com/docs/en/engines/database-engines/replicated

from clickhouse.

@evillique @tavplubix I have never seen this setting 'cluster_secure_connection' and I can't find any documentation about it do you happened to know the issue?

from clickhouse.

Looks like Replicated databases always use TCP port when creating a cluster. We can fix that, but you will have to re-create databases or manually update metadata in ZooKeeper here:

localhost.localdomain :) select * from system.zookeeper where path='/database_path/replicas'

SELECT *
FROM system.zookeeper
WHERE path = '/database_path/replicas'

   ┌─name─┬─value─────────────────────────────────────────────────────────────┬─path──────────────────────┐
1. │ 1|1  │ localhost%2Elocaldomain:9000:37c5d4d7-06d6-44cc-918b-5497d2a104a5 │ /database_path/replicas │
   └──────┴───────────────────────────────────────────────────────────────────┴───────────────────────────┘

Btw, looks like manually updating the port in ZooKeeper is a possible workaround

from clickhouse.

@tavplubix many thanks for looking into it. I was also thinking about updating data in ZooKeeper directly but it looks as wrong way of doing things since there is no guarantee (at least at the documentation) that the metadata wouldn't be overwritten because of new replica joining or anything that triggers changes in replica set. I guess the most safe way is to wait for the proper fix to be sure that everything is under control? Or in that case I still won't be able to change port of already created databases even with this cluster_secure_connection variable?

from clickhouse.

According to the implementation, updating it manually in ZK will work, but yes, new replicas will be added with TCP port.

The fix (#65054) automatically updates the port number in ZK, so it will not require manual intervention

from clickhouse.