MultiDC cluster, observing 100% CPU when connecting kafka debezium source conector about scylla-cdc-source-connector HOT 20 OPEN

It'd be great if you could share with us Scylla version, topology details and logs from affected nodes

from scylla-cdc-source-connector.

Thanks for replying.
We have 32 CPu(With hyperthreading), core 64GB RAM
ScyllaDB DC 1 version is 5.4.0
DC2 version 5.4.1
Create KEYSPACE keyspace_name WITH replication = { 'class' : 'NetworkTopologyStrategy', 'dc1' : 3 , 'dc2' : 3};]
endpoint_snitch: GossipingPropertyFileSnitch

########
-- Logs begin at Mon 2023-02-13 17:40:14 IST, end at Tue 2024-02-06 10:20:16 IST. --
Feb 06 00:00:02 hostname02 scylla[2283]: [shard 0:main] schema_tables - Creating keyspace.tablename_t_240206 id=93992bd0-c454-11ee-b9d4-f39ae583d835 versio>
Feb 06 00:00:02 hostname02 scylla[2283]: [shard 0:main] schema_tables - Creating keyspace.tablename_t_240206_scylla_cdc_log id=939952e1-c454-11ee-b9d4-f39a>
Feb 06 00:00:03 hostname02 scylla[2283]: [shard 0:main] schema_tables - Schema version changed to 2563c4fa-4436-3ad2-801c-c4b986faf9e9
Feb 06 00:00:03 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system_schema.tables 93d71f30-c454-11ee-82fc-f1bce3e05ad7] Compacting [/v>
Feb 06 00:00:03 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system_schema.tables 93d71f30-c454-11ee-82fc-f1bce3e05ad7] Compacted 2 ss>
Feb 06 00:00:03 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.peers 93e74bd0-c454-11ee-82fc-f1bce3e05ad7] Compacting [/var/lib/s>
Feb 06 00:00:03 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.peers 93e74bd0-c454-11ee-82fc-f1bce3e05ad7] Compacted 2 sstables t>
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:main] schema_tables - Creating keyspace.tablename_o id=ab9527c0-c454-11ee-b9d4-f39ae583d835 versio>
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:main] schema_tables - Creating keyspace.tablename_o_scylla_cdc_log id=ab954ed0-c454-11ee-b9d4-f39a>
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:main] schema_tables - Schema version changed to b6159449-b003-3abb-b860-1b75c991099d
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.raft abb99fb0-c454-11ee-82fc-f1bce3e05ad7] Compacting [/var/lib/sc>
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.scylla_table_schema_history abba8a10-c454-11ee-82fc-f1bce3e05ad7] >
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.raft abb99fb0-c454-11ee-82fc-f1bce3e05ad7] Compacted 2 sstables to>
Feb 06 00:00:43 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact system.scylla_table_schema_history abba8a10-c454-11ee-82fc-f1bce3e05ad7] >
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83313b10-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 12:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83313b10-c455-11ee-872e-f1b2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 31:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83316220-c455-11ee-8b33-f1a4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 15:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83316220-c455-11ee-90d5-f1b7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8331b040-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331d750-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 29:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83318930-c455-11ee-a81d-f1a8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 14:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-a26b-f19fe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 4:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-b31f-f1b4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 23:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-b273-f1a7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 9:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-8e16-f1bbe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 27:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-8898-f1abe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 30:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83322570-c455-11ee-b313-f19de3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83318930-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 13:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-9df3-f1aae3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 21:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8b6f-f1ade3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83324c80-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 1:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-b290-f1a2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8332c1b0-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 13:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83329aa0-c455-11ee-9df3-f1aae3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 28:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332e8c0-c455-11ee-aa28-f1ace3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 7:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83324c80-c455-11ee-a34e-f1aee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 22:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-aacb-f1a9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 22:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-aacb-f1a9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 25:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8926-f1b3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 3:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-a5c1-f1a3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 7:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-a34e-f1aee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 12:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-872e-f1b2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83329aa0-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 27:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8898-f1abe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-82fc-f1bce3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 29:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-a81d-f1a8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83327390-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 1:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-b290-f1a2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8332c1b0-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 83335df0-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-82fc-f1bce3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 17:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-9f42-f1b0e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 6:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-978d-f1b9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 6:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83330fd0-c455-11ee-978d-f1b9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 14:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-a26b-f19fe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 5:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-9c9d-f1a0e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 23:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333d320-c455-11ee-b273-f1a7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 25:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8926-f1b3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 16:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83338500-c455-11ee-96dd-f1b8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 16:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333d320-c455-11ee-96dd-f1b8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 24:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-8f94-f1b6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 24:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8f94-f1b6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 11:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-9158-f19ee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 21:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83346f60-c455-11ee-8b6f-f1ade3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 30:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833532b0-c455-11ee-b313-f19de3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8334e490-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 31:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83342140-c455-11ee-8b33-f1a4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833559c0-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 9:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333fa30-c455-11ee-8e16-f1bbe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8333d320-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83349670-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 15:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83316220-c455-11ee-90d5-f1b7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 13:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83329aa0-c455-11ee-9df3-f1aae3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83364420-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8333fa30-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 28:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8334bd80-c455-11ee-aa28-f1ace3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 3:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-a5c1-f1a3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 5:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-9c9d-f1a0e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 17:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-9f42-f1b0e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 4:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8333fa30-c455-11ee-b31f-f1b4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 29:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83318930-c455-11ee-a81d-f1a8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 14:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-a26b-f19fe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 29:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-a81d-f1a8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 25:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8926-f1b3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 1:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-b290-f1a2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 24:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8f94-f1b6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 13:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-9df3-f1aae3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 11:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-9158-f19ee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 7:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-a34e-f1aee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 7:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83324c80-c455-11ee-a34e-f1aee3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 1:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-b290-f1a2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 14:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-a26b-f19fe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 25:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8926-f1b3e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 23:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333d320-c455-11ee-b273-f1a7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83329aa0-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 27:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-8898-f1abe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83318930-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 20:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8332c1b0-c455-11ee-b249-f1a5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 22:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332c1b0-c455-11ee-aacb-f1a9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 24:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-8f94-f1b6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 27:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 833336e0-c455-11ee-8898-f1abe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83324c80-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 22:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-aacb-f1a9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-82fc-f1bce3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 18:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8331b040-c455-11ee-8920-f1a6e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 23:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-b273-f1a7e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 83335df0-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 9:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333fa30-c455-11ee-8e16-f1bbe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 0:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-82fc-f1bce3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8332c1b0-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 9:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83322570-c455-11ee-8e16-f1bbe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833559c0-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 31:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83316220-c455-11ee-8b33-f1a4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 2:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8334e490-c455-11ee-a516-f1b1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 16:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8333d320-c455-11ee-96dd-f1b8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 16:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83338500-c455-11ee-96dd-f1b8e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 31:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83342140-c455-11ee-8b33-f1a4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331d750-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83364420-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 26:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8333fa30-c455-11ee-b7f4-f1afe3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83327390-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 21:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83329aa0-c455-11ee-8b6f-f1ade3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 8:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83330fd0-c455-11ee-a868-f1a1e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 21:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83346f60-c455-11ee-8b6f-f1ade3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 6:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833336e0-c455-11ee-978d-f1b9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_o_240130_scylla_cdc_log 8333d320-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 6:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83330fd0-c455-11ee-978d-f1b9e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83349670-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 12:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8332e8c0-c455-11ee-872e-f1b2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 12:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83313b10-c455-11ee-872e-f1b2e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 10:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 83313b10-c455-11ee-8752-f1b5e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 28:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8334bd80-c455-11ee-aa28-f1ace3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 28:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8332e8c0-c455-11ee-aa28-f1ace3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 30:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 83322570-c455-11ee-b313-f19de3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 30:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 833532b0-c455-11ee-b313-f19de3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 4:comp] compaction - [Compact keyspace.tablename_t_240129_scylla_cdc_log 8331fe60-c455-11ee-b31f-f1b4e3e05a>
Feb 06 00:06:44 hostname02 scylla[2283]: [shard 4:comp] compaction - [Compact keyspace.tablename_o_240129_scylla_cdc_log 8333fa30-c455-11ee-b31f-f1b4e3e05a>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 0:main] schema_tables - Altering keyspace.tenant_config id=b3fa0e10-a575-11ee-8286-727194dae698 versi>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 0:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 29:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 10:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 12:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 31:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 19:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 13:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 5:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 18:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 30:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 17:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 21:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 9:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 7:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 22:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 11:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 3:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 6:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 25:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 23:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 14:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 15:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 27:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 26:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 1:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 16:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 24:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 2:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 28:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 20:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 8:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:19 hostname02 scylla[2283]: [shard 4:main] query_processor - Column definitions for keyspace.tenant_config changed, invalidating related>
Feb 06 00:07:20 hostname02 scylla[2283]: [shard 0:main] schema_tables - Schema version changed to 57e433f2-815d-3cc7-9fe0-79ace24bf0ad
Feb 06 00:11:53 hostname02 scylla[2283]: [shard 0:main] schema_tables - Creating keyspace.tablename_callback_h_240206 id=3aee6e80-c456-11ee-8ccc-48191cf>
Feb 06 00:11:53 hostname02 scylla[2283]: [shard 0:main] schema_tables - Schema version changed to c0b1e63f-e2e1-384c-8c26-5b2ebf985791

#############

Feb 6 00:06:57 hostname01 scylla: [shard 19:stat] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/active/need_cpu#0122416#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#0122744#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125161#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 16#012time_based_evictions: 735#012inactive_reads: 0#012total_successful_reads: 2607877#012total_failed_reads: 3#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 2655579#012reads_enqueued_for_admission: 2277941#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 382798#012reads_queued_because_ready_list: 876070#012reads_queued_because_need_cpu_permits: 1401860#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 2661754#012current_permits: 5161#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:07:27 hostname01 scylla: [shard 19:stat] reader_concurrency_semaphore - (rate limiting dropped 100641 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/active/need_cpu#0122697#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#0122488#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125186#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 16#012time_based_evictions: 735#012inactive_reads: 0#012total_successful_reads: 2716772#012total_failed_reads: 100643#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 2764519#012reads_enqueued_for_admission: 2487503#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 382798#012reads_queued_because_ready_list: 927827#012reads_queued_because_need_cpu_permits: 1559665#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 2871316#012current_permits: 5186#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:07:57 hostname01 scylla: [shard 19:stat] reader_concurrency_semaphore - (rate limiting dropped 116603 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/active/need_cpu#0121#0110#0110B#011keyspace.tablename_t_240206/shard-reader/waiting_for_admission#0122518#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#0122569#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125089#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 16#012time_based_evictions: 735#012inactive_reads: 0#012total_successful_reads: 2819521#012total_failed_reads: 217244#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 2867340#012reads_enqueued_for_admission: 2706760#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 382798#012reads_queued_because_ready_list: 983002#012reads_queued_because_need_cpu_permits: 1723747#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 3090573#012current_permits: 5089#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:08:27 hostname01 scylla: [shard 19:stat] reader_concurrency_semaphore - (rate limiting dropped 79569 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_execution#0122652#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#0122571#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125224#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 16#012time_based_evictions: 735#012inactive_reads: 0#012total_successful_reads: 2971466#012total_failed_reads: 296819#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 3019363#012reads_enqueued_for_admission: 2938436#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 382798#012reads_queued_because_ready_list: 1037745#012reads_queued_because_need_cpu_permits: 1900680#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 3322249#012current_permits: 5224#012need_cpu_permits: 0#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:10:01 hostname01 scylla: [shard 10:stat] reader_concurrency_semaphore - Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/active/need_cpu#01221#0110#0110B#011keyspace.tablename_o_240203_scylla_cdc_log/data-query/waiting_for_admission#0121#0110#0110B#011keyspace.tablename_t_240205_scylla_cdc_log/data-query/waiting_for_admission#01215#0110#0110B#011keyspace.tablename_o_240205_scylla_cdc_log/data-query/waiting_for_admission#01214#0110#0110B#011keyspace.tablename_t_240203_scylla_cdc_log/data-query/waiting_for_admission#01225#0110#0110B#011keyspace.tablename_t_240204_scylla_cdc_log/data-query/waiting_for_admission#0122594#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#01239#0110#0110B#011keyspace.tablename_o_240204_scylla_cdc_log/data-query/waiting_for_admission#0122492#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125202#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 11#012time_based_evictions: 740#012inactive_reads: 0#012total_successful_reads: 15480154#012total_failed_reads: 4#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 15528210#012reads_enqueued_for_admission: 13502365#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 2031046#012reads_queued_because_ready_list: 5602001#012reads_queued_because_need_cpu_permits: 7900353#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 15533708#012current_permits: 5202#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:10:41 hostname01 scylla: [shard 10:stat] reader_concurrency_semaphore - (rate limiting dropped 198054 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/active/need_cpu#0122409#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#0122731#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125141#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 11#012time_based_evictions: 740#012inactive_reads: 0#012total_successful_reads: 16503537#012total_failed_reads: 198059#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 16551737#012reads_enqueued_for_admission: 14723786#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 2031046#012reads_queued_because_ready_list: 5775266#012reads_queued_because_need_cpu_permits: 8948509#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 16755129#012current_permits: 5141#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0
Feb 6 00:13:01 hostname01 scylla: [shard 19:stat] reader_concurrency_semaphore - (rate limiting dropped 2993 similar messages) Semaphore _read_concurrency_sem with 1/100 count and 16384/38042337 memory resources: timed out, dumping permit diagnostics:#012permits#011count#011memory#011table/operation/state#0121#0111#01116K#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/active/need_cpu#0123#0110#0110B#011keyspace.tablename_t_240206/shard-reader/waiting_for_admission#0122270#0110#0110B#011keyspace.tablename_o_240206_scylla_cdc_log/data-query/waiting_for_admission#0122956#0110#0110B#011keyspace.tablename_t_240206_scylla_cdc_log/data-query/waiting_for_admission#012#0125230#0111#01116K#011total#012#012Stats:#012permit_based_evictions: 16#012time_based_evictions: 735#012inactive_reads: 0#012total_successful_reads: 5200366#012total_failed_reads: 299813#012total_reads_shed_due_to_overload: 0#012total_reads_killed_due_to_kill_limit: 0#012reads_admitted: 5248433#012reads_enqueued_for_admission: 4729336#012reads_enqueued_for_memory: 0#012reads_admitted_immediately: 823966#012reads_queued_because_ready_list: 2015437#012reads_queued_because_need_cpu_permits: 2713888#012reads_queued_because_memory_resources: 11#012reads_queued_because_count_resources: 0#012reads_queued_with_eviction: 11#012total_permits: 5554317#012current_permits: 5230#012need_cpu_permits: 1#012awaits_permits: 0#012disk_reads: 0#012sstables_read: 0

from scylla-cdc-source-connector.

It's a bit challenging to decipher, but I think the read concurrency seems to be the issue here - @denesb - is the system overloaded here?

from scylla-cdc-source-connector.

from scylla-cdc-source-connector.

Cpu goes high only when kafka source conn connects at midnight and cools down once Connector starts reading

It's just during the connect phase? Perhaps reading the CDC history? @avelanarius , @kbr-scylla - any ideas?

@amitesh88 - monitoring snapshot could be useful as well.

from scylla-cdc-source-connector.

CPU goes up specifically during connect phase . Hope you can find below snapshot. @Jay-s13 @aryansaraf018 @Amit7083 @mygittryfile @gayatri-s13

from scylla-cdc-source-connector.

@mykaul did you find above clarification helpful
Also, please elaborate when you say
reading from CDC history ?
@avelanarius @kbr-scylla

In our setup , we create a new kafka source connector which reads from a new table’s cdc log everyday at midnight 12 .Keeping older source connectors connected and deleting them after two days

from scylla-cdc-source-connector.

We are using 32 vcpu with hyperthreading (2 threads/core) .On running seastar-cpu-map.sh -n scylla we found scylla using 32 hyperthreaded core.
So we decided to run with --smp 16 then suddnly we found CPU started with 50% utilisation. But half of the CPU is 0% idle.
Please help me understanding the scenario @mykaul

from scylla-cdc-source-connector.

We are using 32 vcpu with hyperthreading (2 threads/core) .On running seastar-cpu-map.sh -n scylla we found scylla using 32 hyperthreaded core. So we decided to run with --smp 16 then suddnly we found CPU started with 50% utilisation. But half of the CPU is 0% idle. Please help me understanding the scenario @mykaul

@amitesh88 - this is probably unrelated to the issue you were discussing earlier. In short, you need to keep some cores for networking, 2-4 cores should be enough, and the rest Scylla should be using. See https://opensource.docs.scylladb.com/stable/operating-scylla/admin-tools/perftune.html#seastar-perftune

from scylla-cdc-source-connector.

Sorry didnt mentioned above ,the above mentioned cpu observation is only when new source connector gets created and goes to connect scylladb to read cdc logs.
the question here is why it is consuming 100% cpu of scylla only when we create source connector and rest of the time it is 95%idle
And once the kafka topic created ,cpu cools down

from scylla-cdc-source-connector.

@avelanarius - any idea?

from scylla-cdc-source-connector.

We have observed that it takes 15 minutes to read CDC data, create a KAFKA topic, and eventually allow the CPU to calm down when we increase scylla.query.time.window.size from 5 to 60 seconds.
We come across another value poll.interval.ms with default value 15s . We couldn't figure out the difference between the two parameters and how they ll affect query performance. @mykaul Your insight please

Is there a link or document where we can find all the parameters used in source connector or Tune the behaviour of source connector setup?
@avikivity @avelanarius Please share your views to unriddle this high CPU mystery .

from scylla-cdc-source-connector.

Are you using CDC streams? Perhaps the the initial scan is inefficient?

from scylla-cdc-source-connector.

Yes , we are using debezium source connector reading cdc logs
Can you elaborate a bit on how initial scan happens?

from scylla-cdc-source-connector.

https://opensource.docs.scylladb.com/stable/using-scylla/cdc/cdc-querying-streams.html#querying-cdc-streams
BTW, you may want to look at https://www.scylladb.com/2023/01/25/how-to-use-change-data-capture-with-apache-kafka-and-scylladb/

from scylla-cdc-source-connector.

Thanks for the help

We manage to reduce 15 mins of High CPU to 6mins by tweaking scylla.query.time.window.size in kafka source connector , but still no clue about 100% cpu utilisation.

Also it is observed , our 6 node dc2 cluster and 4 nodes kafka connector, there is a total of ~40000 active connection on each scylladb node

Any clue about this?? @mykaul @avelanarius

from scylla-cdc-source-connector.

scylladb/scylla-cdc-java#66
@amitesh88 we already encountered this issue, and fixed it.
you can use this pr to reduce load while using cdc source connector.

from scylla-cdc-source-connector.

@hansh0801 thnkyou so much for mentioning the issue, can you please help me understand the cause and elaborate on issue as we are new to ScyllaDb. We want to understand how increasing windowsize reduces the time span of topic creation and cpu surge
And why the cpu goes high, the logic is still not understood

from scylla-cdc-source-connector.

@amitesh88
Knowing how the scylladb cdc works can make it a little easier to understand.

Currently, the scylladb cdc driver uses an in query, and if the IN contains many partition keys (streamids in the case of cdc), the request is not processed on one node, but is requested back to the nodes that have the partition and wait for the response, which increases the overall throughput of the cluster and increases latency.

I encountered this issue about 2 years ago and it is related to a structural issue with the scylladb cdc.

Let's discuss this in more detail in the scylladb cdc slack channel.

from scylla-cdc-source-connector.

@hansh0801 after a long wait, this reply made my day ,it cleared many mysteries
Please add me or send a link of your slack channel for further discussion.
Thanks a lot

from scylla-cdc-source-connector.