liuxinwang / go-mysql-starrocks Goto Github PK

Let me know. [email protected]

亲，是否支持把 mysql 中vachar 转为 starrocks 中的json 字段

亲，发错了。不好意思

新增全量同步，同步完成后，新出现了一个以上错误。而且每次都出现了。重试几次了。
完整错误：前后两个error，都会一起出现
[2024/02/04 14:56:34] [info] api.go:161 resume output write
[2024/02/04 14:56:34] [error] binlogsyncer.go:689 io.CopyN failed. err unexpected EOF, copied 0, expected 1442: connection was bad
[2024/02/04 14:56:35] [info] binlogsyncer.go:614 begin to re-sync from 0bb4dd22-35d1-11eb-8cf4-00155d011502:1-4690822702 (last read GTID=0bb4dd22-35d1-11eb-8cf4-00155d011502:1-4690822703)
[2024/02/04 14:56:35] [error] binlogsyncer.go:875 kill connection 3470732 error ERROR 1094 (HY000): Unknown thread id: 3470732
[2024/02/04 14:56:35] [info] binlogsyncer.go:881 kill last connection id 3470732
[2024/02/04 14:56:35] [info] binlogsyncer.go:808 rotate to (binlog.021163, 4)

[2023/07/13 16:47:37] [info] sync.go:233 table structure changed, clear table cache: daowei.test_tech [2023/07/13 16:47:47] [info] ip_mysql.go:223 ddl event: ALTER TABLE daowei.test_techADD COLUMNd6varchar(2) NULL AFTERd5`
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x9d8454]

goroutine 51 [running]:
github.com/liuxinwang/go-mysql-starrocks/pkg/schema.(*Table).FindColumn(...)
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/schema/schema.go:57
github.com/liuxinwang/go-mysql-starrocks/pkg/schema.(*MysqlTables).UpdateTable(0xc0003de480, {0xc0011766e8?, 0xcb2840?}, {0xc0011766f0, 0x9}, {0xc6faa0, 0xc0006c4c40})
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/schema/sch_mysql.go:103 +0x314
github.com/liuxinwang/go-mysql-starrocks/pkg/input.(*MysqlInputPlugin).parseStmt(0xc0002d9600, {0xf0c880?, 0xc0007b84c0?}, {0xc001176700, 0x6})
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/input/ip_mysql.go:465 +0x523
github.com/liuxinwang/go-mysql-starrocks/pkg/input.(*MysqlInputPlugin).OnDDL(0xc0002d9600, {{0xc000839e50?, 0x6?}, 0x839e56?}, 0xc0003c59d0)
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/input/ip_mysql.go:230 +0x1fb
github.com/go-mysql-org/go-mysql/canal.(*Canal).runSyncBinlog(0xc0002b3b00)
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/liuxinwang/[email protected]/canal/sync.go:162 +0xbb5
github.com/go-mysql-org/go-mysql/canal.(*Canal).run(0xc0002b3b00)
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/liuxinwang/[email protected]/canal/canal.go:230 +0x1c5
github.com/go-mysql-org/go-mysql/canal.(*Canal).Run(...)
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/liuxinwang/[email protected]/canal/canal.go:185
github.com/go-mysql-org/go-mysql/canal.(*Canal).StartFromGTID(0xc0002b3b00, {0xf09078?, 0xc00000e1e0?})
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/liuxinwang/[email protected]/canal/canal.go:198 +0x39
github.com/liuxinwang/go-mysql-starrocks/pkg/input.(*MysqlInputPlugin).StartInput.func1()
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/input/ip_mysql.go:114 +0x25
created by github.com/liuxinwang/go-mysql-starrocks/pkg/input.(*MysqlInputPlugin).StartInput
/Users/wangwuyang/GolandProjects/go-mysql-starrocks/pkg/input/ip_mysql.go:113 +0x495`

请问，对于非int类的主键，有计划支持不。其实配置在配置文件中，他能增量同步，但是直接配置文件启动不支持全量同步。我使用api全量同步报错不支持非int主键。
另外我想咨询下，全量同步时加锁表之类的没有

		if !strings.Contains(primaryKeyDataType, "int") {
			log.Errorf("handling primary key not int type is not currently supported")
			return errors.New("handling primary key not int type is not currently supported")
		}

线上同步过程中，会因为某种原因down掉了。如果能提供监控接口，那就太完美了。

你好，使用了下应用非常方便，提一点建议
能否实现mysql binlog 根据点位或者时间进行回溯
因为在发布或者变更改应用的时候会出现数据遗漏的情况

[2023/07/05 23:40:58] [error] mysql.go:139 do starrocks bulk err Publish timeout. The data will be visible after a whiletxn has not sent publish tasks yet, maybe waiting previous txns on the same table(s) to finish, tableIds: 144103299, targetTable: 库名.表名 visit ErrorURL to view error details, ErrorURL: %!s(), close sync

v0.5.2同步一个大表，从mysql8.0.22到starrocks 3.2.2, 大约400多万数据，老是报以下的一个error错误。有时候几万条就报，最后这一次运行了一个多小时，同步了300多万数据也报错了。

程序报错日志：
[2024/01/31 23:56:04] [info] api.go:81 add schema table meta data: ces.ces_delivery_goods
[2024/01/31 23:56:04] [info] api.go:95 add rule map: {"RuleType":"dynamic add","full_sync":true,"source-schema":"db01","source-table":"t_delivery_goods","target-schema":"p01","target-table":"t_delivery_goods"}
[2024/01/31 23:56:04] [info] api.go:108 add rule includeTableRegex: ^db01.t_delivery_goods$
[2024/01/31 23:56:04] [info] api.go:109 add rule successfully
[2024/01/31 23:56:04] [info] api.go:114 pause output write
[2024/01/31 23:56:04] [info] api.go:320 start handle full data sync...
[2024/02/01 01:07:27] [error] api.go:464 handling execute select streaming failed. err: io.CopyN failed. err unexpected EOF, copied 0, expected 615: connection was bad
[2024/02/01 01:07:27] [info] api.go:144 resume output write

curl添加全量同步那的报错：
result: add rule full sync handle failed err: io.CopyN failed. err unexpected EOF, copied 0, expected 615: connection was bad, full sync rows: 3414723

还有一个疑问，为什么查询数据库要根据主键排序呢。

使用同步软件后，starrocks同步的条数明显多于mysql中的条数，对于mysql的删除操作，同步软件会一并同步过来么？

[root@mysql2es01 go-mysql-sr-v0.5.2-linux-amd64]# curl localhost:6166/api/addRule -d '{"source-schema": "db01","source-table": "t01", "target-schema": "p01_db01","target-table": "t01", "full_sync": true}'
result: add rule full sync handle failed err: too many filtered rows, targetTable: p01_db01.t01, visit ErrorURL to view error details, ErrorURL: http://172.20.3.40:8040/api/_load_error_log?file=error_log_d44b046584ba3cb8_3cbdb1577642c6bc, full sync rows: 40000

加了十几个表了，到这个表报错

您好，咨询两个问题：
1、命令加了-log-file，并没有生成指定日志文件，find整个系统都没找到。
2、配置中的表关系映射如下，但是实际运行时，查看日志中发现他把mysql中所有数据库的表都加载了一遍，报了大量的database not found的错误，而且这两个配置的表并没有同步数据，我后面使用curl localhost:6166/api/addRule -d '{"source-schema": "jcyf_mon_002","source-table": "tbl_test_null", "target-schema": "cloud_db", "target-table": "tbl_test_null", "full_sync": true}'这个方式实现了一个表同步，并且后续新增数据也能自动同步。但是不符合配置文件预期。
[[output.config.rule]]
source-schema = "jcyf_mon_002"
source-table = "tbl_test_null"
target-schema = "cloud_db"
target-table = "tbl_test_null"

[[output.config.rule]]
source-schema = "jcyf_mon_2021"
source-table = "tbl_net_arpinfo"
target-schema = "cloud_db"
target-table = "tbl_net_arpinfo"

time="2023-10-16T14:53:30+08:00" level=warning msg="error running query" connectTime="2023-10-16 14:53:25.783689102 +0800 CST m=+3.071269820" connectionDb= connectionID=1 error="database not found: " query="CREATE TABLE hue.django_admin_log ( id int(11) NOT NULL A
UTO_INCREMENT, action_time datetime(6) NOT NULL, object_id longtext, object_repr varchar(200) NOT NULL, action_flag smallint(5) unsigned NOT NULL, change_message longtext NOT NULL, content_type_id int(11) DEFAULT NULL, user_id int(11) NOT NULL, PRIMARY KEY
(id), KEY django_admin_log_content_type_id_c4bce8eb_fk_django_co (content_type_id), KEY django_admin_log_user_id_c564eba6_fk_auth_user_id (user_id), CONSTRAINT django_admin_log_content_type_id_c4bce8eb_fk_django_co FOREIGN KEY (content_type_id) REFERENCES
django_content_type (id), CONSTRAINT django_admin_log_user_id_c564eba6_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES auth_user (id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8"
[2023/10/16 14:53:30] [warn] sch_mysql_v2.go:404 load tables meta to memDB failed, create ddl: CREATE TABLE hue.django_admin_log (
id int(11) NOT NULL AUTO_INCREMENT,
action_time datetime(6) NOT NULL,
object_id longtext,
object_repr varchar(200) NOT NULL,
action_flag smallint(5) unsigned NOT NULL,
change_message longtext NOT NULL,
content_type_id int(11) DEFAULT NULL,
user_id int(11) NOT NULL,
PRIMARY KEY (id),
KEY django_admin_log_content_type_id_c4bce8eb_fk_django_co (content_type_id),
KEY django_admin_log_user_id_c564eba6_fk_auth_user_id (user_id),
CONSTRAINT django_admin_log_content_type_id_c4bce8eb_fk_django_co FOREIGN KEY (content_type_id) REFERENCES django_content_type (id),
CONSTRAINT django_admin_log_user_id_c564eba6_fk_auth_user_id FOREIGN KEY (user_id) REFERENCES auth_user (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8, err: ERROR 1105 (HY000): database not found:

result: add rule full sync handle failed err: The size of this batch exceed the max size [104857600] of json type data data [ 224800846 ]. Set ignore_json_size to skip the check, although it may lead huge memory consuming., targetTable: p01_db01.t01, visit ErrorURL to view error details, ErrorURL: %!s(), full sync rows: 140000

v0.5.2，我把大字段类型都一 一通过配置filter删除了，但是还是报错这个。数据量也不大，之前400多万的都没报超限？一个批次定的多少条数据，已经没有大字段了，按理不会超过100MB了才对。我查官方说如果要忽略这错误，可以在http头中加入："ignore_json_size:true"。

请问如何规避，缩小channel-size这参数？

你好，请问下项目后续规划是做什么呢，做web ui吗，有没有其他功能方面的优化，例如现在只能是从mysql主库同步数据，但生产更多的是要从从库同步数据而不影响主库的

panic: runtime error: index out of range [23] with length 23

goroutine 4 [running]:
github.com/go-mysql-org/go-mysql/canal.(*RowsEvent).handleUnsigned(0xc0000c2000)
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/go-mysql-org/[email protected]/canal/rows.go:56 +0x5c5
github.com/go-mysql-org/go-mysql/canal.newRowsEvent(0xc000308100, {0xa66468, 0x6}, {0xc000284900, 0x2, 0x2e}, 0xc00061bfc8)
/Users/wangwuyang/GolandProjects/pkg/mod/github.com/go-mysql-org/[email protected]/canal/rows.go:39 +0xe8
github.com/go-mysql-org/go-mysql/canal.(*Canal).handleRowsEvent(0xc000362000, 0xc0004fed20)