flike / kingbus Goto Github PK

A distributed MySQL binlog storage system built on Raft

License: Apache License 2.0

Dockerfile 0.09% Makefile 0.22% Go 96.74% Shell 1.32% PLpgSQL 1.63%

mysql raft golang mysql-binlog distributed-systems storage binlog

kingbus's Introduction

What is kingbus? 中文

Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the slaves in the same way as an intermediate MySQL master does. Kingbus has the following key features:

MySQL replication protocol compatibility, pull the binlog files from the master through gtid mode, and push the binlog file to slave through gtid mode in the same way.
Geo-Replication, kingbus uses Raft to support Geo-Replication. The binlog data written to the cluster is guaranteed to be consistent between multiple nodes, and the order of binlog event is exactly the same as that on the master.
High availability, your mysql binlog replication is always on and continuously available with kingbus.

Why need kingbus?

In a traditional MySQL replication setup a single master server is created and a set of slaves of MySQL servers are configured to pull the binlog files from the master, putting a lot of load on the master.

Introducing a layer between the master server and the slave servers can reduce the load on the master by only serving kingbus instead of all the slaves.
The slaves will only need to be aware of kingbus and not the real master server. Removing the requirement for the slaves to have knowledge of the master also simplifies the process of replacing a failed master within a replication environment.
Kingbus allows us to horizontally scale our slaves without fear of overloading the network interface of the master
Kingbus can also be used to avoide deep nested replication on remote sites, with kingbus you don't need e deeply nested replication.
The size of the binlog storage space on the Master can be reduced, and store the binlog in kingbus.
Support MYSQL database heterogeneous log based replication. Other heterogeneous replication components can be connected to the kingbus, such as canal.

For more usage scenarios of binlog server, please refer:

Booking: blog
Facebook: binlog server at Facebook
Google: mysql-ripple

Quick start

Read the Quick Start

Documentation

1.Kingbus management API introduction

2.Start a kingbus cluster with docker-compose

License

Kingbus is under the Apache 2.0 license. See the LICENSES file for details.

Acknowledgments

Thanks etcd for providing raft library.
Thanks go-mysql for providing mysql replication protocol parser.

kingbus's People

Contributors

Stargazers

Watchers

Forkers

zhanglei lanzengwei igit-cn xuxueyun awesome-archive iweisi sjclijie isgasho bright-spark sryanyuan wangycc gclove leonliu315 0xflotus weiboyiyou hadoop835 dbnski lubaoyilang octopusdb shiftwinting abel-wanglei tang1007 sdtm1016 it-xiaoge jackerzhou zhangf911 skyformat99 iziren liexusong saromanov alongrun awesomegolang yedushusheng terryteng zondahuman harshpandit007 firfly architecturetech alexangelzhang nwsuafzq wuleying mslsoftware huangweili brightgogo weiyanwei412 teazj 154650362 holys tools-env 452562082 banlilin qiuwenhuifx shitouxiaogege lotapp carlziess ymcvalu cokebol justwindy han-ian alaer808yi daihu duzhanyuan tom2jack jedyu1987 foooy chwjbn junftnt yuanddqiao eddyzhou junxie6 sky-9527-github agsyafaat opsfarmer xftbee thinkdb iammuho cbepx haibotop chenglinjava68 luolin0313 woihzh gczheng 1008610010 cherishpy earl86 tangjian021 qdsay jiyulongxu liaokunlin acmcee shaogx sm2x dwwhs leeliked lorddang openark xiaoshangwu yuanqinguo denji showsmall

kingbus's Issues

底层raft采用的是http协议进行交互的吗？

[Question] Howto change kingbus setting when real master mysql is down

Hi there,

Sorry, I don't know where I should ask the question.

I had 3 nodes kingbus and 10 slaves node. Kingbus was binlog server real master A. Master B was a slave master A. Now, we have some issues with master A and we want to change the real master for kingbus cluster. Should we do now? What is the script in the situation?

+--------+                                                         +----------+
|master A| ------------replicated----------------------------------| master B |
+--------+                                                         +----------+
  ^
  |
  |

[ kingbus 1-3] <---------------------replicate [slave 1-10]

Thank you!

关于mysql配置的疑问

通过代码发现：syncer获取mysql master的连接配置，是通过raft协议传递的，而不是通过配置的方式，通过代码暴露出的http接口，也没有找到能通过http设置mysql的连接配置的地方，请问该连接配置，怎么通知集群的？

BinlogServer的启动时通过http接口，用户主动开启，然后等待mysql slave的tcp 连接。

syncer_id、syncer_uuid如何指定

syncer_id、syncer_uuid的值是用主库的，还是自己随机写？另外为啥syncer_id指为0，leader会被干掉？

是否支持增强半同步复制

在异步复制架构中，master宕机时，如果部分binlog event未及时发送，会导致事务丢失。同样的，在使用mysqlbinlog工具做binlog server时，也会存在这种问题，并且由于操作系统机制，可能会有更多的情况产生。
请问，该工具能否支持半同步复制，或者是否有额外的措施用以弥补上述缺陷，谢谢！

启动syncer时报错404 page not found

[root@node04 mysql]# curl -H "Content-Type:application/json" -X PUT --data '{"syncer_id":1023306,"syncer_uuid":"274176a0-e96a-11e8-9645-000c293ddb5","mysql_addr":"192.168.73.102:3306","mysql_user":"repl","mysql_password":"Repl@2018","semi_sync":true}' http://192.168.73.104:9697/binlog/syncer/start
404 page not found

启动BinlogSyncer时报错binlog_server:args are not available

在启动binlog syncer时报错:
curl -H "Content-Type:application/json" -X PUT --data '{"syncer_id":173,"syncer_uuid":"f7e30637-f15a-11e8-accc-244427b6b60e","mysql_addr":"x.x.x.x:3306","mysql_user":"root","mysql_password":"root","semi_sync":false}' http://server_ip:5000/binlog/syncer/start
{"message":"binlog_server:args are not available"}
以下是日志信息:
2019-03-12T11:10:56.675+0800 info replication/binlogsyncer.go:136 create BinlogSyncer with config {173 f7e30637-f15a-11e8-accc-244427b6b60e mysql x.x.x. 3306 root utf8 false false false UTC true 0 10s 20s 3 true}
2019-03-12T11:10:56.677+0800 error server/server.go:869 NewSyncer:binlogProgress reset error,err:invalid interval format, must n[-n]
2019-03-12T11:10:56.677+0800 error server/server.go:813 startSyncerServer error,err:invalid interval format, must n[-n],args:{173 f7e30637-f15a-11e8-accc-244427b6b60e x.x.x.x:3306 root root false}
2019-03-12T11:10:56.677+0800 error api/binlog_syncer_handler.go:86 StartServer error,err:binlog_server:args are not available,args:{173 f7e30637-f15a-11e8-accc-244427b6b60e x.x.x.x:3306 root root false}
2019-03-12T11:10:56.677+0800 error api/binlog_syncer_handler.go:51 StartBinlogSyncer error,err: binlog_server:args are not available
这个问题该如何解决?
另外在查看当前运行的binlog syncer时,发现有个syncer_id=0的同步任务,这个任务的功能是什么?
能否提供删除syncer的删除接口处理?
curl -H "Content-Type:application/json" -X GET http://server_ip:5000/binlog/syncer/status {"message":"success","data":{"syncer_id":0,"syncer_uuid":"","mysql_addr":"","mysql_user":"","mysql_password":"","semi_sync":false,"current_gtid":"","last_binlog_file":"","last_file_position":0,"executed_gtid_set":"","purged_gtid_set":"","status":"stopped"}}

Howto retain binary log in kingbus

Hi there,
I'm a newbie and have been using kingbus for 3 days. Can I rotation kingbus binarylog? I can't find on the docs.
Thank you!
cc: @flike

Leader节点会同时启动syncer和binlogServer，但当Leader变成follower之后binlogServer不会stop对吧

支持mysql半同步复制或无损复制吗？

缺少yaml配置文件说明文档

问题描述:
按照quickstart 进行到第5步, 发现配置文件的链接是空链接.

解决方案:
是否可以补充配置文件的说明和各个配置参数的业务含义.

非常感谢

Mariadb 10.4 Support

Will this work on mariadb 10.4?

是否有类似Arbitrator节点的功能

今天稍微看了下文档.
是不是说binlog会存3份? 那么是否有类似Arbitrator的功能, 这样一个仲裁节点只负责仲裁. 只有两个节点存数据?

what formats of Replica is supported?

Hi
please help me, i have a question:

I don't understanding what type of replica RBR or SRB or Mixed is supported by KingBus

能否同时同步多个实例的binlog

一套kingbus集群中只能有一个syncer，是否表示一套kingbus集群只能同步保存一个实例的binlog？能否做到同时同步多个实例的binlog？

同步延迟不会被放大吗？

看了代码跟设计架构，作为一个中间件直接介入到数据库集群中，感觉存在这样几个问题：

引入运维复杂度跟风险点，维护数据库集群的时候需要再关注kingbus集群的状态。
同步复杂度增加，同样由于一个raft集群的引入，binlog的传输从之前的点对点变为了点到集群再到点，中间还夹杂着raft一致性保证带来的写放大的问题。即使是内网环境，随着kingbus节点和业务的增长，同步性能与数据安全性的平衡怎么保证？写入量大的场景备份延迟如何保证不能被放大？
kingbus集群leader挂掉，等同于所有的从库需要做一次主从切换，并没有减少切换成本。

感觉这个东西做个binlog server来备份、分发数据什么的还可以，至少比canel少了一堆组件要部署，但是作为中间件连接master和slave，感觉稍微有些鸡肋啊。

use dragonboat

https://github.com/lni/dragonboat

通过kingbus作为主库的从库与原主库延迟是什么量级

有实际测试过吗
另外希望有更多的文档.

从库无法获取serverid 导致无法同步

slave 报错
Last_IO_Errno: 1105
Last_IO_Error: The slave I/O thread stops because a fatal error is encountered when it try to get the value of SERVER_ID variable from master. Error: mysql:sql not support

kingbus报错

2019-07-08T15:50:00.397+0800 info mysql/command.go:119 handleQuery sql:SELECT UNIX_TIMESTAMP()
2019-07-08T15:50:00.398+0800 info mysql/command.go:119 handleQuery sql:SHOW VARIABLES LIKE 'SERVER_ID'
2019-07-08T15:50:00.399+0800 debug mysql/command.go:283 show query showVariable:
2019-07-08T15:50:00.399+0800 error mysql/conn.go:151 dispatch error,err:mysql:sql not support,data:[3 83 72 79 87 32 86 65 82 73 65 66 76 69 83 32 76 73 75 69 32 39 83 69 82 86 69 82 95 73 68 39]
2019-07-08T15:50:00.399+0800 debug mysql/conn.go:135 close client connection

如何解决？