qihoo360 / zeppelin Goto Github PK

The data_thread_num is hardcoded in config file, and I think we could make this value is related to number of CPU's on the machine. For example, set data_thread_num to std::hardwared_concurrency()*0.75

Zeppelin calculate key location by hash tag if exist.

Support list and delete operation depends on hash tag.

a Distributed Key-Value Platform which {aim} to {providing} excellent performance
-> aims, provide

@CatKang

Support multi-operation executed together.

Only write or delete operation supported in batch.

Operations in same batch keep atomic.

Requires server provide more information when wrong master is accessed, such as the suitable return code, target server ip and port

Together with request block

👆🏻👆🏻👆🏻

Partition db drop is dangerous and it may happened sometimes as below:

1. Table drop, then all partition dbs under it should be drop;
2. Partition migration, the old node should drop this partition db;
3. Clear command required by manager

We need a more elegant drop mechanism to enable us to restore the dropped db, and then a simple way to delete it forever.

This is similar to recycle bin

Extra null pointer checks are not needed in a destructor like for the class “ZPBinlogSendTask”.

Would you like to wrap any pointer data members with the template class “std::unique_ptr”?

May with Jepsen

创建3个meta在tpd1机器上端口9221,9222,9223
3个node在tpd2机器上，端口13221,13222,13223

执行zk-manager，查看meta，node都正确，但是创建table异常
命令如下：
Zeppelin(tpd1:9221)>> listnode
tpd2:13221 up
tpd2:13222 up
tpd2:13223 up
Zeppelin(tpd1:9221)>> listmeta
Leader:
--- tpd1:9221, status: Up
Followers:
--- tpd1:9222, status: Up
--- tpd1:9223, status: Up
Epoch: -1
Consistency status:

  Node           |    Role    | Term |      Leader      |      VoteFor      | LastLogTerm | LastLogIdx | CommitIndex | LastApplied |
       tpd1:9322    follower     31          tpd1:9321            tpd1:9321          31        13140         13139        13139
       tpd1:9323    follower     31          tpd1:9321            tpd1:9321          31        13140         13139        13139
       tpd1:9321      leader     31          tpd1:9321            tpd1:9321          31        13140         13140        13140

OK
Zeppelin(tpd1:9221)>> createtable test1 hostlist.txt 2
Checkup distribution result
Node Num: 3
Partition Num: 6
3 replicas in 1 node: 0
2 replicas in 1 node: 0
3 replicas in 1 host: 6
2 replicas in 1 host: 0
3 replicas in 1 cabinet: 6
2 replicas in 1 cabinet: 0

6 partitions node num: 3
+++++++++++++++++++++++++++++++++++++
Continue? (Y/N/dump)
dump
Table name: test1
Partition id: 0 master: 10.31.2.242 :13222 slave0: 10.31.2.242 :13221 slave1: 10.31.2.242 :13223
Partition id: 1 master: 10.31.2.242 :13223 slave0: 10.31.2.242 :13221 slave1: 10.31.2.242 :13222
Partition id: 2 master: 10.31.2.242 :13221 slave0: 10.31.2.242 :13222 slave1: 10.31.2.242 :13223
Partition id: 3 master: 10.31.2.242 :13222 slave0: 10.31.2.242 :13221 slave1: 10.31.2.242 :13223
Partition id: 4 master: 10.31.2.242 :13223 slave0: 10.31.2.242 :13221 slave1: 10.31.2.242 :13222
Partition id: 5 master: 10.31.2.242 :13221 slave0: 10.31.2.242 :13222 slave1: 10.31.2.242 :13223
Continue? (Y/N/dump)
Y
Corruption: Node 10.31.2.242 :13221 is down

In current version, the partition distribute algorithm has some problems, such as we can't not allot every node the same number of partition, we can only promise that the 3 copies not be put in same node, but we can't not promise that any two copy will not be put in same machine, since there is many nodes in a machine.

we need improve the partition distribute algorithm, the most elaborate dirstribute algorithm I have seen is ceph's crush algorithm. Maybe we can copy this algorithm and implement it in zeppelin

配置样例：

Server related

meta_addr : xx.xx.xx.xx/9221,xx.xx.xx.xx/10331

代码：zp_data_server.cc 265 auto pos = addr.find(":");

问题：
1.和meta保持一致的话，应该是按照/分隔吧？
2.这个东西你们大规模应用了吗？