These scripts can be used to test replication latency in a multi-region Cassandra cluster
To run the test scripts we have to execute the following commands
# apt update
# apt install python-pip
# pip install cassandra-driver
To create the test tables, run the create-test-tables.cql on your cluster
Start one writer in one local node, with the following parameters
$ python ./cassandra-tester-writer.py LOCAL_ADDRESS_OF_THE_NODE USERNAME PASSWORD NUMBER_OF_READERS
Start readers at the other end of the globe with the following parameters
$ python ./cassandra-tester-reader.py LOCAL_ADDRESS_OF_THE_NODE USERNAME PASSWORD
Output will show measured replication latency on each node
Before restarting the tests, you need to truncate all the tables
CONSISTENCY ALL
TRUNCATE replicated.test;
TRUNCATE replicated.test_count;We have 3 nodes in Belarus and 3 nodes in Asia Pacific (Tokyo). The latency between the two locations are the following TBD Global map
$ traceroute -T -p 7000 ...
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 198.325 ms 205.944 ms 206.179 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 198.464 ms 205.289 ms 197.268 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 196.553 ms 197.667 ms 197.117 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 201.749 ms 203.529 ms 203.689 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 1403.528 ms 210.139 ms 203.983 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 203.817 ms 201.101 ms 199.461 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 197.795 ms 195.290 ms 197.727 ms
15 ec2-...ap-northeast-1.compute.amazonaws.com (...) 208.103 ms 207.926 ms 206.635 ms
When running on real servers, you have to set-up NTP sync between them, or you'll measurements won't be accurate. For example our real severs show around 115 ms latency without any NTP configuration.
sudo yum install ntp
# vi /etc/ntp.conf
TBD
# ntpstat
# ntpq -p
[https://en.wikipedia.org/wiki/Network_Time_Protocol#Clock_synchronization_algorithm] [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html] [http://www.ntp.org/ntpfaq/NTP-a-faq.htm]
We've set NTPd to use the same server for synchronization. ntpstat gave us the following results after one day of continuous synchronization.
[[email protected]@... ~]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 169 ms
polling server every 1024 s
[[email protected]@... ~]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 180 ms
polling server every 1024 s
[cassandra@... bin]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 135 ms
polling server every 1024 s
[ec2-user@... ~]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 96 ms
polling server every 1024 s
[ec2-user@... ~]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 80 ms
polling server every 1024 s
[ec2-user@... ~]$ ntpstat
synchronised to NTP server (117.102.176.202) at stratum 3
time correct to within 50 ms
polling server every 1024 s
Comparing the time correctness with the network latency between the nodes reveals, that the results won't give us the answer precisely. But it can be good enough for an approximation.
The scripts were tested with boot2docker on Windows. Note how widespread are the measured results. This wide distribution of latency can't be found, when running the tests on the real servers.
Use the create-cluster.sh bash script to setup your cluster with docker
You have to start the nodes manually if they failed to start up properly.
docker start by1node3
docker exec -ti --privileged awsnode1 tc qdisc add dev eth1 root netem delay 25ms
docker exec -ti --privileged awsnode1 tc qdisc show
docker exec -ti --privileged awsnode1 tc qdisc del dev eth0 root netem
# add-delay.sh 250ms
docker exec by1node1 nodetool status
docker ps -qa | xargs docker rm -f
Start on by1node1
python ./writer.py 192.168.1.10 cassandra cassandra 3
Start on awsnode1
python ./reader.py 192.168.2.10 cassandra cassandra
Start on awsnode2
python ./reader.py 192.168.2.11 cassandra cassandra
Start on awsnode3
python ./reader.py 192.168.2.12 cassandra cassandra
TBD
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent