I have a small 5-node etcd cluster and a set of 3 processes that listen on an etcd file using eternal_watch, while at the same time repeatedly trying to test_and_set the same etcd file every 29 seconds. The file's TTL is set to 30 seconds. I'm using this to implement master election between the 3 processes.

Everything seems to be running fine for about 40 minutes. After that the processes that do not succeed in the test_and_set operation keep receiving the EtcdEventIndexCleared in eternal_watch, with the error:

EtcdEventIndexCleared: The event in requested index is outdated and cleared : the requested history has been cleared

The CPU utilization spikes to 100% on the python processes, and etcd's CPU utilization increases dramatically too.

The exception actually happens in the read method. It seems to be an error caused by etcd, not by tornetcd. I'm using etcd 2.3.6.

I was wondering if you have seen this problem before and how you fixed it.