The following table shows the versions of elasticsearch and Apache Zookeeper that Zookeeper Plugin was built with.

• Download ZooKeeper 3.3.6 from http://zookeeper.apache.org/releases.html
• Unzip the ZooKeeper archive into a directory, rename conf/zoo-sample.cfg into zoo.cfg and modify dataDir= line to point to a directory on your machine. Start ZooKeeper by running bin/zkServer.sh start.
• Install ZooKeeper plugin to ES by running

for elasticsearch-zookeeper versions 0.90.2 and above:

% bin/plugin -url https://github.com/imotov/elasticsearch-zookeeper/releases/download/v1.0.0/elasticsearch-zookeeper-1.0.0.zip -install zookeeper

for elasticsearch-zookeeper versions 0.90.0 and below:

% bin/plugin -url https://oss-es-plugins.s3.amazonaws.com/elasticsearch-zookeeper/elasticsearch-zookeeper-0.90.0.zip -install zookeeper

• Assuming that you are running ZooKeeper on the port 2181 (default), add the following lines to config/elasticsearch.yml file

discovery:
    type: com.sonian.elasticsearch.zookeeper.discovery.ZooKeeperDiscoveryModule
sonian.elasticsearch.zookeeper:
    settings.enabled: true
    client.host: localhost:2181
    discovery.state_publishing.enabled: true

• Start ES

The only mandatory parameter in the ZooKeeper plugin configuration is client.host. This parameter should contain an address of ZooKeeper server or comma-separate list of addresses if multiple servers are used.

The sonian.elasticsearch.zookeeper.discovery.state_publishing.enabled parameter specifies if the plugin should use ZooKeeper to publish cluster state. By default, the cluster state is published using org.elasticsearch.discovery.zen.publish.PublishClusterStateAction, which causes master to send cluster state to all other nodes. If ZooKeeper-based state publishing is enabled, state is published inside ZooKeeper and all nodes are reading state from the ZooKeeper as soon as it changes.

The sonian.elasticsearch.zookeeper.settings.enabled parameter specifies if the plugin should attempt to load configuration form ZooKeeper.

ZooKeeper Discovery plug-in can load settings from zookeeper. The bin/zookeeper utility can be used to upload new settings to ZooKeeper. The settings can be upload on per cluster basis as well as globally.

Assuming that current directory is ES_HOME, to upload cluster settings use:

$ sh plugins/zookeeper/bin/zookeeper set elasticsearch.json

To upload global settings use:

$ sh plugins/zookeeper/bin/zookeeper set-global elasticsearch.json

To retrieve current global or cluster settings use one of the following commands:

$ sh plugins/zookeeper/bin/zookeeper get
$ sh plugins/zookeeper/bin/zookeeper get-global

The settings are loaded in the following order:

• Local settings
• Global settings
• Cluster settings

The later loaded settings take precedence. In other words, cluster settings override global settings and global settings override local settings.

ES ZooKeeper Discovery plug-in is using ZooKeeper as an atomic, reliable, distributed database for node fault detection, master election and sharing common settings. It uses the fact that ZooKeeper can maintain a tree of nodes in which each node can have an associated value as well as child node, these nodes can be created and read atomically and clients can be notified about updates.

The ZooKeeper discovery process can be divided in 3 logical steps.

Upon ES node start, the ZooKeeper discovery plugin connects to ZooKeeper and creates the following persistent node if it doesn’t already exist:

/es/clusters/cluster name/nodes

Then ES node creates an ephemeral ZK node /es/clusters/cluster name/nodes/node-id with serialized content of local DiscoveryNode.

After registration ES node first tries to read ZK node /es/clusters/cluster name/leader and starts watching this node. Existence of this node indicates that cluster already has a master elected. In which case ES node stops the master discovery process and awaits for the cluster state to be sent to it by the master.

If ZK node /es/clusters/cluster name/leader doesn’t exist and ES node is eligible to be elected as master, ES node tries to create ephemeral ZK node /es/clusters/cluster name/leader. This is an atomic operation and if multiple ES nodes try to create the leader node at the same time only one of them succeeds. If ZK node creation was successful, ES node that created the leader node assumes the role of the cluster master. If leader node creation failed, ES node goes back to trying to read ZK node /es/clusters/cluster name/leader.

If ZK node /es/clusters/cluster name/leader doesn’t exist and ES is not eligible to be elected as a master, it sets NO_MASTER_BLOCK and waits for node creation watcher to be triggered. As soon as the ZK node is created, ES node restarts the master discovery process.

The watcher that is set during read operation on the leader ZK node is also triggered when node disappears. Leader node disappearance is causing the restarts the master discovery process. Because the /es/clusters/cluster name/leader node is ephemeral, it disappears as soon as the ES master node that created it disconnects from the system or stops responding to pings. It ensures that cluster master is always available

If ES node is elected as a master, it reads list of children of the /es/clusters/cluster name/nodes node and starts watching for any changes in the list. Then ES updates the list of nodes in the cluster state to match the list of nodes in ZK. When the watcher on the /es/clusters/cluster name/nodes node is triggered the Node List Update operation is repeated. In other words ephemeral nodes in /es/clusters/cluster name/nodes perform the role of node fault detections.

Join elasticsearch-zookeeper group for questions and discussions related to this plugin.