Make sure you have installed fleet:

brew update & brew install fleetctl

• user-data for Vagrant cluster
• user-data for Development cluster
• user-data for Staging cluster
• user-data for Production cluster
• Ansible tasks to build & deploy containers

These are set by default in the Dockerfiles. They can be adjusted at run time in the service files by appending -e KEY=VALUE in the run docker command.

• PUBLIC_IPV4 0.0.0.0 (This should be set by the systemd unit)
• PRIVATE_IPV4 0.0.0.0 (This should be set by the systemd unit)

All units will broadcast them selfs onto etcd cluster using the following convention:

/services/{{service_name}}/{{service_name}}-{{instance}} 'SomeValue'
# E.g.
/services/elasticsearch/elasticsearch-0 '{"IP":"172.16.1.102","HttpPort":9201,"TransportPort":9301}'

These unit's will auto find each other via zen unicast. They add them selfs to etcd like so /services/elasticsearch/elasticsearch-%i "%i" being the instance name. Each time a new unit is added to the cluster it will add the previous nodes to the unicast hosts. See elasticsearch.yml.

TODO:

• Build a discovery service that will only broadcast the Elasticsearch box once its "avaliable"
• Define meta data so that they only run on specified boxes so they will pick up there exisiting data stores.

Building Elasticsearch docker image and starting it in fleet:

# Build task
docker build -t chrisjenx/elasticsearch ./elasticsearch/
docker push [name]/elasticsearch
# Run on the cluster
fleetctl start ./elasticsearch/systemd/elasticsearch@{0,1}.service

The simple regex "{0,1}" defines the instances to start.

You can check the Elastic cluster status hitting the follow end-point: http://[cluster-ip]:9200/_plugin/kopf/

Kibana and elastic search are seperated, there was no need for these to be on the same box. (Seperation of concerns etc).

This container is Kibana + Nginx. Kibana talks to Elasticsearch directly, the nginx server on this container provides a reverse proxy and a host for kibana.

http://[kibana-host]:9100 will serve the pages then the ajax requests are reverse proxied to the Elasticsearch cluster.

This also means as Elasticsearch boxes are added/removed the reverse proxy is reconfigured so kibana will never know the difference.

Building Kibana docker image and starting it in fleet:

# Build task
docker build -t [name]/kibana ./kibana/
docker push [name]/kibana
# Run on the cluster
fleetctl start ./kibana/systemd/kibana.service

We use HAProxy to load balance between multiple units.

Building haproxy docker image and starting it in fleet:

# Build task
docker build -t chrisjenx/haproxy ./haproxy/
docker push chrisjenx/haproxy
# Run on the cluster
cd haproxy && fleetctl start haproxy.service

Building logstash docker image and starting it in fleet:

# Build task
docker build -t chrisjenx/logstash ./logstash/
docker push chrisjenx/logstash
# Run on the cluster
cd logstash && fleetctl start logstash.service