metaswitch / clearwater-docker Goto Github PK

Docker integration for Project Clearwater

License: Other

Shell 17.25% Python 36.85% Dockerfile 45.90%

clearwater-docker's Introduction

Project Clearwater is backed by Metaswitch Networks. We have discontinued active support for this project as of 1st December 2019. The mailing list archive is available in GitHub. All of the documentation and source code remains available for the community in GitHub. Metaswitch’s Clearwater Core product, built on Project Clearwater, remains an active and successful commercial offering. Please contact [email protected] for more information. Note – this email is for commercial contacts with Metaswitch. We are no longer offering support for Project Clearwater via this contact.

Clearwater Docker

This repository contains instructions and resources for deploying Project Clearwater as Docker containers. It describes three deployment options -- the first two are recommended.

Using Docker Compose for quick deployments on a single host.
Using Kubernetes for orchestrated deployments on clusters of machines.
By deploying the containers manually.

You should follow the "Common Preparation" section, then either the "Using Compose", "Using Kubernetes" or the "Manual Turn-Up" section.

Common Preparation

When using the 3.13.0-74-generic kernel, we've seen an issue which causes Clearwater not to start properly. If you are using this kernel (uname -r will tell you), you should install a newer kernel (e.g. with sudo apt-get install linux-image-3.13.0-87-generic) and reboot.

To prepare your system to deploy Clearwater on Docker, run:

# Install Docker (on Ubuntu).
wget -qO- https://get.docker.com/ | sh

# Checkout clearwater-docker.
# Either:
git clone --recursive [email protected]:Metaswitch/clearwater-docker.git
# Or:
git clone --recursive https://github.com/Metaswitch/clearwater-docker.git

If deploying with compose or manually

edit clearwater-docker/.env so that PUBLIC_IP is set to an IP address that can be used by SIP clients to access the docker host. E.g. if you are running in AWS, this wants to be the public IP of your AWS VM. Note: this file is not used when deploying on Kubernetes.
if you want to be able to monitor your Docker deployment via a web UI then you might like to install and run Weave Scope. This only takes a minute to install and provides real time visualizations showing all of your containers, their resource usage and the connectivity between them.

By default Docker images built using clearwater-docker will pull the latest stable versions of Clearwater packages from http://repo.cw-ngv.com/stable. If you check out the latest stable branch of clearwater-docker e.g. release-XXX then this should be fine. If you are using the master branch of clearwater-docker then it will sometimes be dependent upon the latest Clearwater packages. To use these instead update base/Dockerfile to reference http://repo.cw-ngv.com/latest instead before building the base Docker image.

Limitations

Note that scaling of Docker deployments is a work in progress and there are currently a number of known issues...

Homestead-prov and Ellis don’t load balance across multiple Cassandra nodes.
In general deleting pods from storage clusters (Cassandra, Chronos or Astaire) is not supported. Pods that are deleted will not get removed from the clusters and the clusters will end up broken. The exception is when deployed under Kubernetes -- in this scenario Chronos and Astaire pods can be terminated, so long as this is done gracefully such that their prestop event hook is executed. This means that Astaire and Chronos clusters (under Kubernetes) can be dynamcially scaled up and down.

Using Compose

There is a Compose file to instantiate a minimal (non-fault-tolerant) distributed Clearwater deployment under Docker.

104.155.15.20

Preparation

To prepare your system to deploy Clearwater using Compose, after running the common preparation steps above, run:

# Install Docker Compose (on Ubuntu).
sudo apt-get install python-pip -y
sudo pip install -U docker-compose

# Build the base Clearwater docker image.
cd clearwater-docker
sudo docker build -t clearwater/base base
sudo docker-compose -f minimal-distributed.yaml build

Note that Compose will name each image like ${IMAGE_PREFIX}/<node_name> and IMAGE_PREFIX=clearwater by default. If you need to specify your registry, please edit IMAGE_PREFIX in clearwater-docker/.env like IMAGE_PREFIX=<your_registry>/clearwater.

Starting Clearwater

To start the Clearwater services, run:

# Build all the other Clearwater Docker images and start a deployment.
sudo docker-compose -f minimal-distributed.yaml up -d

Scaling the deployment

Having started up a deployment, it is then possible to scale it by adding or removing additional nodes. E.g. run

sudo docker-compose -f minimal-distributed.yaml scale sprout=2 astaire=2 chronos=2 cassandra=2

Note that it is not possible to scale down storage node clusters -- see limitations above.

If you scale up the clusters of storage nodes, you can monitor progress as new nodes join the clusters by running utils/show_cluster_state.sh.

Using Kubernetes

Instead of using Docker Compose, you can deploy Clearwater in Kubernetes. This requires a Kubernetes cluster, and a Docker repository.

Prepare the images and edit the yaml files

First, build all the required images locally.

  # Build the Clearwater docker images.
  cd clearwater-docker
  for i in base astaire cassandra chronos bono ellis homer homestead homestead-prov ralf sprout ; do docker build -t clearwater/$i $i ; done

Next, push them to your repository (which must be accessible from the Kubernetes deployment)

  for i in base astaire cassandra chronos bono ellis homer homestead homestead-prov ralf sprout
  do
      docker tag clearwater/$i:latest path_to_your_repo/clearwater/$i:latest
      docker push path_to_your_repo/clearwater/$i:latest
  done

Create an env-vars config map.
- At a minimum this must include a ZONE key set to the domain of your Kubernetes cluster e.g. default.svc.cluster.local
- If you wish to use an HSS (instead of Ellis and Homestead-prov) then you can also include an HSS key with a value equal to the name of the HSS node: this will result in the deployment being configured to use the HSS at <HSS key>.<ZONE key>, with an HSS realm of <ZONE key>. If you want to use an HSS in a different domain then you will have to manually configure the appropriate values -- see below.
- It may also include an ADDITIONAL_SHARED_CONFIG key whose value includes additional shared config settings that you want to use. These will override any auto-generated settings. E.g. this can be used to specify an HSS in a different domain. Multiple settings should be separated with \\n
e.g. kubectl create configmap env-vars --from-literal=ZONE=default.svc.cluster.local --from-literal=ADDITIONAL_SHARED_CONFIG=hss_hostname=hss.example.com\\nhss_realm=example.com
If you're using a private container registry (one that requires credentials to pull images from), create a secret with the required credentials. e.g. kubectl create secret docker-registry myregistrykey --docker-server=$DOCKER_REGISTRY_SERVER --docker-username=$DOCKER_USER --docker-password=$DOCKER_PASSWORD --docker-email=$DOCKER_EMAIL
Update the Kubernetes yaml to match your deployment.
- Generate the Kubernetes yaml files from the templates by going to the kubernetes directory and running ./k8s-gencfg --image_path=<path to your repo> --image_tag=<tag for the images you want to use> If you're using a private container registry, add the argument --image_secret=myregistrykey (where myregistrykey matches the secret you made earlier) The script assumes that the Clearwater images that you want to use are located at {{image_path}}/<image name e.g. bono>:{{image_tag}}. It will also generate a helm chart in /kubernetes/clearwater.
- Decide how you want to access Bono and Ellis from outside of the cluster.
  
  The default configuration assumes that you have configured your network such that your SIP endpoints (and whatever devices you will access Ellis from) can
  - connect directly to the IP addresses that Kubernetes assigns to your pods
  - resolve interal kubernetes DNS records (typically as a result of an external DNS server delegating requests for the appropriate domain to the Kubernetes DNS service)
  If this is the case then you do not need to make any changes here. SIP devices can register with bono.:5060 and the Ellis provisioning interface can be accessed at http://ellis.. If these requirements are not met then you will need to arrange alternative means to access Bono and Ellis -- see below for an example of how this can be done on e.g. GKE.

Alternative configuration for exposing Bono and Ellis services

If the above requirements are not met (external devices cannot resolve Kubernetes DNS entries and/or cannot directly access pod IP addresses) then an alternative mechanism must be used to access Ellis and Bono.

To expose Ellis, you can simply set it up as a NodePort service:
- Replace clusterIP: None in ellis-svc.yaml with type: NodePort, and add a line to the "http" port configuration specifying nodePort: <port number>
- Depending on your platform you may need to manually create a firewall rule to allow access to this port. E.g. on GKE this can be done from the command line using gcloud if you have it installed. e.g. gcloud compute firewall-rules create ellis --allow tcp:30080
- The Ellis provisioning interface can then be accessed on http::30080
If you are unable to access the Ellis web interface (e.g. this happens with AKS) then you should instead replace clusterIP: None with type: LoadBalancer.
Bono is more challenging to expose due to the following requirements:
- Each Bono pod must be configured with an externally routable IP address by which that specific pod can be uniquely accessed. Bono will record-route itself in SIP messages using this IP and susbequent SIP messages to that IP address must be guaranteed to hit the same Bono instance.
- Port 5060 in the Bono pod must be accessible via port 5060 on the external IP address. It is not possible to e.g. NAT port 5060 in the pod to port 30060 on the external IP. This is because Bono always record-route's itself in SIP messages as <PUBLIC_IP>:5060.
On e.g. GKE the easiest solution is to use a LoadBalancer with a statically assigned external IP address. This brings in the following limitations:
- you can only have a single Bono instance (as subsequent SIP requests in a session must be guaranteed to be routed back to the same Bono instance so you cannot have the load balancer balance across multiple Bonos)
- the deployment can only support SIP over UDP or SIP over TCP (not both simultaneously) as Bono cannot have separate external IP addresses for each of UDP and TCP, and a single Kubernetes LoadBalancer service can't support multiple protocols. By default bono-svc.yaml is configured to expose SIP over TCP.
If using AKS
- You must disable HTTP Application routing when creating the cluster
- you must replace clusterIP: None with type: LoadBalancer in bono-svc.yaml.
For GKE and other platforms you must:
- Have a static external IP address available that the load balancer can use (e.g. on GKE you must explicitly provision this first)
- Replace clusterIP: None in bono-svc.yaml with type: "LoadBalancer", and add a line following this of loadBalancerIP: <static IP>
- Add the lines name: PUBLIC_IP and value: <static IP> to the env: section of the bono-depl.yaml file

Deploy Clearwater in Kubernetes

Using kubectl

To deploy the images, you should simply run kubectl apply -f clearwater-docker/kubernetes. It may take a minute or so before the deployment is fully established, the load balancer is created, and the deployment is ready to accept calls.

Note this will deploy all containers. If you don't need e.g. Bono, Homestead-prov, Ellis etc. then just move the corresponding svc and depl files out of the directory before running the create command.

Using helm

Run helm install clearwater inside kubernetes directory to deploy.

Run live tests against the deployment

If you have deployed using the standard configuration then you can run the clearwater-live-tests against the deployment using e.g.

rake test[default.svc.cluster.local] PROXY="bono.default.svc.cluster.local" SIGNUP_CODE=secret

If you have had to expose Bono and Ellis in a non-standard manner, you may need to change the PROXY argument, and add an ELLIS argument, so that the test scripts are able to access these services. e.g.

rake test[default.svc.cluster.local] PROXY={{Bono service DNS/IP}} ELLIS={{Ellis service/IP}} SIGNUP_CODE=secret

Scaling the deployment

Most Clearwater services can be dynamically scaled up and down by running e.g. kubectl scale deployment sprout --replicas=3 Exceptions are:

You can only have a single Bono if Bono pods do not have externally routable IP addresses.
Astaire and Chronos clusters can be scaled up and down so long as pods are terminated in such a way that their prestop hook is executed.
The Cassandra cluster can be scaled up but not down. See general limitations above.

After scaling the Chronos, Astaire or Cassandra clusters you should wait for the clusters to stabilise before performing another scaling operation. You can monitor cluster state by running cw-check_cluster_state in a pod belonging to any of the clusters.

Manual Turn-Up

If you can't or don't want to use Compose, you can turn the deployment up manually under Docker.

Preparation

To prepare your system to deploy Clearwater without using Compose, after running the common preparation steps above, run:

# Build the Clearwater docker images.
cd clearwater-docker
for i in base astaire cassandra chronos bono ellis homer homestead homestead-prov ralf sprout ; do sudo docker build -t clearwater/$i $i ; done

Starting Clearwater

To start the Clearwater services, run:

sudo docker network create --driver bridge clearwater_nw
sudo docker run -d --net=clearwater_nw --name etcd quay.io/coreos/etcd:v2.2.5 -name etcd0 -advertise-client-urls http://etcd:2379,http://etcd:4001 -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 -initial-advertise-peer-urls http://etcd:2380 -listen-peer-urls http://0.0.0.0:2380  -initial-cluster etcd0=http://etcd:2380 -initial-cluster-state new
sudo docker run -d --net=clearwater_nw --name astaire -p 22 clearwater/astaire
sudo docker run -d --net=clearwater_nw --name cassandra -p 22 --sysctl net.ipv6.conf.lo.disable_ipv6=0 clearwater/cassandra
sudo docker run -d --net=clearwater_nw --name chronos -p 22 clearwater/chronos
sudo docker run -d --net=clearwater_nw --name homestead -p 22 clearwater/homestead
sudo docker run -d --net=clearwater_nw --name homestead-prov -p 22 clearwater/homestead-prov
sudo docker run -d --net=clearwater_nw --name homer -p 22 clearwater/homer
sudo docker run -d --net=clearwater_nw --name ralf -p 22 clearwater/ralf
sudo docker run -d --net=clearwater_nw --network-alias=icscf.sprout --network-alias=scscf.sprout --name sprout -p 22 clearwater/sprout
sudo docker run -d --net=clearwater_nw --name bono --env-file .env -p 22 -p 3478:3478 -p 3478:3478/udp -p 5060:5060 -p 5060:5060/udp -p 5062:5062 clearwater/bono
sudo docker run -d --net=clearwater_nw --name ellis -p 22 -p 80:80 clearwater/ellis

The Clearwater Docker images use DNS for service discovery - they require, for example, that the name "ellis" should resolve to the Ellis container's IP address. In standard Docker, user-defined networks include an embedded DNS server which guarantees this (and this is why we create the clearwater_nw network) - and this type of DNS server is relatively common (for example, Kubernetes provides something similar).

Scaling the deployment

It is possible to spin up additional Sprout, Cassandra, Astaire and Chronos nodes simply by repeating the relevant command docker run command but providing a different name. E.g.

sudo docker run -d --net=clearwater_nw --name astaire_2 -p 22 clearwater/astaire

Scaling of clearwater-docker deployments is work in progress though, so see the limitations described above (for scaling using Compose).

Exposed Services

The deployment exposes

the Ellis web UI on port 80 (exposed on port 8080) for self-provisioning - the signup key is "secret"
STUN/TURN on port 3478 for media relay
SIP on port 5060 for service
SIP/WebSocket on port 5062 for service.

Additionally, each node exposes SSH - use sudo docker ps to see what port its exposed on. The username/password is root/root. Alternatively you can run a bash session in a container by name using e.g. sudo docker exec -it <container_name> bash

What Next?

Once you've turned up the deployment, you can test it by

making a call - make sure you configure your SIP clients with a proxy, as if it were an All-in-One node
running the live tests - again, set the PROXY and ELLIS elements, as if it were an All-in-One node.

Utilities

There are a few scripts that offer short cuts to querying aspects of your deployment:

# Show an abbreviated version of docker ps that fits without wrapping on smaller terminals
utils/short_ps.sh

# Show the IP addresses of the containers in your deployment
utils/show_ips.sh

# Query Chronos nodes over SNMP to get the number of active registrations
utils/show_registration_count.sh

# Show information about the state of the storage clusters
utils/show_cluster_state.sh

If one of your pods is failing and you can't exec into it because it's in a crash loop, just run kubectl logs on the sidecar instead. e.g.

kubectl logs astaire-7b74ff8df9-5lhz6 -c tailer

Cleaning Up

If you wish to destroy your deployment either to redeploy with a different configuration or version or to free up resources on your docker host, the following may be useful commands:

# To rebuild an image (rather than pull it from the cache), add `--no-cache` or `--force-recreate` to the build commands
sudo docker build --no-cache -t clearwater/base base
sudo docker-compose -f minimal-distributed.yaml build
sudo docker-compose -f minimal-distributed.yaml up --force-recreate

# Remove all docker containers (not just Clearwater ones!)
sudo docker rm $(sudo docker ps -aq)

# Remove all the docker image files (not just Clearwater ones!)
sudo docker rmi $(sudo docker images -aq)

# Remove most of the docker image files, but not the Ubuntu 14.04 base
# image, use this if you intend to redeploy the clearwater deployment
# immediately to save time.
#
# This command will report an error due to a conflict, this can be safely
# ignored.
sudo docker rmi $(sudo docker images -a | tail -n +2 | grep -v "14.04" | tr -s ' ' | cut -f3 -d' ')

clearwater-docker's People

Contributors

Stargazers

Watchers

clearwater-docker's Issues

expose all data dirs (cassandra, mysql etc..) in dockerfiles

Homestead tries to contact cassandra on localhost

Symptoms

Symptom is that homestead terminates, ellis does not work, and the deployment is DOA. This is on a multi-host deployment, so localhost does not resolve to the cassandra node (although cassandra does).

I'm actually using Kubernetes, but believe that the same issue would occur on multi-host with a pure Docker solution.

Impact

Clearwater does not work.

Release and environment

Currently running off the clearwater branch.

Steps to reproduce

Install using Kubernetes. Ellis does not work (happy to help with repro).

build ralf image error

Ralf image has a common problem like bono ,which is tcp kernel params set error in script clearwater-tcp-scalability.postinst

and some more errors occured ,not sure if released with above
Package astaire is not configured yet.
E: Sub-process /usr/bin/dpkg returned an error code (1)

the following is error message
/usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: 40: /usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: sysctl: Permission denied
dpkg: error processing package clearwater-tcp-scalability (--configure):
subprocess installed post-installation script returned error exit status 126
Setting up clearwater-log-cleanup (1.0-160927.094809) ...
Setting up astaire-libs (1.0-160927.103550) ...
Setting up cpulimit (2.0-1) ...
Setting up libc-ares-dev:amd64 (1.10.0-2) ...
dpkg: dependency problems prevent configuration of astaire:
astaire depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.

dpkg: error processing package astaire (--configure):
dependency problems - leaving unconfigured
Setting up snmp (5.7.2dfsg-clearwater4) ...
Setting up snmpd (5.7.2dfsg-clearwater4) ...
update-rc.d: warning: stop runlevel arguments (1) do not match snmpd Default-Stop values (0 1 6)
invoke-rc.d: policy-rc.d denied execution of start.
Setting up memcached (1.6.00-0clearwater0.5) ...
Restarting memcached: memcached.
Setting up clearwater-memcached (1.0-160927.094809) ...
invoke-rc.d: policy-rc.d denied execution of start.
Setting up clearwater-socket-factory (1.0-160927.094809) ...
Setting up lksctp-tools (1.0.15+dfsg-1) ...
Setting up ralf-libs (1.0-160927.095730) ...
Processing triggers for ureadahead (0.100.0-16) ...
dpkg: dependency problems prevent configuration of ralf:
ralf depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.
ralf depends on astaire; however:
Package astaire is not configured yet.

dpkg: error processing package ralf (--configure):
dependency problems - leaving unconfigured
Setting up clearwater-snmpd (1.0-160927.094809) ...

Stopping network management services:
Setting up chronos (1.0-160927.094813) ...
Setting up clearwater-snmp-handler-astaire (1.0-160927.112236) ...
Stopping network management services:
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
clearwater-tcp-scalability
astaire
ralf
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ralf' returned a non-zero code: 100

cassandra start failed

@eleanor-merry @MatMeredith

When I deploy cassandra deployment, it gives the error as follows:

ERROR [main] 2017-10-20 02:43:26,773 DatabaseDescriptor.java:141 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Expecting URI in variable: [cassandra.config].  Please prefix the file    with file:/// for local files or file://<server>/ for remote files. Aborting. If you are executing this from an external tool,    it needs to set Config.setClientMode(true) to avoid loading configuration.
        at org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:73) ~[apache-ca   ssandra-2.1.15.jar:2.1.15]
        at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[apache-cassandra-2   .1.15.jar:2.1.15]
        at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[apache-cassandra-2.1.15.jar   :2.1.15]
        at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) ~[apache-cassandra-2.1.15.jar:2   .1.15]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:170) [apache-cassandra-2.1.15.jar:2.1.15]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.15.jar:2.1.15]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.15.jar:2.1.15]

Then the pod logs are as follow:

2017-10-20 10:45:36,661 CRIT reaped unknown pid 5815)
2017-10-20 10:45:36,997 CRIT reaped unknown pid 5816)
2017-10-20 10:45:37,848 INFO success: cassandra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-20 10:45:37,849 CRIT reaped unknown pid 5860)
2017-10-20 10:45:38,668 INFO exited: cassandra (exit status 232; not expected)
2017-10-20 10:45:39,671 INFO spawned: 'cassandra' with pid 5876
2017-10-20 10:45:39,701 CRIT reaped unknown pid 5892)
2017-10-20 10:45:40,023 CRIT reaped unknown pid 5893)
2017-10-20 10:45:40,675 INFO success: cassandra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-20 10:45:40,676 CRIT reaped unknown pid 5942)
2017-10-20 10:45:41,711 INFO exited: cassandra (exit status 232; not expected)
2017-10-20 10:45:42,715 INFO spawned: 'cassandra' with pid 5958
2017-10-20 10:45:42,744 CRIT reaped unknown pid 5974)
2017-10-20 10:45:43,084 CRIT reaped unknown pid 5975)
2017-10-20 10:45:43,989 INFO success: cassandra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-10-20 10:45:43,989 CRIT reaped unknown pid 6019)
2017-10-20 10:45:44,754 INFO exited: cassandra (exit status 232; not expected)
2017-10-20 10:45:45,759 INFO spawned: 'cassandra' with pid 6035
2017-10-20 10:45:45,804 CRIT reaped unknown pid 6051)

At the same time there is no file : /etc/cassandra/cassandra.yaml

Symptoms

So how can I solve the problem, thanks!

Impact

The kubernetes gives the following error:
Readiness probe failed: Reporting config ADDRESS=192.168.9.16 PORTS=7000 9042 9160 TIMEOUT=5 Testing port 7000

Release and environment

Steps to reproduce

sprout args error

if I set IP addr to param chronos-hostname ,sprout can't startup
--chronos-hostname=169.169.30.160

Ellis starts with wrong configuration

Ellis can start with the wrong IP address configuration for Homestead and Homer. Restarting Ellis fixes the problem.

It looks like the root cause is that supervisord does not guarantee that the clearwater-auto-config-docker and clearwater-infrastructure services are started prior to starting Ellis. Since clearwater-auto-config-docker generates the configuration file, Ellis may start with the wrong configuration (likely the IP address assigned to the image when it is built).

It doesn't look like there is an easy fix to this in supervisord (eg. see http://stackoverflow.com/questions/17443692/python-supervisord-program-dependency). It sounds like priority won't guarantee that the auto-config has completed before starting Ellis, and the recommendation is either to use an event listener or a wrapper shell script which only starts Ellis when the auto-config has completed.

User configurable ports for X service - (mainly ETCD)

Description

When deploying clearwater docker on kubernetes , we are using kubernetes services to decouple containers.
In this way the the kubernetes services provides enhanced decoupling of IPs as well as load balancing.

We have decoupled all of the services expect for ETCD.

Symptoms

We cannot create a service with port 4000,4001,2379,2378 (default etcd ports) as this conflicts with the kubernetes infrastructure (port already open).

Need to be able to configure this

Impact

Problems with scaling on K8s using inbuilt service method.
https://kubernetes.io/docs/user-guide/services/

Release and environment

Centos 7, k8s 1.5 - clearwater docker stable

Steps to reproduce

Failed to open '/etc/clearwater/cluster_settings'

Symptoms

When I deploy the clearwater on kubernetes, the astaire pod gives the error logs:
tail: /var/log/astaire/astaire_current.txt has appeared; following end of new file
19-10-2017 08:54:46.131 UTC Status utils.cpp:686: Log level set to 2
19-10-2017 08:54:46.132 UTC Status main.cpp:251: Astaire starting up
19-10-2017 08:54:46.133 UTC Error memcached_config.cpp:108: Failed to open '/etc/clearwater/cluster_settings'
19-10-2017 08:54:46.133 UTC Error main.cpp:282: Cluster view config is invalid. Exiting
19-10-2017 08:54:47.153 UTC Status utils.cpp:686: Log level set to 2
19-10-2017 08:54:47.154 UTC Status main.cpp:251: Astaire starting up
19-10-2017 08:54:47.155 UTC Error memcached_config.cpp:108: Failed to open '/etc/clearwater/cluster_settings'
19-10-2017 08:54:47.155 UTC Error main.cpp:282: Cluster view config is invalid. Exiting
19-10-2017 08:54:47.155 UTC Status alarm.cpp:244: Reraising all alarms with a known state
19-10-2017 08:54:49.191 UTC Status utils.cpp:686: Log level set to 2
19-10-2017 08:54:49.191 UTC Status main.cpp:251: Astaire starting up
19-10-2017 08:54:49.192 UTC Error memcached_config.cpp:108: Failed to open '/etc/clearwater/cluster_settings'
19-10-2017 08:54:49.192 UTC Error main.cpp:282: Cluster view config is invalid. Exiting

So how can I manage the issus, Thanks!

Impact

Release and environment

Steps to reproduce

Unable to use IP abstraction in front of Bono

Problem:
Unable to use IP abstraction in front of Bono

Many cloud deployment tools such as kubernetes, openstack mesos, etc employ the concept of floating IPs or "IP Abstraction", possibly with NAT and then forward the incoming packets to the required destination.

This does not work with BONO and the user is forced to use a external IP of a physical machine (PUBLIC_IP) for the deployment.
This creates a hard coupling for USER -> Bono IP endpoints.

The client reports a non-routable addresses in SIP Messages received by the server.

As the endpoint behind the Gate/floating/etc is a non routable address

Initial secret is hardcoded

When clearwater is started, ellis has a default secret of "secret". There is no way to change that value (and it is clearly insecure). There should be a documented way to change it.

bono can't connect agentx master agent

19-10-2016 08:34:17.563 UTC Status alarm.cpp:62: sprout issued 1005.4 alarm
19-10-2016 08:34:20.056 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:34:35.072 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:34:47.564 UTC Status alarm.cpp:62: sprout issued 1005.4 alarm
19-10-2016 08:34:50.081 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:34:53.858 UTC Status sip_connection_pool.cpp:447: Recycle TCP connection slot 21
19-10-2016 08:34:57.860 UTC Status sip_connection_pool.cpp:447: Recycle TCP connection slot 2
19-10-2016 08:35:02.863 UTC Status sip_connection_pool.cpp:447: Recycle TCP connection slot 29
19-10-2016 08:35:05.089 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:35:17.564 UTC Status alarm.cpp:62: sprout issued 1005.4 alarm
19-10-2016 08:35:20.103 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:35:35.117 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:35:47.564 UTC Status alarm.cpp:62: sprout issued 1005.4 alarm
19-10-2016 08:35:50.130 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:36:05.139 UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):
19-10-2016 08:36:14.907 UTC Status sip_connection_pool.cpp:447: Recycle TCP connection slot 37
19-10-2016 08:36:17.564 UTC Status alarm.cpp:62: sprout issued 1005.4 alarm

build sprout docker image error

when I build sprout docker image ,there are more errors and I found it try install memcache ,but memcache also a new clearwater image ??

the following is error msgs
Setting up clearwater-tcp-scalability (1.0-160927.094809) ...
/usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: 40: /usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: sysctl: Permission denied
Setting up clearwater-log-cleanup (1.0-160927.094809) ...
dpkg: error processing package clearwater-tcp-scalability (--configure):
subprocess installed post-installation script returned error exit status 126
Setting up astaire-libs (1.0-160927.103550) ...
Setting up cpulimit (2.0-1) ...
Setting up libc-ares-dev:amd64 (1.10.0-2) ...
dpkg: dependency problems prevent configuration of astaire:
astaire depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.

Restarting memcached: memcached.
Setting up clearwater-memcached (1.0-160927.094809) ...
invoke-rc.d: policy-rc.d denied execution of start.
Setting up clearwater-socket-factory (1.0-160927.094809) ...
Setting up sprout-libs (1.0-160927.112438) ...
Processing triggers for ureadahead (0.100.0-16) ...
dpkg: dependency problems prevent configuration of sprout-base:
sprout-base depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.
sprout-base depends on astaire; however:
Package astaire is not configured yet.

dpkg: error processing package sprout-base (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sprout-scscf:
sprout-scscf depends on sprout-base; however:
Package sprout-base is not configured yet.

dpkg: error processing package sprout-scscf (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sprout-icscf:
sprout-icscf depends on sprout-base; however:
Package sprout-base is not configured yet.

dpkg: error processing package sprout-icscf (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sprout-bgcf:
sprout-bgcf depends on sprout-base; however:
Package sprout-base is not configured yet.

dpkg: error processing package sprout-bgcf (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sprout-mmtel-as:
sprout-mmtel-as depends on sprout-base; however:
Package sprout-base is not configured yet.

dpkg: error processing package sprout-mmtel-as (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of sprout:
sprout depends on sprout-base; however:
Package sprout-base is not configured yet.
sprout depends on sprout-scscf; however:
Package sprout-scscf is not configured yet.
sprout depends on sprout-icscf; however:
Package sprout-icscf is not configured yet.
sprout depends on sprout-bgcf; however:
Package sprout-bgcf is not configured yet.
sprout depends on sprout-mmtel-as; however:
Package sprout-mmtel-as is not configured yet.

dpkg: error processing package sprout (--configure):
dependency problems - leaving unconfigured

Setting up clearwater-snmpd (1.0-160927.094809) ...

Stopping network management services:
Setting up chronos (1.0-160927.094813) ...
Setting up clearwater-snmp-handler-astaire (1.0-160927.112236) ...
Stopping network management services:
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
clearwater-tcp-scalability
astaire
sprout-base
sprout-scscf
sprout-icscf
sprout-bgcf
sprout-mmtel-as
sprout
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes sprout' returned a non-zero code: 100

Error loading yaml files in python

Symptoms

yaml.constructor.ConstructorError: while constructing a mapping
in "clearwater-docker/kubernetes/bono-depl.yaml", line 33, column 18
found unacceptable key (unhashable type: 'dict')
in "clearwater-docker/kubernetes/bono-depl.yaml", line 33, column 19

Impact

This makes automation using python harder, since yaml files give error on loading.

Release and environment

On Kubernetes

Steps to reproduce

Try to load the "kubernetes/bono-depl.yaml" file in python3, using yaml (import yaml). It will give an error.

Its a small fix, the earlier <BONO_PUBLIC_IP> worked well instead of the new {{BONO_PUBLIC_IP}}, as a placeholder.
If its okay, I can submit a pull request reverting back to the old format.

Thanks

clearwater image runining problems

ellis nginx can't be started by supervisord
I find clearwater-infrastructure.conf (/etc/supervisor/conf.d/clearwater-infrastructure.conf ) has some problems, clearwater-group can't be started because supervisorctl can't connect supervisord
[program:clearwater-infrastructure]
command=bash -c '/etc/init.d/clearwater-auto-config-docker start && /etc/init.d/clearwater-infrastructure start && /etc/init.d/clearwater-etcd start && /etc/init.d/clearwater-cluster-manager start && supervisorctl start clearwater-group:_'
startsecs=0
root@clearwater-ellis:/# _supervisorctl
unix:///var/run/supervisor.sock refused connection*

this guy (Yossibh) not use supervisorctl to start service ,so all service in supervisor can start ,but can't cotroll start sequnce ,
https://github.com/Yossibh/clearwater-docker/

clearwater-cluster-manager doesn't restart Cassandra under Docker

Symptoms

Spin up a deployment under Docker using an etcd-using version (i.e. commit later than ae892d7).

Cassandra doesn't start, so the deployment is not functional.

Killing Cassandra (using pkill -f cassandra) and then restarting clearwater-infrastructure (using /etc/init.d/clearwater-infrastructure restart on both Homestead and Homer seems to resolve the problem.

Impact

The deployment is not functional.

Release and environment

Seen on release-98.

Steps to reproduce

Simply start up a deployment using an etcd-using version.

icscf.sprout and scscf.sprout host name maybe changed to icscf_sprout and scscf_sprout

icscf.sprout and scscf.sprout host name maybe changed to icscf_sprout and scscf_sprout
or use sprout name ,because kuberntes can't create service which has '.' char in service name,
but I don't now where is config file in all related clearwater components.

build cansandra docker image error

when I build canandra docker image ,some error ocurred and can't pass ,following is err log

Reading package lists...
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_main_binary-amd64_Packages.gz is not what the server reported 667375 667385
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-security_universe_binary-amd64_Packages.gz is not what the server reported 180006 180015
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package homestead-cassandra
E: Unable to locate package homestead-prov-cassandra
E: Unable to locate package homer-cassandra

The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes homestead-cassandra homestead-prov-cassandra homer-cassandra' returned a non-zero code: 100

support automated DNS with skydns

allows containers to discover each other over DNS. you can set homestead to be homestead.cluster.local and have this resolved at runtime with skydns

clearwater dns lookup can't access kuberntes skydns ,if we can disable it

learwater dns lookup can't access kuberntes skydns, if we can disable it

How to run official Stress Testing for the Clearwater-docker deployment

Symptoms

when running the stress testing tool on the clearwater-docker deployment environment which is installed on a host server, runing into the following failures in the log file.

2018-03-06 11:21:09.365930 1520335269.365930: Call-Id: [email protected], receive timeout on message Clearwater registration:1 without label to jump to (ontimeout attribute): aborting call.

All the error messages are the same in the log file.

Impact

I fail to run the stress testing tool on the clearwater-docker deployment environment

Release and environment

Run "sudo docker-compose -f minimal-distributed.yaml up -d" on one server.

Steps to reproduce

install the clearwater-docker project with docker-compose
build a new stress container node, the "Clearwater_stress_testing' docker container
enter into the Clearwater_stress_testing docker container
run '/usr/share/clearwater/bin/run_stress <home_domain> 20 10', I just replace the <home_domain> as ' '.
see the '/var/log/clearwater-sip-stress/xxx_initial_reg_errors.log'

I have create a new project which contains the stress testing tool based on this repo: clearwater-docker-stress

Undocumented ZONE variable

Undocumented $ZONE variable in /etc/init.d/clearwater-auto-config-docker

Action is to update the documentation relevant to this variable.

create trusted build at docker registry

docker pull metaswitch/homestead
etc..

When ellis fails, it reports javascript errors

Symptoms

This was triggered as a side effect of #48. The deployment had got into a broken state where ellis could contact neither homer nor homestead.

Operations on ellis returned javascript errors.

Impact

While the deployment was broken anyway, a fundamental issue was being hidden as a browser incompatibility issue. Correct behaviour would have been to return an error page to the subscriber.

Release and environment

Current master release.

Steps to reproduce

Repros if ellis cannot contact homer or homestead. Bug #48 would cause that, for example.

homer log has some error message

head -100 /var/log/supervisor/homer-stderr---supervisor-FkDgZO.log
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'. Please install it from https://pypi.python.org/pypi/service_identity and make sure all of its dependencies are satisfied. Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification. Many valid certificate/hostname mappings may be rejected.
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'. Please install it from https://pypi.python.org/pypi/service_identity and make sure all of its dependencies are satisfied. Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification. Many valid certificate/hostname mappings may be rejected.
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'. Please install it from https://pypi.python.org/pypi/service_identity and make sure all of its dependencies are satisfied. Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification. Many valid certificate/hostname mappings may be rejected.
:0: UserWarning: You do not have a working installation of the service_identity module: 'No module named service_identity'. Please install it from https://pypi.python.org/pypi/service_identity and make sure all of its dependencies are satisfied. Without the service_identity module and a recent enough pyOpenSSL to support it, Twisted can perform only rudimentary TLS client hostname verification. Many valid certificate/hostname mappings may be rejected.
Unhandled error in Deferred:

Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
Unhandled error in Deferred:

I don't know if this is a problem?

allow log to stdio

logging to stdio is a simpler way to expose logs in container deployments. we can then just grab the docker log and use tools like journalctl https://coreos.com/docs/cluster-management/debugging/reading-the-system-log/

build bono docker images error

When I build bono images, there are some errors ,mainly the tcp kenel params change cmd cause ,which in file
/usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst

docker can't allowed this tcp kenel params be set in docker container
is there any method to Comment out clearwater-tcp-scalability.postinst file content so success file build ?
the following is error message

Unpacking bono (1.0-160927.112438) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libpci3:amd64 (1:3.2.1-1ubuntu5.1) ...
Setting up libboost-system1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-filesystem1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libicu52:amd64 (52.1-3ubuntu0.4) ...
Setting up libboost-regex1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-thread1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libevent-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libevent-core-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libevent-pthreads-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libperl5.18 (5.18.2-2ubuntu1.1) ...
Setting up libsnmp-base (5.7.2dfsg-clearwater4) ...
Setting up libsnmp30:amd64 (5.7.2dfsg-clearwater4) ...
Setting up clearwater-tcp-scalability (1.0-160927.094809) ...
/usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: 40: /usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: sysctl: Permission denied
Setting up clearwater-log-cleanup (1.0-160927.094809) ...
dpkg: error processing package clearwater-tcp-scalability (--configure):
subprocess installed post-installation script returned error exit status 126
Setting up sprout-libs (1.0-160927.112438) ...
Setting up clearwater-socket-factory (1.0-160927.094809) ...
dpkg: dependency problems prevent configuration of bono:
bono depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.

dpkg: error processing package bono (--configure):
dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
clearwater-tcp-scalability
bono
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes bono' returned a non-zero code: 100

ralf log has some warning

19-10-2016 06:11:17.215 UTC Status httpstack.cpp:164: Binding HTTP TCP socket: address=172.17.1.12, port=10888
19-10-2016 06:11:18.433 UTC Status diameterresolver.cpp:62: Created Diameter resolver
19-10-2016 06:11:18.695 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for example.com: DNS server returned answer with no data
19-10-2016 06:11:18.958 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for _diameter._tcp.example.com: Domain name not found
19-10-2016 06:11:18.958 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for _diameter._sctp.example.com: Domain name not found
19-10-2016 06:11:18.958 UTC Error diameterstack.cpp:854: No Diameter peers have been found

Viop client cant register or send a call

192.168.147.130 is my docker host (public ip)
I can't find any log to resolve this prolem .
response VIOP msg SIP/2.0 483 Too Many Hops

following is my sip trace logs

2016-10-25 13:47:04.443760 [blink.exe 6024]: SENDING: Packet 38, +0:04:58.165024
192.168.147.1:52584 -(SIP over TCP)-> 192.168.147.130:5060
REGISTER sip:example.com SIP/2.0

Via: SIP/2.0/TCP 192.168.147.1:52584;rport;branch=z9hG4bKPj1a0cf45e646e4d8f9f88b95647093ee9;alias

Max-Forwards: 70

From: "wuzhihui" sip:[email protected];tag=70809066b7b9417594a10cc28324097a

To: "wuzhihui" sip:[email protected]

Contact: sip:[email protected]:52324;transport=tcp;+sip.instance="urn:uuid:ba6a3e2c-9967-4bad-bdc9-1df857632231"

Call-ID: 8742b45095f44fad9933fec5849c7814

CSeq: 1 REGISTER

Expires: 600

Supported: gruu

User-Agent: Blink 1.4.2 (Windows)

Content-Length: 0

2016-10-25 13:47:04.445205 [blink.exe 6024]: RECEIVED: Packet 39, +0:04:58.166469
192.168.147.130:5060 -(SIP over TCP)-> 192.168.147.1:52584
SIP/2.0 483 Too Many Hops

Via: SIP/2.0/TCP 192.168.147.1:52584;rport=52584;received=172.17.1.1;branch=z9hG4bKPj1a0cf45e646e4d8f9f88b95647093ee9;alias

Call-ID: 8742b45095f44fad9933fec5849c7814

From: "wuzhihui" sip:[email protected];tag=70809066b7b9417594a10cc28324097a

To: "wuzhihui" sip:[email protected];tag=z9hG4bKPjms8X0MkttGIw99ToVrtvlzGFmLtQm6ft

CSeq: 1 REGISTER

Content-Length: 0

Error in script clearwater-auto-config-docker

If environmental variable ZONE is not set, "home_domain" variable should be "example.com".
The error is in setting the variable "home_domain" at row 82 of the script "clearwater-auto-config-docker": there is a blank at both sides of "=" sign.
This ends up in having "home_domain" unset.

Cassandra node failing

My cassandra node appears to be failing, and repeatedly restarting. That in turn leads to other components failing. I'm a little puzzled by this, because I think I once installed and did not see this, but it now consistently gets into this state each time.

The cluster manager logs in /var/log/clearwater-cluster-manager end with something like the following:

15-11-2016 16:15:55.063 UTC ERROR client.py:817 (thread CassandraPlugin): Request to server http://10.0.1.14:4000 failed: MaxRetryError('HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Max retries exceeded with url: /v2/keys/clearwater/node_type_cassandra/clustering/cassandra?waitIndex=19&recursive=false&wait=true (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Read timed out. (read timeout=5)",))',)
15-11-2016 16:16:15.076 UTC ERROR client.py:817 (thread CassandraPlugin): Request to server http://10.0.1.14:4000 failed: MaxRetryError('HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Max retries exceeded with url: /v2/keys/clearwater/node_type_cassandra/clustering/cassandra?waitIndex=19&recursive=false&wait=true (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Read timed out. (read timeout=5)",))',)
15-11-2016 16:16:35.094 UTC ERROR client.py:817 (thread CassandraPlugin): Request to server http://10.0.1.14:4000 failed: MaxRetryError('HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Max retries exceeded with url: /v2/keys/clearwater/node_type_cassandra/clustering/cassandra?waitIndex=19&recursive=false&wait=true (Caused by ReadTimeoutError("HTTPConnectionPool(host=\'10.0.1.14\', port=4000): Read timed out. (read timeout=5)",))',)

However, I cannot see why it is repeatedly failing to read that, since manual reading of the value gives a result:

root@cassandra-884159819-emude:/var/log# etcdctl -C http://10.0.1.14:4000 ls --recursive
/clearwater
/clearwater/node_type_cassandra
/clearwater/node_type_cassandra/clustering
/clearwater/node_type_cassandra/clustering/cassandra
/clearwater/site1
/clearwater/site1/node_type_memcached
/clearwater/site1/node_type_memcached/clustering
/clearwater/site1/node_type_memcached/clustering/memcached
/clearwater/site1/node_type_chronos
/clearwater/site1/node_type_chronos/clustering
/clearwater/site1/node_type_chronos/clustering/chronos
root@cassandra-884159819-emude:/var/log# etcdctl -C http://10.0.1.14:4000 get /clearwater/node_type_cassandra/clustering/cassandra
{"10.0.1.14": "normal"}

Containers, Not Virtual Machines (bloated container images)

There are numerous sources identifying the benefits of containers. In a standard rack you can save upwards of $50000 on memory alone due to the reduced footprint of containers and better "binpacking"

These advantages have been reduced by the current image employed by clearwater.

The images also resists scaling in a typical "micro"service architecture. An orchestration engine such as kubernetes should be responsible for scaling, not the container. Some of the containers have numerous processes running inside via supervisord. In a lot of cases when you want to scale, you don't necessarily want to scale all of the components inside that container.

An container image should have its primary binary as the entry point and the required linked libraries only.

A few interesting sources
cncf/demo#163
https://www.brianchristner.io/docker-is-moving-to-alpine-linux/

Dpkg --list on example ellis container

ii  adduser                           3.113+nmu3ubuntu3     all                   add and remove users and groups                                        
ii  apt                               1.0.1ubuntu2.17       amd64                 commandline package manager                                            
ii  apt-utils                         1.0.1ubuntu2.17       amd64                 package management related utility programs                            
ii  base-files                        7.2ubuntu5.5          amd64                 Debian base system miscellaneous files                                 
ii  base-passwd                       3.5.33                amd64                 Debian base system master password and group files                     
ii  bash                              4.3-7ubuntu1.5        amd64                 GNU Bourne Again SHell                                                 
ii  bind9-host                        1:9.9.5.dfsg-3ubuntu0 amd64                 Version of 'host' bundled with BIND 9.X                                
ii  binutils                          2.24-5ubuntu14.1      amd64                 GNU assembler, linker and binary utilities                             
ii  bsdutils                          1:2.20.1-5.1ubuntu20. amd64                 Basic utilities from 4.4BSD-Lite                                       
ii  build-essential                   11.6ubuntu6           amd64                 Informational list of build-essential packages                         
ii  busybox-initramfs                 1:1.21.0-1ubuntu1     amd64                 Standalone shell setup for initramfs                                   
ii  bzip2                             1.0.6-5               amd64                 high-quality block-sorting file compressor - utilities                 
ii  ca-certificates                   20160104ubuntu0.14.04 all                   Common CA certificates                                                 
ii  clearwater-auto-config-docker     1.0-170110.130703     all                   Package containing the Clearwater auto-configuration tool for Docker   
ii  clearwater-cluster-manager        1.0-170110.151007     all                   Cluster manager for Clearwater                                         
ii  clearwater-config-manager         1.0-170110.151007     all                   Configuration manager for Clearwater                                   
ii  clearwater-diags-monitor          1.0-170110.130703     all                   Diagnostics monitor and bundler for all Clearwater servers             
ii  clearwater-etcd                   1.0-170110.151007     all                   etcd configured for Clearwater                                         
ii  clearwater-infrastructure         1.0-170110.130703     amd64                 Common infrastructure for all Clearwater servers                       
ii  clearwater-log-cleanup            1.0-170110.130703     all                   Script to prevent sprout/bono log files from growing too large         
ii  clearwater-management             1.0-170110.151007     all                   Meta-package for installing all Clearwater management services         
ii  clearwater-monit                  5.18-161212.145429    amd64                 utility for monitoring and managing daemons or similar programs        
ii  clearwater-nginx                  1.0-161215.173010     all                   Nginx configured for Clearwater                                        
ii  clearwater-queue-manager          1.0-170110.151007     all                   Queue manager for Clearwater                                           
ii  clearwater-socket-factory         1.0-170110.130703     amd64                 Enables other processes to establish connections using a different netw
ii  clearwater-tcp-scalability        1.0-170110.130703     all                   TCP scalability improvements for Clearwater                            
ii  console-setup                     1.70ubuntu8           all                   console font and keymap setup program                                  
ii  coreutils                         8.21-1ubuntu5.4       amd64                 GNU core utilities                                                     
ii  cpio                              2.11+dfsg-1ubuntu1.2  amd64                 GNU cpio -- a program to manage archives of files                      
ii  cpp                               4:4.8.2-1ubuntu6      amd64                 GNU C preprocessor (cpp)                                               
ii  cpp-4.8                           4.8.4-2ubuntu1~14.04. amd64                 GNU C preprocessor                                                     
ii  cron                              3.0pl1-124ubuntu2     amd64                 process scheduling daemon                                              
ii  curl                              7.35.0-1ubuntu2.10    amd64                 command line tool for transferring data with URL syntax                
ii  dash                              0.5.7-4ubuntu1        amd64                 POSIX-compliant shell                                                  
ii  debconf                           1.5.51ubuntu2         all                   Debian configuration management system                                 
ii  debconf-i18n                      1.5.51ubuntu2         all                   full internationalization support for debconf                          
ii  debianutils                       4.4                   amd64                 Miscellaneous utilities specific to Debian                             
ii  dh-python                         1.20140128-1ubuntu8.2 all                   Debian helper tools for packaging Python libraries and applications    
ii  diffutils                         1:3.3-1               amd64                 File comparison utilities                                              
ii  dmsetup                           2:1.02.77-6ubuntu2    amd64                 Linux Kernel Device Mapper userspace library                           
ii  dnsmasq                           2.68-1ubuntu0.1       all                   Small caching DNS proxy and DHCP/TFTP server                           
ii  dnsmasq-base                      2.68-1ubuntu0.1       amd64                 Small caching DNS proxy and DHCP/TFTP server                           
ii  dnsutils                          1:9.9.5.dfsg-3ubuntu0 amd64                 Clients provided with BIND                                             
ii  dpkg                              1.17.5ubuntu5.7       amd64                 Debian package management system                                       
ii  dpkg-dev                          1.17.5ubuntu5.7       all                   Debian package development tools                                       
ii  e2fslibs:amd64                    1.42.9-3ubuntu1.3     amd64                 ext2/ext3/ext4 file system libraries                                   
ii  e2fsprogs                         1.42.9-3ubuntu1.3     amd64                 ext2/ext3/ext4 file system utilities                                   
ii  eject                             2.1.5+deb1+cvs2008110 amd64                 ejects CDs and operates CD-Changers under Linux                        
ii  fakeroot                          1.20-3ubuntu2         amd64                 tool for simulating superuser privileges                               
ii  file                              1:5.14-2ubuntu3.3     amd64                 Determines file type using "magic" numbers                             
ii  findutils                         4.4.2-7               amd64                 utilities for finding files--find, xargs                               
ii  fontconfig-config                 2.11.0-0ubuntu4.2     all                   generic font configuration library - configuration                     
ii  fonts-dejavu-core                 2.34-1ubuntu1         all                   Vera font family derivate with additional characters                   
ii  g++                               4:4.8.2-1ubuntu6      amd64                 GNU C++ compiler                                                       
ii  g++-4.8                           4.8.4-2ubuntu1~14.04. amd64                 GNU C++ compiler                                                       
ii  gcc                               4:4.8.2-1ubuntu6      amd64                 GNU C compiler                                                         
ii  gcc-4.8                           4.8.4-2ubuntu1~14.04. amd64                 GNU C compiler                                                         
ii  gcc-4.8-base:amd64                4.8.4-2ubuntu1~14.04. amd64                 GCC, the GNU Compiler Collection (base package)                        
ii  gcc-4.9-base:amd64                4.9.3-0ubuntu4        amd64                 GCC, the GNU Compiler Collection (base package)                        
ii  geoip-database                    20140313-1            all                   IP lookup command line tools that use the GeoIP library (country databa
ii  gnupg                             1.4.16-1ubuntu2.4     amd64                 GNU privacy guard - a free PGP replacement                             
ii  gnutls-bin                        3.0.11+really2.12.23- amd64                 GNU TLS library - commandline utilities                                
ii  gpgv                              1.4.16-1ubuntu2.4     amd64                 GNU privacy guard - signature verification tool                        
ii  grep                              2.16-1                amd64                 GNU grep, egrep and fgrep                                              
ii  gzip                              1.6-3ubuntu1          amd64                 GNU compression utilities                                              
ii  hostname                          3.15ubuntu1           amd64                 utility to set/show the host name or domain name                       
ii  ifupdown                          0.7.47.2ubuntu4.4     amd64                 high level tools to configure network interfaces                       
ii  init-system-helpers               1.14ubuntu1           all                   helper tools for all init systems                                      
ii  initramfs-tools                   0.103ubuntu4.6        all                   tools for generating an initramfs                                      
ii  initramfs-tools-bin               0.103ubuntu4.6        amd64                 binaries used by initramfs-tools                                       
ii  initscripts                       2.88dsf-41ubuntu6.3   amd64                 scripts for initializing and shutting down the system                  
ii  inotify-tools                     3.14-1ubuntu1         amd64                 command-line programs providing a simple interface to inotify          
ii  insserv                           1.14.0-5ubuntu2       amd64                 boot sequence organizer using LSB init.d script dependency information 
ii  iproute2                          3.12.0-2ubuntu1       amd64                 networking and traffic control tools                                   
ii  iputils-ping                      3:20121221-4ubuntu1.1 amd64                 Tools to test the reachability of network hosts                        
ii  isc-dhcp-client                   4.2.4-7ubuntu12.8     amd64                 ISC DHCP client                                                        
ii  isc-dhcp-common                   4.2.4-7ubuntu12.8     amd64                 common files used by all the isc-dhcp* packages                        
ii  kbd                               1.15.5-1ubuntu1       amd64                 Linux console font and keytable utilities                              
ii  keyboard-configuration            1.70ubuntu8           all                   system-wide keyboard preferences                                       
ii  klibc-utils                       2.0.3-0ubuntu1.14.04. amd64                 small utilities built with klibc for early boot                        
ii  kmod                              15-0ubuntu6           amd64                 tools for managing Linux kernel modules                                
ii  krb5-locales                      1.12+dfsg-2ubuntu5.2  all                   Internationalization support for MIT Kerberos                          
ii  ldnsutils                         1.6.17-1              amd64                 ldns library for DNS programming                                       
ii  less                              458-2                 amd64                 pager program similar to more                                          
ii  libacl1:amd64                     2.2.52-1              amd64                 Access control list shared library                                     
ii  libalgorithm-diff-perl            1.19.02-3             all                   module to find differences between files                               
ii  libalgorithm-diff-xs-perl         0.04-2build4          amd64                 module to find differences between files (XS accelerated)              
ii  libalgorithm-merge-perl           0.08-2                all                   Perl module for three-way merge of textual data                        
ii  libapt-inst1.5:amd64              1.0.1ubuntu2.17       amd64                 deb package format runtime library                                     
ii  libapt-pkg4.12:amd64              1.0.1ubuntu2.17       amd64                 package management runtime library                                     
ii  libarchive-extract-perl           0.70-1                all                   generic archive extracting module                                      
ii  libasan0:amd64                    4.8.4-2ubuntu1~14.04. amd64                 AddressSanitizer -- a fast memory error detector                       
ii  libasn1-8-heimdal:amd64           1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - ASN.1 library                                       
ii  libatomic1:amd64                  4.8.4-2ubuntu1~14.04. amd64                 support library providing __atomic built-in functions                  
ii  libattr1:amd64                    1:2.4.47-1ubuntu1     amd64                 Extended attribute shared library                                      
ii  libaudit-common                   1:2.3.2-2ubuntu1      all                   Dynamic library for security auditing - common files                   
ii  libaudit1:amd64                   1:2.3.2-2ubuntu1      amd64                 Dynamic library for security auditing                                  
ii  libbind9-90                       1:9.9.5.dfsg-3ubuntu0 amd64                 BIND9 Shared Library used by BIND                                      
ii  libblas3                          1.2.20110419-7        amd64                 Basic Linear Algebra Reference implementations, shared library         
ii  libblkid1:amd64                   2.20.1-5.1ubuntu20.9  amd64                 block device id library                                                
ii  libboost-filesystem1.54.0:amd64   1.54.0-4ubuntu3.1     amd64                 filesystem operations (portable paths, iteration over directories, etc)
ii  libboost-regex1.54.0:amd64        1.54.0-4ubuntu3.1     amd64                 regular expression library for C++                                     
ii  libboost-system1.54.0:amd64       1.54.0-4ubuntu3.1     amd64                 Operating system (e.g. diagnostics support) library                    
ii  libboost-thread1.54.0:amd64       1.54.0-4ubuntu3.1     amd64                 portable C++ multi-threading                                           
ii  libbsd0:amd64                     0.6.0-2ubuntu1        amd64                 utility functions from BSD systems - shared library                    
ii  libbz2-1.0:amd64                  1.0.6-5               amd64                 high-quality block-sorting file compressor library - runtime           
ii  libc-bin                          2.19-0ubuntu6.9       amd64                 Embedded GNU C Library: Binaries                                       
ii  libc-dev-bin                      2.19-0ubuntu6.9       amd64                 Embedded GNU C Library: Development binaries                           
ii  libc6:amd64                       2.19-0ubuntu6.9       amd64                 Embedded GNU C Library: Shared libraries                               
ii  libc6-dev:amd64                   2.19-0ubuntu6.9       amd64                 Embedded GNU C Library: Development Libraries and Header Files         
ii  libcap2:amd64                     1:2.24-0ubuntu2       amd64                 support for getting/setting POSIX.1e capabilities                      
ii  libcap2-bin                       1:2.24-0ubuntu2       amd64                 basic utility programs for using capabilities                          
ii  libcgmanager0:amd64               0.24-0ubuntu7.5       amd64                 Central cgroup manager daemon (client library)                         
ii  libck-connector0:amd64            0.4.5-3.1ubuntu2      amd64                 ConsoleKit libraries                                                   
ii  libcloog-isl4:amd64               0.18.2-1              amd64                 Chunky Loop Generator (runtime library)                                
ii  libcomerr2:amd64                  1.42.9-3ubuntu1.3     amd64                 common error description library                                       
ii  libcurl3:amd64                    7.35.0-1ubuntu2.10    amd64                 easy-to-use client-side URL transfer library (OpenSSL flavour)         
ii  libdb5.3:amd64                    5.3.28-3ubuntu3       amd64                 Berkeley v5.3 Database Libraries [runtime]                             
ii  libdbus-1-3:amd64                 1.6.18-0ubuntu4.5     amd64                 simple interprocess messaging system (library)                         
ii  libdebconfclient0:amd64           0.187ubuntu1          amd64                 Debian Configuration Management System (C-implementation library)      
ii  libdevmapper1.02.1:amd64          2:1.02.77-6ubuntu2    amd64                 Linux Kernel Device Mapper userspace library                           
ii  libdns100                         1:9.9.5.dfsg-3ubuntu0 amd64                 DNS Shared Library used by BIND                                        
ii  libdpkg-perl                      1.17.5ubuntu5.7       all                   Dpkg perl modules                                                      
ii  libdrm2:amd64                     2.4.67-1ubuntu0.14.04 amd64                 Userspace interface to kernel DRM services -- runtime                  
ii  libedit2:amd64                    3.1-20130712-2        amd64                 BSD editline and history libraries                                     
ii  libestr0                          0.1.9-0ubuntu2        amd64                 Helper functions for handling strings (lib)                            
ii  libevent-2.0-5:amd64              2.0.21-stable-1ubuntu amd64                 Asynchronous event notification library                                
ii  libevent-core-2.0-5:amd64         2.0.21-stable-1ubuntu amd64                 Asynchronous event notification library (core)                         
ii  libevent-pthreads-2.0-5:amd64     2.0.21-stable-1ubuntu amd64                 Asynchronous event notification library (pthreads)                     
ii  libexpat1:amd64                   2.1.0-4ubuntu1.3      amd64                 XML parsing C library - runtime library                                
ii  libexpat1-dev:amd64               2.1.0-4ubuntu1.3      amd64                 XML parsing C library - development kit                                
ii  libfakeroot:amd64                 1.20-3ubuntu2         amd64                 tool for simulating superuser privileges - shared libraries            
ii  libffi-dev:amd64                  3.1~rc1+r3.0.13-12ubu amd64                 Foreign Function Interface library (development files)                 
ii  libffi6:amd64                     3.1~rc1+r3.0.13-12ubu amd64                 Foreign Function Interface library runtime                             
ii  libfile-fcntllock-perl            0.14-2build1          amd64                 Perl module for file locking with fcntl(2)                             
ii  libfontconfig1:amd64              2.11.0-0ubuntu4.2     amd64                 generic font configuration library - runtime                           
ii  libfreetype6:amd64                2.5.2-1ubuntu2.5      amd64                 FreeType 2 font engine, shared library files                           
ii  libfribidi0:amd64                 0.19.6-1              amd64                 Free Implementation of the Unicode BiDi algorithm                      
ii  libgcc-4.8-dev:amd64              4.8.4-2ubuntu1~14.04. amd64                 GCC support library (development files)                                
ii  libgcc1:amd64                     1:4.9.3-0ubuntu4      amd64                 GCC support library                                                    
ii  libgcrypt11:amd64                 1.5.3-2ubuntu4.4      amd64                 LGPL Crypto library - runtime library                                  
ii  libgd3:amd64                      2.1.0-3ubuntu0.5      amd64                 GD Graphics Library                                                    
ii  libgdbm3:amd64                    1.8.3-12build1        amd64                 GNU dbm database routines (runtime version)                            
ii  libgeoip1:amd64                   1.6.0-1               amd64                 non-DNS IP-to-country resolver library                                 
ii  libgmp10:amd64                    2:5.1.3+dfsg-1ubuntu1 amd64                 Multiprecision arithmetic library                                      
ii  libgnutls-openssl27:amd64         2.12.23-12ubuntu2.5   amd64                 GNU TLS library - OpenSSL wrapper                                      
ii  libgnutls26:amd64                 2.12.23-12ubuntu2.5   amd64                 GNU TLS library - runtime library                                      
ii  libgomp1:amd64                    4.8.4-2ubuntu1~14.04. amd64                 GCC OpenMP (GOMP) support library                                      
ii  libgpg-error0:amd64               1.12-0.2ubuntu1       amd64                 library for common error values and messages in GnuPG components       
ii  libgssapi-krb5-2:amd64            1.12+dfsg-2ubuntu5.2  amd64                 MIT Kerberos runtime libraries - krb5 GSS-API Mechanism                
ii  libgssapi3-heimdal:amd64          1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - GSSAPI support library                              
ii  libhcrypto4-heimdal:amd64         1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - crypto library                                      
ii  libheimbase1-heimdal:amd64        1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - Base library                                        
ii  libheimntlm0-heimdal:amd64        1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - NTLM support library                                
ii  libhx509-5-heimdal:amd64          1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - X509 support library                                
ii  libicu52:amd64                    52.1-3ubuntu0.4       amd64                 International Components for Unicode                                   
ii  libidn11:amd64                    1.28-1ubuntu2.1       amd64                 GNU Libidn library, implementation of IETF IDN specifications          
ii  libinotifytools0                  3.14-1ubuntu1         amd64                 utility wrapper around inotify                                         
ii  libisc95                          1:9.9.5.dfsg-3ubuntu0 amd64                 ISC Shared Library used by BIND                                        
ii  libisccc90                        1:9.9.5.dfsg-3ubuntu0 amd64                 Command Channel Library used by BIND                                   
ii  libisccfg90                       1:9.9.5.dfsg-3ubuntu0 amd64                 Config File Handling Library used by BIND                              
ii  libisl10:amd64                    0.12.2-1              amd64                 manipulating sets and relations of integer points bounded by linear con
ii  libitm1:amd64                     4.8.4-2ubuntu1~14.04. amd64                 GNU Transactional Memory Library                                       
ii  libjbig0:amd64                    2.0-2ubuntu4.1        amd64                 JBIGkit libraries                                                      
ii  libjpeg-turbo8:amd64              1.3.0-0ubuntu2        amd64                 IJG JPEG compliant runtime library.                                    
ii  libjpeg8:amd64                    8c-2ubuntu8           amd64                 Independent JPEG Group's JPEG runtime library (dependency package)     
ii  libjson-c2:amd64                  0.11-3ubuntu1.2       amd64                 JSON manipulation library - shared library                             
ii  libjson0:amd64                    0.11-3ubuntu1.2       amd64                 JSON manipulation library (transitional package)                       
ii  libk5crypto3:amd64                1.12+dfsg-2ubuntu5.2  amd64                 MIT Kerberos runtime libraries - Crypto Library                        
ii  libkeyutils1:amd64                1.5.6-1               amd64                 Linux Key Management Utilities (library)                               
ii  libklibc                          2.0.3-0ubuntu1.14.04. amd64                 minimal libc subset for use with initramfs                             
ii  libkmod2:amd64                    15-0ubuntu6           amd64                 libkmod shared library                                                 
ii  libkrb5-26-heimdal:amd64          1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - libraries                                           
ii  libkrb5-3:amd64                   1.12+dfsg-2ubuntu5.2  amd64                 MIT Kerberos runtime libraries                                         
ii  libkrb5support0:amd64             1.12+dfsg-2ubuntu5.2  amd64                 MIT Kerberos runtime libraries - Support library                       
ii  libldap-2.4-2:amd64               2.4.31-1+nmu2ubuntu8. amd64                 OpenLDAP libraries                                                     
ii  libldns1                          1.6.17-1              amd64                 ldns library for DNS programming                                       
ii  liblinear-tools                   1.8+dfsg-1ubuntu1     amd64                 Standalone applications for LIBLINEAR                                  
ii  liblinear1                        1.8+dfsg-1ubuntu1     amd64                 Library for Large Linear Classification                                
ii  liblocale-gettext-perl            1.05-7build3          amd64                 module using libc functions for internationalization in Perl           
ii  liblockfile-bin                   1.09-6ubuntu1         amd64                 support binaries for and cli utilities based on liblockfile            
ii  liblockfile1:amd64                1.09-6ubuntu1         amd64                 NFS-safe locking library                                               
ii  liblog-message-simple-perl        0.10-1                all                   simplified interface to Log::Message                                   
ii  liblua5.2-0:amd64                 5.2.3-1               amd64                 Shared library for the Lua interpreter version 5.2                     
ii  liblwres90                        1:9.9.5.dfsg-3ubuntu0 amd64                 Lightweight Resolver Library used by BIND                              
ii  liblzma5:amd64                    5.1.1alpha+20120614-2 amd64                 XZ-format compression library                                          
ii  libmagic1:amd64                   1:5.14-2ubuntu3.3     amd64                 File type determination library using "magic" numbers                  
ii  libmnl0:amd64                     1.0.3-3ubuntu1        amd64                 minimalistic Netlink communication library                             
ii  libmodule-pluggable-perl          5.1-1                 all                   module for giving  modules the ability to have plugins                 
ii  libmount1:amd64                   2.20.1-5.1ubuntu20.9  amd64                 block device id library                                                
ii  libmpc3:amd64                     1.0.1-1ubuntu1        amd64                 multiple precision complex floating-point library                      
ii  libmpdec2:amd64                   2.4.0-6               amd64                 library for decimal floating point arithmetic (runtime library)        
ii  libmpfr4:amd64                    3.1.2-1               amd64                 multiple precision floating-point computation                          
ii  libncurses5:amd64                 5.9+20140118-1ubuntu1 amd64                 shared libraries for terminal handling                                 
ii  libncursesw5:amd64                5.9+20140118-1ubuntu1 amd64                 shared libraries for terminal handling (wide character support)        
ii  libnet1:amd64                     1.1.6+dfsg-2ubuntu1   amd64                 library for the construction and handling of network packets           
ii  libnetfilter-conntrack3:amd64     1.0.4-1               amd64                 Netfilter netlink-conntrack library                                    
ii  libnewt0.52:amd64                 0.52.15-2ubuntu5      amd64                 Not Erik's Windowing Toolkit - text mode windowing with slang          
ii  libnfnetlink0:amd64               1.0.1-2               amd64                 Netfilter netlink library                                              
ii  libnih-dbus1:amd64                1.0.3-4ubuntu25       amd64                 NIH D-Bus Bindings Library                                             
ii  libnih1:amd64                     1.0.3-4ubuntu25       amd64                 NIH Utility Library                                                    
ii  libopts25:amd64                   1:5.18-2ubuntu2       amd64                 automated option processing library based on autogen                   
ii  libp11-kit0:amd64                 0.20.2-2ubuntu2       amd64                 Library for loading and coordinating access to PKCS#11 modules - runtim
ii  libpam-cap:amd64                  1:2.24-0ubuntu2       amd64                 PAM module for implementing capabilities                               
ii  libpam-modules:amd64              1.1.8-1ubuntu2.2      amd64                 Pluggable Authentication Modules for PAM                               
ii  libpam-modules-bin                1.1.8-1ubuntu2.2      amd64                 Pluggable Authentication Modules for PAM - helper binaries             
ii  libpam-runtime                    1.1.8-1ubuntu2.2      all                   Runtime support for the PAM library                                    
ii  libpam0g:amd64                    1.1.8-1ubuntu2.2      amd64                 Pluggable Authentication Modules library                               
ii  libpcap0.8:amd64                  1.5.3-2               amd64                 system interface for user-level packet capture                         
ii  libpci3:amd64                     1:3.2.1-1ubuntu5.1    amd64                 Linux PCI Utilities (shared library)                                   
ii  libpcre3:amd64                    1:8.31-2ubuntu2.3     amd64                 Perl 5 Compatible Regular Expression Library - runtime files           
ii  libperl5.18                       5.18.2-2ubuntu1.1     amd64                 shared Perl library                                                    
ii  libpgm-5.1-0:amd64                5.1.118-1~dfsg-0.1ubu amd64                 OpenPGM shared library                                                 
ii  libplymouth2:amd64                0.8.8-0ubuntu17.1     amd64                 graphical boot animation and logger - shared libraries                 
ii  libpng12-0:amd64                  1.2.50-1ubuntu2.14.04 amd64                 PNG library - runtime                                                  
ii  libpod-latex-perl                 0.61-1                all                   module to convert Pod data to formatted LaTeX                          
ii  libpopt0:amd64                    1.16-8ubuntu1         amd64                 lib for parsing cmdline parameters                                     
ii  libprocps3:amd64                  1:3.3.9-1ubuntu2.2    amd64                 library for accessing process information from /proc                   
ii  libpython-dev:amd64               2.7.5-5ubuntu3        amd64                 header files and a static library for Python (default)                 
ii  libpython-stdlib:amd64            2.7.5-5ubuntu3        amd64                 interactive high-level object-oriented language (default python version
ii  libpython2.7:amd64                2.7.6-8ubuntu0.3      amd64                 Shared Python runtime library (version 2.7)                            
ii  libpython2.7-dev:amd64            2.7.6-8ubuntu0.3      amd64                 Header files and a static library for Python (v2.7)                    
ii  libpython2.7-minimal:amd64        2.7.6-8ubuntu0.3      amd64                 Minimal subset of the Python language (version 2.7)                    
ii  libpython2.7-stdlib:amd64         2.7.6-8ubuntu0.3      amd64                 Interactive high-level object-oriented language (standard library, vers
ii  libpython3-stdlib:amd64           3.4.0-0ubuntu2        amd64                 interactive high-level object-oriented language (default python3 versio
ii  libpython3.4-minimal:amd64        3.4.3-1ubuntu1~14.04. amd64                 Minimal subset of the Python language (version 3.4)                    
ii  libpython3.4-stdlib:amd64         3.4.3-1ubuntu1~14.04. amd64                 Interactive high-level object-oriented language (standard library, vers
ii  libquadmath0:amd64                4.8.4-2ubuntu1~14.04. amd64                 GCC Quad-Precision Math Library                                        
ii  libreadline6:amd64                6.3-4ubuntu2          amd64                 GNU readline and history libraries, run-time libraries                 
ii  libroken18-heimdal:amd64          1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - roken support library                               
ii  librtmp0:amd64                    2.4+20121230.gitdf6c5 amd64                 toolkit for RTMP streams (shared library)                              
ii  libsasl2-2:amd64                  2.1.25.dfsg1-17build1 amd64                 Cyrus SASL - authentication abstraction library                        
ii  libsasl2-modules:amd64            2.1.25.dfsg1-17build1 amd64                 Cyrus SASL - pluggable authentication modules                          
ii  libsasl2-modules-db:amd64         2.1.25.dfsg1-17build1 amd64                 Cyrus SASL - pluggable authentication modules (DB)                     
ii  libselinux1:amd64                 2.2.2-1ubuntu0.1      amd64                 SELinux runtime shared libraries                                       
ii  libsemanage-common                2.2-1                 all                   Common files for SELinux policy management libraries                   
ii  libsemanage1:amd64                2.2-1                 amd64                 SELinux policy management library                                      
ii  libsensors4:amd64                 1:3.3.4-2ubuntu1      amd64                 library to read temperature/voltage/fan sensors                        
ii  libsepol1:amd64                   2.2-1ubuntu0.1        amd64                 SELinux library for manipulating binary security policies              
ii  libslang2:amd64                   2.2.4-15ubuntu1       amd64                 S-Lang programming library - runtime version                           
ii  libsnmp-base                      5.7.2~dfsg-clearwater all                   SNMP configuration script, MIBs and documentation                      
ii  libsnmp30:amd64                   5.7.2~dfsg-clearwater amd64                 SNMP (Simple Network Management Protocol) library                      
ii  libsqlite3-0:amd64                3.8.2-1ubuntu2.1      amd64                 SQLite 3 shared library                                                
ii  libss2:amd64                      1.42.9-3ubuntu1.3     amd64                 command-line interface parsing library                                 
ii  libssl-dev:amd64                  1.0.1f-1ubuntu2.21    amd64                 Secure Sockets Layer toolkit - development files                       
ii  libssl-doc                        1.0.1f-1ubuntu2.21    all                   Secure Sockets Layer toolkit - development documentation               
ii  libssl1.0.0:amd64                 1.0.1f-1ubuntu2.21    amd64                 Secure Sockets Layer toolkit - shared libraries                        
ii  libstdc++-4.8-dev:amd64           4.8.4-2ubuntu1~14.04. amd64                 GNU Standard C++ Library v3 (development files)                        
ii  libstdc++6:amd64                  4.8.4-2ubuntu1~14.04. amd64                 GNU Standard C++ Library v3                                            
ii  libtasn1-6:amd64                  3.4-3ubuntu0.4        amd64                 Manage ASN.1 structures (runtime)                                      
ii  libterm-ui-perl                   0.42-1                all                   Term::ReadLine UI made easy                                            
ii  libtext-charwidth-perl            0.04-7build3          amd64                 get display widths of characters on the terminal                       
ii  libtext-iconv-perl                1.7-5build2           amd64                 converts between character sets in Perl                                
ii  libtext-soundex-perl              3.4-1build1           amd64                 implementation of the soundex algorithm                                
ii  libtext-wrapi18n-perl             0.06-7                all                   internationalized substitute of Text::Wrap                             
ii  libtiff5:amd64                    4.0.3-7ubuntu0.4      amd64                 Tag Image File Format (TIFF) library                                   
ii  libtimedate-perl                  2.3000-1              all                   collection of modules to manipulate date/time information              
ii  libtinfo5:amd64                   5.9+20140118-1ubuntu1 amd64                 shared low-level terminfo library for terminal handling                
ii  libtsan0:amd64                    4.8.4-2ubuntu1~14.04. amd64                 ThreadSanitizer -- a Valgrind-based detector of data races (runtime)   
ii  libudev1:amd64                    204-5ubuntu20.20      amd64                 libudev shared library                                                 
ii  libusb-0.1-4:amd64                2:0.1.12-23.3ubuntu1  amd64                 userspace USB programming library                                      
ii  libustr-1.0-1:amd64               1.0.4-3ubuntu2        amd64                 Micro string library: shared library                                   
ii  libuuid1:amd64                    2.20.1-5.1ubuntu20.9  amd64                 Universally Unique ID library                                          
ii  libvpx1:amd64                     1.3.0-2               amd64                 VP8 video codec (shared library)                                       
ii  libwind0-heimdal:amd64            1.6~git20131207+dfsg- amd64                 Heimdal Kerberos - stringprep implementation                           
ii  libwrap0:amd64                    7.6.q-25              amd64                 Wietse Venema's TCP wrappers library                                   
ii  libx11-6:amd64                    2:1.6.2-1ubuntu2      amd64                 X11 client-side library                                                
ii  libx11-data                       2:1.6.2-1ubuntu2      all                   X11 client-side library                                                
ii  libxau6:amd64                     1:1.0.8-1             amd64                 X11 authorisation library                                              
ii  libxcb1:amd64                     1.10-2ubuntu1         amd64                 X C Binding                                                            
ii  libxdmcp6:amd64                   1:1.1.1-1             amd64                 X11 Display Manager Control Protocol library                           
ii  libxext6:amd64                    2:1.3.2-1ubuntu0.0.14 amd64                 X11 miscellaneous extension library                                    
ii  libxml2:amd64                     2.9.1+dfsg1-3ubuntu4. amd64                 GNOME XML library                                                      
ii  libxmuu1:amd64                    2:1.1.1-1             amd64                 X11 miscellaneous micro-utility library                                
ii  libxpm4:amd64                     1:3.5.10-1            amd64                 X11 pixmap library                                                     
ii  libxslt1.1:amd64                  1.1.28-2build1        amd64                 XSLT 1.0 processing library - runtime library                          
ii  libzmq3:amd64                     4.0.4+dfsg-2          amd64                 lightweight messaging kernel (shared library)                          
ii  libzmq3-dev:amd64                 4.0.4+dfsg-2          amd64                 lightweight messaging kernel (development files)                       
ii  linux-libc-dev:amd64              3.13.0-107.154        amd64                 Linux Kernel Headers for development                                   
ii  locales                           2.13+git20120306-12.1 all                   common files for locale support                                        
ii  lockfile-progs                    0.1.17                amd64                 Programs for locking and unlocking files and mailboxes                 
ii  login                             1:4.1.5.1-1ubuntu9.2  amd64                 system login tools                                                     
ii  logrotate                         3.8.7-1ubuntu1        amd64                 Log rotation utility                                                   
ii  lsb-base                          4.1+Debian11ubuntu6.2 all                   Linux Standard Base 4.1 init script functionality                      
ii  lsb-release                       4.1+Debian11ubuntu6.2 all                   Linux Standard Base version reporting utility                          
ii  make                              3.81-8.2ubuntu3       amd64                 An utility for Directing compilation.                                  
ii  makedev                           2.3.1-93ubuntu1       all                   creates device files in /dev                                           
ii  manpages                          3.54-1ubuntu1         all                   Manual pages about using a GNU/Linux system                            
ii  manpages-dev                      3.54-1ubuntu1         all                   Manual pages about using GNU/Linux for development                     
ii  mawk                              1.3.3-17ubuntu2       amd64                 a pattern scanning and text processing language                        
ii  mime-support                      3.54ubuntu1.1         all                   MIME files 'mime.types' & 'mailcap', and support programs              
ii  module-init-tools                 15-0ubuntu6           all                   transitional dummy package (module-init-tools to kmod)                 
ii  mount                             2.20.1-5.1ubuntu20.9  amd64                 Tools for mounting and manipulating filesystems                        
ii  mountall                          2.53                  amd64                 filesystem mounting tool                                               
ii  multiarch-support                 2.19-0ubuntu6.9       amd64                 Transitional package to ensure multiarch compatibility                 
ii  ncurses-base                      5.9+20140118-1ubuntu1 all                   basic terminal type definitions                                        
ii  ncurses-bin                       5.9+20140118-1ubuntu1 amd64                 terminal-related programs and man pages                                
ii  ncurses-term                      5.9+20140118-1ubuntu1 all                   additional terminal type definitions                                   
ii  net-tools                         1.60-25ubuntu2.1      amd64                 The NET-3 networking toolkit                                           
ii  netbase                           5.2                   all                   Basic TCP/IP networking system                                         
ii  netcat-openbsd                    1.105-7ubuntu1        amd64                 TCP/IP swiss army knife                                                
ii  nginx                             1.4.6-1ubuntu3.7      all                   small, powerful, scalable web/proxy server                             
ii  nginx-common                      1.4.6-1ubuntu3.7      all                   small, powerful, scalable web/proxy server - common files              
ii  nginx-core                        1.4.6-1ubuntu3.7      amd64                 nginx web/proxy server (core version)                                  
ii  ngrep                             1.45.ds2-12           amd64                 grep for network traffic                                               
ii  nmap                              6.40-0.2ubuntu1       amd64                 The Network Mapper                                                     
ii  ntp                               1:4.2.6.p5+dfsg-3ubun amd64                 Network Time Protocol daemon and utility programs                      
ii  ntpdate                           1:4.2.6.p5+dfsg-3ubun amd64                 client for setting system time from NTP servers                        
ii  openssh-client                    1:6.6p1-2ubuntu2.8    amd64                 secure shell (SSH) client, for secure access to remote machines        
ii  openssh-server                    1:6.6p1-2ubuntu2.8    amd64                 secure shell (SSH) server, for secure access from remote machines      
ii  openssh-sftp-server               1:6.6p1-2ubuntu2.8    amd64                 secure shell (SSH) sftp server module, for SFTP access from remote mach
ii  openssl                           1.0.1f-1ubuntu2.21    amd64                 Secure Sockets Layer toolkit - cryptographic utility                   
ii  passwd                            1:4.1.5.1-1ubuntu9.2  amd64                 change and administer password and group data                          
ii  patch                             2.7.1-4ubuntu2.3      amd64                 Apply a diff file to an original                                       
ii  perl                              5.18.2-2ubuntu1.1     amd64                 Larry Wall's Practical Extraction and Report Language                  
ii  perl-base                         5.18.2-2ubuntu1.1     amd64                 minimal Perl system                                                    
ii  perl-modules                      5.18.2-2ubuntu1.1     all                   Core Perl modules                                                      
ii  plymouth                          0.8.8-0ubuntu17.1     amd64                 graphical boot animation and logger - main package                     
ii  procps                            1:3.3.9-1ubuntu2.2    amd64                 /proc file system utilities                                            
ii  python                            2.7.5-5ubuntu3        amd64                 interactive high-level object-oriented language (default version)      
ii  python-chardet                    2.0.1-2build2         all                   universal character encoding detector                                  
ii  python-chardet-whl                2.2.1-2~ubuntu1       all                   universal character encoding detector                                  
ii  python-colorama                   0.2.5-0.1ubuntu2      all                   Cross-platform colored terminal text in Python - Python 2.x            
ii  python-colorama-whl               0.2.5-0.1ubuntu2      all                   Cross-platform colored terminal text in Python - Wheels                
ii  python-dev                        2.7.5-5ubuntu3        amd64                 header files and a static library for Python (default)                 
ii  python-distlib                    0.1.8-1ubuntu1        all                   low-level components of python distutils2/packaging                    
ii  python-distlib-whl                0.1.8-1ubuntu1        all                   low-level components of python distutils2/packaging                    
ii  python-html5lib                   0.999-3~ubuntu1       all                   HTML parser/tokenizer based on the WHATWG HTML5 specification (Python 2
ii  python-html5lib-whl               0.999-3~ubuntu1       all                   HTML parser/tokenizer based on the WHATWG HTML5 specification          
ii  python-meld3                      0.6.10-1              amd64                 HTML/XML templating system for Python                                  
ii  python-minimal                    2.7.5-5ubuntu3        amd64                 minimal subset of the Python language (default version)                
ii  python-pip                        1.5.4-1ubuntu4        all                   alternative Python package installer                                   
ii  python-pip-whl                    1.5.4-1ubuntu4        all                   alternative Python package installer                                   
ii  python-pkg-resources              3.3-1ubuntu2          all                   Package Discovery and Resource Access using pkg_resources              
ii  python-requests                   2.2.1-1ubuntu0.3      all                   elegant and simple HTTP library for Python, built for human beings     
ii  python-requests-whl               2.2.1-1ubuntu0.3      all                   elegant and simple HTTP library for Python, built for human beings     
ii  python-setuptools                 3.3-1ubuntu2          all                   Python Distutils Enhancements                                          
ii  python-setuptools-whl             3.3-1ubuntu2          all                   Python Distutils Enhancements (wheel package)                          
ii  python-six                        1.5.2-1ubuntu1        all                   Python 2 and 3 compatibility library (Python 2 interface)              
ii  python-six-whl                    1.5.2-1ubuntu1        all                   Python 2 and 3 compatibility library (universal wheel)                 
ii  python-urllib3                    1.7.1-1ubuntu4        all                   HTTP library with thread-safe connection pooling for Python            
ii  python-urllib3-whl                1.7.1-1ubuntu4        all                   HTTP library with thread-safe connection pooling                       
ii  python-virtualenv                 1.11.4-1ubuntu1       all                   Python virtual environment creator                                     
ii  python-wheel                      0.24.0-1~ubuntu1      all                   built-package format for Python                                        
ii  python2.7                         2.7.6-8ubuntu0.3      amd64                 Interactive high-level object-oriented language (version 2.7)          
ii  python2.7-dev                     2.7.6-8ubuntu0.3      amd64                 Header files and a static library for Python (v2.7)                    
ii  python2.7-minimal                 2.7.6-8ubuntu0.3      amd64                 Minimal subset of the Python language (version 2.7)                    
ii  python3                           3.4.0-0ubuntu2        amd64                 interactive high-level object-oriented language (default python3 versio
ii  python3-minimal                   3.4.0-0ubuntu2        amd64                 minimal subset of the Python language (default python3 version)        
ii  python3-pkg-resources             3.3-1ubuntu2          all                   Package Discovery and Resource Access using pkg_resources              
ii  python3.4                         3.4.3-1ubuntu1~14.04. amd64                 Interactive high-level object-oriented language (version 3.4)          
ii  python3.4-minimal                 3.4.3-1ubuntu1~14.04. amd64                 Minimal subset of the Python language (version 3.4)                    
ii  readline-common                   6.3-4ubuntu2          all                   GNU readline and history libraries, common files                       
ii  realpath                          1.19                  amd64                 Return the canonicalized absolute pathname
ii  resolvconf                        1.69ubuntu1.1         all                   name server information handler
ii  rsyslog                           7.4.4-1ubuntu2.6      amd64                 reliable system and kernel logging daemon
ii  sed                               4.2.2-4ubuntu1        amd64                 The GNU sed stream editor
ii  sensible-utils                    0.0.9                 all                   Utilities for sensible alternative selection
ii  sgml-base                         1.26+nmu4ubuntu1      all                   SGML infrastructure and SGML catalog file support
ii  sprout                            1.0-170110.131406     amd64                 sprout, the SIP Router
ii  sprout-base                       1.0-170110.131406     amd64                 sprout-base, the SIP Router basic executable
ii  sprout-bgcf                       1.0-170110.131406     amd64                 sprout-bgcf, the SIP Router BGCF plug-in
ii  sprout-icscf                      1.0-170110.131406     amd64                 sprout-icscf, the SIP Router I-CSCF plug-in
ii  sprout-libs                       1.0-170110.131406     amd64                 Libraries for sprout, bono and restund
ii  sprout-mmtel-as                   1.0-170110.131406     amd64                 sprout-mmtel-as, the SIP Router MMTEL Application Server plug-in
ii  sprout-scscf                      1.0-170110.131406     amd64                 sprout-scscf, the SIP Router S-CSCF plug-in
ii  ssh-import-id                     3.21-0ubuntu1         all                   securely retrieve an SSH public key and install it locally
ii  sudo                              1.8.9p5-1ubuntu1.3    amd64                 Provide limited super user privileges to specific users
ii  supervisor                        3.0b2-1               all                   A system for controlling process state
ii  sysstat                           10.2.0-1              amd64                 system performance tools for Linux
ii  sysv-rc                           2.88dsf-41ubuntu6.3   all                   System-V-like runlevel change mechanism
ii  sysvinit-utils                    2.88dsf-41ubuntu6.3   amd64                 System-V-like utilities
ii  tar                               1.27.1-1ubuntu0.1     amd64                 GNU version of the tar archiving utility
ii  tcpd                              7.6.q-25              amd64                 Wietse Venema's TCP wrapper utilities
ii  tcpdump                           4.5.1-2ubuntu1.2      amd64                 command-line network traffic analyzer
ii  tzdata                            2016j-0ubuntu0.14.04  all                   time zone and daylight-saving time data
ii  ubuntu-keyring                    2012.05.19            all                   GnuPG keys of the Ubuntu archive
ii  ubuntu-minimal                    1.325                 amd64                 Minimal core of Ubuntu
ii  ucf                               3.0027+nmu1           all                   Update Configuration File(s): preserve user changes to config files
ii  udev                              204-5ubuntu20.20      amd64                 /dev/ and hotplug management daemon
ii  upstart                           1.12.1-0ubuntu4.2     amd64                 event-based init daemon
ii  ureadahead                        0.100.0-16            amd64                 Read required files in advance
ii  util-linux                        2.20.1-5.1ubuntu20.9  amd64                 Miscellaneous system utilities
ii  vim-common                        2:7.4.052-1ubuntu3.1  amd64                 Vi IMproved - Common files
ii  vim-tiny                          2:7.4.052-1ubuntu3.1  amd64                 Vi IMproved - enhanced vi editor - compact version
ii  wget                              1.15-1ubuntu1.14.04.2 amd64                 retrieves files from the web
ii  whiptail                          0.52.15-2ubuntu5      amd64                 Displays user-friendly dialog boxes from shell scripts
ii  xauth                             1:1.0.7-1ubuntu1      amd64                 X authentication utility
ii  xkb-data                          2.10.1-1ubuntu1       all                   X Keyboard Extension (XKB) configuration data
ii  xml-core                          0.13+nmu2             all                   XML infrastructure and XML catalog file support
ii  xz-utils                          5.1.1alpha+20120614-2 amd64                 XZ-format compression utilities
ii  zlib1g:amd64                      1:1.2.8.dfsg-1ubuntu1 amd64                 compression library - runtime
ii  zlib1g-dev:amd64                  1:1.2.8.dfsg-1ubuntu1 amd64                 compression library - development

expose all log dirs as volumes, allow single current hotfile to tail with fixed name {service}_current.log

Ellis stops replying to requests

I spin up a Clearwater Docker deployment and run our live test suite against (see the live-test-docker CI job). The live tests fail reliably. Specifically, about midway through the run, an Ellis request times out. The remaining tests continue to run as normal once the request times out.

I've no idea what's going on here really. There's nothing obvious in the nginx logs or ellis logs. There had been no code changes in any of our repos when this first started happening. I thought we might be using a newer version of another package that ellis depends on, and that was bugged, but I spun up a normal AIO node (no Docker) and didn't reproduce this problem.

Setting to medium priority because it damages our docker story.

Error Building Ellis - Starting MySQL database server mysqld ...fail!

@plwhite @MatMeredith

Symptoms

The ellis container image fails to build with mysql error "Starting MySQL database server mysqld ...fail!"

Impact

Unable to build and deploy clearwater-docker

Release and environment

$ git clone https://github.com/Metaswitch/clearwater-docker.git
$ git branch = master

Steps to reproduce

git clone https://github.com/Metaswitch/clearwater-docker.git
cd clearwater-docker/
run this command: for i in base astaire cassandra chronos bono ellis homer homestead homestead-prov ralf sprout ; do docker build -t clearwater/$i $i ; done
build fails for ellis container with mysql error (Starting MySQL database server mysqld ...fail!) logs below:

$ docker build -t clearwater/ellis ellis/
Sending build context to Docker daemon 8.704 kB
Step 1 : FROM clearwater/base
 ---> 4b2d82dcaf16
Step 2 : MAINTAINER [email protected]
 ---> Running in 74cfd2e788fa
 ---> c348a07b5da8
Removing intermediate container 74cfd2e788fa
Step 3 : RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes mysql-server
 ---> Running in af69d37044a1
Hit http://security.ubuntu.com trusty-security InRelease
Ign http://archive.ubuntu.com trusty InRelease
Hit http://archive.ubuntu.com trusty-updates InRelease
Get:1 http://archive.ubuntu.com trusty-backports InRelease [65.9 kB]
Hit http://archive.ubuntu.com trusty Release.gpg
Hit http://security.ubuntu.com trusty-security/universe Sources
Get:2 http://archive.ubuntu.com trusty Release [58.5 kB]
Ign http://repo.cw-ngv.com binary/ InRelease
Hit http://archive.ubuntu.com trusty-updates/universe Sources
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/restricted amd64 Packages
Hit http://archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://repo.cw-ngv.com binary/ Release.gpg
Hit http://archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/multiverse amd64 Packages
Hit http://archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://repo.cw-ngv.com binary/ Release
Hit http://archive.ubuntu.com trusty-backports/restricted amd64 Packages
Get:3 http://archive.ubuntu.com trusty-backports/universe amd64 Packages [52.6 kB]
Hit http://archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://repo.cw-ngv.com binary/ Packages
Get:4 http://archive.ubuntu.com trusty/universe Sources [7,926 kB]
Get:5 http://archive.ubuntu.com trusty/main amd64 Packages [1,743 kB]
Get:6 http://archive.ubuntu.com trusty/restricted amd64 Packages [16.0 kB]
Get:7 http://archive.ubuntu.com trusty/universe amd64 Packages [7,589 kB]
Get:8 http://archive.ubuntu.com trusty/multiverse amd64 Packages [169 kB]
Fetched 17.6 MB in 2s (6,591 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  libaio1 libdbd-mysql-perl libdbi-perl libhtml-template-perl
  libterm-readkey-perl mysql-client-5.5 mysql-client-core-5.5 mysql-server-5.5
  mysql-server-core-5.5 psmisc
Suggested packages:
  libclone-perl libmldbm-perl libnet-daemon-perl libplrpc-perl
  libsql-statement-perl libipc-sharedcache-perl tinyca mailx
The following NEW packages will be installed:
  libaio1 libdbd-mysql-perl libdbi-perl libhtml-template-perl
  libterm-readkey-perl mysql-client-5.5 mysql-client-core-5.5 mysql-server
  mysql-server-5.5 mysql-server-core-5.5 psmisc
0 upgraded, 11 newly installed, 0 to remove and 1 not upgraded.
Need to get 9,057 kB of archives.
After this operation, 93.8 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty/main libaio1 amd64 0.3.109-4 [6,364 B]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty/main libdbi-perl amd64 1.630-1 [879 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/main libdbd-mysql-perl amd64 4.025-1ubuntu0.1 [87.6 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty/main libterm-readkey-perl amd64 2.31-1 [27.4 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-client-core-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [707 kB]
Get:6 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-client-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [1,588 kB]
Get:7 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server-core-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [3,766 kB]
Get:8 http://archive.ubuntu.com/ubuntu/ trusty/main psmisc amd64 22.20-1ubuntu2 [53.2 kB]
Get:9 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server-5.5 amd64 5.5.57-0ubuntu0.14.04.1 [1,866 kB]
Get:10 http://archive.ubuntu.com/ubuntu/ trusty/main libhtml-template-perl all 2.95-1 [65.5 kB]
Get:11 http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-server all 5.5.57-0ubuntu0.14.04.1 [11.3 kB]
Preconfiguring packages ...
Fetched 9,057 kB in 1s (7,879 kB/s)
Selecting previously unselected package libaio1:amd64.
(Reading database ... 20043 files and directories currently installed.)
Preparing to unpack .../libaio1_0.3.109-4_amd64.deb ...
Unpacking libaio1:amd64 (0.3.109-4) ...
Selecting previously unselected package libdbi-perl.
Preparing to unpack .../libdbi-perl_1.630-1_amd64.deb ...
Unpacking libdbi-perl (1.630-1) ...
Selecting previously unselected package libdbd-mysql-perl.
Preparing to unpack .../libdbd-mysql-perl_4.025-1ubuntu0.1_amd64.deb ...
Unpacking libdbd-mysql-perl (4.025-1ubuntu0.1) ...
Selecting previously unselected package libterm-readkey-perl.
Preparing to unpack .../libterm-readkey-perl_2.31-1_amd64.deb ...
Unpacking libterm-readkey-perl (2.31-1) ...
Selecting previously unselected package mysql-client-core-5.5.
Preparing to unpack .../mysql-client-core-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
Unpacking mysql-client-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Selecting previously unselected package mysql-client-5.5.
Preparing to unpack .../mysql-client-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
Unpacking mysql-client-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Selecting previously unselected package mysql-server-core-5.5.
Preparing to unpack .../mysql-server-core-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
Unpacking mysql-server-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Selecting previously unselected package psmisc.
Preparing to unpack .../psmisc_22.20-1ubuntu2_amd64.deb ...
Unpacking psmisc (22.20-1ubuntu2) ...
Selecting previously unselected package mysql-server-5.5.
Preparing to unpack .../mysql-server-5.5_5.5.57-0ubuntu0.14.04.1_amd64.deb ...
Unpacking mysql-server-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Selecting previously unselected package libhtml-template-perl.
Preparing to unpack .../libhtml-template-perl_2.95-1_all.deb ...
Unpacking libhtml-template-perl (2.95-1) ...
Selecting previously unselected package mysql-server.
Preparing to unpack .../mysql-server_5.5.57-0ubuntu0.14.04.1_all.deb ...
Unpacking mysql-server (5.5.57-0ubuntu0.14.04.1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libaio1:amd64 (0.3.109-4) ...
Setting up libdbi-perl (1.630-1) ...
Setting up libdbd-mysql-perl (4.025-1ubuntu0.1) ...
Setting up libterm-readkey-perl (2.31-1) ...
Setting up mysql-client-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Setting up mysql-client-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Setting up mysql-server-core-5.5 (5.5.57-0ubuntu0.14.04.1) ...
Setting up psmisc (22.20-1ubuntu2) ...
Setting up mysql-server-5.5 (5.5.57-0ubuntu0.14.04.1) ...
invoke-rc.d: policy-rc.d denied execution of stop.
invoke-rc.d: policy-rc.d denied execution of start.
Setting up libhtml-template-perl (2.95-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up mysql-server (5.5.57-0ubuntu0.14.04.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
 ---> eb16f31d9bec
Removing intermediate container af69d37044a1
Step 4 : RUN /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis
 ---> Running in faaa8ad8d99b
 * Starting MySQL database server mysqld
   ...fail!
The command '/bin/sh -c /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis' returned a non-zero code: 1

502 Bad Gateway nginx/1.4.6 (Ubuntu)

I am, working on Clearwater Docker images. When ever I try to make some changes to my containers (like adding application server in the json file) my Clearwater use to stop working(I can generate numbers on ellis but I could not register them on Clearwater using zoipher).

So, by using "commit"(sudo docker commit ) command I saved the status of my containers. Then I started using my new images. But , when ever I tried to log into ellis I am , getting a message on my browser "502 Bad Gateway nginx/1.4.6 (Ubuntu)".

Then I even crossed checked my new running container by connecting to them. Where I could even get the data in the database, of my older containers.

I need a solution for this issue.

build homestead docker image error

when build homestead docker image error there are some errors
first problem is dnsmasq ,it seems dnsmasq should removed ? because docker or kubernetes's dns name service ??

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.3592?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.3893?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!
zmq_msg_recv: Resource temporarily unavailable

then ,I'm confused that homestead still need install cansandra ??(cansandra already a new docker images of clearwater )

Cassandra is not installed yet, skipping schema addition for now
Cassandra is not installed yet, skipping schema addition for now
hostname: you must be root to change the host name
mv: cannot move ?.tmp/hosts.4037?.to ?.etc/hosts?. Device or resource busy
Configuring monit for only localhost access

next ,homestead-prov install errror

Processing dependencies for homestead-prov==0.1
Finished processing dependencies for homestead-prov==0.1

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.4164?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!
zmq_msg_recv: Resource temporarily unavailable
No process matching value from pidfile:

following is install logs

Setting up sgml-base (1.26+nmu4ubuntu1) ...
Setting up clearwater-cassandra (1.0-160927.095924) ...
Adding 'diversion of /etc/default/cassandra to /etc/default/cassandra.clearwater-orig by clearwater-cassandra'
Adding 'diversion of /etc/init.d/cassandra to /etc/init.d/cassandra.clearwater-orig by clearwater-cassandra'
Adding 'diversion of /usr/share/cassandra/cassandra.in.sh to /usr/share/cassandra/cassandra.in.sh.clearwater-orig by clearwater-cassandra'
Adding 'diversion of /etc/cassandra/cassandra.yaml to /usr/share/clearwater-cassandra/etc++cassandra++cassandra.yaml by clearwater-cassandra'
Adding 'diversion of /etc/cassandra/cassandra-rackdc.properties to /usr/share/clearwater-cassandra/etc++cassandra++cassandra-rackdc.properties by clearwater-cassandra'
Adding 'diversion of /etc/cassandra/cassandra-env.sh to /usr/share/clearwater-cassandra/etc++cassandra++cassandra-env.sh by clearwater-cassandra'

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.3592?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!
zmq_msg_recv: Resource temporarily unavailable
mv: cannot move ?.tmp/local_settings.py.3640?.to ?.usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/clearwater_prov_tools-0.1-py2.7.egg/metaswitch/ellis/local_settings.py?. No such file or directory
nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)
nginx: configuration file /etc/nginx/nginx.conf test failed
** Note: Please use the --sec-param instead of --bits
Generating a 1024 bit RSA private key...
Generating a self signed certificate...
X.509 Certificate Information:
Version: 3
Serial Number (hex): 57fe1917
Validity:
Not Before: Wed Oct 12 11:05:59 UTC 2016
Not After: Sat Oct 10 11:05:59 UTC 2026
Subject: CN=172.17.1.2
Subject Public Key Algorithm: RSA
Certificate Security Level: Weak
Modulus (bits 1024):
00:c6:ec:d6:2d:1a:e8:51:da:86:65:b5:bb:a0:8c:d8
92:33:47:7b:f7:cf:22:5f:b1:15:82:cb:24:67:74:74
c4:88:85:62:ef:79:85:0e:e1:99:75:f8:1d:43:0f:2f
75:4a:f7:47:47:43:be:42:b6:82:2b:60:ac:44:fb:68
89:de:48:c3:0a:69:3d:7f:db:7b:ed:b9:09:bd:f7:b7
4c:c5:48:ae:ed:f6:8c:16:f4:3f:69:ce:c7:53:54:d6
2d:ca:cf:dd:03:b3:30:02:b4:6f:db:19:b5:8b:d0:c5
b6:95:09:62:35:a0:3d:b8:84:22:22:bf:ef:79:6d:13
35
Exponent (bits 24):
01:00:01
Extensions:
Basic Constraints (critical):
Certificate Authority (CA): FALSE
Subject Key Identifier (not critical):
fd6cbc8053b879e30a111b874d0c48b52d3ad8ba
Other Information:
Public Key Id:
fd6cbc8053b879e30a111b874d0c48b52d3ad8ba

Signing certificate...
Cassandra is not installed yet, skipping schema addition for now
Cassandra is not installed yet, skipping schema addition for now
hostname: you must be root to change the host name
mv: cannot move ?.tmp/hosts.3743?.to ?.etc/hosts?. Device or resource busy
Configuring monit for only localhost access
...done.
Setting up libboost-system1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libboost-filesystem1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libicu52:amd64 (52.1-3ubuntu0.4) ...
Setting up libboost-regex1.54.0:amd64 (1.54.0-4ubuntu3.1) ...
Setting up libevent-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libevent-core-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libevent-pthreads-2.0-5:amd64 (2.0.21-stable-1ubuntu1.14.04.1) ...
Setting up libjbig0:amd64 (2.0-2ubuntu4.1) ...
Setting up libtiff5:amd64 (4.0.3-7ubuntu0.4) ...
Setting up libvpx1:amd64 (1.3.0-2) ...
Setting up libxpm4:amd64 (1:3.5.10-1) ...
Setting up libgd3:amd64 (2.1.0-3ubuntu0.3) ...
Setting up libperl5.18 (5.18.2-2ubuntu1.1) ...
Setting up libsnmp-base (5.7.2dfsg-clearwater4) ...
Setting up libsnmp30:amd64 (5.7.2dfsg-clearwater4) ...
Setting up libxslt1.1:amd64 (1.1.28-2build1) ...
Setting up geoip-database (20140313-1) ...
Setting up libglib2.0-data (2.40.2-0ubuntu1) ...
Setting up shared-mime-info (1.2-0ubuntu3) ...
Setting up xml-core (0.13+nmu2) ...
Setting up clearwater-log-cleanup (1.0-160927.094809) ...
Setting up nginx-common (1.4.6-1ubuntu3.5) ...
Setting up clearwater-prov-tools (1.0-160927.104257) ...
Already using interpreter /usr/bin/python
New python executable in /usr/share/clearwater/clearwater-prov-tools/env/bin/python
Installing setuptools, pip...done.
Processing backports.ssl_match_hostname-3.4.0.2-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/backports.ssl_match_hostname-3.4.0.2-py2.7.egg
Extracting backports.ssl_match_hostname-3.4.0.2-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding backports.ssl-match-hostname 3.4.0.2 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/backports.ssl_match_hostname-3.4.0.2-py2.7.egg
Processing dependencies for backports.ssl-match-hostname==3.4.0.2
Finished processing dependencies for backports.ssl-match-hostname==3.4.0.2
Processing backports_abc-0.4-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/backports_abc-0.4-py2.7.egg
Extracting backports_abc-0.4-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding backports-abc 0.4 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/backports_abc-0.4-py2.7.egg
Processing dependencies for backports-abc==0.4
Finished processing dependencies for backports-abc==0.4
Processing certifi-2015.11.20.1-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/certifi-2015.11.20.1-py2.7.egg
Extracting certifi-2015.11.20.1-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding certifi 2015.11.20.1 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/certifi-2015.11.20.1-py2.7.egg
Processing dependencies for certifi==2015.11.20.1
Finished processing dependencies for certifi==2015.11.20.1
Processing cffi-1.3.1-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/cffi-1.3.1-py2.7-linux-x86_64.egg
Extracting cffi-1.3.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding cffi 1.3.1 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/cffi-1.3.1-py2.7-linux-x86_64.egg
Processing dependencies for cffi==1.3.1
Searching for pycparser
Best match: pycparser 2.14
Processing pycparser-2.14-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Extracting pycparser-2.14-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding pycparser 2.14 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Finished processing dependencies for cffi==1.3.1
Processing cffi-1.5.2-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/cffi-1.5.2-py2.7-linux-x86_64.egg
Extracting cffi-1.5.2-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Removing cffi 1.3.1 from easy-install.pth file
Adding cffi 1.5.2 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/cffi-1.5.2-py2.7-linux-x86_64.egg
Processing dependencies for cffi==1.5.2
Finished processing dependencies for cffi==1.5.2
Processing clearwater_prov_tools-0.1-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/clearwater_prov_tools-0.1-py2.7.egg
Extracting clearwater_prov_tools-0.1-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding clearwater-prov-tools 0.1 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/clearwater_prov_tools-0.1-py2.7.egg
Processing dependencies for clearwater-prov-tools==0.1
Searching for tornado
Best match: tornado 4.3
Processing tornado-4.3-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg
Extracting tornado-4.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding tornado 4.3 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg
Searching for singledispatch
Best match: singledispatch 3.4.0.3
Processing singledispatch-3.4.0.3-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg
Extracting singledispatch-3.4.0.3-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding singledispatch 3.4.0.3 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg
Searching for six
Best match: six 1.10.0
Processing six-1.10.0-py2.7.egg
Copying six-1.10.0-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding six 1.10.0 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Finished processing dependencies for clearwater-prov-tools==0.1
Processing metaswitchcommon-0.1-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/metaswitchcommon-0.1-py2.7-linux-x86_64.egg
Extracting metaswitchcommon-0.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding metaswitchcommon 0.1 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/metaswitchcommon-0.1-py2.7-linux-x86_64.egg
Processing dependencies for metaswitchcommon==0.1
Searching for monotonic==0.6
Best match: monotonic 0.6
Processing monotonic-0.6-py2.7.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Extracting monotonic-0.6-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding monotonic 0.6 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Searching for pyzmq==15.2
Best match: pyzmq 15.2.0
Processing pyzmq-15.2.0-py2.7-linux-x86_64.egg
Copying pyzmq-15.2.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding pyzmq 15.2.0 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg
Searching for pycrypto==2.6.1
Best match: pycrypto 2.6.1
Processing pycrypto-2.6.1-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Extracting pycrypto-2.6.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding pycrypto 2.6.1 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Searching for py-bcrypt
Best match: py-bcrypt 0.4
Processing py_bcrypt-0.4-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Extracting py_bcrypt-0.4-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Adding py-bcrypt 0.4 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Finished processing dependencies for metaswitchcommon==0.1
Processing monotonic-0.6-py2.7.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Extracting monotonic-0.6-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
monotonic 0.6 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Processing dependencies for monotonic==0.6
Finished processing dependencies for monotonic==0.6
Processing py_bcrypt-0.4-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Extracting py_bcrypt-0.4-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
py-bcrypt 0.4 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Processing dependencies for py-bcrypt==0.4
Finished processing dependencies for py-bcrypt==0.4
Processing pycparser-2.14-py2.7.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Extracting pycparser-2.14-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
pycparser 2.14 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Processing dependencies for pycparser==2.14
Finished processing dependencies for pycparser==2.14
Processing pycrypto-2.6.1-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Extracting pycrypto-2.6.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
pycrypto 2.6.1 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Processing dependencies for pycrypto==2.6.1
Finished processing dependencies for pycrypto==2.6.1
Processing pyzmq-15.1.0-py2.7-linux-x86_64.egg
Copying pyzmq-15.1.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Removing pyzmq 15.2.0 from easy-install.pth file
Adding pyzmq 15.1.0 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pyzmq-15.1.0-py2.7-linux-x86_64.egg
Processing dependencies for pyzmq==15.1.0
Finished processing dependencies for pyzmq==15.1.0
Processing pyzmq-15.2.0-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg' (and everything under it)
Copying pyzmq-15.2.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
Removing pyzmq 15.1.0 from easy-install.pth file
Adding pyzmq 15.2.0 to easy-install.pth file

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg
Processing dependencies for pyzmq==15.2.0
Finished processing dependencies for pyzmq==15.2.0
Processing singledispatch-3.4.0.3-py2.7.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg
Extracting singledispatch-3.4.0.3-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
singledispatch 3.4.0.3 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg
Processing dependencies for singledispatch==3.4.0.3
Finished processing dependencies for singledispatch==3.4.0.3
Processing six-1.10.0-py2.7.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg' (and everything under it)
Copying six-1.10.0-py2.7.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
six 1.10.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Processing dependencies for six==1.10.0
Finished processing dependencies for six==1.10.0
Processing tornado-4.3-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg' (and everything under it)
creating /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg
Extracting tornado-4.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages
tornado 4.3 is already the active version in easy-install.pth

Installed /usr/share/clearwater/clearwater-prov-tools/env/lib/python2.7/site-packages/tornado-4.3-py2.7-linux-x86_64.egg
Processing dependencies for tornado==4.3
Finished processing dependencies for tornado==4.3

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.3893?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!
zmq_msg_recv: Resource temporarily unavailable
Cassandra is not installed yet, skipping schema addition for now
Cassandra is not installed yet, skipping schema addition for now
hostname: you must be root to change the host name
mv: cannot move ?.tmp/hosts.4037?.to ?.etc/hosts?. Device or resource busy
Configuring monit for only localhost access
...done.
Setting up clearwater-socket-factory (1.0-160927.094809) ...
Setting up clearwater-tcp-scalability (1.0-160927.094809) ...
/usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: 40: /usr/share/clearwater/infrastructure/install/clearwater-tcp-scalability.postinst: sysctl: Permission denied
Setting up libxml2-dev:amd64 (2.9.1+dfsg1-3ubuntu4.8) ...
dpkg: error processing package clearwater-tcp-scalability (--configure):
subprocess installed post-installation script returned error exit status 126
Setting up libxslt1-dev:amd64 (1.1.28-2build1) ...
Setting up python-zmq (14.0.1-1build2) ...
Setting up homestead-libs (1.0-160927.095833) ...
dpkg: dependency problems prevent configuration of homestead:
homestead depends on clearwater-tcp-scalability; however:
Package clearwater-tcp-scalability is not configured yet.

dpkg: error processing package homestead (--configure):
dependency problems - leaving unconfigured
Setting up lksctp-tools (1.0.15+dfsg-1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up nginx-core (1.4.6-1ubuntu3.5) ...
invoke-rc.d: policy-rc.d denied execution of start.
Setting up nginx (1.4.6-1ubuntu3.5) ...
Setting up clearwater-nginx (1.0-160927.112632) ...
Testing nginx configuration...
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Site ping has been enabled. Run /etc/init.d/nginx reload to apply the changes.
Generating a 2048 bit RSA private key
........................................+++
................................................................................................+++

writing new private key to 'nginx.key'

Signature ok
subject=/C=UK/ST=Unknown state/L=Unknown locality/O=Unknown organization/OU=Unknown organizational unit/CN=unknown.example/emailAddress=Unknown
Getting Private key
Setting up crest (1.0-160927.105344) ...
Already using interpreter /usr/bin/python
New python executable in /usr/share/clearwater/crest/env/bin/python
Installing setuptools, pip...done.
Processing Twisted-15.5.0-py2.7-linux-x86_64.egg
Copying Twisted-15.5.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding Twisted 15.5.0 to easy-install.pth file
Installing cftp script to /usr/share/clearwater/crest/env/bin
Installing tap2rpm script to /usr/share/clearwater/crest/env/bin
Installing tkconch script to /usr/share/clearwater/crest/env/bin
Installing twistd script to /usr/share/clearwater/crest/env/bin
Installing ckeygen script to /usr/share/clearwater/crest/env/bin
Installing trial script to /usr/share/clearwater/crest/env/bin
Installing pyhtmlizer script to /usr/share/clearwater/crest/env/bin
Installing tap2deb script to /usr/share/clearwater/crest/env/bin
Installing conch script to /usr/share/clearwater/crest/env/bin
Installing manhole script to /usr/share/clearwater/crest/env/bin
Installing mailmail script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/Twisted-15.5.0-py2.7-linux-x86_64.egg
Processing dependencies for Twisted==15.5.0
Searching for zope.interface>=3.6.0
Best match: zope.interface 4.1.3
Processing zope.interface-4.1.3-py2.7-linux-x86_64.egg
Copying zope.interface-4.1.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding zope.interface 4.1.3 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/zope.interface-4.1.3-py2.7-linux-x86_64.egg
Finished processing dependencies for Twisted==15.5.0
Processing cffi-1.5.2-py2.7-linux-x86_64.egg
creating /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cffi-1.5.2-py2.7-linux-x86_64.egg
Extracting cffi-1.5.2-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding cffi 1.5.2 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cffi-1.5.2-py2.7-linux-x86_64.egg
Processing dependencies for cffi==1.5.2
Searching for pycparser
Best match: pycparser 2.14
Processing pycparser-2.14-py2.7.egg
Copying pycparser-2.14-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pycparser 2.14 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Finished processing dependencies for cffi==1.5.2
Processing cql-1.4.0-py2.7.egg
Copying cql-1.4.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding cql 1.4.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cql-1.4.0-py2.7.egg
Processing dependencies for cql==1.4.0
Searching for thrift
Best match: thrift 0.9.3
Processing thrift-0.9.3-py2.7-linux-x86_64.egg
Copying thrift-0.9.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding thrift 0.9.3 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/thrift-0.9.3-py2.7-linux-x86_64.egg
Finished processing dependencies for cql==1.4.0
Processing crest-0.1-py2.7.egg
creating /usr/share/clearwater/crest/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg
Extracting crest-0.1-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding crest 0.1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg
Processing dependencies for crest==0.1
Searching for monotonic==0.6
Best match: monotonic 0.6
Processing monotonic-0.6-py2.7.egg
Copying monotonic-0.6-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding monotonic 0.6 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Searching for prctl
Best match: prctl 1.0.1
Processing prctl-1.0.1-py2.7-linux-x86_64.egg
Copying prctl-1.0.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding prctl 1.0.1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/prctl-1.0.1-py2.7-linux-x86_64.egg
Searching for pure-sasl
Best match: pure-sasl 0.2.0
Processing pure_sasl-0.2.0-py2.7.egg
Copying pure_sasl-0.2.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pure-sasl 0.2.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pure_sasl-0.2.0-py2.7.egg
Searching for msgpack-python==0.4.7
Best match: msgpack-python 0.4.7
Processing msgpack_python-0.4.7-py2.7-linux-x86_64.egg
Copying msgpack_python-0.4.7-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding msgpack-python 0.4.7 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/msgpack_python-0.4.7-py2.7-linux-x86_64.egg
Searching for lxml
Best match: lxml 3.5.0
Processing lxml-3.5.0-py2.7-linux-x86_64.egg
Copying lxml-3.5.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding lxml 3.5.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/lxml-3.5.0-py2.7-linux-x86_64.egg
Searching for cyclone==1.0
Best match: cyclone 1.0
Processing cyclone-1.0-py2.7.egg
Copying cyclone-1.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding cyclone 1.0 to easy-install.pth file
Installing cyclone script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cyclone-1.0-py2.7.egg
Searching for py-bcrypt
Best match: py-bcrypt 0.4
Processing py_bcrypt-0.4-py2.7-linux-x86_64.egg
Copying py_bcrypt-0.4-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding py-bcrypt 0.4 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Searching for pyzmq==15.2
Best match: pyzmq 15.2.0
Processing pyzmq-15.2.0-py2.7-linux-x86_64.egg
Copying pyzmq-15.2.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pyzmq 15.2.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg
Searching for pyopenssl
Best match: pyOpenSSL 0.15.1
Processing pyOpenSSL-0.15.1-py2.7.egg
Copying pyOpenSSL-0.15.1-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pyOpenSSL 0.15.1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyOpenSSL-0.15.1-py2.7.egg
Searching for six>=1.5.2
Best match: six 1.10.0
Processing six-1.10.0-py2.7.egg
Copying six-1.10.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding six 1.10.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Searching for cryptography>=0.7
Best match: cryptography 1.2.2
Processing cryptography-1.2.2-py2.7-linux-x86_64.egg
Copying cryptography-1.2.2-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding cryptography 1.2.2 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cryptography-1.2.2-py2.7-linux-x86_64.egg
Searching for ipaddress
Best match: ipaddress 1.0.16
Processing ipaddress-1.0.16-py2.7.egg
Copying ipaddress-1.0.16-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding ipaddress 1.0.16 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/ipaddress-1.0.16-py2.7.egg
Searching for enum34
Best match: enum34 1.1.2
Processing enum34-1.1.2-py2.7.egg
Copying enum34-1.1.2-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding enum34 1.1.2 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/enum34-1.1.2-py2.7.egg
Searching for pyasn1>=0.1.8
Best match: pyasn1 0.1.9
Processing pyasn1-0.1.9-py2.7.egg
Copying pyasn1-0.1.9-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pyasn1 0.1.9 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg
Searching for idna>=2.0
Best match: idna 2.0
Processing idna-2.0-py2.7.egg
Copying idna-2.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding idna 2.0 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/idna-2.0-py2.7.egg
Finished processing dependencies for crest==0.1
Processing cryptography-1.2.2-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/cryptography-1.2.2-py2.7-linux-x86_64.egg' (and everything under it)
Copying cryptography-1.2.2-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
cryptography 1.2.2 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cryptography-1.2.2-py2.7-linux-x86_64.egg
Processing dependencies for cryptography==1.2.2
Finished processing dependencies for cryptography==1.2.2
Processing cyclone-1.0-py2.7.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/cyclone-1.0-py2.7.egg' (and everything under it)
Copying cyclone-1.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
cyclone 1.0 is already the active version in easy-install.pth
Installing cyclone script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/cyclone-1.0-py2.7.egg
Processing dependencies for cyclone==1.0
Finished processing dependencies for cyclone==1.0
Processing enum34-1.1.2-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/enum34-1.1.2-py2.7.egg
Copying enum34-1.1.2-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
enum34 1.1.2 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/enum34-1.1.2-py2.7.egg
Processing dependencies for enum34==1.1.2
Finished processing dependencies for enum34==1.1.2
Processing idna-2.0-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/idna-2.0-py2.7.egg
Copying idna-2.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
idna 2.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/idna-2.0-py2.7.egg
Processing dependencies for idna==2.0
Finished processing dependencies for idna==2.0
Processing ipaddress-1.0.16-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/ipaddress-1.0.16-py2.7.egg
Copying ipaddress-1.0.16-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
ipaddress 1.0.16 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/ipaddress-1.0.16-py2.7.egg
Processing dependencies for ipaddress==1.0.16
Finished processing dependencies for ipaddress==1.0.16
Processing lxml-3.5.0-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/lxml-3.5.0-py2.7-linux-x86_64.egg' (and everything under it)
Copying lxml-3.5.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
lxml 3.5.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/lxml-3.5.0-py2.7-linux-x86_64.egg
Processing dependencies for lxml==3.5.0
Finished processing dependencies for lxml==3.5.0
Processing metaswitchcommon-0.1-py2.7-linux-x86_64.egg
Copying metaswitchcommon-0.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding metaswitchcommon 0.1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/metaswitchcommon-0.1-py2.7-linux-x86_64.egg
Processing dependencies for metaswitchcommon==0.1
Searching for pycrypto==2.6.1
Best match: pycrypto 2.6.1
Processing pycrypto-2.6.1-py2.7-linux-x86_64.egg
Copying pycrypto-2.6.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding pycrypto 2.6.1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Finished processing dependencies for metaswitchcommon==0.1
Processing monotonic-0.6-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Copying monotonic-0.6-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
monotonic 0.6 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/monotonic-0.6-py2.7.egg
Processing dependencies for monotonic==0.6
Finished processing dependencies for monotonic==0.6
Processing msgpack_python-0.4.7-py2.7-linux-x86_64.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/msgpack_python-0.4.7-py2.7-linux-x86_64.egg
Copying msgpack_python-0.4.7-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
msgpack-python 0.4.7 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/msgpack_python-0.4.7-py2.7-linux-x86_64.egg
Processing dependencies for msgpack-python==0.4.7
Finished processing dependencies for msgpack-python==0.4.7
Processing prctl-1.0.1-py2.7-linux-x86_64.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/prctl-1.0.1-py2.7-linux-x86_64.egg
Copying prctl-1.0.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
prctl 1.0.1 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/prctl-1.0.1-py2.7-linux-x86_64.egg
Processing dependencies for prctl==1.0.1
Finished processing dependencies for prctl==1.0.1
Processing pure_sasl-0.2.0-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pure_sasl-0.2.0-py2.7.egg
Copying pure_sasl-0.2.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pure-sasl 0.2.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pure_sasl-0.2.0-py2.7.egg
Processing dependencies for pure-sasl==0.2.0
Finished processing dependencies for pure-sasl==0.2.0
Processing pyOpenSSL-0.15.1-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyOpenSSL-0.15.1-py2.7.egg
Copying pyOpenSSL-0.15.1-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pyOpenSSL 0.15.1 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyOpenSSL-0.15.1-py2.7.egg
Processing dependencies for pyOpenSSL==0.15.1
Finished processing dependencies for pyOpenSSL==0.15.1
Processing py_bcrypt-0.4-py2.7-linux-x86_64.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Copying py_bcrypt-0.4-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
py-bcrypt 0.4 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/py_bcrypt-0.4-py2.7-linux-x86_64.egg
Processing dependencies for py-bcrypt==0.4
Finished processing dependencies for py-bcrypt==0.4
Processing pyasn1-0.1.9-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg
Copying pyasn1-0.1.9-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pyasn1 0.1.9 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg
Processing dependencies for pyasn1==0.1.9
Finished processing dependencies for pyasn1==0.1.9
Processing pycparser-2.14-py2.7.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Copying pycparser-2.14-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pycparser 2.14 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycparser-2.14-py2.7.egg
Processing dependencies for pycparser==2.14
Finished processing dependencies for pycparser==2.14
Processing pycrypto-2.6.1-py2.7-linux-x86_64.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Copying pycrypto-2.6.1-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pycrypto 2.6.1 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg
Processing dependencies for pycrypto==2.6.1
Finished processing dependencies for pycrypto==2.6.1
Processing pyzmq-15.2.0-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg' (and everything under it)
Copying pyzmq-15.2.0-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
pyzmq 15.2.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/pyzmq-15.2.0-py2.7-linux-x86_64.egg
Processing dependencies for pyzmq==15.2.0
Finished processing dependencies for pyzmq==15.2.0
Processing setuptools-20.2.1-py2.7.egg
Copying setuptools-20.2.1-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding setuptools 20.2.1 to easy-install.pth file
Installing easy_install script to /usr/share/clearwater/crest/env/bin
Installing easy_install-2.7 script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/setuptools-20.2.1-py2.7.egg
Processing dependencies for setuptools==20.2.1

Note: Bypassing https://pypi.python.org/packages/source/c/certifi/certifi-2015.11.20.tar.gz#md5=25134646672c695c1ff1593c2dd75d08 (disallowed host; see http://bit.ly/1dg9ijs for details).

Note: Bypassing https://pypi.python.org/packages/source/w/wincertstore/wincertstore-0.2.zip#md5=ae728f2f007185648d0c7a8679b361e2 (disallowed host; see http://bit.ly/1dg9ijs for details).

Finished processing dependencies for setuptools==20.2.1
Processing setuptools-20.2.2-py2.7.egg
Copying setuptools-20.2.2-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Removing setuptools 20.2.1 from easy-install.pth file
Adding setuptools 20.2.2 to easy-install.pth file
Installing easy_install script to /usr/share/clearwater/crest/env/bin
Installing easy_install-2.7 script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/setuptools-20.2.2-py2.7.egg

Note: Bypassing https://pypi.python.org/packages/source/c/certifi/certifi-2015.11.20.tar.gz#md5=25134646672c695c1ff1593c2dd75d08 (disallowed host; see http://bit.ly/1dg9ijs for details).

Note: Bypassing https://pypi.python.org/packages/source/w/wincertstore/wincertstore-0.2.zip#md5=ae728f2f007185648d0c7a8679b361e2 (disallowed host; see http://bit.ly/1dg9ijs for details).

Processing dependencies for setuptools==20.2.2
Finished processing dependencies for setuptools==20.2.2
Processing setuptools-21.0.0-py2.7.egg
Copying setuptools-21.0.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Removing setuptools 20.2.2 from easy-install.pth file
Adding setuptools 21.0.0 to easy-install.pth file
Installing easy_install script to /usr/share/clearwater/crest/env/bin
Installing easy_install-2.7 script to /usr/share/clearwater/crest/env/bin

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/setuptools-21.0.0-py2.7.egg

Note: Bypassing https://pypi.python.org/packages/source/c/certifi/certifi-2016.2.28.tar.gz#md5=5d672aa766e1f773c75cfeccd02d3650 (disallowed host; see http://bit.ly/1dg9ijs for details).

Note: Bypassing https://pypi.python.org/packages/source/w/wincertstore/wincertstore-0.2.zip#md5=ae728f2f007185648d0c7a8679b361e2 (disallowed host; see http://bit.ly/1dg9ijs for details).

Processing dependencies for setuptools==21.0.0
Finished processing dependencies for setuptools==21.0.0
Processing six-1.10.0-py2.7.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg' (and everything under it)
Copying six-1.10.0-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
six 1.10.0 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/six-1.10.0-py2.7.egg
Processing dependencies for six==1.10.0
Finished processing dependencies for six==1.10.0
Processing telephus-1.0.0b1-py2.7.egg
Copying telephus-1.0.0b1-py2.7.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
Adding telephus 1.0.0b1 to easy-install.pth file

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/telephus-1.0.0b1-py2.7.egg
Processing dependencies for telephus==1.0.0b1
Finished processing dependencies for telephus==1.0.0b1
Processing thrift-0.9.3-py2.7-linux-x86_64.egg
Removing /usr/share/clearwater/crest/env/lib/python2.7/site-packages/thrift-0.9.3-py2.7-linux-x86_64.egg
Copying thrift-0.9.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
thrift 0.9.3 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/thrift-0.9.3-py2.7-linux-x86_64.egg
Processing dependencies for thrift==0.9.3
Finished processing dependencies for thrift==0.9.3
Processing zope.interface-4.1.3-py2.7-linux-x86_64.egg
removing '/usr/share/clearwater/crest/env/lib/python2.7/site-packages/zope.interface-4.1.3-py2.7-linux-x86_64.egg' (and everything under it)
Copying zope.interface-4.1.3-py2.7-linux-x86_64.egg to /usr/share/clearwater/crest/env/lib/python2.7/site-packages
zope.interface 4.1.3 is already the active version in easy-install.pth

Installed /usr/share/clearwater/crest/env/lib/python2.7/site-packages/zope.interface-4.1.3-py2.7-linux-x86_64.egg
Processing dependencies for zope.interface==4.1.3
Finished processing dependencies for zope.interface==4.1.3
Setting up homestead-prov (1.0-160927.105344) ...
Creating /usr/share/clearwater/homestead/python/packages/site.py
Processing homestead_prov-0.1-py2.7.egg
creating /usr/share/clearwater/homestead/python/packages/homestead_prov-0.1-py2.7.egg
Extracting homestead_prov-0.1-py2.7.egg to /usr/share/clearwater/homestead/python/packages
Adding homestead-prov 0.1 to easy-install.pth file

Installed /usr/share/clearwater/homestead/python/packages/homestead_prov-0.1-py2.7.egg
Processing dependencies for homestead-prov==0.1
Finished processing dependencies for homestead-prov==0.1

Restarting clearwater-infrastructure clearwater-infrastructure
mv: cannot move ?.tmp/hosts.4164?.to ?.etc/hosts?. Device or resource busy
Restarting DNS forwarder and DHCP server dnsmasq

dnsmasq: setting capabilities failed: Operation not permitted
...fail!
zmq_msg_recv: Resource temporarily unavailable
No process matching value from pidfile:
Cassandra is not configured yet, skipping schema addition for now
Cassandra is not configured yet, skipping schema addition for now
hostname: you must be root to change the host name
mv: cannot move ?.tmp/hosts.4309?.to ?.etc/hosts?. Device or resource busy
Configuring monit for only localhost access
...done.
Cassandra is not configured yet, skipping schema addition for now
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
clearwater-tcp-scalability
homestead
E: Sub-process /usr/bin/dpkg returned an error code (1)

Issues with homestead and homestead-prov under Kubernetes

Hi, I wanted to first float these issues on the clearwater list but am getting no response to list subscription requests. So reaching out here for help. I'm working on deploying clearwater-docker under Kubernetes for use in the OPNFV Models and VES projects. I'm having issues with homestead and homestead-prov crashing, and never completely coming up. It appears to be related to the snmp daemon, for which some binaries are missing. See below for the logs (output of "kubectl logs --namespace default homestead-1026420941-q3lk4 homestead", similar output for homestead-prov). I've looked into the Dockerfiles and AFAICT the issue must be somewhere in the package build process for homestead, vs the packages used for the other containers (for which this snmp issue does not occur). I have tried this with the latest (master) and stable repos.

I can provide build output for the container image as needed. Nothing unusual there, beyond a couple of errors that do not fail the build, and which are also present for the containers that do work.

From comparing the bono and homestead containers live, it's clear that there are files in /usr/share/clearwater/bin/ named clearwater-socket-factory* (a set of files) that are not present in the homestead container, as built. I believe side-effects of this error are that the homestead container never fully initializes, and the necessary DNS name for homestead not registered (as are all the other container names, even homestead-prov curiously), and thus sprout rejects registration requests as it can't find the homestead server.

Here is the log.

$ kubectl logs --namespace default homestead-1026420941-q3lk4 homestead
2018-01-28 22:55:22,011 CRIT Supervisor running as root (no user in config file)
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/socket-factory.supervisord.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/clearwater-group.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/homestead-prov.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/snmpd.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/clearwater-infrastructure.conf" during parsing
2018-01-28 22:55:22,011 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2018-01-28 22:55:22,032 INFO RPC interface 'supervisor' initialized
2018-01-28 22:55:22,032 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-01-28 22:55:22,032 INFO supervisord started with pid 7980
2018-01-28 22:55:24,504 INFO spawned: 'snmpd' with pid 7988
2018-01-28 22:55:24,505 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-sig-wrapper'
2018-01-28 22:55:24,506 INFO spawned: 'clearwater-infrastructure' with pid 7989
2018-01-28 22:55:24,507 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-mgmt-wrapper'
2018-01-28 22:55:24,516 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:24,516 INFO success: clearwater-infrastructure entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:24,914 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:27,466 INFO spawned: 'snmpd' with pid 8036
2018-01-28 22:55:27,467 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-sig-wrapper'
2018-01-28 22:55:27,467 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-mgmt-wrapper'
2018-01-28 22:55:27,477 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:29,130 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:29,132 INFO spawned: 'snmpd' with pid 8044
2018-01-28 22:55:29,133 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:30,139 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-sig-wrapper'
2018-01-28 22:55:30,140 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-mgmt-wrapper'
2018-01-28 22:55:30,591 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:31,594 INFO spawned: 'snmpd' with pid 8096
2018-01-28 22:55:31,607 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:33,269 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-sig-wrapper'
2018-01-28 22:55:33,269 INFO gave up: socket-factory-sig entered FATAL state, too many start retries too quickly
2018-01-28 22:55:33,269 INFO spawnerr: can't find command '/usr/share/clearwater/bin/clearwater-socket-factory-mgmt-wrapper'
2018-01-28 22:55:33,269 INFO gave up: socket-factory-mgmt entered FATAL state, too many start retries too quickly
2018-01-28 22:55:33,270 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:33,300 INFO spawned: 'snmpd' with pid 8187
2018-01-28 22:55:33,306 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:33,715 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:34,707 INFO spawned: 'snmpd' with pid 8239
2018-01-28 22:55:34,712 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:34,717 INFO spawned: 'homestead-prov' with pid 8242
2018-01-28 22:55:34,720 INFO spawned: 'nginx' with pid 8243
2018-01-28 22:55:35,738 INFO success: homestead-prov entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-28 22:55:35,738 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-01-28 22:55:36,239 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:36,241 INFO spawned: 'snmpd' with pid 8260
2018-01-28 22:55:36,241 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:36,246 INFO exited: clearwater-infrastructure (exit status 0; expected)
2018-01-28 22:55:37,733 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:39,376 INFO spawned: 'snmpd' with pid 8273
2018-01-28 22:55:39,381 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:55:39,696 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:55:40,698 INFO spawned: 'snmpd' with pid 8280
... (last  2 lines repeated a bunch of times, every ~2 secs)
2018-01-28 22:56:39,585 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:56:39,610 INFO spawned: 'snmpd' with pid 8402
2018-01-28 22:56:39,610 WARN received SIGTERM indicating exit request
2018-01-28 22:56:39,610 INFO waiting for snmpd, homestead-prov, nginx to die
2018-01-28 22:56:39,620 INFO success: snmpd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-01-28 22:56:39,942 INFO exited: snmpd (exit status 1; not expected)
2018-01-28 22:56:42,373 INFO stopped: homestead-prov (terminated by SIGTERM)
2018-01-28 22:56:42,374 INFO stopped: nginx (exit status 0)

Failed requestes due to DSN record retrieve

Symptoms

sprout.log

25-11-2016 21:52:37.626 UTC Warning scscfsproutlet.cpp:1856: No charging role in Route header, assume originating
25-11-2016 21:52:37.729 UTC Warning scscfsproutlet.cpp:1856: No charging role in Route header, assume originating
25-11-2016 21:52:37.845 UTC Error dnscachedresolver.cpp:846: Failed to retrieve record for example.com: DNS server returned answer with no data
25-11-2016 21:52:37.866 UTC Warning scscfsproutlet.cpp:1856: No charging role in Route header, assume originating
25-11-2016 21:52:37.938 UTC Status load_monitor.cpp:277: Maximum incoming request rate/second increased to 734.414795 (based on a smoothed mean latency of 20028, 0 upstream overload responses, 2000ms time passing, 803 accepted requests, and 420 rejected requests).
25-11-2016 21:52:37.951 UTC Warning scscfsproutlet.cpp:1856: No charging role in Route header, assume originating
25-11-2016 21:52:37.958 UTC Error dnscachedresolver.cpp:846: Failed to retrieve record for _sip._udp.example.com: Domain name not found
25-11-2016 21:52:37.958 UTC Error dnscachedresolver.cpp:846: Failed to retrieve record for _sip._tcp.example.com: Domain name not found

Impact

Some requests failed

Release and environment

Last clearwater-docker release

Steps to reproduce

For example by a sipp stress workload.

The problem happens because by default we use the docker embedded DSN service. Create an external DSN service can be a solution, but there is another solution/workaround?

clearwater-live-test SUBSCRIBE timeout tests unreliable against Docker deployment

We create a deployment by running the following commands...

docker build --no-cache -t clearwater/base base
docker-compose -f minimal-distributed.yaml up --force-recreate -d
docker-compose -f minimal-distributed.yaml scale memcached=2 chronos=2 cassandra=2

We then sleep for 4 minutes and run the live tests against our new deployment. They all pass, except the SUBSCRIBE - Subscription timeout and SUBSCRIBE - Registration timeout tests which timeout "waiting for NOTIFY". It seems to be the expiry NOTIFY that we don't receive.

We run this regularly using Jenkins (the live-test-docker job), and these tests seem to fail most of the time. We collect some diags, but they're currently not very useful. I've excluded these tests from that job now, so we should turn them back on when this is fixed.

In order to repro, I'd just run through the live-test-docker job manually (with the tests not excluded!).

'clearwater-auto-config-docker' Service Stores Ingress Network IP in local_config in Swarm Mode

Symptoms

When deploying clearwater images in docker swarm, the local_config file is updated by clearwater-auto-config-docker service (by the following code: ip=$(hostname -I | sed -e 's/$^\|^[0-9A-Fa-f: ]* $$[0-9.][0-9.]*$$ .*$\|$$/\2/g' -e 's/$^$$^[0-9A-Fa-f:]*$$ .*$\|$$/\2/g')) with ingress network IPs instead of overlay network IP because of the position of the IP after running hostname -I command.

Impact

This prevents clearwater-cluster-manager and component services (e.g. ellis, bono, ...) from starting and communicating.

Release and environment

Environment: VMs in OpenStack environment, ubuntu 16.04 OS

Steps to reproduce

1- Initialize swarm and join nodes
2- Create an overlay network
3- Create docker services using the overlay network
4- Check /etc/clearwater/local_config file. It is updated with ingress network IPs because of its position in hostname -I command.

By manually changing the IPs in local_config files in each container and restarting services result in successful live tests.

Homestead does not work if SCTP not installed

Symptoms

Homestead hangs on startup and does not function (hitting bug #49). Underlying cause appears to be that the node kernel does not have SCTP installed (and the OS does not have it available - this is a standard debian release on GKE which may not support SCTP).

Error message is as follows.

    10-11-2016 17:44:47.064 UTC Status diameterstack.cpp:573: Starting Diameter stack
    10-11-2016 17:44:47.066 UTC Error freeDiameter: ERROR: in '(*sock = socket(family, SOCK_STREAM, IPPROTO_SCTP))' :       Protocol not supported
    10-11-2016 17:44:47.066 UTC Error freeDiameter: ERROR: in '(fd_sctp_create_bind_server( &cnx->cc_socket, cnx->cc_family, ep_list, port ))' :    Protocol not supported
    10-11-2016 17:44:47.066 UTC Error freeDiameter: ERROR: in '(s->conn = fd_cnx_serv_sctp(fd_g_config->cnf_port, empty_conf_ep ? ((void *)0) : &fd_g_config->cnf_endpoints))' :    Protocol not supported
    10-11-2016 17:44:47.066 UTC Error freeDiameter: ERROR: in '((fd_servers_start()))' :    Protocol not supported
    10-11-2016 17:44:47.066 UTC Error main.cpp:803: Failed to initialize Diameter stack - function fd_core_start, rc 93
    10-11-2016 17:44:47.066 UTC Status main.cpp:804: Homestead is shutting down

Probable resolution is to disable SCTP.

Impact

Deployment DOA.

Release and environment

GKE.

Steps to reproduce

See above - install on GKE.

Building ellis fails due to mysql error

Symptoms

Building ellis fails with

Building ellis
Step 1/12 : FROM clearwater/base
 ---> 89622cf6d6c2
Step 2/12 : MAINTAINER [email protected]
 ---> Using cache
 ---> 3330d6638bc9
Step 3/12 : RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes mysql-server
 ---> Using cache
 ---> 3f4ebd787719
Step 4/12 : RUN /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis
 ---> Running in 380b4d2a037b
 * Starting MySQL database server mysqld
   ...fail!
ERROR: Service 'ellis' failed to build: The command '/bin/sh -c /etc/init.d/mysql start && apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --force-yes ellis' returned a non-zero code: 1

Impact

docker-compose cannot be used to create deployment.

Release and environment

Docker 17.0.3.1 on Mac OS X 10.12.6

Steps to reproduce

Follow steps on homepage:

git clone --recursive [email protected]:Metaswitch/clearwater-docker.git
sudo docker build -t clearwater/base base
sudo docker-compose -f minimal-distributed.yaml up -d

extract database servers to separate containers

need in order to support large scale deployment

homestead docker conainter running error ,sames can't connect cassandra

17-10-2016 22:39:43.367 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for cassandra: Could not contact DNS servers
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:181: Configuring store connection
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:182: Hostname: cassandra
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:183: Port: 9160
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:211: Configuring store worker pool
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:212: Threads: 10
17-10-2016 22:40:51.998 UTC Status cassandra_store.cpp:213: Max Queue: 0
17-10-2016 22:40:51.999 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for cassandra: Could not contact DNS servers
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:181: Configuring store connection
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:182: Hostname: cassandra
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:183: Port: 9160
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:211: Configuring store worker pool
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:212: Threads: 10
17-10-2016 22:42:01.164 UTC Status cassandra_store.cpp:213: Max Queue: 0
17-10-2016 22:42:01.165 UTC Error dnscachedresolver.cpp:607: Failed to retrieve record for cassandra: Could not contact DNS servers

in container ,can connect it (kuberntes servivce name is cassandra )

root@clearwater-homestead:/# wget cassandra:9160
--2016-10-17 22:42:36-- http://cassandra:9160/
Resolving cassandra (cassandra)... 169.169.193.170
Connecting to cassandra (cassandra)|169.169.193.170|:9160... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

bono not connect chronos

bono log show it connect 127.0.0.1 for chronos ,not remote chronos address

21-10-2016 06:57:59.940 UTC Status bono.cpp:3322: Create list of PBXes
21-10-2016 06:57:59.940 UTC Status main.cpp:1910: Creating connection to Chronos 127.0.0.1:7253 using 127.0.0.1:9888 as the callback URI
21-10-2016 06:57:59.941 UTC Status http_connection_pool.cpp:50: Connection pool will use a response timeout of 500ms
21-10-2016 06:57:59.941 UTC Status httpconnection.h:71: Configuring HTTP Connection
21-10-2016 06:57:59.941 UTC Status httpconnection.h:72: Connection created for server 127.0.0.1:7253
21-10-2016 06:57:59.941 UTC Status main.cpp:1957: Using local store

Could not add PSTN number

I am working on a VM ubuntu :
Linux clearwater2 3.13.0-87-generic #133-Ubuntu SMP Tue May 24 18:32:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I have followed the installation with docker compose. Everything seems OK. But when I want to add a new PSTN number, I have this error:
Failed to update the server (see detailed diagnostics in developer console). Please refresh the page.

In ellis the log :

WARNING _base.py:163: 503 POST /accounts/TATA%40TITI.org/numbers/?cb=1fa846b0bfN4 (0.0.0.0): No available numbers
30-08-2016 13:18:20.420 UTC ERROR web.py:1447: 503 POST /accounts/TATA%40TITI.org/numbers/?cb=1fa846b0bfN4 (0.0.0.0) 13.87ms

in sprout :

Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL]):

In homer :
Sending 200 response to localhost for PUT http://http_homer/org.etsi.ngn.simservs/users/sip%3A6505550170%40example.com/simservs.xml

in homestead :
UTC Warning (Net-SNMP): Warning: Failed to connect to the agentx master agent ([NIL])

Homestead neither terminates nor recovers on error

Symptoms

Homestead failed. It turns out that this was because of #48. Homestead reported that it was terminating but failed to do so, and continued listening on its port. Homer appeared to be in a similar state

Logs are below.

    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:181: Configuring store connection
    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:182:   Hostname:  localhost
    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:183:   Port:      9160
    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:211: Configuring store worker pool
    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:212:   Threads:   10
    09-11-2016 11:20:23.576 UTC Status cassandra_store.cpp:213:   Max Queue: 0
    09-11-2016 11:20:23.577 UTC Error main.cpp:745: Failed to initialize the Cassandra cache with error code 3.
    09-11-2016 11:20:23.577 UTC Status main.cpp:746: Homestead is shutting down

Impact

Deployment fails completely.

This is because the orchestration (Docker Compose / Kubernetes) cannot terminate and restart the failed container or report that the container has failed.

Correct behaviour is for the container to exit when it has failed and is shutting down the service. Wrong but possibly acceptable behaviour (i.e. code workaround) might be for it to stop listening on well known ports so that the orchestration can detect the failure.

Release and environment

Current master release.

Steps to reproduce

Happens at same time as #48. Misconfiguring cassandra address should do it once #48 is fixed.

issue with web interface register new user, and issue cqlsh in Cassandra node

Symptoms

after of step :
-Build all the other Clearwater Docker images and start a deployment.
sudo docker-compose -f minimal-distributed.yaml up -d

the nodes are running
and I can go to web interface and register new user
but always get the error ( after click register and redirect to dashboard ) :

Failed to update the server ( see detailed diagnostics in developer console). Please refresh the page

I ssh to Cassandra node and run cqlsh to view data
$cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Impact

Release and environment

Vmware workstation 12 pro.
Docker on Ubuntu 14.04 server

Steps to reproduce

Clearwater video call will be hang up after 30 secondes

We try Clearwater video call with X-lite, but we found video call will be hang up after 30 secondes .
Is there any param to controll ?
Thanks .

dns.json missing in homestead and homestead-prov

Symptoms

When I spin up a kubernetes deployment, homestead and homestead-prov fail to come up properly.

kubectl get pods

NAME                              READY     STATUS             RESTARTS   AGE
cassandra-7777c47568-ddxnj        1/1       Running            0          4h
homestead-568c574587-d8h8r        1/2       CrashLoopBackOff   41         4h
homestead-prov-7f9d9bc455-lmbhl   0/1       CrashLoopBackOff   41         4h
...

kubectl logs homestead-568c574587-d8h8r homestead

2018-05-15 07:01:33,214 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:01:47,228 INFO spawned: 'homestead' with pid 2685
2018-05-15 07:01:47,809 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:02,824 INFO spawned: 'homestead' with pid 2862
2018-05-15 07:02:03,400 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:19,417 INFO spawned: 'homestead' with pid 3039
2018-05-15 07:02:19,996 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:37,018 INFO spawned: 'homestead' with pid 3216
2018-05-15 07:02:37,632 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:02:55,650 INFO spawned: 'homestead' with pid 3393
2018-05-15 07:02:56,225 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:03:15,245 INFO spawned: 'homestead' with pid 3580
2018-05-15 07:03:15,821 INFO exited: homestead (exit status 0; not expected)
2018-05-15 07:03:21,188 WARN received SIGTERM indicating exit request
2018-05-15 07:03:21,188 INFO waiting for nginx, homestead, socket-factory-sig, socket-factory-mgmt to die
2018-05-15 07:03:21,189 INFO stopped: socket-factory-mgmt (terminated by SIGTERM)
2018-05-15 07:03:22,190 INFO stopped: socket-factory-sig (terminated by SIGTERM)
2018-05-15 07:03:22,195 INFO stopped: nginx (exit status 0)

When I docker exec into the container (hard to do because it keeps being restarted by k8s) and look in /var/log/homestead/homestead_current.txt, I see:

15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status dnscachedresolver.cpp:123: Creating Cached Resolver using servers:
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status dnscachedresolver.cpp:133:     10.0.0.10
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Error static_dns_cache.cpp:89: DNS config file /etc/clearwater/dns.json mis
sing
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status a_record_resolver.cpp:29: Created ARecordResolver
15-05-2018 07:05:40.126 UTC [7f6379c8e7c0] Status main.cpp:653: Using local impu store: astaire.2991678f-f9dd-4098-8fe
8-1395c572a307.westeurope.aksapp.io
15-05-2018 07:05:40.129 UTC [7f6379c8e7c0] Status http_connection_pool.cpp:37: Connection pool will use calculated res
ponse timeout of 550ms
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status httpconnection.h:35: Configuring HTTP Connection
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status httpconnection.h:36:   Connection created for server sprout.2991678f
-f9dd-4098-8fe8-1395c572a307.westeurope.aksapp.io:9888
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status main.cpp:1013: No HSS configured - using Homestead-prov
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status a_record_resolver.cpp:29: Created ARecordResolver
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:266: Configuring store connection
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:267:   Hostname:  cassandra.2991678f-f9dd-4098-8
fe8-1395c572a307.westeurope.aksapp.io
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:268:   Port:      9160
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:296: Configuring store worker pool
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:297:   Threads:   10
15-05-2018 07:05:40.130 UTC [7f6379c8e7c0] Status cassandra_store.cpp:298:   Max Queue: 0
15-05-2018 07:05:40.131 UTC [7f6366ffd700] Status alarm.cpp:244: Reraising all alarms with a known state
15-05-2018 07:05:40.636 UTC [7f6379c8e7c0] Error main.cpp:1056: Failed to initialize the Cassandra store with error co
de 3.
15-05-2018 07:05:40.636 UTC [7f6379c8e7c0] Status main.cpp:1057: Homestead is shutting down

I think the crucial error message is DNS config file /etc/clearwater/dns.json missing

What is that file? Where does it come from?

Impact

Severe: It is not possible to deploy clearwater on kubernetes.

homestead and homestead-prov do not instantiate sucessfully.

Release and environment

Using Azure Kubernetes Service (AKS).

I've enabled HTTP Application Routing, which means that Azure manages DNS for me. The only unusual step is that Azure requires that I add ingresses to do so. The DNS records point to each service (bono, ellis, sprout etc).

Steps to reproduce

Apply pull requests #88 and #89. Then do a kubernetes deployment, on AKS.

I did not do this optional step because I do not think it is necessary for this case.

metaswitch / clearwater-docker Goto Github PK

clearwater-docker's Introduction

Clearwater Docker

Common Preparation

Limitations

Using Compose

Preparation

Starting Clearwater

Scaling the deployment

Using Kubernetes

Prepare the images and edit the yaml files

Alternative configuration for exposing Bono and Ellis services

Deploy Clearwater in Kubernetes

Using kubectl

Using helm

Run live tests against the deployment

Scaling the deployment

Manual Turn-Up

Preparation

Starting Clearwater

Scaling the deployment

Exposed Services

What Next?

Utilities

Cleaning Up

clearwater-docker's People

Contributors

Stargazers

Watchers

Forkers

clearwater-docker's Issues

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Description

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

writing new private key to 'nginx.key'

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment

Steps to reproduce

Symptoms

Impact

Release and environment