From @mikedanese on October 11, 2016 18:25

Example of current output:

root@2f0adb170448:/# kubeadm init --cloud-provider=aws
<cmd/init> cloud provider "aws" initialized for the control plane. Remember to set the same cloud provider flag on the kubelet.
<master/tokens> generated token: "3a83a1.cd513749f90a4514"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2016-10-11 18:20:57 +0000 UTC Not After: 2026-10-09 18:20:57 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2016-10-11 18:20:57 +0000 UTC Not After: 2017-10-11 18:20:57 +0000 UTC
Alternate Names: [192.168.9.3 10.0.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready

I don't know what is important. There's no indentation. There's no color. Let's add some structure and convention to this output.

cc @kubernetes/sig-cluster-lifecycle

Copied from original issue: kubernetes/kubernetes#34559

From @bulletRush on October 21, 2016 7:50

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):
No.

**What keywords did you search in Kubernetes issues before filing this one?:
kubeadm, kubeadm pull, kubeadm image

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:06:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: vmware virtual machine 2c4g
• OS (e.g. from /etc/os-release): centos 7.2
• Kernel (e.g. uname -a): Linux 130 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

Hi, I have implement pre pull image and configurable pods.
so you can:

• pre download kube-apiserver/kcm and so on images before write static manifests.
• you can download image from docker hub instead of "gcr.io". so you can use a docker hub mirror now.
• use custom configuration in pod(not fully test).

To use this feature, you can run kubeadm in this command ./kubeadm init --config=config.json. a short sample about config.json :

{
    "podSpecs": {
        "dummy": {
            "containers": [
                {
                    "image": "barrettwu/pause-amd64:3.0",
                    "imagePullPolicy": "IfNotPresent",
                    "name": "dummy",
                    "resources": {}
                }
            ],
            "securityContext": {
                "hostNetwork": true
            },
            "serviceAccountName": "",
            "volumes": null
        }
    },
    "prePullPods": [ 
        "dummy"
    ]
}

when you run kubeadm by ./kubeadm --config=config.json, the output will show you the image pull process:

[root@130 kubeadm]# ./kubeadm init --config=config.json 
Running pre-flight checks
<master/tokens> generated token: "79284f.3e6b319b0ae9ad6f"
prePullPods: [etcd kube-apiserver kube-scheduler pause kube-controller-manager kube-discovery dummy addonImages]
pre pull image "weaveworks/weave-npc:1.7.2" in pod "addonImages"
1.7.2: Pulling from weaveworks/weave-npc

2aff618222e8: Pull complete 
b6f276ea2686: Downloading [=======================>                           ] 6.106 MB/12.92 MB
a3ed95caeb02: Download complete 
^Ced95caeb02: Pulling fs layer 

If you config kube-apiserver, kube-controller-manager in this config file, the static pod manifests genereted by kubeadm will use podSpec definition in this file.

but I still have some confuse about my implementation:

• what's your idea to implement this feature? my implementation has so much of shortage.
  • conflict with pre-flight check if you use custom pod config.
  • only images listed both in prePullPods and podSpecs can be download.
• should I remove the configurable pod feature to make a pr? because this have a lot of limitation and will cause kubelet failed to start some pod if some static manifest generated rule changed.

Copied from original issue: kubernetes/kubernetes#35264

From @kelseyhightower on October 25, 2016 18:15

Some people are having trouble authenticating to the Kubernetes dashboard when the cluster is built with kubeadm. Seems like the deployed api server does not support basic auth, which makes it impossible to hit the Dashboard UI without using kubectl proxy.

Copied from original issue: kubernetes/kubernetes#35536

From @linfan on October 30, 2016 16:26

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): nodePort, kubeadm

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"5a0a696437ad35c133c0c8493f7e9d22b0f9b81b", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS
• OS (e.g. from ca): Ubuntu 16.04.1 LTS
• Kernel (e.g. uname -a): 4.4.0-31-generic x86_64 GNU/Linux
• Install tools: kubeadm
• Others: single node cluster, with calico network plugin

What happened:

nodePort attribute in pod does not map container port to host.

What you expected to happen:

Yaml file like:

ports:
- containerPort: 80
   hostPort: 80

should expose the container port 80 to host port 80.

How to reproduce it (as minimally and precisely as possible):

Create a pod use below yaml file:

apiVersion: v1
kind: Pod
metadata:
  name: demo
spec:
  containers:
  - name: nginx-demo
    image: nginx
    ports:
    - containerPort: 80
      hostPort: 80

Then use curl <node-ip>:80 try access it, will get Connection refused error.

Check the docker container via docker inspect <container-id>, the container port is actually not exposed:

"HostConfig": {
  "PortBindings": null
}
"Config": {
  "ExposedPorts": {
    "443/tcp": {},
    "80/tcp": {}
  }
}
"NetworkSettings": {
  "Ports": null,
}

Anything else do we need to know:

Issue was found originally because the rc-default.yaml file in ingress example require nodePort.

Copied from original issue: kubernetes/kubernetes#35875

From @errordeveloper on October 3, 2016 11:48

Copied from original issue: kubernetes/kubernetes#33923

From @andersla on October 27, 2016 18:8

When trying to install Kubeadm inside Ubuntu 16.04 docker container it fails.

BUG REPORT

Kubernetes version (use kubectl version):
latest

Environment:
Ubuntu 16.04 Docker container

What happened:
When trying to install Kubeadm inside Ubuntu 16.04 docker container it fails.
My Idea was to use one docker container as master "node" and a second container as a worker "node" (kubernetes in docker)
Is this a systemd issue? (something I came across when "googling" for answers)

Inside Ubuntu 16.04 docker image I install with : apt-get install -y kubeadm

setup log:

...
...
...
all: Setting up socat (1.7.3.1-1) ...
    all: Setting up kubelet (1.4.3-00) ...
    all: /var/lib/dpkg/info/kubelet.postinst: 38: /var/lib/dpkg/info/kubelet.postinst: [[: not found
    all: Setting up kubectl (1.4.3-00) ...
    all: Setting up kubeadm (1.5.0-alpha.0-1534-gcf7301f-00) ...
    all: Failed to connect to bus: No such file or directory
    **all: dpkg: error processing package kubeadm (--configure):**
    all: subprocess installed post-installation script returned error exit status 1
    all: Setting up netcat-traditional (1.10-41) ...
    all: update-alternatives: using /bin/nc.traditional to provide /bin/nc (nc) in auto mode
    all: Setting up netcat (1.10-41) ...
    all: Setting up patch (2.7.5-1) ...
    all: Setting up rename (0.20-4) ...
    all: update-alternatives: using /usr/bin/file-rename to provide /usr/bin/rename (rename) in auto mode
    all: Setting up tcpd (7.6.q-25) ...
    all: Setting up ubuntu-fan (0.9.1) ...
    all: invoke-rc.d: could not determine current runlevel
    all: invoke-rc.d: policy-rc.d denied execution of start.
    all: Setting up xz-utils (5.1.1alpha+20120614-2ubuntu2) ...
    all: update-alternatives: using /usr/bin/xz to provide /usr/bin/lzma (lzma) in auto mode
    all: Setting up python3 (3.5.1-3) ...
    all: running python rtupdate hooks for python3.5...
    all: running python post-rtupdate hooks for python3.5...
    all: Setting up apparmor (2.10.95-0ubuntu2.2) ...
    all: update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
    all: Setting up dh-python (2.20151103ubuntu1.1) ...
    all: Processing triggers for libc-bin (2.23-0ubuntu4) ...
    all: Processing triggers for systemd (229-4ubuntu11) ...
    all: Processing triggers for initramfs-tools (0.122ubuntu8.5) ...
    all: Processing triggers for dbus (1.10.6-1ubuntu3) ...
    all: Errors were encountered while processing:
    all: kubeadm
    all: E: Sub-process /usr/bin/dpkg returned an error code (1)
==> all: Killing the container: 93babb5045461c343a803109ba683a2acf68f1f453447a336b09171a1b190f38
Build 'all' errored: Script exited with non-zero exit status: 100

==> Some builds didn't complete successfully and had errors:
--> all: Script exited with non-zero exit status: 100

Copied from original issue: kubernetes/kubernetes#35712

From @bruceauyeung on October 9, 2016 2:32

currently kubeadm command provides a command join to join a cluster. but there isn't a command to leave that cluster. maybe kubeadm leave is suitable for this circumstance

kubeadm version

version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Copied from original issue: kubernetes/kubernetes#34404

From @errordeveloper on September 29, 2016 11:22

It may take quite long for API server to come up, as this involves pulling Docker images in the background.

We have already discussed adding a progress bar, and it'd would be great. Also, we should consider pulling directly from kubeadm, so we can provide user with a real progress bar. Also, we could consider adding kubeadm util pull-images and allow alternative registries, as it would help users in air-gapped environments.

Copied from original issue: kubernetes/kubernetes#33744

From @errordeveloper on September 30, 2016 10:30

We should allow users to use their own private registry instead of GCR. We should also make it clear what images need to be mirrored, ideally we should implement a helper for doing pull/retag/push (i.e. kubeadm util mirror-images), as we have a handful of images and users will have to write scripts to do it. People will have to mirror images continuously, when we provide upgrades, so helper would be very handy.

Copied from original issue: kubernetes/kubernetes#33828

From @errordeveloper on October 4, 2016 17:14

Things like service CIDR and pod network CIDR, cluster DNS domain are useful for network and other similar type of add-ons, also there has been some ideas around general accessibility of key parameters. Other example may include cloud provider name, CNI config path and other bits of information that are scatered in kubelet, kube-apiserver and kube-controller-manager flags. Let's discussed this more specifically.

Copied from original issue: kubernetes/kubernetes#34025

From @danielschonfeld on October 28, 2016 10:49

Wanted to make a wish. Being a long time kubernetes user and having stitched clusters by hand, I am convinced kubeadm is the best thing that happened to kubernetes in terms of mass adoption. I think it would be nice if we could do version upgrades with kubeadm. I imagine it wouldn't be too hard needing only to update the versions in the manifests and download a new version of kubelet?

Is there something else I'm missing? Could this become a reality?

Copied from original issue: kubernetes/kubernetes#35779

From @ndtreviv on November 17, 2016 12:13

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

• kubectl join
• preflight
• kubelet is not empty

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  AWS EC2 Classic ami-45b69e52 m3.2xlarge instance type in us-east-1 region.

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux ip-10-159-146-253 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
  N/A
• Others:
  N/A

What happened:
I was standing up a 3-node cluster following the instructions here: http://kubernetes.io/docs/getting-started-guides/kubeadm/

Commands I ran on each slave:

sudo su -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y docker.io
apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Additional commands run on the master:

kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f flannel.yml

flannel.yml:

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: quay.io/coreos/flannel-git:v0.6.1-28-g5dde68d-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

ssh-d onto slave node and ran - as per the documentation - (anonymised to protect the not-so-innocent):

sudo su - 
kubeadm join --token=[TOKEN] [MASTER IP]

Output was:

Running pre-flight checks
preflight check errors:
	/var/lib/kubelet is not empty

What you expected to happen:
I expected the node to join the cluster and output data according to the docs: http://kubernetes.io/docs/getting-started-guides/kubeadm/

As I understand the architecture, each node has to have a kubelet on it. I have a feeling that some preflight checks have been added for something else, reused for this and aren't quite appropriate. I may be wrong, though!

How to reproduce it (as minimally and precisely as possible):
Create three AWS EC2 Classic ami-45b69e52 m3.2xlarge instance type in us-east-1 region.
Run the commands as I did.

Anything else do we need to know:
No.

Copied from original issue: kubernetes/kubernetes#36987

From @errordeveloper on October 3, 2016 12:0

It'd help to implement dry-run mode, tarball export or other things.

Copied from original issue: kubernetes/kubernetes#33926

From @kilmarnock on November 15, 2016 17:33

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kubernetes/kubernetes#33729

they have a different bug

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
This is a BUG report

Kubernetes version (use kubectl version):
1.4.4
kubeadm init --use-kubernetes-version v1.4.6
kubeadm init --use-kubernetes-version v1.4.1
etc.

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  ubuntu 16.04
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

the containers stay in pause mode: docker.ps shows (sorry, i gave my best for table layout)
COMMAND | NAMES
------------------ | ---------------------
"kube-controller-mana" | k8s_kube-controller-manager.
"kube-scheduler --v=2" | k8s_kube-scheduler.
"/pause" | k8s_POD.d8dbe16c_kube-controller-manager-
"/pause" | k8s_POD.d8dbe16c_kube-apiserver-
"/pause" | k8s_POD.d8dbe16c_etcd-
"/pause" | k8s_POD.d8dbe16c_kube-scheduler

What you expected to happen:
start the api container at port 8080 / 6443

How to reproduce it (as minimally and precisely as possible):

I just followed the tutorials.
Anything else do we need to know:

I have nothing listening at port 8080 / 443 / 6443. I do not know which parts of the syslog could be helpful. The first part is here:
http://pastebin.com/TUk1qsms

then come a lot of xy.service not handled by systemd

and then the useless attempts to connect to the (paused) node:

26360 manager.go:843] ignoring container "/system.slice/dev-hugepages.mount"
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.800147 26360 conversion.go:133] failed to handle multiple devices for container. Skipping Filesystem stats
Nov 15 18:15:52 laptop kubelet[26360]: E1115 18:15:52.825236 26360 eviction_manager.go:162] eviction manager: unexpected err: failed GetNode: node 'laptop.tu' not found
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.849907 26360 kubelet_node_status.go:203] Setting node annotation to enable volume controller attach/detach
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854325 26360 kubelet_node_status.go:354] Recording NodeHasSufficientDisk event message for node laptop.tu
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854354 26360 kubelet_node_status.go:354] Recording NodeHasSufficientMemory event message for node laptop.tu
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854367 26360 kubelet_node_status.go:354] Recording NodeHasNoDiskPressure event message for node laptop.tu
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854385 26360 kubelet_node_status.go:73] Attempting to register node laptop.tu
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854410 26360 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"laptop.tu", UID:"laptop.tu", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientDisk' Node laptop.tu status is now: NodeHasSufficientDisk
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854466 26360 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"laptop.tu", UID:"laptop.tu", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientMemory' Node laptop.tu status is now: NodeHasSufficientMemory
Nov 15 18:15:52 laptop kubelet[26360]: I1115 18:15:52.854594 26360 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"laptop.tu", UID:"laptop.tu", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasNoDiskPressure' Node laptop.tu status is now: NodeHasNoDiskPressure
Nov 15 18:15:52 laptop kubelet[26360]: E1115 18:15:52.854691 26360 kubelet_node_status.go:97] Unable to register node "laptop.tu" with API server: Post https://192.168.0.2:6443/api/v1/nodes: dial tcp 192.168.0.2:6443: getsockopt: connection refused

It does this for the next couple of hours. Then I kick my laptop out of the window. Usually this stops the entrys.

What can I do? Where is the one parameter to rule the stuff? Do you need more information?

Copied from original issue: kubernetes/kubernetes#36825

From @samnissen on October 13, 2016 13:39

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
Possibly the same issue as #33729 -- different OS.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:
  privately hosted 4-core, 4gb VM
• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"

• Kernel (e.g. uname -a):

Linux abc-1234 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

What happened:
I followed this guide precisely: http://kubernetes.io/docs/getting-started-guides/kubeadm/

At the step "(2/4) Initializing your master" the command kubeadm init stops at the line:

<master/apiclient> created API client, waiting for the control plane to become ready

and never moves forward.

What you expected to happen:
The command would proceed as detailed in the guide.

How to reproduce it (as minimally and precisely as possible):
Follow the guide for CentOS 7 exactly.

Anything else do we need to know

Copied from original issue: kubernetes/kubernetes#34720

From @errordeveloper on October 3, 2016 11:38

Right now we do it an ad-host fashion, but should do it properly instead, may be we just need to make a PR to cobra.

Copied from original issue: kubernetes/kubernetes#33921

From @codablock on November 3, 2016 10:52

What keywords did you search in Kubernetes issues before filing this one?
controller /etc/ssl/certs
kubeadm /etc/ssl/certs

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):
1.5 custom built master
I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ...

Environment:

• Cloud provider or hardware configuration: azure
• OS (e.g. from /etc/os-release): CentOS 7.2
• Kernel (e.g. uname -a): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others: deployed with Azure RM template and ansible

What happened:
controller is not able to do https, including calls to the Azure RM API

What you expected to happen:
https should work as usual

How to reproduce it (as minimally and precisely as possible):
Use kubeadm to install kubernetes on CentOS
exec into the controller and try "curl https://google.com"

Anything else do we need to know:
It looks like kubernetes/kubernetes#33681 caused this issue. On CentOS, I see these files on the host OS:

[devops@ma-kub8ms0 ~]$ ls -lah /etc/ssl/certs/
total 12K
drwxr-xr-x. 2 root root  112 Oct 26 20:48 .
drwxr-xr-x. 5 root root   76 Oct 26 20:47 ..
lrwxrwxrwx. 1 root root   49 Oct 26 20:47 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Oct 26 20:47 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rwxr-xr-x. 1 root root  610 Sep 27 13:40 make-dummy-cert
-rw-r--r--. 1 root root 2.4K Sep 27 13:40 Makefile
-rwxr-xr-x. 1 root root  829 Sep 27 13:40 renew-dummy-cert

I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image.

I also tried to verify this with gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4. It seems to not be based on Debian and thus gives a different error:

[devops@ma-kub8ms0 ~]$ docker run -ti --rm -v /etc/ssl/certs:/etc/ssl/certs gcr.io/google_containers/kube-controller-manager-amd64:v1.4.4 sh
/ # wget https://google.de
Connecting to google.de (172.217.21.227:443)
wget: can't execute 'ssl_helper': No such file or directory
wget: error getting response: Connection reset by peer

I see 2 possible solutions:

1. Instead of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image. This is already done for hyperkube and would work out of the box when the host mount would be omitted.
2. kubeadm could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required.

I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.

Copied from original issue: kubernetes/kubernetes#36150

From @errordeveloper on October 17, 2016 7:39

<cmd/init> Running pre-flight checks
<preflight/checks> WARNING: kubelet service is not enabled, please run 'systemctl enable kubelet.service'

kubeadm: I am an alpha version, my authors welcome your feedback and bug reports
kubeadm: please create issue an using https://github.com/kubernetes/kubernetes/issues/new
kubeadm: and make sure to mention @kubernetes/sig-cluster-lifecycle. Thank you!
<cmd/init> preflight check error
  count: 2 
  msg: 
    ebtables not found in system path
    ethtool not found in system path

Firstly, it has some typos. Secondly, it doesn't tell the user how to skip the checks.

I'd like it to look more like this:

<cmd/init> running pre-flight checks
<preflight/checks> WARNING: kubelet service is not enabled, please run 'systemctl enable kubelet.service'

kubeadm: I am an alpha version, my authors welcome your feedback and bug reports
kubeadm: please create an issue using https://github.com/kubernetes/kubernetes/issues/new
kubeadm: and make sure to mention @kubernetes/sig-cluster-lifecycle. Thank you!
<cmd/init> preflight check error - 2 check(s) failed:
    - ebtables not found in system path
    - ethtool not found in system path
<cmd/init> if you know what you are doing, you can skip pre-flight checks with `--skip-preflight`

Copied from original issue: kubernetes/kubernetes#34927

From @luxas on November 4, 2016 19:22

ref: http://stackoverflow.com/questions/40294423/kubeadm-why-does-my-node-not-show-up-though-kubelet-says-it-joined/40416796#40416796

@kubernetes/sig-cluster-lifecycle

Copied from original issue: kubernetes/kubernetes#36255

From @michelleN on October 27, 2016 21:52

Not sure if this is a bug or a feature.

What happened:
I configured two GCE nodes with Ubuntu 16.04 images. I used kubeadm's getting started guide to make one node the master. I then used the kubeadm join command using the token from my master on my other node. I tried to run kubectl get nodes but got
The connection to the server localhost:8080 was refused - did you specify the right host or port?.

root@michelle-testing-uz4v:~# kubectl config view
apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

To fix this problem I had to set my $KUBECONFIG to /etc/kubernetes/kubelet.conf
However, I didn't have to do that on master and my kubectl worked fine.

What you expected to happen:
I expected my non-master node to work as the getting started guide pointed out. Am I doing something wrong here? I understand that to control your cluster from your laptop, you have to copy your KubeConfig. I thought that since I already have the config on my non-master node as a result of kubeadm join that I wouldn't have to set the env variable. I'm a bit confused on what to expect here.

How to reproduce it (as minimally and precisely as possible):
Run a kubeadm join (with token) on a non-master node and run kubectl get nodes

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", GitCommit:"4957b090e9a4f6a68b4a40375408fdc74
a212260", GitTreeState:"clean", BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linu
x/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675c
d97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linu
x/amd64"}

Environment:

• Cloud provider or hardware configuration: GCE
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a): Linux michelle-testing-uz4v 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Li
  nux
• Install tools: kubeadm
• Others:

Copied from original issue: kubernetes/kubernetes#35729

From @jberkus on October 28, 2016 20:9

Neither Kubeadm reference nor kubeadm getting-started-guide contain a list of ports which need to be open within the cluster for a kubeadm deployment to function. Kubeadm appears to require additional port access beyond what older kubernetes does. This causes a lot of troubleshooting pain in getting things working while exercising reasonable port security.

For my part, I was only able to get it to work once I threw open all TCP ports in the cluster; there are apparently ports required which don't show up in the list of "listening" ports on the master.

The list of ports and port ranges should be documented.

Copied from original issue: kubernetes/kubernetes#35818

From @errordeveloper on October 3, 2016 11:31

It could prevent MITM, where attacker has guessed the JWS token, yet hasn't provided the same CA certificate.

Copied from original issue: kubernetes/kubernetes#33918

From @luxas on October 27, 2016 20:16

We want to default to etcd3 in etcd2 mode for kubeadm now I think, and eventually default to etcd3 in etcd3 mode as well (I suppose that will be the default for v1.5?)

I don't think we have to deal with migration and upgrades from etcd2 to etcd3, as we're in alpha.
We can just say: create a new cluster.

Of course, when we're in beta, etc. we can't do that, but I think it's best to "fully support" etcd3 and higher when we're that far.

WDYT?
@kubernetes/sig-cluster-lifecycle @wojtek-t @lavalamp

Copied from original issue: kubernetes/kubernetes#35723

From @pires on October 27, 2016 17:54

Currently, one kubeadm {init,join} can only be configured through flags or configuration file. So:

kubeadm init --config master.yaml

or

kubeadm init <flags>

Both configuration options don't coexist as users may expect it to (#35424).

Copied from original issue: kubernetes/kubernetes#35710

From @mikedanese on October 18, 2016 22:47

The token discovery interface is not properly factored to extend to the other two discovery methods. Discovery should return a kubeconfig. TLS bootstrap should accept a kubeconfig. The internals of the token discovery should be hidden.

cc @dgoodwin @kubernetes/sig-cluster-lifecycle

Copied from original issue: kubernetes/kubernetes#35073

From @axsuul on October 7, 2016 7:57

I provisioned my kubernetes cluster using kubeadm but I would like to change the option to --secure-port=8081 for the API server. It seems like changing this manifest file

/etc/kubernetes/manifests/kube-apiserver.json

would give me what I want but how would I apply the changes? Running

# kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.json

doesn't appear to update it properly.

Copied from original issue: kubernetes/kubernetes#34307

From @pbecotte on October 28, 2016 18:9

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

Environment:

• AWS
• Xenial
• Terraform

What happened:
I am setting up a K8s cluster using Terraform + kubeadm in AWS. Etcd, kube master, and all kube nodes are in auto-scaling groups and behind load balancers, with the idea being that if some or all nodes fail, the cluster will heal itself. Most recently, I have been working to get the aws cloud-provider to work. At one point, I corrected my kubelet systemd file and deleted the kmaster and knode instances. When they came back up, the discovery service failed to start on the master node.

<master/discovery> failed to create "kube-discovery" deployment [deployments.extensions "kube-discovery" already exists]

With the existing deployment, I was surprised that a new discovery pod wasn't scheduled, but it stayed in pending.

root@ip-10-253-128-238:~# kubectl describe pod/kube-discovery-1150918428-aca0v --namespace=kube-system
Name:       kube-discovery-1150918428-aca0v
Namespace:  kube-system
Node:       /
Labels:     component=kube-discovery
        k8s-app=kube-discovery
        kubernetes.io/cluster-service=true
        name=kube-discovery
        pod-template-hash=1150918428
        tier=node
Status:     Pending
IP:     
Controllers:    ReplicaSet/kube-discovery-1150918428
Containers:
  kube-discovery:
    Image:  gcr.io/google_containers/kube-discovery-amd64:1.0
    Port:   9898/TCP
    Command:
      /usr/local/bin/kube-discovery
    Volume Mounts:
      /tmp/secret from clusterinfo (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-lhygh (ro)
    Environment Variables:  <none>
Conditions:
  Type      Status
  PodScheduled  False 
Volumes:
  clusterinfo:
    Type:   Secret (a volume populated by a Secret)
    SecretName: clusterinfo
  default-token-lhygh:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-lhygh
QoS Class:  BestEffort
Tolerations:    dedicated=master:NoSchedule
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  59m       20s     147 {default-scheduler }            Warning     FailedScheduling    pod (kube-discovery-1150918428-aca0v) failed to fit in any node
fit failure on node (ip-10-253-128-238.ec2.internal): MatchNodeSelector

What you expected to happen:
Either for kubeadm to delete the existing deployment, or for that deployment to sucessfully schedule new pods on the new master node.

How to reproduce it (as minimally and precisely as possible):
Bring up a cluster using kubeadm and external etcd. After the cluster is up, delete the master node and try to bring it back up (using the same command as the first time).

Copied from original issue: kubernetes/kubernetes#35807

From @spacexnice on October 25, 2016 8:17

Support for adding extra Kubelet config

kubeadm should support for adding extra kubelet configuration while doing kubeadm init. i want to add an extra [ --pod-infra-container-image] parameter to kubelet cmdline ,but cant find a parameter doing so.

Copied from original issue: kubernetes/kubernetes#35502

From @luxas on October 26, 2016 16:51

I'm leaning towards yes, because the dashboard is an integrated part of k8s and our focus is on good UX and an UI is great for that purpose.

cc @kubernetes/sig-cluster-lifecycle

Copied from original issue: kubernetes/kubernetes#35642

From @errordeveloper on October 3, 2016 11:36

Copied from original issue: kubernetes/kubernetes#33920

From @tonyclam on November 1, 2016 23:56

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Features Request

Kubernetes version (use kubectl version):
[root@localhost ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1877+3f4a66f3d6892b-dirty", GitCommit:"3f4a66f3d6892b8d8831a8a60b91fd1afbefee4d", GitTreeState:"dirty", BuildDate:"2016-10-31T20:47:46Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

[root@localhost ~]# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:x89_64
• OS (e.g. from /etc/os-release):Ubuntu 16.04
• Kernel (e.g. uname -a): Linux kubenode01 4.4.0-38-generic #57-Ubuntu SMP Tue Sep 6 15:42:33 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:kubeadm
• Others:

What happened:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else do we need to know:

Copied from original issue: kubernetes/kubernetes#36026

From @errordeveloper on October 18, 2016 14:55

As of #34607, we have the following code:

if len(s.API.AdvertiseAddresses) > 0 {
    command = append(command, fmt.Sprintf("--advertise-address=%s", s.API.AdvertiseAddresses[0]))
}

It turns out that the condition will always be true, which was not what I desired when I wrote #34607, however it may be what we want, let's discuss here.

When I wrote this, I have overlooked the fact that when kubeadm auto-detects the primary IP , it just sets it in s.API.AdvertiseAddresses and doesn't indicate that it's an auto-detected IP.

I may be desirable to leave API server's --advertise-address flag unset in the case when kubeadm auto-detects the IP and only set it when user has explicitly specified --api-advertise-addresses (and document clearly that only first value will be used for advertising internally, i.e. what kubernetes service IP will get pointed at). This way we can leave API server to us it's own logic to set address it should advertise.

Alternatively, we can keep having strong preference here, which may be a good thing. But it will probably conflict with --bind-address, whenever we decide to add support for it.

Initially, --api-advertise-addresses flag was only mean for certificates and completely independent of --bind-address, but that was simply about picking less preferences. We also still need to have the discussion about --bind-address as well, I think the main requirement there is to allow user to have any kind of external load-balancer, DNS or whatever NAT situation they might have...

ATTN: @kubernetes/sig-cluster-lifecycle

Copied from original issue: kubernetes/kubernetes#35043

From @errordeveloper on October 25, 2016 17:38

Some users reported this, and it looks like disabling it solves the problem, but I'd like to find out what rules need to be added to make things work. I've been able to make most things work (see errordeveloper/kubernetes-ansible-vagrant@67a4d87), but lacking pod to master right now.

Copied from original issue: kubernetes/kubernetes#35535

From @jimmyjones2 on October 9, 2016 20:42

CentOS 7.1.1503 minimal VM, using kubeadm guide with weave networking. kube-dns has issues starting up:

I1009 20:09:37.093825       1 dns.go:173] Ignoring error while waiting for service default/kubernetes: Get https://100.64.0.1:443/api/v1/namespaces/default/services/kubernetes: dial tcp 100.64.0.1:443: getsockopt: no route to host. Sleeping 1s before retrying.
E1009 20:09:39.098145       1 reflector.go:214] pkg/dns/dns.go:155: Failed to list *api.Endpoints: Get https://100.64.0.1:443/api/v1/endpoints?resourceVersion=0: dial tcp 100.64.0.1:443: getsockopt: no route to host
I1009 20:09:39.522095       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.2:48292->192.168.1.1:53: i/o timeout"

Disabling firewalld resolves the issue.

Copied from original issue: kubernetes/kubernetes#34420

From @luxas on October 5, 2016 19:26

We should use Kubelet Dynamic Settings by pushing values given to kubeadm init to the apiserver when started, and that will make all kubelets in the cluster reload and pick up those configuration values.

That's much better than having to modify every 10-kubeadm.conf systemd dropin file in the whole cluster.

Copied from original issue: kubernetes/kubernetes#34132

From @dragoscirjan on October 22, 2016 21:17

Hi,

I'm trying to build a test environment which in the end will be deployed to our production server; however I encountered some issues that, for now, I consider bugs.

Please find my install scripts @ https://github.com/itmcdev/solaris.

Following the valid code from provision.sh, I followed the instructions from http://kubernetes.io/docs/getting-started-guides/kubeadm/ running:

# for master
kubeadm init --api-advertise-addresses=10.0.3.10

# for workers
kubeadm join --token e36420.92c6e1d4959fb8eb 10.0.3.10

# than, for master
kubectl apply -f https://git.io/weave-kube

Following this, I noticed that kube-dns and weave-kube images were persisting in ContainerCreating and CrashLoopBackOff statuses. I will let you decide further, whether this is a bug, or a mistake generated by me. To note, I'm a total noob under the process of learning.

root@solaris-master:~# kubectl get nodes
NAME               STATUS    AGE
solaris-master     Ready     52m
solaris-worker-1   Ready     50m
solaris-worker-2   Ready     50m

root@solaris-master:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                     READY     STATUS              RESTARTS   AGE
kube-system   etcd-solaris-master                      1/1       Running             0          51m
kube-system   kube-apiserver-solaris-master            1/1       Running             0          52m
kube-system   kube-controller-manager-solaris-master   1/1       Running             0          52m
kube-system   kube-discovery-982812725-wm3o2           1/1       Running             0          29m
kube-system   kube-dns-2247936740-ubzbm                0/3       ContainerCreating   0          51m
kube-system   kube-proxy-amd64-3jabb                   1/1       Running             0          50m
kube-system   kube-proxy-amd64-rt2f9                   1/1       Running             0          50m
kube-system   kube-proxy-amd64-ygc2p                   1/1       Running             0          51m
kube-system   kube-scheduler-solaris-master            1/1       Running             0          51m
kube-system   weave-net-4c3a4                          1/2       CrashLoopBackOff    11         37m
kube-system   weave-net-yvv71                          2/2       Running             0          37m
kube-system   weave-net-z9aqe                          1/2       CrashLoopBackOff    11         37m

root@solaris-master:~# kubectl describe pod weave-net-4c3a4 --namespace=kube-system
Name:           weave-net-4c3a4
Namespace:      kube-system
Node:           solaris-worker-1/10.0.2.15
Start Time:     Sat, 22 Oct 2016 16:33:32 -0400
Labels:         name=weave-net
Status:         Running
IP:             10.0.2.15
Controllers:    DaemonSet/weave-net
Containers:
  weave:
    Container ID:       docker://a254dce85d8a027d9d3de1de080ca6322806e05fda2bc72a87f1d5b3cc936f35
    Image:              weaveworks/weave-kube:1.7.2
    Image ID:           docker://sha256:9a2aa48020f51f3bc736e63a888441979dbcba394debdc1abb8e43cf95449168
    Port:
    Command:
      /home/weave/launch.sh
    Requests:
      cpu:              10m
    State:              Waiting
      Reason:           CrashLoopBackOff
    Last State:         Terminated
      Reason:           Error
      Exit Code:        1
      Started:          Sat, 22 Oct 2016 17:11:32 -0400
      Finished:         Sat, 22 Oct 2016 17:11:32 -0400
    Ready:              False
    Restart Count:      12
    Liveness:           http-get http://127.0.0.1:6784/status delay=30s timeout=1s period=10s #success=1 #failure=3
    Volume Mounts:
      /etc from cni-conf (rw)
      /host_home from cni-bin2 (rw)
      /opt from cni-bin (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-eejd4 (ro)
      /weavedb from weavedb (rw)
    Environment Variables:      <none>
  weave-npc:
    Container ID:       docker://2395225b64692a0a4caf63903ec5cd4ca55b0fc60989623c165a747acd456068
    Image:              weaveworks/weave-npc:1.7.2
    Image ID:           docker://sha256:63a7347dde435cccc849d0eacc1a11c17733b6e557e67ce56efd8264f11e5d8c
    Port:
    Requests:
      cpu:              10m
    State:              Running
      Started:          Sat, 22 Oct 2016 16:34:49 -0400
    Ready:              True
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-eejd4 (ro)
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  weavedb:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  cni-bin:
    Type:       HostPath (bare host directory volume)
    Path:       /opt
  cni-bin2:
    Type:       HostPath (bare host directory volume)
    Path:       /home
  cni-conf:
    Type:       HostPath (bare host directory volume)
    Path:       /etc
  default-token-eejd4:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-eejd4
QoS Class:      Burstable
Tolerations:    dedicated=master:Equal:NoSchedule
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath                   Type            Reason          Message
  ---------     --------        -----   ----                            -------------                   --------        ------          -------
  38m           38m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Pulling         pulling image "weaveworks/weave-kube:1.7.2"
  37m           37m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Pulled          Successfully pulled image "weaveworks/weave-kube:1.7.2"
  37m           37m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Created         Created container with docker id 06f090f2775f; Security:[seccomp=unconfined]
  37m           37m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Started         Started container with docker id 06f090f2775f
  37m           37m             1       {kubelet solaris-worker-1}      spec.containers{weave-npc}      Normal          Pulling         pulling image "weaveworks/weave-npc:1.7.2"
  36m           36m             1       {kubelet solaris-worker-1}      spec.containers{weave-npc}      Normal          Pulled          Successfully pulled image "weaveworks/weave-npc:1.7.2"

  36m           36m             1       {kubelet solaris-worker-1}      spec.containers{weave-npc}      Normal          Created         Created container with docker id 2395225b6469; Security:[seccomp=unconfined]
  36m           36m             1       {kubelet solaris-worker-1}      spec.containers{weave-npc}      Normal          Started         Started container with docker id 2395225b6469
  36m           36m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Created         Created container with docker id 8c4256f19b85; Security:[seccomp=unconfined]
  36m           36m             1       {kubelet solaris-worker-1}      spec.containers{weave}          Normal          Started         Started container with docker id 8c4256f19b85
  36m           36m             3       {kubelet solaris-worker-1}                                      Warning         FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 10s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  36m   36m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id 8bba1d69d16b
  36m   36m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id 8bba1d69d16b; Security:[seccomp=unconfined]
  36m   36m     2       {kubelet solaris-worker-1}                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 20s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  36m   36m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id 88c6006d13c3
  36m   36m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id 88c6006d13c3; Security:[seccomp=unconfined]
  36m   35m     5       {kubelet solaris-worker-1}                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 40s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  35m   35m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id 06dad5fefd0c; Security:[seccomp=unconfined]
  35m   35m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id 06dad5fefd0c
  35m   34m     7       {kubelet solaris-worker-1}                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 1m20s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  34m   34m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id d8e315b09473; Security:[seccomp=unconfined]
  34m   34m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id d8e315b09473
  33m   31m     14      {kubelet solaris-worker-1}                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 2m40s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  31m   31m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id 49ca69fe4aa7; Security:[seccomp=unconfined]
  31m   31m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id 49ca69fe4aa7
  26m   26m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         Created container with docker id 5f4f6e79bf1b; Security:[seccomp=unconfined]
  26m   26m     1       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         Started container with docker id 5f4f6e79bf1b
  20m   16s     5       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Started         (events with common reason combined)
  20m   16s     5       {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Created         (events with common reason combined)
  36m   16s     12      {kubelet solaris-worker-1}      spec.containers{weave}  Normal  Pulled          Container image "weaveworks/weave-kube:1.7.2" already present on machine
  31m   6s      142     {kubelet solaris-worker-1}                              Warning FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=weave pod=weave-net-4c3a4_kube-system(cc47b290-9896-11e6-b6af-080027d4fd28)"

  36m   6s      173     {kubelet solaris-worker-1}      spec.containers{weave}  Warning BackOff Back-off restarting failed docker container

root@solaris-master:~# kubectl logs weave-net-4c3a4 --namespace=kube-system
Error from server: a container name must be specified for pod weave-net-4c3a4, choose one of: [weave weave-npc]

root@solaris-master:~# kubectl describe pod kube-dns-2247936740-ubzbm --namespace=kube-system
Name:           kube-dns-2247936740-ubzbm
Namespace:      kube-system
Node:           solaris-master/10.0.2.15
Start Time:     Sat, 22 Oct 2016 16:19:37 -0400
Labels:         component=kube-dns
                k8s-app=kube-dns
                kubernetes.io/cluster-service=true
                name=kube-dns
                pod-template-hash=2247936740
                tier=node
Status:         Pending
IP:
Controllers:    ReplicaSet/kube-dns-2247936740
Containers:
  kube-dns:
    Container ID:
    Image:              gcr.io/google_containers/kubedns-amd64:1.7
    Image ID:
    Ports:              10053/UDP, 10053/TCP
    Args:
      --domain=cluster.local
      --dns-port=10053
    Limits:
      cpu:      100m
      memory:   170Mi
    Requests:
      cpu:              100m
      memory:           170Mi
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Liveness:           http-get http://:8080/healthz delay=60s timeout=5s period=10s #success=1 #failure=1
    Readiness:          http-get http://:8081/readiness delay=30s timeout=5s period=10s #success=1 #failure=3
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-eejd4 (ro)
    Environment Variables:      <none>
  dnsmasq:
    Container ID:
    Image:              gcr.io/google_containers/kube-dnsmasq-amd64:1.3
    Image ID:
    Ports:              53/UDP, 53/TCP
    Args:
      --cache-size=1000
      --no-resolv
      --server=127.0.0.1#10053
    Limits:
      cpu:      100m
      memory:   170Mi
    Requests:
      cpu:              100m
      memory:           170Mi
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-eejd4 (ro)
    Environment Variables:      <none>
  healthz:
    Container ID:
    Image:              gcr.io/google_containers/exechealthz-amd64:1.1
    Image ID:
    Port:               8080/TCP
    Args:
      -cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:53 >/dev/null && nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null
      -port=8080
      -quiet
    Limits:
      cpu:      10m
      memory:   50Mi
    Requests:
      cpu:              10m
      memory:           50Mi
    State:              Waiting
      Reason:           ContainerCreating
    Ready:              False
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-eejd4 (ro)
    Environment Variables:      <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  default-token-eejd4:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-eejd4
QoS Class:      Guaranteed
Tolerations:    dedicated=master:NoSchedule
Events:
  FirstSeen     LastSeen        Count   From                            SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                            -------------   --------        ------          -------
  55m           55m             1       {default-scheduler }                            Normal          Scheduled       Successfully assigned kube-dns-2247936740-ubzbm to solaris-master
  55m           39m             694     {kubelet solaris-master}                        Warning         FailedSync      Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-2247936740-ubzbm_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-2247936740-ubzbm_kube-system(dac6ff63-9894-11e6-b6af-080027d4fd28)\" using network plugins \"cni\": cni config unintialized; Skipping pod"

root@solaris-master:~# kubectl logs kube-dns-2247936740-ubzbm --namespace=kube-system
Error from server: a container name must be specified for pod kube-dns-2247936740-ubzbm, choose one of: [kube-dns dnsmasq healthz]

Copied from original issue: kubernetes/kubernetes#35378

From @superdump on October 24, 2016 9:56

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): PKI

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
FEATURE REQUEST

The kubernetes API server can now act as a PKI to service Certificate Signing Requests for kubelets and set up secured and authenticated TLS communications between the API server and kubelets. kubeadm makes use of this API endpoint. In order to provide this endpoint, the CA key must be present on the master (or API server) to be able to sign certificates.

It should be clear that having a CA key, or more clearly the ability to sign new certificates, as a permanent part of the cluster could be considered weaker security than not. If the cluster gets hacked, one admittedly already has problems, but in terms of security it is common to think in terms of having layers of security so that if one barrier gets broken, there are still more to overcome.

It could make sense then to support somehow provisioning certificates in a way that the cluster does not retain the capability to provision new certificates while a cluster operator/administrator is not actively managing the cluster, so to speak. This could be in the form of kubeadm using a temporary token to interact with an external PKI such that when the kubeadm provisioning session is completed, no further certificates can be provisioned from the PKI with that token. Something along those lines.

Kubernetes version (use kubectl version):

1.4.3

Environment:

• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: kubeadm
• Others: N/A

What happened:
kubeadm generated a CA key on the master and retained it. kubelets sent their CSRs to the master API server in order to provision certificates.

What you expected to happen:
I expected the above to happen, but the proposal is that if security is particularly important, the CA key and ability to provision new TLS certificates does not remain part of the cluster but is a separate concern that is separately managed.

How to reproduce it (as minimally and precisely as possible):
Use kubeadm to deploy a cluster.

Anything else do we need to know:

Copied from original issue: kubernetes/kubernetes#35413

From @bogdando on October 19, 2016 16:32

FEATURE REQUEST:

Add runtime/config feature flags for granular control of kubeadm init tasks. Like:

• etcd clustering,
• certs management,
• container images pulling,
• configs generation,
• static pods definitions for system components,
• essential addons installation.

Use case:
One may want to use hyperkube and do not pull images for apiserver and other components.
Or one may want to only run certs management tasks, or only re-generate configs or static manifests pods or only install addons. Having feature flags eases idempotent runs, co-existing with external CM systems like ansible or puppet and integration testing efforts a lot.

The feature flags, either CLI or config file based, should ease integration efforts and make the tool more flexible as well.

Related feature #11

Kubernetes version (use kubectl version): next

Environment:

• Cloud provider or hardware configuration: any
• OS (e.g. from /etc/os-release): any
• Kernel (e.g. uname -a): any
• Install tools: n/a
• Others: n/a

Copied from original issue: kubernetes/kubernetes#35133

From @vganapathy1 on October 26, 2016 6:50

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:13:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

`root@Ubuntu14041-mars08:/# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.1.409+714f816a349e79", GitCommit:"714f816a349e7978bc93b35c67ce7b9851e53a6f", GitTreeState:"clean", BuildDate:"2016-10-17T13:01:29Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
`

Environment:
VMware vCloud Air VM

• Cloud provider or hardware configuration:
  7.797 GiB RAM, 2 CPUs each
• OS (e.g. from /etc/os-release):

 NAME="Ubuntu"
 VERSION="16.04.1 LTS (Xenial Xerus)"

• Kernel (e.g. uname -a):
  Kernel Version: 4.4.0-43-generic
• Install tools:
• Others:

What happened:
We used kubeadm and the procedure in Installing Kubernetes on Linux with kubeadm and for the most part the installation went well.

weve-cube installation failed with the Peer name collision,

INFO: 2016/10/26 05:24:32.585405 ->[10.63.33.46:6783] attempting connection INFO: 2016/10/26 05:24:32.587778 ->[10.63.33.46:6783|72:47:96:69:16:bb(Ubuntu14041-mars09)]: connection shutting down due to error: local "72:47:96:69:16:bb(Ubuntu14041-mars09)" and remote "72:47:96:69:16:bb(Ubuntu14041-mars08)" peer names collision

What you expected to happen:
weve-cube installation should have successful and brought the kube-dns up!

How to reproduce it (as minimally and precisely as possible):

On master node:
kubeadm init --api-advertise-addresses=$IP

On Node:
kubeadm join --token $actualtoken $IP

Installed wave-cube as below,
`# kubectl apply -f https://git.io/weave-kube

daemonset "weave-net" created`

kube-dns didn't not start as expected,

Both master and node gets assigned with the same HWaddr causing name collision
` on Master
docker logs a65253346635
INFO: 2016/10/26 05:24:20.719919 Command line options: map[ipalloc-range:10.32.0.0/12 nickname:Ubuntu14041-mars08 no-dns:true docker-api: http-addr:127.0.0.1:6784 ipalloc-init:consensus=2 datapath:datapath name:72:47:96:69:16:bb port:6783]
INFO: 2016/10/26 05:24:20.730839 Communication between peers is unencrypted.
INFO: 2016/10/26 05:24:20.971010 Our name is 72:47:96:69:16:bb(Ubuntu14041-mars08)

On Node,
docker logs a65253346635
INFO: 2016/10/26 05:23:39.312294 Command line options: map[datapath:datapath ipalloc-range:10.32.0.0/12 name:72:47:96:69:16:bb port:6783 docker-api: http-addr:127.0.0.1:6784 ipalloc-init:consensus=2 nickname:Ubuntu14041-mars09 no-dns:true]
INFO: 2016/10/26 05:23:39.314095 Communication between peers is unencrypted.
INFO: 2016/10/26 05:23:39.323302 Our name is 72:47:96:69:16:bb(Ubuntu14041-mars09)
`

CUrling kube-apiserver from a node:
root@Ubuntu14041-mars09:~# curl -k https://10.0.0.1 Unauthorized

nslookup on both master & node
root@Ubuntu14041-mars08:/# curl -k https://10.0.0.1 Unauthorized root@Ubuntu14041-mars08:/# nslookup kubernetes.default Server: 10.30.48.37 Address: 10.30.48.37#53 ** server can't find kubernetes.default: NXDOMAIN

Anything else do we need to know:
iptables -S

`root@Ubuntu14041-mars08:/# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION
-N KUBE-FIREWALL
-N KUBE-SERVICES
-N WEAVE-NPC
-N WEAVE-NPC-DEFAULT
-N WEAVE-NPC-INGRESS
-A INPUT -j KUBE-FIREWALL
-A INPUT -d 172.17.0.1/32 -i docker0 -p tcp -m tcp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6784 -j DROP
-A INPUT -i docker0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i docker0 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -i docker0 -o weave -j DROP
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o weave -j WEAVE-NPC
-A FORWARD -o weave -j LOG --log-prefix "WEAVE-NPC:"
-A FORWARD -o weave -j DROP
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-SERVICES -d 10.0.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]<@p dst -j ACCEPT
`

kube-proxy-amd logs had the following entries,

`I1026 06:39:50.083990       1 iptables.go:339] running iptables-restore [--noflush --counters]
I1026 06:39:50.093036       1 proxier.go:751] syncProxyRules took 58.207586ms
I1026 06:39:50.093063       1 proxier.go:523] OnEndpointsUpdate took 58.262934ms for 4 endpoints
I1026 06:39:50.970922       1 config.go:99] Calling handler.OnEndpointsUpdate()
I1026 06:39:50.974755       1 proxier.go:758] Syncing iptables rules
I1026 06:39:50.974769       1 iptables.go:362] running iptables -N [KUBE-SERVICES -t filter]
I1026 06:39:50.976635       1 healthcheck.go:86] LB service health check mutation request Service: default/kubernetes - 0 Endpoints []
I1026 06:39:50.978146       1 iptables.go:362] running iptables -N [KUBE-SERVICES -t nat]
I1026 06:39:50.980501       1 iptables.go:362] running iptables -C [OUTPUT -t filter -m comment --comment kubernetes service portals -j KUBE-SERVICES]
I1026 06:39:50.982778       1 iptables.go:362] running iptables -C [OUTPUT -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
I1026 06:39:50.984762       1 iptables.go:362] running iptables -C [PREROUTING -t nat -m comment --comment kubernetes service portals -j KUBE-SERVICES]
I1026 06:39:50.986536       1 iptables.go:362] running iptables -N [KUBE-POSTROUTING -t nat]
I1026 06:39:50.988244       1 iptables.go:362] running iptables -C [POSTROUTING -t nat -m comment --comment kubernetes postrouting rules -j KUBE-POSTROUTING]
I1026 06:39:50.990022       1 iptables.go:298] running iptables-save [-t filter]
I1026 06:39:50.992581       1 iptables.go:298] running iptables-save [-t nat]
I1026 06:39:50.995184       1 proxier.go:1244] Restoring iptables rules: *filter
:KUBE-SERVICES - [0:0]
-A KUBE-SERVICES -m comment --comment "kube-system/kube-dns:dns has no endpoints" -m udp -p udp -d 10.0.0.10/32 --dport 53 -j REJECT
-A KUBE-SERVICES -m comment --comment "kube-system/kube-dns:dns-tcp has no endpoints" -m tcp -p tcp -d 10.0.0.10/32 --dport 53 -j REJECT
COMMIT
`
 @errordeveloper, please refer the previous conversations,
#https://github.com/kubernetes/kubernetes/issues/34884

Copied from original issue: kubernetes/kubernetes#35591

From @druidsbane on November 10, 2016 11:57

Is this a request for help? No

What keywords did you search in Kubernetes issues before filing this one? http proxy, http_proxy, https_proxy, AWS, kubeadm, kubectl, cloud-provider

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): v1.4.4

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.421+a6bea3d79b8bba", GitCommit:"a6bea3d79b8bbaa5e8b57482c9fff9265d402708", GitTreeState:"clean", BuildDate:"2016-11-03T06:54:50Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS
• OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS
• Kernel (e.g. uname -a): Linux ip-10-0-133-99 4.4.0-45-generic #66-Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm
• Others: http_proxy, https_proxy and no_proxy set in /etc/environment, Docker config as well on startup scripts that run kubeadm init

What happened:
kubeadm init --cloud-provider aws won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to http.ProxyFromEnvironment and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set http_proxy and https_proxy and no_proxy I am able to use the AWS CLI without any issues.

root@ip-10-2-221-240:/home/ubuntu# kubectl logs kube-controller-manager-ip-10-2-221-240 --namespace=kube-system
E1108 13:58:14.493903       1 event.go:258] Could not construct reference to: '&api.Endpoints{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"kube-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, Subsets:[]api.EndpointSubset(nil)}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' '%v became leader' 'ip-10-2-221-240'
I1108 13:58:14.494179       1 leaderelection.go:214] sucessfully acquired lease kube-system/kube-controller-manager
I1108 13:58:14.499388       1 aws.go:745] Building AWS cloudprovider
I1108 13:58:14.499484       1 aws.go:699] Zone not specified in configuration file; querying AWS metadata service
I1108 13:58:14.499651       1 replication_controller.go:219] Starting RC Manager
I1108 13:58:14.605138       1 endpoints_controller.go:326] Waiting for pods controller to sync, requeuing service default/kubernetes`

[10:25]  
I1108 15:24:09.602100       1 endpoints_controller.go:326] Waiting for pods controller to sync, requeuing service default/kubernetes
I1108 15:24:09.602100       1 endpoints_controller.go:326] Waiting for pods controller to sync, requeuing service default/kubernetes
I1108 15:24:09.602100       1 endpoints_controller.go:326] Waiting for pods controller to sync, requeuing service default/kubernetes
I1108 15:24:09.602100       1 endpoints_controller.go:326] Waiting for pods controller to sync, requeuing service default/kubernetes
....
F1108 15:24:09.625221       1 controllermanager.go:227] Cloud provider could not be initialized: could not init cloud provider "aws": error finding instance i-052a60ad4cc805154: error listing AWS instances: RequestError: send request failed
caused by: Post https://ec2.us-west-1.amazonaws.com/: dial tcp 176.32.118.53:443: i/o timeout

What you expected to happen:
I would expect kubeadm init --cloud-provider aws to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access.

How to reproduce it (as minimally and precisely as possible):

• Set up a public VPC with Squid set up as a proxy
• Set up a private VPC peered to the public VPC
• Set up a master node and run kubeadm init --cloud-provider aws
• Set up a regular node and run kubeadm join

kubeadm will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register.

Anything else do we need to know:

Copied from original issue: kubernetes/kubernetes#36573

From @avkonst on October 5, 2016 14:12

Is this a request for help?

I think it is an issue either with software or documentation, but I am not quite sure. I have started with a question on stackoverflow: http://stackoverflow.com/questions/39872332/how-to-fix-weave-net-crashloopbackoff-for-the-second-node

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

I think it is a bug or request to improve documentation

Kubernetes version (use kubectl version):
1.4.0

Environment:

• Cloud provider or hardware configuration: Vagrant
• OS (e.g. from /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a):
• Install tools: kubeadm init/join
• Others:

What happened:

I have got 2 VMs nodes. Both see each other either by hostname (through /etc/hosts) or by ip address. One has been provisioned with kubeadm as a master. Another as a worker node. Following the instructions (http://kubernetes.io/docs/getting-started-guides/kubeadm/) I have added weave-net. The list of pods looks like the following:

vagrant@vm-master:~$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
kube-system   etcd-vm-master                          1/1       Running            0          3m
kube-system   kube-apiserver-vm-master                1/1       Running            0          5m
kube-system   kube-controller-manager-vm-master       1/1       Running            0          4m
kube-system   kube-discovery-982812725-x2j8y          1/1       Running            0          4m
kube-system   kube-dns-2247936740-5pu0l               3/3       Running            0          4m
kube-system   kube-proxy-amd64-ail86                  1/1       Running            0          4m
kube-system   kube-proxy-amd64-oxxnc                  1/1       Running            0          2m
kube-system   kube-scheduler-vm-master                1/1       Running            0          4m
kube-system   kubernetes-dashboard-1655269645-0swts   1/1       Running            0          4m
kube-system   weave-net-7euqt                         2/2       Running            0          4m
kube-system   weave-net-baao6                         1/2       CrashLoopBackOff   2          2m

CrashLoopBackOff appears for each worker node connected. I have spent several ours playing with network interfaces, but it seems the network is fine. I have found similar question on stackoverflow, where the answer advised to look into the logs and no follow up. So, here are the logs:

vagrant@vm-master:~$ kubectl logs weave-net-baao6 -c weave --namespace=kube-system
2016-10-05 10:48:01.350290 I | error contacting APIServer: Get https://100.64.0.1:443/api/v1/nodes: dial tcp 100.64.0.1:443: getsockopt: connection refused; trying with blank env vars
2016-10-05 10:48:01.351122 I | error contacting APIServer: Get http://localhost:8080/api: dial tcp [::1]:8080: getsockopt: connection refused
Failed to get peers

What you expected to happen:

I would expect the weave-net to be in Running state

How to reproduce it (as minimally and precisely as possible):

I have not done anything special, just followed the documentation on Getting Started. If it is essencial, I can share Vagrant project, which I used to provision everything. Please, let me know if you need one.

Copied from original issue: kubernetes/kubernetes#34101

From @jberkus on November 6, 2016 4:30

Platform: CentOS 7.2.1511 on AWS EC2 (t2.medium)

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+85fe0f1aadf91e", GitCommit:"85fe0f1aadf91e134102cf3c01a9eed11a7e257f", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

Symptoms:

While following installation instructions at http://kubernetes.io/docs/getting-started-guides/kubeadm/, I am finding that I need to always disable preflight checks due to spurious failures.

With kubeadmin --init:

preflight check errors:
	ebtables not found in system path

This error appears on all CentOS machines on AWS I've tried. It doesn't seem to relate to anything at all that I can find; the only google hit I can get on it is a Kubernetes issue explaining how to use --skip-preflight-checks. It does not seem to block kubeadmin setup.

On the nodes I get an additional error:

preflight check errors:
	/etc/kubernetes is not empty
	ebtables not found in system path

Indeed, /etc/kubernetes is not empty ... because the RPMs distributed by Kubernetes.io include a sub-directory in that dir, "manifests".

Copied from original issue: kubernetes/kubernetes#36301

From @jicki on November 3, 2016 6:3

[root@k8s-master ~]#kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.0.0.10 53/UDP,53/TCP 2d

kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.12.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_EXTRA_ARGS=--v=4"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_EXTRA_ARGS

svc cluster ip = 10.0.0.10

kubeadm.conf cluster-dns = 10.12.0.10

Copied from original issue: kubernetes/kubernetes#36135

From @peterkellyonline on October 7, 2016 12:43

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kubeadm master join

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

FEATURE REQUEST

A convenient command such as kubeadm config or kubeadm show to return the command for a node to join the cluster i.e. kubeadm join --token <token> <master-ip>.

(I am happy to work on this myself and contribute)

Copied from original issue: kubernetes/kubernetes#34324

From @sandys on November 12, 2016 8:17

I would also request that kubeadm have the ability to build and install k8s. At the pace that k8s is changing...we would like to dogfood a lot of upcoming k8s changes. If kubeadm can be pointed to a git URL and have it build that k8s version and use it for install/upgrade...that would be a godsend.

For example, we have been blocking on several changes in ingress controllers (for ssl passthrough) and waiting for packaging takes too long. We would love to be able to build and install in one step.

Please note, I'm not referring to building k8s inside a single docker image. I'm talking about building and installing components of k8s (using kubeadm) by pointing to a git url (branch/tag/version/etc) and having the install and upgrade to work

Copied from original issue: kubernetes/kubernetes#36697

@mikedanese can you create the kubeadm-maintainers and kubeadm-collaborator groups and give the people access as decided in kubernetes/kubernetes#36393?

From @JasonHubS on October 19, 2016 2:33

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
no

Kubernetes version (use kubectl version):

1. kubernetes version:

   Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:06:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
   Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

2. kubeadm,kubelet,kubectl and kubenetes-cni version:

kubeadm-1.5.0-0.alpha.0.1403.gc19e08e.x86_64.rpm   28-Sep-2016 17:28             7256298
kubectl-1.4.0-0.beta.8.x86_64.rpm                  28-Sep-2016 17:28             9532646
kubelet-1.4.0-0.beta.8.x86_64.rpm                  28-Sep-2016 17:28            13807494
kubernetes-cni-0.3.0.1-0.07a8a2.x86_64.rpm         28-Sep-2016 17:28            10282814

the url above :http://files.rm-rf.ca/rpms/kubelet/

Environment:

• Cloud provider or hardware configuration:

virtual machine and the nat net-work

• OS (e.g. from /etc/os-release):

CentOS Linux release 7.2.1511 (Core)

• Kernel (e.g. uname -a):

Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

• docker version

Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.14.x86_64
 Go version:      go1.6.3
 Git commit:      cb079f6-unsupported
 Built:           Fri Sep 16 13:24:25 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.14.x86_64
 Go version:      go1.6.3
 Git commit:      cb079f6-unsupported
 Built:           Fri Sep 16 13:24:25 2016
 OS/Arch:         linux/amd64

What happened:

After I exec kubeadm init and it block at <master/apiclient> created API client, waiting for the control plane to become ready
and i setenforce 0 and stop the firewalld and make it disable before ;so i docker ps and found that i can't fount kube-discovery:

71a15f961e5f        gcr.io/google_containers/kube-controller-manager-amd64:v1.4.0   "/usr/local/bin/kube-"   23 minutes ago      Up 23 minutes                                   k8s_kube-controller-manager.a2978680_kube-controller-manager-localhost.localdomain_kube-system_210def2c1c363f691a1bd89e41d56592_a55623fe
faa693818ffc        gcr.io/google_containers/kube-apiserver-amd64:v1.4.0            "/usr/local/bin/kube-"   23 minutes ago      Up 23 minutes                                   k8s_kube-apiserver.c44dda3f_kube-apiserver-localhost.localdomain_kube-system_0446d05fb9406214210e8d29397f8bf2_cc5f6cf8
8d54be04c80c        gcr.io/google_containers/kube-scheduler-amd64:v1.4.0            "/usr/local/bin/kube-"   23 minutes ago      Up 23 minutes                                   k8s_kube-scheduler.1b5cde04_kube-scheduler-localhost.localdomain_kube-system_9d0a8510feaf631fdab3304a555e1aa9_175bc6a5
78ee5fb6de88        gcr.io/google_containers/etcd-amd64:2.2.5                       "etcd --listen-client"   23 minutes ago      Up 23 minutes                                   k8s_etcd.4ffa9846_etcd-localhost.localdomain_kube-system_42857e4bd57d261fc438bcb2a87572b9_16d0f09c
9dfb222354b2        gcr.io/google_containers/pause-amd64:3.0                        "/pause"                 23 minutes ago      Up 23 minutes                                   k8s_POD.d8dbe16c_kube-controller-manager-localhost.localdomain_kube-system_210def2c1c363f691a1bd89e41d56592_d0f74174
9ee1e1722377        gcr.io/google_containers/pause-amd64:3.0                        "/pause"                 23 minutes ago      Up 23 minutes                                   k8s_POD.d8dbe16c_kube-apiserver-localhost.localdomain_kube-system_0446d05fb9406214210e8d29397f8bf2_965a6703
aa71f671bc35        gcr.io/google_containers/pause-amd64:3.0                        "/pause"                 23 minutes ago      Up 23 minutes                                   k8s_POD.d8dbe16c_etcd-localhost.localdomain_kube-system_42857e4bd57d261fc438bcb2a87572b9_5fbe7c3b
869d85a000f1        gcr.io/google_containers/pause-amd64:3.0                        "/pause"                 23 minutes ago      Up 23 minutes                                   k8s_POD.d8dbe16c_kube-scheduler-localhost.localdomain_kube-system_9d0a8510feaf631fdab3304a555e1aa9_066b3e15

What you expected to happen:
After I exec the command: kubeadm init ,and it shout start it sucessceful.

How to reproduce it (as minimally and precisely as possible):
The os and environment above,and the container and the kubeadm,docker .
stop firewalld and setenforce 0

Anything else do we need to know:
Kubelet apparently require the network-name to be eth0

Copied from original issue: kubernetes/kubernetes#35084

From @derailed on October 26, 2016 17:23

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kubeadm arm

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Bug

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.3", 
GitCommit:"4957b090e9a4f6a68b4a40375408fdc74a212260", GitTreeState:"clean", 
BuildDate:"2016-10-16T06:36:33Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/arm"}

Environment:

Raspberry Pi 3

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

HypriotOS v 1.0

• Kernel (e.g. uname -a):

Linux m10 4.4.15-hypriotos-v7+ #1 SMP PREEMPT Mon Jul 25 08:46:52 UTC 2016 armv7l GNU/Linux

• Install tools:
• Others:

What happened:

Following kubeadm installation docs. I've installed the prereqs and proceeded with the init as follows:

kubeadm init --use-kubernetes-version v1.4.1 --pod-network-cidr=10.244.0.0/16

The command is stuck on:

master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready

Looking at docker on this node expecting to see images being installed for etcd, api-server, etc..
but docker images reports nothing. Guessing kubeadm is somehow no able to connect with local docker daemon? or the default Hypriot docker configuration is not jiving with kubeadm??

What you expected to happen:

kubeadm init to complete successfully

How to reproduce it (as minimally and precisely as possible):

o Download Hypriot v1.0 image and flash on SD card
o Boot rp with SD card
o curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
o cat < /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
o apt-get update
o apt-get install -y kubelet kubeadm kubectl kubernetes-cni
o kubeadm init --use-kubernetes-version v1.4.1 --pod-network-cidr=10.244.0.0/16

Anything else do we need to know:

Looking at /etc/kubernetes/manifests shows:

etcd.json kube-apiserver.json kube-controller-manager.json kube-scheduler.json

Inspecting journalctl on kubelet shows:

Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.467776    2787 kubelet_node_status.go:203] Setting node annotation to enable volume controller attach/detach
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.469033    2787 interface.go:248] Default route transits interface "eth0"
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.470261    2787 interface.go:93] Interface eth0 is up
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471170    2787 interface.go:138] Interface "eth0" has 2 addresses :[192.168.0.92/24 fe80::ba27:ebff:fe24:986f/64].
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471484    2787 interface.go:105] Checking addr  192.168.0.92/24.
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471664    2787 interface.go:114] IP found 192.168.0.92
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471831    2787 interface.go:144] valid IPv4 address for interface "eth0" found as 192.168.0.92.
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.471960    2787 interface.go:254] Choosing IP 192.168.0.92
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497415    2787 kubelet_node_status.go:354] Recording NodeHasSufficientDisk event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497616    2787 kubelet_node_status.go:354] Recording NodeHasSufficientMemory event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.497905    2787 kubelet_node_status.go:354] Recording NodeHasNoDiskPressure event message for node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498077    2787 kubelet_node_status.go:73] Attempting to register node m10
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498243    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientDisk' Node m10 status is now: NodeHasSufficientDisk
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498554    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasSufficientMemory' Node m10 status is now: NodeHasSufficientMemory
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.498823    2787 server.go:608] Event(api.ObjectReference{Kind:"Node", Namespace:"", Name:"m10", UID:"m10", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeHasNoDiskPressure' Node m10 status is now: NodeHasNoDiskPressure
Oct 26 17:19:31 m10 kubelet[2787]: E1026 17:19:31.501189    2787 kubelet_node_status.go:97] Unable to register node "m10" with API server: Post https://192.168.0.92:443/api/v1/nodes: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:31 m10 kubelet[2787]: I1026 17:19:31.527656    2787 reflector.go:249] Listing and watching *api.Pod from pkg/kubelet/config/apiserver.go:43
Oct 26 17:19:31 m10 kubelet[2787]: E1026 17:19:31.531088    2787 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://192.168.0.92:443/api/v1/pods?fieldSelector=spec.nodeName%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.416745    2787 reflector.go:249] Listing and watching *api.Service from pkg/kubelet/kubelet.go:384
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.418995    2787 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://192.168.0.92:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.455119    2787 reflector.go:249] Listing and watching *api.Node from pkg/kubelet/kubelet.go:403
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.457560    2787 reflector.go:203] pkg/kubelet/kubelet.go:403: Failed to list *api.Node: Get https://192.168.0.92:443/api/v1/nodes?fieldSelector=metadata.name%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:32 m10 kubelet[2787]: I1026 17:19:32.531641    2787 reflector.go:249] Listing and watching *api.Pod from pkg/kubelet/config/apiserver.go:43
Oct 26 17:19:32 m10 kubelet[2787]: E1026 17:19:32.534040    2787 reflector.go:203] pkg/kubelet/config/apiserver.go:43: Failed to list *api.Pod: Get https://192.168.0.92:443/api/v1/pods?fieldSelector=spec.nodeName%3Dm10&resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused
Oct 26 17:19:33 m10 kubelet[2787]: I1026 17:19:33.419424    2787 reflector.go:249] Listing and watching *api.Service from pkg/kubelet/kubelet.go:384
Oct 26 17:19:33 m10 kubelet[2787]: E1026 17:19:33.421922    2787 reflector.go:203] pkg/kubelet/kubelet.go:384: Failed to list *api.Service: Get https://192.168.0.92:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.92:443: getsockopt: connection refused

Copied from original issue: kubernetes/kubernetes#35643

From @jberkus on November 11, 2016 21:45

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No.

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

PetSet, Statefulset, PV, persistent volume, AWS

This is similar to issue #36589 except that (a) it's a StatefulSet, and (b) the kubelets were not restarted.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: AWS, EC2, t2.medium
• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Kernel (e.g. uname -a):

Linux ip-172-31-47-16.us-west-2.compute.internal 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Install tools: kubeadm

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.2.380+85fe0f1aadf91e", GitCommit:"85fe0f1aadf91e134102cf3c01a9eed11a7e257f", GitTreeState:"clean", BuildDate:"2016-11-02T14:58:17Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"}

• Others:

What happened:

All EC2 nodes are set up in an IAM role with EBS management access. To test this, I created a regular PV, which created successfully.

1. Built 4-node cluster on EC2 using the Kubeadm instructions for CentOS, including kubeadm init --cloud-provider=aws

2. Had to manually fix SSL certs by creating hard links, due to this bug: #36150

3. Added weave network

4. Created this StatefulSet: https://github.com/jberkus/atomicdb/blob/master/patroni_petset/kubernetes/ps-patroni.yaml
   (supporting objects are in same dir)

5. Service, PVCs, PVs created successfully. Checking AWS, the EBS volumes were created and are available.

6. Pods failed, due to an inability to mount the PVs:

FirstSeen LastSeen Count From SubobjectPath Type Reason Message

20m 20m 1 {default-scheduler } Normal Scheduled Successfully assigned patroni-0 to ip-172-31-43-85
18m 1m 9 {kubelet ip-172-31-43-85} Warning FailedMount Unable to mount volumes for pod "patroni-0_default(a8e441ac-a854-11e6-bb5c-06027ce5ae69)": timeout expired waiting for volumes to attach/mount for pod "patroni-0"/"default". list of unattached/unmounted volumes=[pgdata]
18m 1m 9 {kubelet ip-172-31-43-85} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "patroni-0"/"default". list of unattached/unmounted volumes=[pgdata]
20m 41s 18 {kubelet ip-172-31-43-85} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/aws-ebs/a8e441ac-a854-11e6-bb5c-06027ce5ae69-pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69" (spec.Name: "pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69") pod "a8e441ac-a854-11e6-bb5c-06027ce5ae69" (UID: "a8e441ac-a854-11e6-bb5c-06027ce5ae69") with: mount failed: exit status 32
Mounting arguments: /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-2a/vol-b893ce0c /var/lib/kubelet/pods/a8e441ac-a854-11e6-bb5c-06027ce5ae69/volumes/kubernetes.io~aws-ebs/pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69 [bind]
Output: mount: special device /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-west-2a/vol-b893ce0c does not exist
20m 39s 18 {controller-manager } Warning FailedMount Failed to attach volume "pvc-a1c46983-a854-11e6-bb5c-06027ce5ae69" on node "ip-172-31-43-85" with: error finding instance ip-172-31-43-85: instance not found

What you expected to happen:

PVs should have mounted on the pod containers, and the remaining pods should have come up.

How to reproduce it (as minimally and precisely as possible):

Follow steps above.

Anything else do we need to know:

• kubectl logs and journalctl -xe don't show any errors related to pvs or mounts
• I can't kube exec into the pod because it's down (to check paths)
• the same StatefulSet, using only EmptyDir(), works fine.

Copied from original issue: kubernetes/kubernetes#36670

From @msplayer on October 13, 2016 2:13

I have 2 machine:
10.86.8.109 (master) CentOS7
10.86.8.110 (cluster node) CentOS7

I use kubeadm to install master, it it works well, "kubectl get pods --all-namespaces" get all pods are running.
When I join a node to cluster, "kubectl get nodes" is still get only one localhost node. and pod "kube-apiserver", "etcd-localhost" restart endless.

I stop and disable firewall on both node as well.
It always like that:

kube-system   etcd-localhost.localdomain                      0/1       Pending   0          0s
kube-system   kube-controller-manager-localhost.localdomain   0/1       Pending   0          0s
kube-system   kube-discovery-982812725-wkb9w                  1/1       Running   0          17h
kube-system   kube-dns-2247936740-jk73s                       3/3       Running   0          16h
kube-system   kube-proxy-amd64-jgchq                          1/1       Running   0          16h
kube-system   weave-net-35a1t                                 2/2       Running   0          16h

if I get nodes:
localhost.localdomain Ready 17h

Thanks

Copied from original issue: kubernetes/kubernetes#34677