Hi there, Can some please tell me, why the pod deployed through GKE autopilot mode throws this error, when describe events suggests that init containers were successfully created, started, and have done the task of passing "sysctl -w vm.max_map_count=262144".
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
gke cluster throws me below stack-trace error for the deployed pod in GKE autopilot mode. These are logs of elasticsearch-pod-0
ubuntu@ubuntu$ kubectl logs elasticsearch-pod-0
2021-03-04T18:57:19,302][INFO ][o.e.n.Node ] [] initializing ...
[2021-03-04T18:57:19,520][INFO ][o.e.e.NodeEnvironment ] [gbOK4rP] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sdb)]], net usable_space [9.7gb], net total_space [9.7gb], spins? [unknown], types [ext4]
[2021-03-04T18:57:19,521][INFO ][o.e.e.NodeEnvironment ] [gbOK4rP] heap size [1.9gb], compressed ordinary object pointers [true]
[2021-03-04T18:57:19,523][INFO ][o.e.n.Node ] [gbOK4rP] node name [gbOK4rP] derived from node ID; set [node.name] to override
[2021-03-04T18:57:19,525][INFO ][o.e.n.Node ] [gbOK4rP] version[5.0.0], pid[1], build[253032b/2016-10-26T05:11:34.737Z], OS[Linux/5.4.49+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b14]
[2021-03-04T18:57:22,709][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [aggs-matrix-stats]
[2021-03-04T18:57:22,710][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [ingest-common]
[2021-03-04T18:57:22,710][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [lang-expression]
[2021-03-04T18:57:22,710][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [lang-groovy]
[2021-03-04T18:57:22,772][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [lang-mustache]
[2021-03-04T18:57:22,772][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [lang-painless]
[2021-03-04T18:57:22,772][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [percolator]
[2021-03-04T18:57:22,773][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [reindex]
[2021-03-04T18:57:22,773][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [transport-netty3]
[2021-03-04T18:57:22,773][INFO ][o.e.p.PluginsService ] [gbOK4rP] loaded module [transport-netty4]
[2021-03-04T18:57:22,774][INFO ][o.e.p.PluginsService ] [gbOK4rP] no plugins loaded
[2021-03-04T18:57:23,193][WARN ][o.e.d.s.g.GroovyScriptEngineService] [groovy] scripts are deprecated, use [painless] scripts instead
[2021-03-04T18:57:31,491][INFO ][o.e.n.Node ] [gbOK4rP] initialized
[2021-03-04T18:57:31,492][INFO ][o.e.n.Node ] [gbOK4rP] starting ...
[2021-03-04T18:57:32,001][INFO ][o.e.t.TransportService ] [gbOK4rP] publish_address {10.42.0.130:9300}, bound_addresses {[::]:9300}
[2021-03-04T18:57:32,008][INFO ][o.e.b.BootstrapCheck ] [gbOK4rP] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
[2021-03-04T18:57:32,086][INFO ][o.e.n.Node ] [gbOK4rP] stopping ...
[2021-03-04T18:57:32,184][INFO ][o.e.n.Node ] [gbOK4rP] stopped
[2021-03-04T18:57:32,184][INFO ][o.e.n.Node ] [gbOK4rP] closing ...
[2021-03-04T18:57:32,208][INFO ][o.e.n.Node ] [gbOK4rP] closed
Here's events of elasticsearch deployed pod
kubectl describe pod/elasticsearch-0
ubuntu@ubuntu$ kubectl describe pod/elasticsearch-0
Name: elasticsearch-0
Namespace: default
Priority: 0
Node: gk3-autopilot-cluster-1-nap-1535p6rr-5687b4d4-2w8t/10.142.0.22
Start Time: Fri, 05 Mar 2021 00:25:11 +0530
Labels: app=elasticsearch
controller-revision-hash=elasticsearch-6c57546b4b
statefulset.kubernetes.io/pod-name=elasticsearch-0
Annotations: seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status: Running
IP: 10.42.0.130
IPs:
IP: 10.42.0.130
Controlled By: StatefulSet/elasticsearch
Init Containers:
increase-vm-max-map:
Container ID: containerd://bfda3431afee52e71a789cb7d6f612f4bb2ea5d81f9cb74bdc0733d8aa64a29f
Image: busybox
Image ID: docker.io/library/busybox@sha256:c6b45a95f932202dbb27c31333c4789f45184a744060f6e569cc9d2bf1b9ad6f
Port: <none>
Host Port: <none>
Command:
sh
-c
sysctl
-p
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 05 Mar 2021 00:25:30 +0530
Finished: Fri, 05 Mar 2021 00:25:30 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 750m
ephemeral-storage: 1Gi
memory: 3000Mi
Requests:
cpu: 750m
ephemeral-storage: 1Gi
memory: 3000Mi
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p7f8t (ro)
Containers:
elasticsearch:
Container ID: containerd://ea6d1f14ec03b683fab51feb0cb0e540f7299250d973fcfc07301bd51128f865
Image: elasticsearch:5.0.0
Image ID: docker.io/library/elasticsearch@sha256:29f6b68a0088238f4a108e6c725163130e382a0f34ed62159c82e3961f0639fa
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Terminated
Reason: Error
Exit Code: 78
Started: Fri, 05 Mar 2021 00:27:15 +0530
Finished: Fri, 05 Mar 2021 00:27:32 +0530
Last State: Terminated
Reason: Error
Exit Code: 78
Started: Fri, 05 Mar 2021 00:26:31 +0530
Finished: Fri, 05 Mar 2021 00:26:49 +0530
Ready: False
Restart Count: 3
Limits:
cpu: 750m
ephemeral-storage: 1Gi
memory: 3000Mi
Requests:
cpu: 750m
ephemeral-storage: 1Gi
memory: 3000Mi
Environment Variables from:
elastic-config ConfigMap Optional: false
Environment: <none>
Mounts:
/usr/share/elasticsearch/config/elasticsearch.yml from elasticsearch-configfile (rw,path="elasticsearch.yml")
/usr/share/elasticsearch/data from elastic-pvc (rw,path="elastic-data")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p7f8t (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elastic-pvc:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elastic-pvc-elasticsearch-0
ReadOnly: false
elasticsearch-configfile:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch-config
Optional: false
kube-api-access-p7f8t:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal TriggeredScaleUp 3m43s cluster-autoscaler pod triggered scale-up: [{https://content.googleapis.com/compute/v1/projects/lexical-pattern-305514/zones/us-east1-c/instanceGroups/gk3-autopilot-cluster-1-nap-1535p6rr-5687b4d4-grp 0->1 (max: 1000)}]
Warning FailedScheduling 3m34s (x3 over 4m39s) gke.io/optimize-utilization-scheduler 0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory.
Warning FailedScheduling 3m2s (x2 over 3m2s) gke.io/optimize-utilization-scheduler 0/3 nodes are available: 1 Insufficient ephemeral-storage, 2 Insufficient cpu, 2 Insufficient memory.
Warning FailedScheduling 2m42s (x3 over 2m54s) gke.io/optimize-utilization-scheduler 0/3 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate, 2 Insufficient cpu, 2 Insufficient memory.
Normal Scheduled 2m32s gke.io/optimize-utilization-scheduler Successfully assigned default/elasticsearch-0 to gk3-autopilot-cluster-1-nap-1535p6rr-5687b4d4-2w8t
Normal SuccessfulAttachVolume 2m17s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-d9ab9683-a408-421f-96f3-dd4eba1b871d"
Normal Pulling 2m13s kubelet Pulling image "busybox"
Normal Pulled 2m13s kubelet Successfully pulled image "busybox"
Normal Created 2m13s kubelet Created container increase-vm-max-map
Normal Started 2m12s kubelet Started container increase-vm-max-map
Normal Pulling 2m12s kubelet Pulling image "elasticsearch:5.0.0"
Normal Pulled 2m4s kubelet Successfully pulled image "elasticsearch:5.0.0"
Normal Created 27s (x4 over 2m1s) kubelet Created container elasticsearch
Normal Started 27s (x4 over 2m1s) kubelet Started container elasticsearch
Normal Pulled 27s (x3 over 103s) kubelet Container image "elasticsearch:5.0.0" already present on machine
Warning BackOff 10s (x4 over 86s) kubelet Back-off restarting failed container
Here's the statefulset of elasticsearch yaml resource.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch
spec:
selector:
matchLabels:
app: elasticsearch
serviceName: "elasticsearch"
replicas: 1
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: elasticsearch:5.0.0
resources:
limits:
memory: "4000Mi"
cpu: 1000m
requests:
memory: "3000Mi"
cpu: 600m
imagePullPolicy: IfNotPresent
envFrom:
- configMapRef:
name: fx-elastic-config
ports:
- containerPort: 9200
name: http-port
- containerPort: 9300
name: transport
volumeMounts:
- name: elastic-pvc
mountPath: /usr/share/elasticsearch/data
subPath: elastic-data
- mountPath: "/usr/share/elasticsearch/config/elasticsearch.yml"
subPath: elasticsearch.yml
name: elasticsearch-configfile
volumes:
- name: elasticsearch-configfile
configMap:
name: elasticsearch-config
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: elastic-data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'sysctl', '-w', 'vm.max_map_count=262144']
command: ["sh", "-c", "echo", "vm.max_map_count=262144", ">>", "/etc/sysctl.conf"]
command: ["sh", "-c", "sysctl", "-p"]
securityContext:
privileged: false
- name: increase-fd-ulimit
image: busybox
command: ['sh', '-c', 'ulimit -n 65536']
securityContext:
privileged: false
volumeClaimTemplates:
- metadata:
name: elastic-pvc
# annotations:
# ...
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "premium-rwo"
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: elasticsearch
labels:
app: elasticsearch
spec:
type: ClusterIP
selector:
app: elasticsearch
ports:
- port: 9200
targetPort: 9200
name: elasticsearch-http
- port: 9300
targetPort: 9300
Same elasticsearch statefulset k8s resource file, when deployed in GKE standard mode, runs successfully without any error.
Can some please tell me why this kind of behaviour and how to fix this issue?