Giter VIP home page Giter VIP logo

zeebe-full-helm's Introduction

MERGED INTO MONO REPO


Community ExtensionLifecycle: IncubatingLicense

Zeebe + Operate + TaskList and ZeeQS Ingress Controller HELM Chart

This chart is hosted at: http://helm.camunda.io

These charts install:

  • A Zeebe Cluster
  • Operate configured to talk with the Zeebe Cluster
  • Ingress Controller which
    • Expose Operate HTTP endpoint under /
    • If Kibana is enabled also expose Kibana HTTP endpoint under /logs
zeebe-cluster: 
  kibana:
    enabled: true     
    healthCheckPath: "/logs/app/kibana"
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      path: /logs
      hosts:
        - ""
    extraEnvs:
      - name: SERVER_BASEPATH
        value: /logs
      - name: SERVER_REWRITEBASEPATH
        value: "true"

zeebe-full-helm's People

Contributors

celanthe avatar falko avatar pow-devops2020 avatar salaboy avatar strawhat5 avatar zelldon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

zeebe-full-helm's Issues

zeeqs pod crashing after http graphql request.

  • Running on GKE
  • Deploy command: helm install zeebe ./zeebe-full-helm -f ./zeebe-full-helm/values.yaml -n zeebe
  • I enabled zeeqs and tasklist on values.yaml

I did a port fowarding from the zeeqs service running on port 9000 to my local machine, but when I sent a request (POST query or /graphiql), the zeeqs pod crashes with a Error with exit code 1 and restart.

Following the logs I found a error stacktrace.

Storage Class Not Found on Deployment

Hey, I'm not 100% sure if this is the right place to ask this, however I'm currently trying to get this chart deployed on a dev cluster. Basically after doing a helm install for zeebe/zeebe-full, the PVC's are stuck in pending state with this:

no persistent volumes available for this claim and no storage class is set.

I think the issues is that the deployment is attempting to use the default storage class, however on this cluster it is a different storage class. I'm wondering if there is a flag I can pass in during the helm install --name= zeebe/zeebe-full that will allow me to specify which storage class to use.

Readiness probe failed: Get "http://172.17.0.12:9600/ready": dial tcp 172.17.0.12:9600: connect: connection refused

Info of versions.

# helm version
version.BuildInfo{Version:"v3.3.1", GitCommit:"249e5215cde0c3fa72e27eb7a30e8d55c9696144", GitTreeState:"clean", GoVersion:"go1.14.7"}
# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:23:04Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
# minikube version
minikube version: v1.13.0
commit: 0c5e9de4ca6f9c55147ae7f90af97eff5befef5f-dirty

Then create a new namespace.

# kubectl create ns my-zeebe
namespace/my-zeebe created

Then add helm repo and update.

# helm repo add zeebe https://helm.zeebe.io
"zeebe" has been added to your repositories
# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "zeebe" chart repository
Update Complete. ⎈Happy Helming!⎈
# helm repo list
NAME 	URL                  
zeebe	https://helm.zeebe.io

And install Zeebe Full Helm Chart.

# helm install zeebe/zeebe-full --namespace my-zeebe --generate-name
NAME: zeebe-full-1600416712
LAST DEPLOYED: Fri Sep 18 16:12:01 2020
NAMESPACE: my-zeebe
STATUS: deployed
REVISION: 1
NOTES:
______     ______     ______     ______     ______    
/\___  \   /\  ___\   /\  ___\   /\  == \   /\  ___\   
\/_/  /__  \ \  __\   \ \  __\   \ \  __<   \ \  __\   
  /\_____\  \ \_____\  \ \_____\  \ \_____\  \ \_____\ 
  \/_____/   \/_____/   \/_____/   \/_____/   \/_____/                                                                                                                                      

(zeebe-full - 0.0.105)

- Cluster Name: zeebe-full-1600416712-zeebe

kubectl get pod -n my-zeebe
NAME                                                              READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                                            0/1     Running   0          71s
elasticsearch-master-1                                            0/1     Pending   0          71s
elasticsearch-master-2                                            0/1     Pending   0          71s
zeebe-full-1600416712-nginx-ingress-controller-7bc56d4bb6-nhf7q   1/1     Running   0          71s
zeebe-full-1600416712-nginx-ingress-default-backend-d8879c4pf7d   1/1     Running   0          71s
zeebe-full-1600416712-operate-76db4dc49d-gqtx8                    1/1     Running   0          71s
zeebe-full-1600416712-zeebe-0                                     0/1     Running   0          71s
zeebe-full-1600416712-zeebe-1                                     0/1     Running   0          71s
zeebe-full-1600416712-zeebe-2                                     0/1     Running   0          71s
zeebe-full-1600416712-zeebe-gateway-86b45b5997-x58ph              1/1     Running   0          71s

But there are some errors.

# kubectl describe pod zeebe-full-1600416712-zeebe-0 -n my-zeebe
Name:         zeebe-full-1600416712-zeebe-0
Namespace:    my-zeebe
Priority:     0
Node:         zhaoyj-hp-elitebook-830-g5/10.9.54.253
Start Time:   Fri, 18 Sep 2020 16:12:04 +0800
Labels:       app.kubernetes.io/component=broker
              app.kubernetes.io/instance=zeebe-full-1600416712
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=zeebe
              controller-revision-hash=zeebe-full-1600416712-zeebe-7bbc44d84
              statefulset.kubernetes.io/pod-name=zeebe-full-1600416712-zeebe-0
Annotations:  <none>
Status:       Running
IP:           172.17.0.11
IPs:
  IP:           172.17.0.11
Controlled By:  StatefulSet/zeebe-full-1600416712-zeebe
Containers:
  zeebe:
    Container ID:   docker://ac0c9df749bb061af45ce8a6a4ed1f44b97de97552719b99b3a4dd373401485e
    Image:          camunda/zeebe:0.24.1
    Image ID:       docker-pullable://camunda/zeebe@sha256:46097cc5f8dda917837ca8b22a38cf8a3c48e0ed55e651ee277f360d27640003
    Ports:          9600/TCP, 26501/TCP, 26502/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 18 Sep 2020 16:12:06 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:9600/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                zeebe-full-1600416712-zeebe
      ZEEBE_LOG_LEVEL:                                 
      ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:            3
      ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                3
      ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:          3
      ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:             2
      ZEEBE_BROKER_THREADS_IOTHREADCOUNT:              2
      ZEEBE_BROKER_GATEWAY_ENABLE:                     false
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:  io.zeebe.exporter.ElasticsearchExporter
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:   http://elasticsearch-master:9200
      ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:            26501
      ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:           26502
      ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:         9600
      K8S_POD_NAME:                                    zeebe-full-1600416712-zeebe-0 (v1:metadata.name)
      JAVA_TOOL_OPTIONS:                               -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
    Mounts:
      /exporters from exporters (rw)
      /usr/local/bin/startup.sh from config (rw,path="startup.sh")
      /usr/local/zeebe/config/application.yaml from config (rw,path="application.yaml")
      /usr/local/zeebe/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qskn6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-zeebe-full-1600416712-zeebe-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      zeebe-full-1600416712
    Optional:  false
  exporters:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  default-token-qskn6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qskn6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  7m14s                  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling  7m14s                  default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled         7m11s                  default-scheduler  Successfully assigned my-zeebe/zeebe-full-1600416712-zeebe-0 to zhaoyj-hp-elitebook-830-g5
  Normal   Pulled            7m10s                  kubelet            Container image "camunda/zeebe:0.24.1" already present on machine
  Normal   Created           7m9s                   kubelet            Created container zeebe
  Normal   Started           7m9s                   kubelet            Started container zeebe
  Warning  Unhealthy         6m57s (x2 over 7m7s)   kubelet            Readiness probe failed: Get "http://172.17.0.11:9600/ready": dial tcp 172.17.0.11:9600: connect: connection refused
  Warning  Unhealthy         6m46s                  kubelet            Readiness probe failed: Get "http://172.17.0.11:9600/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy         2m6s (x28 over 6m37s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

And here is the log of the pod.

kubectl logs -f zeebe-full-1600416712-zeebe-0 -n my-zeebe
++ hostname -f
+ export ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local
+ ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local
+ export ZEEBE_BROKER_CLUSTER_NODEID=0
+ ZEEBE_BROKER_CLUSTER_NODEID=0
+ export ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ contactPointPrefix=zeebe-full-1600416712-zeebe
+ contactPoints=
+ [[ -z '' ]]
+ (( i=0 ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-1.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-1.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-2.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
+ export ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-1.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-2.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502
+ ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-1.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502,zeebe-full-1600416712-zeebe-2.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502
++ ls -A /exporters/
No exporters available.
+ '[' '' ']'
+ echo 'No exporters available.'
+ exec /usr/local/zeebe/bin/broker
Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
2020-09-18 08:12:09,123 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
	at org.apache.logging.log4j.Level.valueOf(Level.java:320)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
	at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
	at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
	at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
	at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
	at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
	at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
	at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
	at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
	at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
	at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
	at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
	at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:196)
	at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

2020-09-18 08:12:10,574 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
	at org.apache.logging.log4j.Level.valueOf(Level.java:320)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
	at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
	at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
	at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
	at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
	at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
	at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
	at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
	at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.reinitialize(Log4J2LoggingSystem.java:204)
	at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:73)
	at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
	at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.initialize(Log4J2LoggingSystem.java:160)
	at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:306)
	at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:281)
	at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:239)
	at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:216)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
	at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80)
	at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
	at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
	at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

  ______  ______   ______   ____    ______     ____    _____     ____    _  __  ______   _____  
 |___  / |  ____| |  ____| |  _ \  |  ____|   |  _ \  |  __ \   / __ \  | |/ / |  ____| |  __ \ 
    / /  | |__    | |__    | |_) | | |__      | |_) | | |__) | | |  | | | ' /  | |__    | |__) |
   / /   |  __|   |  __|   |  _ <  |  __|     |  _ <  |  _  /  | |  | | |  <   |  __|   |  _  / 
  / /__  | |____  | |____  | |_) | | |____    | |_) | | | \ \  | |__| | | . \  | |____  | | \ \ 
 /_____| |______| |______| |____/  |______|   |____/  |_|  \_\  \____/  |_|\_\ |______| |_|  \_\
                                                                                                
2020-09-18 08:12:10.932 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v0.24.1 on zeebe-full-1600416712-zeebe-0 with PID 6 (/usr/local/zeebe/lib/zeebe-distribution-0.24.1.jar started by root in /usr/local/zeebe)
2020-09-18 08:12:10.958 [] [main] INFO  io.zeebe.broker.StandaloneBroker - No active profile set, falling back to default profiles: default
2020-09-18 08:12:17.869 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 9600 (http)
2020-09-18 08:12:17.909 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-09-18 08:12:17.911 [] [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-09-18 08:12:17.948 [] [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-09-18 08:12:18.487 [] [main] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-09-18 08:12:18.490 [] [main] INFO  org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 7406 ms
2020-09-18 08:12:20.371 [] [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-09-18 08:12:22.073 [] [main] INFO  org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 2 endpoint(s) beneath base path '/actuator'
2020-09-18 08:12:22.264 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-09-18 08:12:22.407 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 9600 (http) with context path ''
2020-09-18 08:12:22.452 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 12.784 seconds (JVM running for 16.116)
2020-09-18 08:12:22.656 [] [main] INFO  io.zeebe.broker.system - Version: 0.24.1
2020-09-18 08:12:22.900 [] [main] INFO  io.zeebe.broker.system - Starting broker 0 with configuration {
  "network" : {
    "host" : "0.0.0.0",
    "portOffset" : 0,
    "maxMessageSize" : "4MB",
    "advertisedHost" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local",
    "commandApi" : {
      "host" : "0.0.0.0",
      "port" : 26501,
      "advertisedHost" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local",
      "advertisedPort" : 26501,
      "advertisedAddress" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26501",
      "address" : "0.0.0.0:26501"
    },
    "internalApi" : {
      "host" : "0.0.0.0",
      "port" : 26502,
      "advertisedHost" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local",
      "advertisedPort" : 26502,
      "advertisedAddress" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502",
      "address" : "0.0.0.0:26502"
    },
    "monitoringApi" : {
      "host" : "0.0.0.0",
      "port" : 9600,
      "advertisedHost" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local",
      "advertisedPort" : 9600,
      "advertisedAddress" : "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:9600",
      "address" : "0.0.0.0:9600"
    },
    "maxMessageSizeInBytes" : 4194304
  },
  "cluster" : {
    "initialContactPoints" : [ "zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502", "zeebe-full-1600416712-zeebe-1.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502", "zeebe-full-1600416712-zeebe-2.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502" ],
    "partitionIds" : [ 1, 2, 3 ],
    "nodeId" : 0,
    "partitionsCount" : 3,
    "replicationFactor" : 3,
    "clusterSize" : 3,
    "clusterName" : "zeebe-full-1600416712-zeebe",
    "membership" : {
      "broadcastUpdates" : false,
      "broadcastDisputes" : true,
      "notifySuspect" : false,
      "gossipInterval" : "PT0.25S",
      "gossipFanout" : 2,
      "probeInterval" : "PT1S",
      "probeTimeout" : "PT2S",
      "suspectProbes" : 3,
      "failureTimeout" : "PT10S",
      "syncInterval" : "PT10S"
    }
  },
  "threads" : {
    "cpuThreadCount" : 2,
    "ioThreadCount" : 2
  },
  "data" : {
    "directories" : [ "/usr/local/zeebe/data" ],
    "logSegmentSize" : "512MB",
    "snapshotPeriod" : "PT15M",
    "logIndexDensity" : 100,
    "logSegmentSizeInBytes" : 536870912,
    "atomixStorageLevel" : "DISK"
  },
  "exporters" : {
    "elasticsearch" : {
      "jarPath" : null,
      "className" : "io.zeebe.exporter.ElasticsearchExporter",
      "args" : {
        "url" : "http://elasticsearch-master:9200"
      },
      "external" : false
    }
  },
  "gateway" : {
    "network" : {
      "host" : "0.0.0.0",
      "port" : 26500,
      "minKeepAliveInterval" : "PT30S"
    },
    "cluster" : {
      "contactPoint" : "0.0.0.0:26502",
      "requestTimeout" : "PT15S",
      "clusterName" : "zeebe-cluster",
      "memberId" : "gateway",
      "host" : "0.0.0.0",
      "port" : 26502,
      "membership" : {
        "broadcastUpdates" : false,
        "broadcastDisputes" : true,
        "notifySuspect" : false,
        "gossipInterval" : "PT0.25S",
        "gossipFanout" : 2,
        "probeInterval" : "PT1S",
        "probeTimeout" : "PT2S",
        "suspectProbes" : 3,
        "failureTimeout" : "PT10S",
        "syncInterval" : "PT10S"
      }
    },
    "threads" : {
      "managementThreads" : 1
    },
    "monitoring" : {
      "enabled" : false,
      "host" : "0.0.0.0",
      "port" : 9600
    },
    "security" : {
      "enabled" : false,
      "certificateChainPath" : null,
      "privateKeyPath" : null
    },
    "longPolling" : {
      "enabled" : true
    },
    "initialized" : true,
    "enable" : false
  },
  "backpressure" : {
    "enabled" : true,
    "algorithm" : "VEGAS",
    "aimd" : {
      "requestTimeout" : "PT1S",
      "initialLimit" : 100,
      "minLimit" : 1,
      "maxLimit" : 1000,
      "backoffRatio" : 0.9
    },
    "fixedLimit" : {
      "limit" : 20
    },
    "vegas" : {
      "alpha" : 3,
      "beta" : 6,
      "initialLimit" : 20
    },
    "gradient" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0
    },
    "gradient2" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0,
      "longWindow" : 600
    }
  },
  "stepTimeout" : "PT5M",
  "executionMetricsExporterEnabled" : false
}
2020-09-18 08:12:22.992 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [1/10]: actor scheduler
2020-09-18 08:12:23.012 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [2/10]: membership and replication protocol
2020-09-18 08:12:29.593 [] [http-nio-0.0.0.0-9600-exec-2] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-09-18 08:12:29.595 [] [http-nio-0.0.0.0-9600-exec-2] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-09-18 08:12:29.692 [] [http-nio-0.0.0.0-9600-exec-2] INFO  org.springframework.web.servlet.DispatcherServlet - Completed initialization in 96 ms
2020-09-18 08:12:36.279 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [3/10]: command api transport
2020-09-18 08:12:36.862 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [4/10]: command api handler
2020-09-18 08:12:37.177 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [5/10]: subscription api
2020-09-18 08:12:37.287 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services
2020-09-18 08:12:41.893 [] [raft-server-0-raft-partition-partition-2] WARN  io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-2}{role=FOLLOWER} - Poll request to 2 failed: java.net.ConnectException: Expected to send a message with subject 'raft-partition-partition-2-poll' to member '2', but member is not known. Known members are '[Member{id=0, address=zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502, properties={}}]'.
2020-09-18 08:12:41.893 [] [raft-server-0-raft-partition-partition-2] WARN  io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-2}{role=FOLLOWER} - Poll request to 1 failed: java.net.ConnectException: Expected to send a message with subject 'raft-partition-partition-2-poll' to member '1', but member is not known. Known members are '[Member{id=0, address=zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502, properties={}}]'.
2020-09-18 08:12:42.149 [] [raft-server-0-raft-partition-partition-3] WARN  io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-3}{role=FOLLOWER} - Poll request to 2 failed: java.net.ConnectException: Expected to send a message with subject 'raft-partition-partition-3-poll' to member '2', but member is not known. Known members are '[Member{id=0, address=zeebe-full-1600416712-zeebe-0.zeebe-full-1600416712-zeebe.my-zeebe.svc.cluster.local:26502, properties={}}]'.

Chart version for legacy 0.26

Zeebe 1.0.0 has numerous breaking changes.

Please maintain a version of this chart that can be pinned to Zeebe 0.26.

Hazelcast exporter not working. Zeeqs cannot fetch any data.

I'm running on GKE.

  • I suspect that the hazelcast exporter is not delivering the data to the hazelcast, and because of this, the zeeqs queries returns empty results.

To enable the hazelcast exporter I modified the helm chart to add the extraInitContainers responsible for the exporter mounting:

harts/zeebe-cluster-helm/values.yaml

extraInitContainers: |
    - name: init-exporters-hazelcast
      image: busybox:1.28
      command: ['/bin/sh', '-c']
      args: ['wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/0.8.0-alpha1/zeebe-hazelcast-exporter-0.8.0-alpha1-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al']
      volumeMounts:
      - name: exporters
        mountPath: /exporters/

I've also added the env variables for hazelcast config:

zeebe-full-helm/charts/zeebe-cluster-helm/templates/statefulset.yaml

      env:
        - name: ZEEBE_BROKER_CLUSTER_CLUSTERNAME
          value: {{ tpl .Values.global.zeebe . }}
        - name: ZEEBE_LOG_LEVEL
          value: {{ .Values.logLevel | quote }}
        - name: ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT
          value: {{ .Values.partitionCount | quote }}
        - name: ZEEBE_BROKER_CLUSTER_CLUSTERSIZE
          value: {{ .Values.clusterSize | quote }}
        - name: ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR
          value: {{ .Values.replicationFactor | quote }}
        - name: ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
          value: {{ .Values.cpuThreadCount  | quote }}
        - name: ZEEBE_BROKER_THREADS_IOTHREADCOUNT
          value: {{ .Values.ioThreadCount  | quote }}
        - name: ZEEBE_BROKER_GATEWAY_ENABLE
          value: "false"
        {{- if not .Values.global.elasticsearch.disableExporter }}
        - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME
          value: "io.zeebe.exporter.ElasticsearchExporter"
        - name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL
          value: "http://{{ .Values.global.elasticsearch.host }}:{{ .Values.global.elasticsearch.port }}"
        {{- end }}
        - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH 
          value: exporters/zeebe-hazelcast-exporter.jar
        - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME 
          value: io.zeebe.hazelcast.exporter.HazelcastExporter
        - name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
          value: "{{ .Release.Name }}-hazelcast:5701"  
        - name: ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT
          value: {{ .Values.serviceCommandPort  | quote }}
        - name: ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT
          value: {{  .Values.serviceInternalPort | quote }}
        - name: ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT
          value: {{  .Values.serviceHttpPort | quote }}         
        - name: K8S_POD_NAME

From zeebe logs, I verified that the exporter was loaded without errors:


(...)

},
"logSegmentSizeInBytes":536870912,
"atomixStorageLevel":"MAPPED",
"freeDiskSpaceCommandWatermark":315053138,
"freeDiskSpaceReplicationWatermark":105017713
},
"exporters":{
   "elasticsearch":{
      "jarPath":null,
      "className":"io.zeebe.exporter.ElasticsearchExporter",
      "args":{
         "url":"http://elasticsearch-master:9200"
      },
      "external":false
   },
   "hazelcast":{
      "jarPath":"/usr/local/zeebe/exporters/zeebe-hazelcast-exporter.jar",
      "className":"io.zeebe.hazelcast.exporter.HazelcastExporter",
      "args":null,
      "external":true
   }
},

(...)

2021-03-17 17:25:37.937 BRT
2021-03-17 20:25:37.935 [] [hz._hzInstance_1_dev.IO.thread-in-2] INFO
2021-03-17 17:25:37.937 BRT
com.hazelcast.nio.tcp.TcpIpConnection - [10.24.0.44]:5701 [dev] [3.12.6] Initialized new cluster connection between /10.24.0.44:5701 and /10.24.0.44:49313
2021-03-17 17:25:37.939 BRT
2021-03-17 20:25:37.939 [] [hz._hzInstance_1_dev.IO.thread-in-1] INFO
2021-03-17 17:25:37.939 BRT
com.hazelcast.nio.tcp.TcpIpConnection - [10.24.0.44]:5703 [dev] [3.12.6] Initialized new cluster connection between /10.24.0.44:49313 and /10.24.0.44:5701
2021-03-17 17:25:37.941 BRT
2021-03-17 20:25:37.940 [] [hz._hzInstance_1_dev.IO.thread-in-1] INFO
2021-03-17 17:25:37.941 BRT
com.hazelcast.nio.tcp.TcpIpConnection - [10.24.0.44]:5701 [dev] [3.12.6] Initialized new cluster connection between /10.24.0.44:33603 and /10.24.0.44:5703
2021-03-17 17:25:38.125 BRT
2021-03-17 20:25:38.125 [] [hz._hzInstance_1_dev.migration] INFO
2021-03-17 17:25:38.125 BRT
com.hazelcast.internal.partition.impl.MigrationManager - [10.24.0.44]:5702 [dev] [3.12.6] Re-partitioning cluster data... Migration queue size: 542
2021-03-17 17:25:38.730 BRT
2021-03-17 20:25:38.729 [Broker-2-Exporter-3] [Broker-2-zb-fs-workers-1] INFO
2021-03-17 17:25:38.730 BRT
com.hazelcast.core.LifecycleService - [10.24.0.44]:5701 [dev] [3.12.6] [10.24.0.44]:5701 is STARTED
2021-03-17 17:25:38.738 BRT
2021-03-17 20:25:38.737 [Broker-2-Exporter-3] [Broker-2-zb-fs-workers-1] INFO
2021-03-17 17:25:38.738 BRT
com.hazelcast.internal.partition.impl.PartitionStateManager - [10.24.1.60]:5701 [dev] [3.12.6] Initializing cluster partition table arrangement... 
2021-03-17 17:25:38.946 BRT
2021-03-17 20:25:38.946 [Broker-2-Exporter-1] [Broker-2-zb-fs-workers-0] INFO
2021-03-17 17:25:39.025 BRT
io.zeebe.broker.exporter.hazelcast - Export records to ringbuffer with name 'zeebe' [head: 0, tail: -1, size: 0, capacity: 10000]

On hazelcast-mancenter I verified that there is only 1 client named hz.client_1 (zeeqs) connected. I think the hazelcast exporter is also a client and should be listed.

Maybe the hazelcast exporter is not connecting with the hazelcast broker for some reason that I haven't been able to find out yet.

Cannot pull image "gcr.io/camunda-researchanddevelopment/zeebe-cluster-helm:0.0.211"

I was trying to deploy zeebe on GKE with the new helm chart release zeebe-full-helm-0.0.133, but GKE was not able to pull the image from: gcr.io/camunda-researchanddevelopment/zeebe-cluster-helm:0.0.211, causing a ImagePullBackOff.

I browsed into gcr.io/camunda-researchanddevelopment, but did not find the zeebe-cluster-helm or any image related.

I was also unable to pull the image locally using docker. The docker output:

Error response from daemon: manifest for gcr.io/camunda-researchanddevelopment/zeebe-cluster-helm:0.0.211 not found: manifest unknown: Failed to fetch "0.0.211" from request "/v2/camunda-researchanddevelopment/zeebe-cluster-helm/manifests/0.0.211".

I don't know if this is a problem with the gcr.io or with the helm chart, so I'm reporting it.

Zeebe on kubernetes: container keeps restarting

Zeebe has been running on my Kubernetes for a while, and it has been working very well. Until recently, I found that a Pod kept reporting errors and restarting.
I tried to use helm to delete the chart, then use kubectl to delete all pvc/pv, even enter the node and use docker to remove the image, and use helm to reinstall zeebe, I repeated this step many times, but the problem still exists.

➜  ~ kubectl describe pods zeebe-zeebe-0
Name:         zeebe-zeebe-0
Namespace:    default
Priority:     0
Node:         server4/192.168.137.124
Start Time:   Tue, 20 Oct 2020 16:38:54 +0800
Labels:       app.kubernetes.io/component=broker
              app.kubernetes.io/instance=zeebe
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=zeebe
              controller-revision-hash=zeebe-zeebe-555c6877cd
              statefulset.kubernetes.io/pod-name=zeebe-zeebe-0
Annotations:  <none>
Status:       Running
IP:           10.244.1.108
IPs:
  IP:           10.244.1.108
Controlled By:  StatefulSet/zeebe-zeebe
Containers:
  zeebe:
    Container ID:   docker://8f3a116d0fa70298a240a6c92d3213025b235cdfe772a7660e47291cff228659
    Image:          camunda/zeebe:0.24.2
    Image ID:       docker-pullable://camunda/zeebe@sha256:795ace31c498ad4bc37b7b0fab612307c34852f4187766e3f777a509821c9fb3
    Ports:          9600/TCP, 26501/TCP, 26502/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 20 Oct 2020 16:47:55 +0800
      Finished:     Tue, 20 Oct 2020 16:48:01 +0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:9600/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                zeebe-zeebe
      ZEEBE_LOG_LEVEL:
      ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:            3
      ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                3
      ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:          3
      ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:             2
      ZEEBE_BROKER_THREADS_IOTHREADCOUNT:              2
      ZEEBE_BROKER_GATEWAY_ENABLE:                     false
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:  io.zeebe.exporter.ElasticsearchExporter
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:   http://elasticsearch-master:9200
      ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:            26501
      ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:           26502
      ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:         9600
      K8S_POD_NAME:                                    zeebe-zeebe-0 (v1:metadata.name)
      JAVA_TOOL_OPTIONS:                               -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
    Mounts:
      /exporters from exporters (rw)
      /usr/local/bin/startup.sh from config (rw,path="startup.sh")
      /usr/local/zeebe/config/application.yaml from config (rw,path="application.yaml")
      /usr/local/zeebe/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mt74h (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-zeebe-zeebe-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      zeebe
    Optional:  false
  exporters:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-mt74h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mt74h
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned default/zeebe-zeebe-0 to server4
  Normal   Pulled     10m (x4 over 12m)     kubelet            Container image "camunda/zeebe:0.24.2" already present on machine
  Normal   Created    10m (x4 over 12m)     kubelet            Created container zeebe
  Normal   Started    10m (x4 over 12m)     kubelet            Started container zeebe
  Warning  Unhealthy  10m (x5 over 12m)     kubelet            Readiness probe failed: Get "http://10.244.1.108:9600/ready": dial tcp 10.244.1.108:9600: connect: connection refused
  Warning  Unhealthy  10m (x3 over 12m)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  BackOff    2m50s (x33 over 11m)  kubelet            Back-off restarting failed container

...and for the log

➜  ~ kubectl logs zeebe-zeebe-0
++ hostname -f
+ export ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local
+ ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local
+ export ZEEBE_BROKER_CLUSTER_NODEID=0
+ ZEEBE_BROKER_CLUSTER_NODEID=0
+ export ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ contactPointPrefix=zeebe-zeebe
+ contactPoints=
+ [[ -z '' ]]
+ (( i=0 ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
+ export ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
+ ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
++ ls -A /exporters/
+ '[' '' ']'
No exporters available.
+ echo 'No exporters available.'
+ exec /usr/local/zeebe/bin/broker
Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
2020-10-20 08:53:11,231 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
        at org.apache.logging.log4j.Level.valueOf(Level.java:320)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
        at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
        at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
        at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
        at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
        at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
        at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
        at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
        at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:196)
        at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

2020-10-20 08:53:12,913 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
        at org.apache.logging.log4j.Level.valueOf(Level.java:320)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
        at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
        at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
        at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.reinitialize(Log4J2LoggingSystem.java:204)
        at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:73)
        at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
        at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.initialize(Log4J2LoggingSystem.java:160)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:306)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:281)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:239)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:216)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
        at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

  ______  ______   ______   ____    ______     ____    _____     ____    _  __  ______   _____
 |___  / |  ____| |  ____| |  _ \  |  ____|   |  _ \  |  __ \   / __ \  | |/ / |  ____| |  __ \
    / /  | |__    | |__    | |_) | | |__      | |_) | | |__) | | |  | | | ' /  | |__    | |__) |
   / /   |  __|   |  __|   |  _ <  |  __|     |  _ <  |  _  /  | |  | | |  <   |  __|   |  _  /
  / /__  | |____  | |____  | |_) | | |____    | |_) | | | \ \  | |__| | | . \  | |____  | | \ \
 /_____| |______| |______| |____/  |______|   |____/  |_|  \_\  \____/  |_|\_\ |______| |_|  \_\

2020-10-20 08:53:13.232 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v0.24.2 on zeebe-zeebe-0 with PID 6 (/usr/local/zeebe/lib/zeebe-distribution-0.24.2.jar started by root in /usr/local/zeebe)
2020-10-20 08:53:13.300 [] [main] INFO  io.zeebe.broker.StandaloneBroker - No active profile set, falling back to default profiles: default
2020-10-20 08:53:18.335 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 9600 (http)
2020-10-20 08:53:18.405 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-10-20 08:53:18.406 [] [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-10-20 08:53:18.407 [] [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-10-20 08:53:18.797 [] [main] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-10-20 08:53:18.798 [] [main] INFO  org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 5368 ms
2020-10-20 08:53:20.219 [] [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-10-20 08:53:21.126 [] [main] INFO  org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 2 endpoint(s) beneath base path '/actuator'
2020-10-20 08:53:21.218 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-10-20 08:53:21.394 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 9600 (http) with context path ''
2020-10-20 08:53:21.415 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 9.509 seconds (JVM running for 13.396)
2020-10-20 08:53:21.536 [] [main] INFO  io.zeebe.broker.system - Version: 0.24.2
2020-10-20 08:53:21.706 [] [main] INFO  io.zeebe.broker.system - Starting broker 0 with configuration {
  "network" : {
    "host" : "0.0.0.0",
    "portOffset" : 0,
    "maxMessageSize" : "4MB",
    "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
    "commandApi" : {
      "host" : "0.0.0.0",
      "port" : 26501,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 26501,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26501",
      "address" : "0.0.0.0:26501"
    },
    "internalApi" : {
      "host" : "0.0.0.0",
      "port" : 26502,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 26502,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502",
      "address" : "0.0.0.0:26502"
    },
    "monitoringApi" : {
      "host" : "0.0.0.0",
      "port" : 9600,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 9600,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:9600",
      "address" : "0.0.0.0:9600"
    },
    "maxMessageSizeInBytes" : 4194304
  },
  "cluster" : {
    "initialContactPoints" : [ "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502", "zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502", "zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502" ],
    "partitionIds" : [ 1, 2, 3 ],
    "nodeId" : 0,
    "partitionsCount" : 3,
    "replicationFactor" : 3,
    "clusterSize" : 3,
    "clusterName" : "zeebe-zeebe",
    "membership" : {
      "broadcastUpdates" : false,
      "broadcastDisputes" : true,
      "notifySuspect" : false,
      "gossipInterval" : "PT0.25S",
      "gossipFanout" : 2,
      "probeInterval" : "PT1S",
      "probeTimeout" : "PT2S",
      "suspectProbes" : 3,
      "failureTimeout" : "PT10S",
      "syncInterval" : "PT10S"
    }
  },
  "threads" : {
    "cpuThreadCount" : 2,
    "ioThreadCount" : 2
  },
  "data" : {
    "directories" : [ "/usr/local/zeebe/data" ],
    "logSegmentSize" : "512MB",
    "snapshotPeriod" : "PT15M",
    "logIndexDensity" : 100,
    "logSegmentSizeInBytes" : 536870912,
    "atomixStorageLevel" : "DISK"
  },
  "exporters" : {
    "elasticsearch" : {
      "jarPath" : null,
      "className" : "io.zeebe.exporter.ElasticsearchExporter",
      "args" : {
        "url" : "http://elasticsearch-master:9200"
      },
      "external" : false
    }
  },
  "gateway" : {
    "network" : {
      "host" : "0.0.0.0",
      "port" : 26500,
      "minKeepAliveInterval" : "PT30S"
    },
    "cluster" : {
      "contactPoint" : "0.0.0.0:26502",
      "requestTimeout" : "PT15S",
      "clusterName" : "zeebe-cluster",
      "memberId" : "gateway",
      "host" : "0.0.0.0",
      "port" : 26502,
      "membership" : {
        "broadcastUpdates" : false,
        "broadcastDisputes" : true,
        "notifySuspect" : false,
        "gossipInterval" : "PT0.25S",
        "gossipFanout" : 2,
        "probeInterval" : "PT1S",
        "probeTimeout" : "PT2S",
        "suspectProbes" : 3,
        "failureTimeout" : "PT10S",
        "syncInterval" : "PT10S"
      }
    },
    "threads" : {
      "managementThreads" : 1
    },
    "monitoring" : {
      "enabled" : false,
      "host" : "0.0.0.0",
      "port" : 9600
    },
    "security" : {
      "enabled" : false,
      "certificateChainPath" : null,
      "privateKeyPath" : null
    },
    "longPolling" : {
      "enabled" : true
    },
    "initialized" : true,
    "enable" : false
  },
  "backpressure" : {
    "enabled" : true,
    "algorithm" : "VEGAS",
    "aimd" : {
      "requestTimeout" : "PT1S",
      "initialLimit" : 100,
      "minLimit" : 1,
      "maxLimit" : 1000,
      "backoffRatio" : 0.9
    },
    "fixedLimit" : {
      "limit" : 20
    },
    "vegas" : {
      "alpha" : 3,
      "beta" : 6,
      "initialLimit" : 20
    },
    "gradient" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0
    },
    "gradient2" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0,
      "longWindow" : 600
    }
  },
  "stepTimeout" : "PT5M",
  "executionMetricsExporterEnabled" : false
}
2020-10-20 08:53:21.733 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [1/10]: actor scheduler
2020-10-20 08:53:21.810 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [2/10]: membership and replication protocol
2020-10-20 08:53:23.806 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-10-20 08:53:23.814 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-10-20 08:53:24.003 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Completed initialization in 189 ms
2020-10-20 08:53:30.219 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [3/10]: command api transport
2020-10-20 08:53:30.725 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [4/10]: command api handler
2020-10-20 08:53:31.199 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [5/10]: subscription api
2020-10-20 08:53:31.239 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services
2020-10-20 08:53:36.339 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager
2020-10-20 08:53:36.341 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: monitoring services
2020-10-20 08:53:36.349 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [9/10]: leader management request handler
2020-10-20 08:53:36.357 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [10/10]: zeebe partitions
2020-10-20 08:53:36.360 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [1/3]: partition 3
2020-10-20 08:53:36.796 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [2/3]: partition 2
2020-10-20 08:53:36.816 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [3/3]: partition 1
2020-10-20 08:53:36.903 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions succeeded. Started 3 steps in 543 ms.
2020-10-20 08:53:36.904 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 succeeded. Started 10 steps in 15172 ms.

Version Info:

➜  ~ kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
server1   Ready    master   23d   v1.19.2
server2   Ready    <none>   23d   v1.19.2
server3   Ready    <none>   23d   v1.19.2
server4   Ready    <none>   23d   v1.19.2
➜  ~ helm install zeebe zeebe/zeebe-full
NAME: zeebe
LAST DEPLOYED: Wed Oct 21 09:56:10 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
______     ______     ______     ______     ______
/\___  \   /\  ___\   /\  ___\   /\  == \   /\  ___\
\/_/  /__  \ \  __\   \ \  __\   \ \  __<   \ \  __\
  /\_____\  \ \_____\  \ \_____\  \ \_____\  \ \_____\
  \/_____/   \/_____/   \/_____/   \/_____/   \/_____/


(zeebe-full - 0.0.107)

- Cluster Name: zeebe-zeebe

Can't deploy full helm charts without values file

With just running helm install zeebe-cluster zeebe/zeebe-full I exepct to be able to deploy the complete full helm chart, but it turns out that this fails.

The standalone gateway keeps in the container creating phase.

[zell zell-test-gw/ ns:zell-test-gw]$ kgpo
NAME                                                           READY   STATUS              RESTARTS   AGE
elasticsearch-master-0                                         1/1     Running             0          82s
elasticsearch-master-1                                         1/1     Running             0          82s
elasticsearch-master-2                                         1/1     Running             0          82s
zeebe-cluster-nginx-ingress-controller-89b6fcbb6-j8l72         1/1     Running             0          82s
zeebe-cluster-nginx-ingress-default-backend-7c987dd75d-kr8hp   1/1     Running             0          82s
zeebe-cluster-operate-6cbff9f89d-wx7jv                         1/1     Running             1          82s
zeebe-cluster-zeebe-0                                          1/1     Running             0          82s
zeebe-cluster-zeebe-1                                          1/1     Running             0          82s
zeebe-cluster-zeebe-2                                          1/1     Running             0          82s
zeebe-cluster-zeebe-gateway-d59f5845-2cvbb                     0/1     ContainerCreating   0          82s
[zell zell-test-gw/ ns:zell-test-gw]$ k describe pod zeebe-cluster-zeebe-gateway-d59f5845-2cvbb 
Name:           zeebe-cluster-zeebe-gateway-d59f5845-2cvbb
Namespace:      zell-test-gw
Priority:       0
Node:           gke-zeebe-cluster-default-pool-1f44de48-j12j/10.132.0.75
Start Time:     Mon, 03 Aug 2020 14:29:17 +0200
Labels:         app.kubernetes.io/component=gateway
                app.kubernetes.io/instance=zeebe-cluster
                app.kubernetes.io/managed-by=Helm
                app.kubernetes.io/name=zeebe
                app.kubernetes.io/version=0.23.4
                helm.sh/chart=zeebe-0.0.127
                pod-template-hash=d59f5845
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/zeebe-cluster-zeebe-gateway-d59f5845
Containers:
  zeebe:
    Container ID:   
    Image:          camunda/zeebe:0.23.4
    Image ID:       
    Ports:          9600/TCP, 26500/TCP, 26502/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Readiness:      tcp-socket :gateway delay=20s timeout=1s period=5s #success=1 #failure=3
    Environment:
      ZEEBE_STANDALONE_GATEWAY:            true
      ZEEBE_GATEWAY_CLUSTER_CLUSTERNAME:   zeebe-cluster-zeebe
      ZEEBE_GATEWAY_CLUSTER_MEMBERID:      zeebe-cluster-zeebe-gateway-d59f5845-2cvbb (v1:metadata.name)
      ZEEBE_LOG_LEVEL:                     info
      JAVA_TOOL_OPTIONS:                   -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
      ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT:  zeebe-cluster-zeebe:26502
      ZEEBE_GATEWAY_NETWORK_HOST:          0.0.0.0
      ZEEBE_GATEWAY_NETWORK_PORT:          26500
      ZEEBE_GATEWAY_CLUSTER_HOST:           (v1:status.podIP)
      ZEEBE_GATEWAY_CLUSTER_PORT:          26502
      ZEEBE_GATEWAY_MONITORING_HOST:       0.0.0.0
      ZEEBE_GATEWAY_MONITORING_PORT:       9600
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-p8rvq (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      zeebe-cluster-zeebe
    Optional:  false
  default-token-p8rvq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-p8rvq
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age               From                                                   Message
  ----     ------       ----              ----                                                   -------
  Normal   Scheduled    69s               default-scheduler                                      Successfully assigned zell-test-gw/zeebe-cluster-zeebe-gateway-d59f5845-2cvbb to gke-zeebe-cluster-default-pool-1f44de48-j12j
  Warning  FailedMount  5s (x8 over 69s)  kubelet, gke-zeebe-cluster-default-pool-1f44de48-j12j  MountVolume.SetUp failed for volume "config" : configmap "zeebe-cluster-zeebe" not found

It search's for the configmap which seem not to be specified as default.

With the zeebe-cluster chart this seem to be possible.

[zell zell-test-gw/ ns:zell-test-gw]$ helm install zeebe-cluster zeebe/zeebe-cluster 
NAME: zeebe-cluster
LAST DEPLOYED: Mon Aug  3 14:34:25 2020
NAMESPACE: zell-test-gw
STATUS: deployed
REVISION: 1
NOTES:
______     ______     ______     ______     ______    
/\___  \   /\  ___\   /\  ___\   /\  == \   /\  ___\   
\/_/  /__  \ \  __\   \ \  __\   \ \  __<   \ \  __\   
  /\_____\  \ \_____\  \ \_____\  \ \_____\  \ \_____\ 
  \/_____/   \/_____/   \/_____/   \/_____/   \/_____/                                                                                                                                      

(zeebe-cluster - 0.0.128)

- Docker Image used for Broker: camunda/zeebe:0.23.4
- Cluster Name: "zeebe-cluster-zeebe"
- ElasticSearch Enabled: true - URL: http://elasticsearch-master:9200
- Kibana Enabled: false
- Prometheus Enabled: false
- Prometheus ServiceMonitor Enabled: false

The Cluster itself is not exposed as a service that means that you can use kubectl port-forward to access the Zeebe cluster from outside Kubernetes:

> kubectl port-forward svc/zeebe-cluster-zeebe-gateway 26500:26500 -n zell-test-gw

Now you can connect your workers and clients to `localhost:26500`
[zell zell-test-gw/ ns:zell-test-gw]$ kgpo -w
NAME                                           READY   STATUS              RESTARTS   AGE
elasticsearch-master-0                         0/1     Init:0/1            0          3s
elasticsearch-master-1                         0/1     Init:0/1            0          3s
elasticsearch-master-2                         0/1     Init:0/1            0          3s
zeebe-cluster-zeebe-0                          0/1     ContainerCreating   0          2s
zeebe-cluster-zeebe-1                          0/1     ContainerCreating   0          2s
zeebe-cluster-zeebe-gateway-6fcc4d5949-8vlnt   0/1     Running             0          3s
zeebe-cluster-zeebe-2                          0/1     Pending             0          0s
zeebe-cluster-zeebe-2                          0/1     Pending             0          1s
zeebe-cluster-zeebe-2                          0/1     ContainerCreating   0          1s
elasticsearch-master-2                         0/1     PodInitializing     0          10s
elasticsearch-master-2                         0/1     Running             0          11s
zeebe-cluster-zeebe-2                          0/1     Running             0          11s
elasticsearch-master-1                         0/1     PodInitializing     0          18s
zeebe-cluster-zeebe-0                          0/1     Running             0          18s
zeebe-cluster-zeebe-1                          0/1     Running             0          18s
elasticsearch-master-1                         0/1     Running             0          19s
zeebe-cluster-zeebe-gateway-6fcc4d5949-8vlnt   1/1     Running             0          24s

Operate returns 502 Bad Gateway

I installed the zeebe-full-helm chart in my k8s cluster. using the command :
"""
helm install zeebe-homologation zeebe/zeebe-full-helm
"""
The cluster was successfully installed
""""
NAME: zeebe-homologation
LAST DEPLOYED: Wed May 26 10:33:50 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:


/___ \ /\ \ /\ \ /\ == \ /\
/
/ / \ \ __\ \ \ __\ \ \ __< \ \ __
/_
\ \ _
\ \ _\ \ _\ \ ___
// // // // /_____/

(zeebe-full-helm - 0.0.143)
""""
However when I open the operate web page I have "502 Bad Gateway".
The operate logs show the following
""""

This Operate Trial Version is for testing and non-production use only.
The General Terms and Conditions for the Operate Trial Version are available here: https://zeebe.io/legal/operate-evaluation-license/

Login with the credentials:
User: demo
Password: demo


/ | /\ | / | | | | | | \ /\ / _ | _ \ | _ \ /_ | |
| (
/ _ | |/| | |
| | .` | |) / _ \ | (
) | _/ _|| / / _ | | | _|
_// __| ||_
/||_|// _\ _/|| |||// __| |___|

						   1.0.0

2021-05-26 10:34:08.670 INFO 7 --- [ main] i.c.o.Application : Starting Application using Java 11.0.11 on zeebe-homologation-zeebe-operate-helm-58745f69cf-g964q with PID 7 (/usr/local/operate/lib/operate-webapp-1.0.0-exec.jar started by root in /usr/local/operate)
2021-05-26 10:34:08.845 INFO 7 --- [ main] i.c.o.Application : The following profiles are active: auth
2021-05-26 10:34:23.701 INFO 7 --- [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-05-26 10:34:23.785 INFO 7 --- [ main] o.a.c.h.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"]
2021-05-26 10:34:23.788 INFO 7 --- [ main] o.a.c.c.StandardService : Starting service [Tomcat]
2021-05-26 10:34:23.790 INFO 7 --- [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.45]
2021-05-26 10:34:24.278 INFO 7 --- [ main] o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext
2021-05-26 10:34:24.278 INFO 7 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 15117 ms
2021-05-26 10:34:26.125 INFO 7 --- [ main] i.c.o.WebappModuleConfiguration : Starting module: webapp
2021-05-26 10:34:28.199 WARN 7 --- [ main] i.c.o.e.ElasticsearchConnector : Username and/or password for are empty. Basic authentication for elasticsearch is not used.
2021-05-26 10:34:31.704 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:35.723 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:40.731 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:45.735 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:50.738 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:54.742 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:34:58.746 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:35:02.750 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:35:07.754 WARN 7 --- [ main] i.c.o.u.RetryOperation : Timeout connecting to [elasticsearch-master/10.43.236.4:9200]
2021-05-26 10:35:11.777 INFO 7 --- [ main] o.s.s.c.ThreadPoolTaskExecutor : Initializing ExecutorService
2021-05-26 10:35:11.779 INFO 7 --- [ main] o.s.s.c.ThreadPoolTaskExecutor : Initializing ExecutorService 'migrationThreadPoolExecutor'
2021-05-26 10:35:11.780 INFO 7 --- [ main] i.c.o.s.SchemaStartup : SchemaStartup started.
2021-05-26 10:35:11.780 INFO 7 --- [ main] i.c.o.s.SchemaStartup : SchemaStartup: validate schema.
2021-05-26 10:35:11.869 WARN 7 --- [ main] o.e.c.RestClient : request [GET http://elasticsearch-master:9200/operate*?master_timeout=30s&ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-6.8.5-78990e9 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
2021-05-26 10:35:12.361 WARN 7 --- [ main] o.e.c.RestClient : request [GET http://elasticsearch-master:9200/operate*?master_timeout=30s&ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false] returned 1 warnings: [299 Elasticsearch-6.8.5-78990e9 "[types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions."]
2021-05-26 10:35:12.365 INFO 7 --- [ main] i.c.o.s.SchemaStartup : SchemaStartup: schema is empty or not complete. Indices will be created.
2021-05-26 10:35:12.369 INFO 7 --- [ main] i.c.o.s.ElasticsearchSchemaManager : Create default settings from 'operate_template' with 1 shards and 0 replicas per index.
2021-05-26 10:35:12.399 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:14.403 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:16.409 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:18.414 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:20.418 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:22.422 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:24.426 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:26.431 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:28.435 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:30.439 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:32.444 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:34.448 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:36.453 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:38.457 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:40.461 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]
2021-05-26 10:35:42.465 WARN 7 --- [ main] i.c.o.u.RetryOperation : method [HEAD], host [http://elasticsearch-master:9200], URI [/_index_template/operate_template?master_timeout=30s], status line [HTTP/1.1 405 Method Not Allowed]

"""
.
PS: the docker-compose version works fine

helm install zeebe/zeebe-full --version 0.0.74 is returning an error

Using helm install zeebe-cluster-production zeebe/zeebe-full --version 0.0.74 is returning the following error:

Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: unknown object type "nil" in ConfigMap.data.application.yaml

Environment context:
This is a fresh helm install on a GKE cluster with nodes running Kubernetes 1.15

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.