Comments (13)
yanivbh1
commented on June 16, 2024
Moved to https://github.com/memphisdev/memphis-k8s
from memphis-k8s.
valeraBr
commented on June 16, 2024
Hi @Cave-Johnson,
Can you share the exact helm command you used to deploy Memphis?
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
The command I am using is below, sourced from the documentation
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update
helm install memphis memphis/memphis \
--create-namespace \
--namespace memphis \
--set cluster.enabled="true" \
# --analytics="false" \
--wait
EDIT: when the analytics line is removed, the wait fails with Error: INSTALLATION FAILED: timed out waiting for the condition (the commented out also removes the --wait
running the helm install command with --debug shows
install.go:209: [debug] CHART PATH: /home/bootstrap/.cache/helm/repository/memphis-0.4.2.tgz
client.go:128: [debug] creating 1 resource(s)
client.go:128: [debug] creating 16 resource(s)
wait.go:48: [debug] beginning wait for 16 resources with timeout of 5m0s
ready.go:277: [debug] Deployment is not ready: memphis/memphis-http-proxy. 0 out of 2 expected pods are ready
ready.go:277: [debug] Deployment is not ready: memphis/memphis-http-proxy. 0 out of 2 expected pods are ready
...
from memphis-k8s.
valeraBr
commented on June 16, 2024
Thanks @Cave-Johnson, can you share the output of
kubectl get nodes -o wide
Just want to understand your environment to reproduce it.
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k3s-master-01 Ready control-plane,etcd,master 71d v1.24.6+k3s1 172.16.1.91 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
k3s-master-02 Ready control-plane,etcd,master 71d v1.24.6+k3s1 172.16.1.92 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
k3s-master-03 Ready control-plane,etcd,master 71d v1.24.6+k3s1 172.16.1.93 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
k3s-node-01 Ready <none> 71d v1.24.6+k3s1 172.16.1.94 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
k3s-node-02 Ready <none> 71d v1.24.6+k3s1 172.16.1.95 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
k3s-node-03 Ready <none> 71d v1.24.6+k3s1 172.16.1.96 <none> Ubuntu 22.04.1 LTS 5.15.0-53-generic containerd://1.6.8-k3s1
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
I have built some install scripts based off the work here:
- https://github.com/techno-tim/k3s-ansible - should get you a small k3s cluster up and running really quicky
- https://youtu.be/CbkEWcUZ7zM - video with more details
from memphis-k8s.
valeraBr
commented on June 16, 2024
@Cave-Johnson thanks for the reference it definitely will speed up the reproduction process.
I need some additional details.
Please share the output of the following:
- kubectl get svc -n memphis
- kubectl describe svc memphis-cluster -n memphis
- kubectl get pods -o wide -n memphis
I want to mention once again, we really appreciate your help!
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
kubectl get svc -n memphis
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
memphis-cluster ClusterIP None <none> 9000/TCP,7770/TCP,6666/TCP,8222/TCP 66s
memphis-http-proxy ClusterIP 10.43.15.83 <none> 4444/TCP 66s
mongo ClusterIP None <none> 27017/TCP 66s
kubectl describe svc memphis-cluster -n memphis
Name: memphis-cluster
Namespace: memphis
Labels: app.kubernetes.io/instance=memphis
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=memphis
app.kubernetes.io/version=0.4.2
helm.sh/chart=memphis-0.4.2
Annotations: meta.helm.sh/release-name: memphis
meta.helm.sh/release-namespace: memphis
Selector: app.kubernetes.io/instance=memphis,app.kubernetes.io/name=memphis
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: memphis-cp-management 9000/TCP
TargetPort: 9000/TCP
Endpoints: <none>
Port: memphis-ws 7770/TCP
TargetPort: 7770/TCP
Endpoints: <none>
Port: memphis-cp-tcp 6666/TCP
TargetPort: 6666/TCP
Endpoints: <none>
Port: monitor 8222/TCP
TargetPort: 8222/TCP
Endpoints: <none>
Session Affinity: None
Events: <none>
kubectl get pods -o wide -n memphis
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
memphis-http-proxy-59bfc88644-9mj2m 0/1 Init:0/1 0 113s 10.42.5.121 k3s-node-03 <none> <none>
memphis-http-proxy-59bfc88644-w8pfk 0/1 Init:0/1 0 113s 10.42.3.132 k3s-node-01 <none> <none>
from memphis-k8s.
valeraBr
commented on June 16, 2024
@Cave-Johnson Looks like the statefulsets that are responsible for memphis-broker failed to do their job.
Memphis use statefulset resources for broker and DB pods therefore there are some dependencies that should be solved.
Can you please provide the following:
- kubectl get sts -n memphis
- kubectl describe sts memphis-broker -n memphis
- kubectl describe sts memphis-mongodb -n memphis
- kubectl get pvc -n memphis
- kubectl get pv -n memphis
I believe we will find the RC with these details.
Thanks.
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
kubectl get sts -n memphis
NAME READY AGE
memphis-broker 0/3 24h
memphis-mongodb 0/2 24h
kubectl describe sts memphis-broker -n memphis
Name: memphis-broker
Namespace: memphis
CreationTimestamp: Sun, 11 Dec 2022 22:49:17 +0000
Selector: app.kubernetes.io/instance=memphis,app.kubernetes.io/name=memphis
Labels: app.kubernetes.io/instance=memphis
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=memphis
app.kubernetes.io/version=0.4.2
helm.sh/chart=memphis-0.4.2
Annotations: meta.helm.sh/release-name: memphis
meta.helm.sh/release-namespace: memphis
Replicas: 3 desired | 0 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/instance=memphis
app.kubernetes.io/name=memphis
Service Account: memphis-memphis-cp-account
Init Containers:
k8s-busybox-waits4mongo-cluster:
Image: busybox:1.28
Port: <none>
Host Port: <none>
Command:
sh
-c
until nslookup memphis-mongodb-1.mongo.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for memphis-mongodb-1; sleep 2; done
Environment:
NAMESPACE: memphis
Mounts: <none>
k8s-busybox-initiliaze-mongodb:
Image: memphisos/k8s-busybox:latest
Port: <none>
Host Port: <none>
Environment:
NAMESPACE: memphis
Mounts: <none>
Containers:
memphis-broker:
Image: memphisos/memphis-broker:latest
Ports: 9000/TCP, 7770/TCP, 6666/TCP, 8222/TCP, 6222/TCP
Host Ports: 9000/TCP, 7770/TCP, 6666/TCP, 0/TCP, 0/TCP
Command:
nats-server
--config
/etc/nats-config/nats.conf
--auth=$(CONNECTION_TOKEN)
Liveness: http-get http://:8222/ delay=10s timeout=5s period=60s #success=1 #failure=3
Startup: http-get http://:8222/ delay=10s timeout=5s period=10s #success=1 #failure=30
Environment:
POD_NAME: (v1:metadata.name)
SERVER_NAME: $(POD_NAME)
POD_NAMESPACE: (v1:metadata.namespace)
CLUSTER_ADVERTISE: $(POD_NAME).memphis-cluster.$(POD_NAMESPACE).svc.cluster.local
ANALYTICS: true
LOGS_RETENTION_IN_DAYS: 3
K8S_NAMESPACE: memphis
ROOT_PASSWORD: <set to the key 'ROOT_PASSWORD' in secret 'memphis-creds'> Optional: false
CONNECTION_TOKEN: <set to the key 'CONNECTION_TOKEN' in secret 'memphis-creds'> Optional: false
JWT_SECRET: <set to the key 'JWT_SECRET' in secret 'memphis-creds'> Optional: false
REFRESH_JWT_SECRET: <set to the key 'REFRESH_JWT_SECRET' in secret 'memphis-creds'> Optional: false
Mounts:
/data from memphis-broker-js-pvc (rw)
/etc/nats-config from config-volume (rw)
/var/run/nats from pid (rw)
reloader:
Image: memphisos/memphis-config-reloader:latest
Port: <none>
Host Port: <none>
Command:
memphis-config-reloader
-pid
/var/run/nats/nats.pid
-config
/etc/nats-config/nats.conf
Environment: <none>
Mounts:
/etc/nats-config from config-volume (rw)
/var/run/nats from pid (rw)
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: memphis-broker-config
Optional: false
pid:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
Volume Claims:
Name: memphis-broker-js-pvc
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 6m42s (x107 over 24h) statefulset-controller create Claim memphis-broker-js-pvc-memphis-broker-0 for Pod memphis-broker-0 in StatefulSet memphis-broker failed error: persistentvolumeclaims "memphis-broker-js-pvc-memphis-broker-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
Warning FailedCreate 6m42s (x107 over 24h) statefulset-controller create Pod memphis-broker-0 in StatefulSet memphis-broker failed error: failed to create PVC memphis-broker-js-pvc-memphis-broker-0: persistentvolumeclaims "memphis-broker-js-pvc-memphis-broker-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
kubectl describe sts memphis-mongodb -n memphis
Name: memphis-mongodb
Namespace: memphis
CreationTimestamp: Sun, 11 Dec 2022 22:49:17 +0000
Selector: app=mongo
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: memphis
meta.helm.sh/release-namespace: memphis
Replicas: 2 desired | 0 total
Update Strategy: RollingUpdate
Partition: 0
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app=mongo
selector=mongo
Service Account: memphis-memphis-mongo-account
Containers:
mongodb:
Image: memphisos/mongo:4.2
Port: 27017/TCP
Host Port: 0/TCP
Command:
/bin/sh
Args:
-c
mongod --replSet=rs0 --bind_ip_all
Limits:
cpu: 1
memory: 1500Mi
Requests:
cpu: 1
memory: 1000Mi
Environment: <none>
Mounts:
/data/db from memphis-memphis-mongo-data (rw)
/mnt/scripts from mongo-startup-script (rw)
Volumes:
mongo-startup-script:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mongo-startup-script
Optional: false
Volume Claims:
Name: memphis-memphis-mongo-data
StorageClass:
Labels: <none>
Annotations: <none>
Capacity: 10Gi
Access Modes: [ReadWriteOnce]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 7m5s (x107 over 24h) statefulset-controller create Claim memphis-memphis-mongo-data-memphis-mongodb-0 for Pod memphis-mongodb-0 in StatefulSet memphis-mongodb failed error: persistentvolumeclaims "memphis-memphis-mongo-data-memphis-mongodb-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
Warning FailedCreate 7m5s (x107 over 24h) statefulset-controller create Pod memphis-mongodb-0 in StatefulSet memphis-mongodb failed error: failed to create PVC memphis-memphis-mongo-data-memphis-mongodb-0: persistentvolumeclaims "memphis-memphis-mongo-data-memphis-mongodb-0" is forbidden: Internal error occurred: 2 default StorageClasses were found
kubectl get pvc -n memphis
No resources found in memphis namespace.
kubectl get pv -n memphis
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-04e9dd0b-7f48-488e-866e-06d0106df828 10Gi RWO Delete Bound kafka/data-project-kafka-zookeeper-2 longhorn 72d
pvc-1559f425-7c95-4621-ad27-fc59bc2e56f7 10Gi RWO Delete Bound kafka/data-project-kafka-zookeeper-0 longhorn 72d
pvc-61041be7-7147-4a4e-b08c-7da302a358dc 10Gi RWO Delete Bound kafka/data-0-project-kafka-kafka-0 longhorn 72d
pvc-6e657ae0-7b2d-48a2-b520-2cdefaca412f 10Gi RWO Delete Bound kafka/data-0-project-kafka-kafka-1 longhorn 72d
pvc-a004f50e-d114-4465-857d-31f71a021302 10Gi RWO Delete Bound kafka/data-0-project-kafka-kafka-2 longhorn 72d
pvc-f04ceeed-c773-415d-878e-4be6bc981a87 10Gi RWO Delete Bound kafka/data-project-kafka-zookeeper-1 longhorn 72d
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
looks like the problem was with my StorageClasses Internal error occurred: 2 default StorageClasses were found
Not sure how that happened (or when) but i have weirdly been able to deploy other services (like kafka)
I have fixed the issue of having 2 default storageclasses and memphis is now deploying as expected. Thanks for the help debugging!
from memphis-k8s.
valeraBr
commented on June 16, 2024
You welcome. Any missing information that we might need to add to our docs?
from memphis-k8s.
Cave-Johnson
commented on June 16, 2024
No i don't believe so - other than maybe a debugging section to include some of the above commands.
I don't know if its possible for the helm chart to do things like check there is only one default storageclass, could stop some headaches and time debugging (the original DNS error didn't make it clear where the problem was).
from memphis-k8s.
Related Issues (20)
- Add plain K8S yamls, without HELM
- Set pullPolicy for busybox top IfNotPresent HOT 1
- An ability to provide custom secret for memphis creds. HOT 3
- Unable to specify a secret as external PostgreSQL credential source HOT 3
- Add global value for metadata pvc configuration HOT 1
- Why does the memphis statefulset use host ports? HOT 4
- Can't install using flux due to duplicate key HOT 4
- Add support for automatic protocol detection for most common service meshes HOT 2
- ServiceMonitor resource endpoint spec is using port instead of targetPort HOT 1
- Move user configuration to a secret HOT 5
- Helm deployment fails on K8S Cluster HOT 9
- REST Gateway Pods missing tolerations
- Create k8s operator
- Create k8s operator
- Check Memphis deployment on OCP HOT 1
- Memphis broker should run on separate nodes HOT 1
- busy box not working well HOT 1
- External mongodb HOT 4
- Refreshing creds every deployment HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from memphis-k8s.