Comments (6)
cescoferraro
commented on July 25, 2024
Strange thing I notice is that if I enable auth on the sidecar and not on mongod, the replicas are able to connect,
2018-05-02T21:08:03.850+0000 I ACCESS [conn266] Successfully authenticated as principal cesco on admin
2018-05-02T21:08:03.851+0000 I NETWORK [conn266] end connection 127.0.0.1:36076 (6 connections now open)
2018-05-02T21:08:08.876+0000 I NETWORK [listener] connection accepted from 127.0.0.1:36094 #267 (7 connections now open)
2018-05-02T21:08:08.876+0000 I NETWORK [conn267] received client metadata from 127.0.0.1:36094 conn267: { driver: { name: "nodejs", version: "2.2.35" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.4.115-k8s" }, platform: "Node.js v9.8.0, LE, mongodb-core: 2.1.19" }
2018-05-02T21:08:08.880+0000 I ACCESS [conn267] Successfully authenticated as principal cesco on admin
2018-05-02T21:08:08.881+0000 I NETWORK [conn267] end connection 127.0.0.1:36094 (6 connections now open)
but the sidecar logs get full of
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
As soon as I enable --auth flag on the mongod I replicas fail to connect to master
from mongo-k8s-sidecar.
cescoferraro
commented on July 25, 2024
Finally got it to work with https://gist.github.com/thilinapiy/0c5abc2c0c28efe1bbe2165b0d8dc115
Only thing I change is the password and the replica count. I using 3 without any issues.
I just cannot realize why it does not work the way I was doing. Enabling auth after setting a sudo user seems to be the mongo way to do this 🤷♂️🤷♂️🤷♂️
from mongo-k8s-sidecar.
supereagle
commented on July 25, 2024
Strange thing I notice is that if I enable auth on the sidecar and not on mongod, the replicas are able to connect,
@cescoferraro I meet the same problem, how do you enable auth on the sidecar?
My yaml is:
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
helm.sh/namespace: default
helm.sh/path: mongo
helm.sh/release: infra-mongo
creationTimestamp: 2018-05-21T04:40:31Z
generation: 23
labels:
controller.caicloud.io/chart: mongo
controller.caicloud.io/release: infra-mongo
name: infra-mongo-mongo-v1-0
namespace: default
ownerReferences:
- apiVersion: release.caicloud.io/v1alpha1
kind: Release
name: infra-mongo
uid: 10519cea-5cb1-11e8-8fec-5254000a3441
resourceVersion: "604588"
selfLink: /apis/apps/v1/namespaces/default/statefulsets/infra-mongo-mongo-v1-0
uid: 1804b652-5cb1-11e8-94c2-52540017abeb
spec:
podManagementPolicy: OrderedReady
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
controller.caicloud.io/chart: mongo
controller.caicloud.io/name: infra-mongo-mongo-v1-0
controller.caicloud.io/release: infra-mongo
serviceName: mgo-cluster
template:
metadata:
annotations:
helm.sh/namespace: default
helm.sh/path: mongo
helm.sh/release: infra-mongo
creationTimestamp: null
labels:
controller.caicloud.io/chart: mongo
controller.caicloud.io/name: infra-mongo-mongo-v1-0
controller.caicloud.io/release: infra-mongo
spec:
containers:
- args:
- mongod
- --replSet
- rs0
- --bind_ip
- 0.0.0.0
- --smallfiles
- --noprealloc
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: MONGO_INITDB_ROOT_USERNAME
value: admin
- name: MONGO_INITDB_ROOT_PASSWORD
value: Pwd123456
image: mongo:3.6.4
imagePullPolicy: Always
name: mongo
ports:
- containerPort: 27017
name: tcp-27017
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /docker-entrypoint-initdb.d
name: init-js
- env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
- name: NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: MONGODB_USERNAME
value: admin
- name: MONGODB_PASSWORD
value: Pwd123456
- name: MONGODB_DATABASE
value: admin
- name: MONGO_SIDECAR_POD_LABELS
value: controller.caicloud.io/release=infra-mongo
- name: MONGO_PORT
value: "27017"
- name: KUBERNETES_MONGO_SERVICE_NAME
value: mgo-cluster
image: mongo-k8s-sidecar:v0.0.1
imagePullPolicy: Always
name: mongo-sidecar
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /data/db
type: ""
name: mongo-storage
updateStrategy:
type: RollingUpdate
Log from mongo:
2018-05-24T01:41:55.058+0000 I NETWORK [listener] connection accepted from 192.168.65.40:37036 #61 (7 connections now open)
2018-05-24T01:41:55.059+0000 I NETWORK [conn61] received client metadata from 192.168.65.40:37036 conn61: { driver: { name: "nodejs", version: "2.2.35" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-693.el7.x86_64" }, platform: "Node.js v9.8.0, LE, mongodb-core: 2.1.19" }
2018-05-24T01:41:55.067+0000 I ACCESS [conn61] Successfully authenticated as principal admin on admin
2018-05-24T01:41:55.068+0000 I NETWORK [conn61] end connection 192.168.65.40:37036 (6 connections now open)
2018-05-24T01:41:55.893+0000 I NETWORK [listener] connection accepted from 127.0.0.1:57002 #62 (7 connections now open)
2018-05-24T01:41:55.923+0000 I NETWORK [conn62] received client metadata from 127.0.0.1:57002 conn62: { driver: { name: "nodejs", version: "2.2.35" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-693.el7.x86_64" }, platform: "Node.js v9.8.0, LE, mongodb-core: 2.1.19" }
2018-05-24T01:41:55.930+0000 I ACCESS [conn62] Successfully authenticated as principal admin on admin
2018-05-24T01:41:55.934+0000 I REPL [conn62] replSetReconfig admin command received from client; new config: { _id: "rs0", version: 4, protocolVersion: 1, members: [ { _id: 0, host: "infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017" }, { _id: 2, host: "infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017" } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: -1, catchUpTakeoverDelayMillis: 30000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5b0618077dac1be32b9185d1') } }
2018-05-24T01:41:55.942+0000 I REPL [conn62] replSetReconfig config object with 3 members parses ok
2018-05-24T01:41:55.944+0000 W REPL [replexec-2] Got error (KeyNotFound: Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126115, 1) } with id: 6558956334245478401) response on heartbeat request to infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017; { ok: 1.0, hbmsg: "" }
2018-05-24T01:41:55.944+0000 W REPL [replexec-0] Got error (KeyNotFound: Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126115, 1) } with id: 6558956334245478401) response on heartbeat request to infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017; { ok: 1.0, hbmsg: "" }
2018-05-24T01:41:55.944+0000 E REPL [conn62] replSetReconfig failed; NodeNotFound: Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017; the following nodes did not respond affirmatively: infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126115, 1) } with id: 6558956334245478401, infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126115, 1) } with id: 6558956334245478401
2018-05-24T01:41:55.946+0000 I NETWORK [conn62] end connection 127.0.0.1:57002 (6 connections now open)
Log from sidecar of primary instance:
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
Addresses to add: [ 'infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017',
'infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017' ]
Addresses to remove: []
replSetReconfig { _id: 'rs0',
version: 3,
protocolVersion: 1,
members:
[ { _id: 0,
host: 'infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
slaveDelay: 0,
votes: 1 },
{ _id: 1,
host: 'infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017' },
{ _id: 2,
host: 'infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017' } ],
settings:
{ chainingAllowed: true,
heartbeatIntervalMillis: 2000,
heartbeatTimeoutSecs: 10,
electionTimeoutMillis: 10000,
catchUpTimeoutMillis: -1,
catchUpTakeoverDelayMillis: 30000,
getLastErrorModes: {},
getLastErrorDefaults: { w: 1, wtimeout: 0 },
replicaSetId: 5b0618077dac1be32b9185d1 } }
Error in workloop { MongoError: Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017; the following nodes did not respond affirmatively: infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401, infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401
at Function.MongoError.create (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/error.js:31:11)
at /opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:497:72
at authenticateStragglers (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:443:16)
at Connection.messageHandler (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/pool.js:477:5)
at Socket.<anonymous> (/opt/cvallance/mongo-k8s-sidecar/node_modules/mongodb-core/lib/connection/connection.js:333:22)
at Socket.emit (events.js:180:13)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:578:20)
name: 'MongoError',
message: 'Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017; the following nodes did not respond affirmatively: infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401, infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401',
ok: 0,
errmsg: 'Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: infra-mongo-mongo-v1-0-1.mgo-cluster.default.svc.cluster.local:27017; the following nodes did not respond affirmatively: infra-mongo-mongo-v1-0-0.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401, infra-mongo-mongo-v1-0-2.mgo-cluster.default.svc.cluster.local:27017 failed with Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1527126185, 1) } with id: 6558956334245478401',
code: 74,
codeName: 'NodeNotFound',
operationTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1527126185 },
'$clusterTime':
{ clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1527126185 },
signature: { hash: [Binary], keyId: [Long] } } }
Error log from sidecar of other Mongos:
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
Db.prototype.authenticate method will no longer be available in the next major release 3.x as MongoDB 3.6 will only allow auth against users in the admin db and will no longer allow multiple credentials on a socket. Please authenticate using MongoClient.connect with auth credentials.
from mongo-k8s-sidecar.
supereagle
commented on July 25, 2024
Maybe it is caused by the bug https://jira.mongodb.org/browse/SERVER-31916, my MongoDB is 3.6.4.
from mongo-k8s-sidecar.
cescoferraro
commented on July 25, 2024
The gist on my last comment works just fine. I cannot find the reason why
from mongo-k8s-sidecar.
fairlight12
commented on July 25, 2024
Replica sets and sharded clusters require internal authentication between members when access control is enabled. For more details, please see Internal Authentication.
from mongo-k8s-sidecar.
Related Issues (20)
- Unable to configure master slave in Azure K8s HOT 1
- Issue with headless service connection string
- Alternating add/remove addresses HOT 1
- Connect to database from outside kubernetes HOT 3
- k8s 添加验证报错
- Tag docker image with version HOT 1
- Is this project still a PoC? Is it production ready? HOT 10
- Connection timed out (worked well before) HOT 1
- sidecar correctly does config on first mongo cluster only HOT 2
- Can not work when server require client present a CA file.
- Error in workloop RangeError [ERR_SOCKET_BAD_PORT] HOT 1
- Kubernetes API version (apps/v1beta1) should be updated to apps/v1
- (Error in workloop { MongoError: failed to connect to server [127.0.0.1:27017]) - After scaling up HOT 4
- Trying to deploy the mongodb using kubernetes HOT 1
- Unable to scale up more than 7 replicas HOT 1
- Using this sidecar disables running docker-entrypoint scripts HOT 2
- errro due to RBAC. how to change default user in setup
- Permission error HOT 4
- unable to get sidecar initialize my mongo-pod
- Error in workloop { MongoError: Unsupported OP_QUERY command: replSetGetStatus. The client driver may require an upgrade.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mongo-k8s-sidecar.