Comments (8)
That is a good point. Indeed, redis cluster management is a bit tedious. If a self-managing way would have existed, I don't think you would even need a StatefulSet
...
So if I understand you correctly you want to do something of the following. (let's assume a 1:1 master-slave ratio):
if podNr == 0 {
set up new cluster
} else if podNr % 2 == 0 {
join cluster
} else {
join cluster as slave
}
I like the idea, but it has some complications:
- We need to balance the keys across masters. This can be done when a new master is added, but I'm not sure how to automate balancing if a pod is removed (e.g. scaling down)
- As I said, this assumes a specific master-slave ratio. The downside is that you cannot easily change this after deploying, you'd have to start from scratch or make the pods much more complex.
That being said, I wrote this how-to with the assumption that you do not scale your redis cluster up and down multiple times a day. Compared to spinning up new VMs and configuring them by hand, having to perform a handful of copy-and-paste commands to scale up your cluster whenever you need seems like a fair compromise, don't you think?
from redis-cluster.
redis cluster management is a bit tedious
Yeah. I had not run redis in k8s before (lots of kube, lots of redis, never together), so I had never gone down the full automation path. The assumption that a human will, at least once, manually create the cluster and add/remove nodes is not exactly cloud-friendly.
So if I understand you correctly you want to do something of the following. (let's assume a 1:1 master-slave ratio):
Actually, I would like something even a step beyond. FYI, this is how I automate etcd
setup for kube (and zookeeper for Kafka).
findAllNodesInMyCluster()
if (no other nodes) {
iAmFirstMaster_Initialize()
} else {
findOtherMasters_Join()
}
We can weave master-vs-slave logic in here, or have a separate StatefulSet
for slaves.
For the complications:
- "We need to balance the keys across masters" - well, the whole removal automation thing is non-trivial. I would be more than happy if we can start with a number of masters, or even a fixed number, as long as no human intervention is required.
( - "As I said, this assumes a specific master-slave ratio." Agreed. For now, though, I am happy with fixed ratios, even fixed numbers of masters and slaves, then can improve.
From my perspective, the biggest stumbling block to getting it fully automated is the initial cluster setup and join.
Compared to spinning up new VMs and configuring them by hand, having to perform a handful of copy-and-paste commands to scale up your cluster whenever you need seems like a fair compromise, don't you think?
As long as I can do them via CI (kube config files) and not logging in, sure.
I think I am going to fork your repo and make some suggestions. Good idea? You have MIT license, so I figure you don't mind?
from redis-cluster.
I don't have much time to look into your comment now, sorry. But:
I think I am going to fork your repo and make some suggestions. Good idea? You have MIT license, so I figure you don't mind?
Of course! That's why it's open source after all. I'm curious what you'll come up with π
from redis-cluster.
Ugh, I am coming up against all sorts of limitations. See my SO question here
basically:
- When a master goes down and comes back, unless there is external persistent storage volume, the master will not rejoin the cluster, but the other nodes won't know it. Cluster is broken.
- When a master goes down and comes back, unless there is external persistent storage volume, the master comes back with no data, so the slave loses all data too
- In a completely new startup, if A comes online before B (which can happen), A will start a cluster without B, but fail trying for it
The last one is solvable with timeouts (wait for all other nodes), but that is fragile
There is a "right" solution to this, which is the Kafka model (or consul's but with sharding):
- Each node starts up in cluster mode
- Each node is told the name of at least one peer, connects to it, and joins the cluster, even an existing one
- Each node detects the existence or failure of all others and adjusts accordingly (resharding)
- Each node contains its own shard data and replicas for some other nodes
Essentially, there only are masters, they are self-adjusting and self-healing and self-joining. But Redis is built in a different way entirely. Sigh.
from redis-cluster.
Right, that's kind of what I also encountered. Essentially, all of your points are the main reason I used StatefulSets in the first place. I admire your courage in trying to find an all-automated solution though π .
from redis-cluster.
Yeah but even statefulsets don't solve it. They make the hostname consistent and if you map external persistent volumes (I prefer not to) consistently mounted, but the fundamental protocol and cluster problems continue.
Basically, redis is a great KV tech that was built pre cloud.
from redis-cluster.
@sanderploegsma Does this project support k8s 1.8+?
from redis-cluster.
@KeithTt not sure why you're hijacking the thread, but yeah, it should.
from redis-cluster.
Related Issues (16)
- Storage class missing HOT 2
- Waiting for the cluster to joinοΌk8s1.10,flannel-v0.10.0οΌ HOT 2
- if you use Flannel network in k8s,you can't success HOT 1
- need requirepass HOT 1
- It should be $(HOSTNAME) but not $(hostname) HOT 3
- Waiting for the cluster to join takes forever.... HOT 4
- Auto-Scaling HOT 1
- redis5.0.4 problem with GOSSIP
- Docker build fails (redis requires Ruby version >= 2.2.2) HOT 4
- error building docker image, HOT 9
- clusterIP is None HOT 3
- Docker image missing? HOT 2
- statefulset failover IP change HOT 2
- Issue with Initializing the cluster. HOT 1
- ${POD_IP} in /data/nodes.conf HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis-cluster.