wuxibin89 / redis-go-cluster Goto Github PK
View Code? Open in Web Editor NEWredis cluster client implementation in Go
License: Apache License 2.0
redis cluster client implementation in Go
License: Apache License 2.0
redis version: redis-5.0.5
redis-go-cluster version: 1.0.0
my code
cluster, err := redis.NewCluster(
&redis.Options{
StartNodes: nodes,
ConnTimeout: 50 * time.Millisecond,
ReadTimeout: 50 * time.Millisecond,
WriteTimeout: 50 * time.Millisecond,
KeepAlive: 16,
AliveTime: 60 * time.Second,
})
if err != nil {
return nil, err
}
fmt.Println(redis.String(cluster.Do("GET", "name")))
but when i go run, have this error
=== RUN TestNewRedisPool
--- FAIL: TestNewRedisPool (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x544f2a]
goroutine 7 [running]:
testing.tRunner.func1(0xc0000ae100)
/home/liangjf/app/go/src/testing/testing.go:874 +0x3a3
panic(0x5749a0, 0x6d8890)
/home/liangjf/app/go/src/runtime/panic.go:679 +0x1b2
github.com/chasex/redis-go-cluster.(*redisConn).shutdown(...)
/home/liangjf/ljf_home/code/go_home/pkg/mod/github.com/chasex/[email protected]/node.go:146
github.com/chasex/redis-go-cluster.(*redisNode).do(0xc0000b5200, 0x5ab7fa, 0x3, 0xc00004c760, 0x1, 0x1, 0x10, 0x10, 0xc00004c760, 0x546b0a)
/home/liangjf/ljf_home/code/go_home/pkg/mod/github.com/chasex/[email protected]/node.go:192 +0x23a
github.com/chasex/redis-go-cluster.(*redisCluster).Do(0xc0000ac080, 0x5ab7fa, 0x3, 0xc00004c760, 0x1, 0x1, 0x1372e4f5, 0x1372e4f50000000f, 0x5ed5b3ba, 0xc00003af60)
/home/liangjf/ljf_home/code/go_home/pkg/mod/github.com/chasex/[email protected]/cluster.go:254 +0x20f
command-line-arguments.(*RedisPool).Get(...)
/home/liangjf/opensource/gpusher/common/db/redis_string.go:40
command-line-arguments.TestNewRedisPool(0xc0000ae100)
/home/liangjf/opensource/gpusher/common/db/redis_test.go:23 +0x16b
testing.tRunner(0xc0000ae100, 0x5b5de8)
/home/liangjf/app/go/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
/home/liangjf/app/go/src/testing/testing.go:960 +0x350
FAIL command-line-arguments 0.006s
at last i find the code:
func (node *redisNode) getConn() (*redisConn, error) {
...
c, err := net.DialTimeout("tcp", node.address, node.connTimeout)
if err != nil {
return nil, err
}
...
}
i find that, when panic happend, the node.address is "172.16.7.16:8002@18002"
but when is normal, the node.address is "172.16.7.16:8002"
so when run in debug mode, i modified it manually "172.16.7.16:8002@18002" to "172.16.7.16:8002", it is normal
one by one to follow the code call stack, i find that
//github.com/chasex/[email protected]/cluster.go:609
func (cluster *redisCluster) updateClustrInfo(node *redisNode) error {
info, err := String(node.do("CLUSTER", "NODES"))
infos := strings.Split(strings.Trim(info, "\n"), "\n")
for i := range fields {
fields[i] = strings.Split(infos[i], " ")
if len(fields[i]) < kFieldSlot {
return fmt.Errorf("missing field: %s [%d] [%d]", infos[i], len(fields[i]), kFieldSlot)
}
nodes[fields[i][kFieldAddr]] = &redisNode {
name: fields[i][kFieldName],
address: fields[i][kFieldAddr],
slaves: make([]*redisNode, 0),
connTimeout: cluster.connTimeout,
readTimeout: cluster.readTimeout,
writeTimeout: cluster.writeTimeout,
keepAlive: cluster.keepAlive,
aliveTime: cluster.aliveTime,
}
}
...
}
the function well update the cluster info after connect to cluster, it Mainly process the cluster information returned by the cluster nodes command, and then get the address of each node
such as:
172.16.7.16:8001> cluster nodes
78a8926d4ad002e88f7d499ff4e590f6385b8ffd 172.16.7.16:8005@18005 slave 37a21de453a1118217fed1ec3535463d1bfe5244 0 1591076153884 4 connected
37a21de453a1118217fed1ec3535463d1bfe5244 172.16.7.16:8002@18002 master - 0 1591076151881 2 connected 5462-10922
f83fc4f5b61e4886be681e3b19b952781de9d1bc 172.16.7.16:8003@18003 master - 0 1591076152000 0 connected 10923-16383
b1a7ab40c42a0e8c9c5037aa3df7a701377567cd 172.16.7.16:8006@18006 slave f83fc4f5b61e4886be681e3b19b952781de9d1bc 0 1591076153000 5 connected
3e1c4883e904adde6c4e917c9929954cf14a1a57 172.16.7.16:8001@18001 myself,master - 0 1591076152000 1 connected 0-5461
4e6f4de2fab077ea612b05a4e264cb85a6a9eda7 172.16.7.16:8004@18004 slave 3e1c4883e904adde6c4e917c9929954cf14a1a57 0 1591076151000 3 connected
at this fields[i][kFieldAddr] is 172.16.7.16:8002@18002
, so client run Do() well error, becase the address error
i modified as follows:
//github.com/chasex/[email protected]/cluster.go:622
for i := range fields {
fields[i] = strings.Split(infos[i], " ")
if len(fields[i]) < kFieldSlot {
return fmt.Errorf("missing field: %s [%d] [%d]", infos[i], len(fields[i]), kFieldSlot)
}
//added by me: Handling the address field
if strings.Contains(fields[i][kFieldAddr], "@") {
fields[i][kFieldAddr] = strings.Split(fields[i][kFieldAddr], "@")[0]
}
nodes[fields[i][kFieldAddr]] = &redisNode {
name: fields[i][kFieldName],
address: fields[i][kFieldAddr],
slaves: make([]*redisNode, 0),
connTimeout: cluster.connTimeout,
readTimeout: cluster.readTimeout,
writeTimeout: cluster.writeTimeout,
keepAlive: cluster.keepAlive,
aliveTime: cluster.aliveTime,
}
}
or you can use the master, it had modify the way to updateClusterInfo:
cluster.go:314
func (cluster *Cluster) update(node *redisNode) error {
info, err := Values(node.do("CLUSTER", "SLOTS"))
....
}
i test against the 6 nodes instance made followed by https://redis.io/topics/cluster-tutorial
and my code is
`package main
import (
"fmt"
"time"
redis "github.com/chasex/redis-go-cluster"
)
func main() {
cluster, err := redis.NewCluster(
&redis.Options{
StartNodes: []string{
"192.168.212.173:30001",
"192.168.212.173:30002",
"192.168.212.173:30003",
"192.168.212.173:30004",
"192.168.212.173:30005",
"192.168.212.173:30006",
},
ConnTimeout: 50 * time.Millisecond,
ReadTimeout: 50 * time.Millisecond,
WriteTimeout: 50 * time.Millisecond,
KeepAlive: 16,
AliveTime: 60 * time.Second,
})
fmt.Println("redis new ", err)
reply, err2 := cluster.Do("SET", "foo", "bar")
// reply, err2 := redis.String(cluster.Do("GET", "foo"))
// reply, err2 := redis.Int(cluster.Do("INCR", "mycount", 1))
fmt.Println("reply", reply)
fmt.Println("err2", err2)
}`
And i got the results below
redis new nil
reply nil
err2 ECONNTIMEOUT
hello,can you add a new tag update to master?
when I use go mod to import your library,it always download v1.0.0 tag which is old version code, and get value from redis6.0+ would be crash like this:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4f82e3]
goroutine 22 [running]:
github.com/chasex/redis-go-cluster.(*redisConn).shutdown(...)
/home/danwei/rtb/golang/gopath/pkg/mod/github.com/chasex/[email protected]/node.go:146
github.com/chasex/redis-go-cluster.(*redisNode).do(0xc000272400, 0x532324, 0x3, 0xc0000125b0, 0x1, 0x1, 0x8, 0x10, 0x7f05dd263108, 0xc0000125c0)
/home/danwei/rtb/golang/gopath/pkg/mod/github.com/chasex/[email protected]/node.go:192 +0x243
github.com/chasex/redis-go-cluster.(*redisCluster).Do(0xc0000bc000, 0x532324, 0x3, 0xc0000125b0, 0x1, 0x1, 0x4, 0x0, 0x0, 0x0)
/home/danwei/rtb/golang/gopath/pkg/mod/github.com/chasex/[email protected]/cluster.go:254 +0x21d
main.main.func1(0xc00018e010, 0x55b920, 0xc0000bc000, 0x3)
/home/danwei/rtb/golang/test/test2.go:34 +0x1e4
created by main.main
/home/danwei/rtb/golang/test/test2.go:29 +0x1aa
would you please add a new tag and push the master code to it
var s string
reply, err = redis.Scan(reply, &s)
if err != nil {
fmt.Println(err)
}
it print error:
redigo: Scan cannot convert from string to *string
step 1:
sadd one key, like this:
test_set_key_one := "{xxx}.test_set_key_1"
v1 := 12
v2 :="test_v2"
v3 := 81231.13
sadd(test_set_key_one, v1, v2, v3)
step 2:
sadd second key, like this:
test_set_key_second := "{xxx}.test_set_key_2"
v4 := 12
v5 := "aaaa"
v6 := 90.12
sadd(test_set_key_second , v4, v5, v6)
step3:
sdiff(test_set_key_one, test_set_key_second)
return the same of test_set_key_one set:
12, "test_v2", 81231.13
not is :
"test_v2", 81231.13
you can repeat process above test case, the result is the same to my.
but when i iput the process in redis-cli terminal, the result is no problem.
when use redis batch and redis cluster failover( master node become slave, slave become master node), this package not reconnect the new master node, still use invalid slave node.
package main
import (
"github.com/chasex/redis-go-cluster"
"time"
)
var ClusterPool redis.Cluster
var Err error
func main() {
host := "127.0.0.1"
ClusterPool, Err = redis.NewCluster(
&redis.Options{
StartNodes: []string{
host+":6379",
host+":6380",
host+":6381",
host+":6382",
host+":6383",
host+":6384"},
ConnTimeout: 50 * time.Millisecond,
ReadTimeout: 50 * time.Millisecond,
WriteTimeout: 50 * time.Millisecond,
KeepAlive: 16,
AliveTime: 60 * time.Second,
})
run main.go
console:
no invalid node in [127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384]
where?
code like this: redisResp, err := redisCluster.Do("HGET", redisHashKey, redisFiled)
I'm not intimately aware of any intricacies that might be involved in supporting Lua scripts, is it not supported simply because there's been no need for the feature? I'm looking at migrating this project to use Redis Cluster and the Lua scripts are something that are desperately needed.
Do you know if there'd be any blocking issues with adding support, or would it be possible? I'd be happy to do the legwork, just wanted to ask whether it would be a welcome contribution or not.
https://github.com/chasex/redis-go-cluster/blob/master/cluster.go#L96
”no invalid node“ should be “no valid node”?
I use this code to create cluster, but it has a problem:
cluster, err := clust.NewCluster(
&clust.Options{
StartNodes: []string{"10.151.3.24:6379", "10.151.3.24:6383", "10.151.3.24:6381"},
ConnTimeout: 5 * time.Second,
ReadTimeout: 5 * time.Second,
WriteTimeout: 5 * time.Second,
KeepAlive: 16,
AliveTime: 60 * time.Second,
})
NewCluster: no valid node in [10.151.3.24:6379 10.151.3.24:6383 10.151.3.24:6381]
but when I use "github.com/gomodule/redigo/redis" to create Redis.conn it was worked.
this is codes:
var RedisPool *redis.Pool
var ip = "10.151.3.24:6381"
func NewRedisPool(ip string) *redis.Pool {
return &redis.Pool{
MaxIdle: 6,
IdleTimeout: 240 * time.Second,
Dial: func() (redis.Conn, error) {
c, err := redis.Dial("tcp", ip)
if err != nil {
return nil, err
}
return c, nil
},
TestOnBorrow: func(c redis.Conn, t time.Time) error {
if time.Since(t) < time.Minute {
return nil
}
_, err := c.Do("PING")
return err
},
}
}
RedisPool = NewRedisPool(ip)
redisConn := RedisPool.Get()
defer redisConn.Close()
redisConn.Do("AUTH", "123456")
reply, err := redisConn.Do("set", "demo_key", "666")
I don't know what cause this problem. What Redis cluster need? I use Virtualmachine+Docker to create Redis cluster.What should I do to handle this problem. Please help me.Thanks
-get mykey3500000: ECONNTIMEOUT
-get mykey3800000: ECONNTIMEOUT
-get mykey3200000: ECONNTIMEOUT
-get mykey1500000: ECONNTIMEOUT
-get mykey1300000: ECONNTIMEOUT
-get mykey3700000: ECONNTIMEOUT
-get mykey3900000: ECONNTIMEOUT
-get mykey3600000: ECONNTIMEOUT
-get mykey1200000: ECONNTIMEOUT
The documentation says
Mutiple keys command - MGET/MSET are supported using result aggregation. Processing steps are as follows:
First, split the keys into multiple nodes according to their hash slot.
Then, start a goroutine for each node to excute MGET/MSET commands and wait them finish.
Last, collect and rerange all replies, return back to caller.
Then, start a goroutine for each node to excute MGET/MSET commands and wait them finish. is misleading
number of goroutines is equal to no of hash-slots for the n keys,
number of goroutines is not equal to number of nodes in cluster
Thought might be useful to add the ActiveCount() int
method (or IdleCount() int
maybe, seems active conns are not counted).
redigo has ActiveCount() int
and IdleCount() int
, which we use to do client side pool monitoring.
please helpe me tell How to change db
what is the default size of the cluster pool? Can i change it ?
set($key, $value, array('nx', 'ex' => $ttl));
https://redis.io/commands/set
like this
Options
Starting with Redis 2.6.12 SET supports a set of options that modify its behavior:
EX seconds -- Set the specified expire time, in seconds.
PX milliseconds -- Set the specified expire time, in milliseconds.
NX -- Only set the key if it does not already exist.
XX -- Only set the key if it already exist.
Note: Since the SET command options can replace SETNX, SETEX, PSETEX, it is possible that in future versions of Redis these three commands will be deprecated and finally removed.
I'm trying to do a MSET and MGET on cluster configuration, but seems like the keys are not getting set when done through BATCH
cluster.go:314: ERR unknown command 'CLUSTER'
How does one authenticate a connection to a Redis cluster?
Hi,
We have a use case when we running N ( where N could be 10, 20 ....) redis servers with dynamic ip addresses and dns based service discovery. So we have a hostname like redis-privatenet.com which resolves to N servers.
My proposal is the following:
I am ready to make a patch just wanted to discuss the approach here first.
Any thoughts?
In the top-level README.md
, it lists "Pub/Sub" under the "NOT supported" heading.
But, in my reading of the code, at least the "PUBLISH" command would be supported the way most other commands would be:
If this is the case, then I'd be willing to submit a PR for the documentation... But I just wanted to verify my understanding.
if i use domain name to init the client instead of ip address, like this:
cluster, err := redis.NewCluster(
&redis.Options{
StartNodes: []string{"1xxx.com:7000", "2xxx.com:7001", "3xxx.com:7002"},
ConnTimeout: 50 * time.Millisecond,
ReadTimeout: 50 * time.Millisecond,
WriteTimeout: 50 * time.Millisecond,
KeepAlive: 16,
AliveTime: 60 * time.Second,
})
it will be panic when my application running, i find out the problem in this file:node.go.
if conn.c.RemoteAddr().String() != node.address {
panic("unreachable")
}
the result of "conn.c.RemoteAddr().String()" is ip address, but the return of node.address is domain name
I want to exec redis module command in redis cluster.
Does this cluster client support redis module command, for example redisearch?
Thank you very much.
Code use the string "/r/n" to decide the end of the response, but when I set a bytes to the cluster and read it, I got an ERROR because the bytes has a "/r/n" inside.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.