Comments (6)
Thanks @RodionGork for raising this issue.
@ivanvc we need to backport #15835 to 3.4, basically it's the same as your proposal above. Previously the reason why the PR was't backported to 3.4 was that 3.4 has different implementation in terms of gRPC name resolver/load balancer, refer to #15835 (comment). But the reason isn't valid anymore after #16800 is merged.
from etcd.
File for reproducing the issue
from etcd.
I could replicate this with any version >= 3.4.28, as @RodionGork stated. Another way of replicating this issue is with a Procfile
with:
etcd1: bin/etcd --listen-client-urls=http://0.0.0.0:2379 --listen-peer-urls=http://0.0.0.0:2380 --advertise-client-urls=http://127.0.0.1:2379 --initial-advertise-peer-urls=http://127.0.0.1:2380 --initial-cluster=default=http://127.0.0.1:2380
etcd2: bin/etcd grpc-proxy start --endpoints=127.0.0.1:2379 --listen-addr=0.0.0.0:22379 --advertise-client-url=http://127.0.0.1:22379 --resolver-prefix=etcd-discovery --resolver-ttl=10
etcd3: bin/etcd grpc-proxy start --endpoints=127.0.0.1:2379 --listen-addr=0.0.0.0:23379 --advertise-client-url=http://127.0.0.1:23379 --resolver-prefix=etcd-discovery --resolver-ttl=10
Then, running:
$ goreman run stop etcd3
$ ./bin/etcdctl --endpoints=127.0.0.1:22379 mem l
0, started, <hostname>, , http://127.0.0.1:22379, false
0, started, <nostname>, , http://127.0.0.1:23379, false
After the 10s timeout, it still shows the two hosts.
I think the issue comes from this backport: e61f1d8
The right implementation is the following patch:
diff --git a/proxy/grpcproxy/cluster.go b/proxy/grpcproxy/cluster.go
index 338827d46..cd25e1867 100644
--- a/proxy/grpcproxy/cluster.go
+++ b/proxy/grpcproxy/cluster.go
@@ -105,9 +105,9 @@ func (cp *clusterProxy) monitor(wc endpoints.WatchChannel) {
for _, up := range updates {
switch up.Op {
case endpoints.Add:
- cp.umap[up.Endpoint.Addr] = up.Endpoint
+ cp.umap[up.Key] = up.Endpoint
case endpoints.Delete:
- delete(cp.umap, up.Endpoint.Addr)
+ delete(cp.umap, up.Key)
}
}
cp.umu.Unlock()
@@ -162,12 +162,12 @@ func (cp *clusterProxy) membersFromUpdates() ([]*pb.Member, error) {
cp.umu.RLock()
defer cp.umu.RUnlock()
mbs := make([]*pb.Member, 0, len(cp.umap))
- for addr, upt := range cp.umap {
+ for _, upt := range cp.umap {
m, err := decodeMeta(fmt.Sprint(upt.Metadata))
if err != nil {
return nil, err
}
- mbs = append(mbs, &pb.Member{Name: m.Name, ClientURLs: []string{addr}})
+ mbs = append(mbs, &pb.Member{Name: m.Name, ClientURLs: []string{upt.Addr}})
}
return mbs, nil
}
After applying this patch, I confirmed that the issue is no longer reproducible. My question to @ahrtr / @serathius is whether we want to write an integration or e2e test along with the fix. If we do, can you advise which one would be?
from etcd.
/assign
I'll backport it tomorrow.
from etcd.
Opened PR #17896. I'll update the CHANGELOG once it's merged.
from etcd.
Related Issues (20)
- Downgrade to `3.4.31` Fails in Single-Node Etcd Cluster Using `--next-cluster-version-compatible=true`; Retains Version `3.5` HOT 3
- auto-compaction-retention in logs displays nanoseconds when auto-compaction-mode set to revision HOT 2
- Peer rtt unreasonably large
- Add comaptibility matrix HOT 2
- Plan to release etcd 3.4.32 HOT 1
- Panic occurs when etcd (ver 3.5.13) new node joins the cluster HOT 5
- Setup a consistent way to manage go version HOT 7
- Support AllowedCN and AllowedHostname through config file HOT 1
- [test] keep test framework update on release-3.4 and release-3.5 HOT 3
- fatal error: fuse3/fuse.h: No such file or directory HOT 12
- Add better verbosity to the etcdctl endpoint status output that includes percentage of DB used (to illustrate Defrag rate) HOT 10
- Add SLSA provenance to your releases HOT 7
- add etcd_debugging_lease_total gauge metric HOT 2
- Error when running sample robustness tests in a pristine developer machine HOT 4
- Enable Prow slack messages to SIG-etcd channel HOT 7
- Use of deprecated grpc-ecosystem/go-grpc-middleware v1 and grpc-ecosystem/go-grpc-prometheus
- The `endpoint` commands should either work with `--discovery-srv` or should indicate that they ignore the flag HOT 1
- add snapshot step in `etcdutl migrate` HOT 13
- Don't run robustness tests on user forks
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from etcd.