Grpc-proxy cannot handle requests from grpc-gateway correctly about etcd HOT 13 OPEN

@ivanvc The source code you mentioned above is where etcdserver receives gRPC requests, this issue is about receiving REST(HTTP) requests.

Note that etcdserver receives REST(HTTP) request via the grpc-gateway.

I think the reason why the build-in gateway doesn't have such issue is that the gateway carries the http request header over to the following gRPC request,

Please read

• etcd/api/etcdserverpb/gw/rpc.pb.gw.go

  Line 2354 in fc337b0

  annotatedContext, err = runtime.AnnotateContext(ctx, mux, req, "/etcdserverpb.KV/Range", runtime.WithHTTPPathPattern("/v3/kv/range"))

• https://github.com/grpc-ecosystem/grpc-gateway/blob/0bcc6bf00e0bf6ac71caab131df54e13af377603/runtime/context.go#L153-L155

The existing e2e test cases should have already verified it.

from etcd.

Here is the related PR #18012.

from etcd.

It seems like a valid issue. We don't see much use case on grpcProxy, so I don't think this is a major issue.

• etcdserver not only supports grpc requests, but also REST requests (see below), so we need to double check whether the build-in gateway has similar issue.
  

  etcd/server/embed/serve.go

  Lines 312 to 326 in 97a9af9

  	handlers := []registerHandlerFunc{
  	etcdservergw.RegisterKVHandler,
  	etcdservergw.RegisterWatchHandler,
  	etcdservergw.RegisterLeaseHandler,
  	etcdservergw.RegisterClusterHandler,
  	etcdservergw.RegisterMaintenanceHandler,
  	etcdservergw.RegisterAuthHandler,
  	v3lockgw.RegisterLockHandler,
  	v3electiongw.RegisterElectionHandler,
  	}
  	for _, h := range handlers {
  	if err := h(ctx, gwmux, conn); err != nil {
  	return nil, err
  	}
  	}

Can anyone double check this? @ivanvc @siyuanfoundation @chaochn47 @tjungblu

• For this issue, we should add an simple e2e test to reproduce the issue firstly in etcd_grpcproxy_test.go in the first commit (ensure you can reproduce the issue with the e2e test), then apply your fix as the second commit.

from etcd.

I also checked the build-in gateway, it seems OK. I think the implementation is different in etcdserver and grpcproxy server, which leads to the different behavior.
We mainly use grpcproxy to reduce the watch pressure from apisix to etcdserver.
I will add an simple e2e test soon.
Thanks for your reply.

from etcd.

Can anyone double check this? @ivanvc @siyuanfoundation @chaochn47 @tjungblu

I think the built-in gateway (if I'm correct, it should be in server/etcdserver/api/v3rpc/grpc.go), is forwarding the metadata. Refer to:

And newUnaryInterceptor(...):

The source code for release 3.5 is very similar, it doesn't appear to happen there either:

https://github.com/etcd-io/etcd/blob/v3.5.13/server/etcdserver/api/v3rpc/grpc.go#L45-L49 / https://github.com/etcd-io/etcd/blob/v3.5.13/server/etcdserver/api/v3rpc/interceptor.go#L46-L73

Caveats: I'm still not too familiar with etcdserver, so I apologize if my answer is misleading.

from etcd.

The existing e2e test cases should have already verified it.

https://github.com/etcd-io/etcd/blob/main/tests/e2e/v3_curl_auth_test.go

from etcd.

Thanks for your reply, Do we have any comments for the watch cancled error? This is another problem which I think is more important than auth. I try to fix the watch canceled error in #18012 . Looking forward for reviewing. The discussions above seem focusing on the auth problem. @ahrtr @ivanvc

from etcd.

Do we have any comments for the watch cancled error?

I am not sure what issue you are talking about here. Please feel free to raise another issue to track it if not present.

Please let's only fix one thing/issue in one PR. Please add an e2e test to reproduce the issue firstly, please read the second point in #18011 (comment)

from etcd.

Do we have any comments for the watch cancled error?

I am not sure what issue you are talking about here. Please feel free to raise another issue to track it if not present.

Please let's only fix one thing/issue in one PR. Please add an e2e test to reproduce the issue firstly, please read the second point in #18011 (comment)

Thanks for your reply, I'll add e2e tests in a fews days and recommit the PR.

The watch context cancled error is described as the first part of this issue, maybe not clearly. I copied the part below.

Start etcd(v3.5.13) and grpc-proxy locally by commands below:

nohup etcd&
nohup etcd grpc-proxy start --endpoints=localhost:2379 &

Then start grpc-gateway locally(compiled by the code above).
Send watch command to grpc-gateway with curl:

curl -X POST -d '{"create_request": {"key":"dGVzdGtleQo=","range_end":"dGVzdGtleTEK"}}' 'localhost:8887/v3/watch'

The curl command returns directly instead of waiting for watch response,with the error repsose below:

{"result":{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"created":true}} {"error":{"grpc_code":1,"http_code":408,"message":"context canceled","http_status":"Request Timeout"}}

from etcd.

Then start grpc-gateway locally(compiled by the code above). Send watch command to grpc-gateway with curl:

curl -X POST -d '{"create_request": {"key":"dGVzdGtleQo=","range_end":"dGVzdGtleTEK"}}' 'localhost:8887/v3/watch'

The curl command returns directly instead of waiting for watch response,with the error repsose below:

{"result":{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"created":true}} {"error":{"grpc_code":1,"http_code":408,"message":"context canceled","http_status":"Request Timeout"}}

The error is returned by your grpc-gateway, so you need to debug it firstly. Please let me know if you see any issue when you watch a key via etcd or grpc-proxy directly.

from etcd.

Then start grpc-gateway locally(compiled by the code above). Send watch command to grpc-gateway with curl:
curl -X POST -d '{"create_request": {"key":"dGVzdGtleQo=","range_end":"dGVzdGtleTEK"}}' 'localhost:8887/v3/watch'
The curl command returns directly instead of waiting for watch response,with the error repsose below:
{"result":{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"created":true}} {"error":{"grpc_code":1,"http_code":408,"message":"context canceled","http_status":"Request Timeout"}}

The error is returned by your grpc-gateway, so you need to debug it firstly. Please let me know if you see any issue when you watch a key via etcd or grpc-proxy directly.

The watch error happens in this scenario：

Take grpc proxy away, all worked OK:

So I guess this indicates that the problem is with the grpc proxy.

from etcd.

The watch error happens in this scenario：

Do you still see the issue after applying the #18012?

from etcd.

The watch error happens in this scenario：

Do you still see the issue after applying the #18012?

After applying the #18012, all worked OK.

from etcd.