How to catch the panic of sendResponse? about grpc-go HOT 5 CLOSED

sendResponse should not be causing a panic. I also noticed that a very old version of gRPC is being used ([email protected]). Could you also tell me in what situations are you facing the panic?

I need to check if there is best practices around handling panics this way using interceptors. I will check with the team and get back on that.

from grpc-go.

@arvindbr8 , Thank you very much for your response.

Here is the gRPC used on our side.

client -> gRPC gateway -> gRPC server

Unfortunately, this panic happened occasionally and we did NOT catch the real reason why this panic happened. Sorry for that no more information could be given to you except the panic information

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4822ef]
goroutine 1190261159 [running]:
unicode/utf8.ValidString({0x0?, 0xc001541398?})
        /usr/local/Cellar/go/1.21.4/libexec/src/unicode/utf8/utf8.go:545 +0x4f
github.com/golang/protobuf/proto.appendUTF8StringValueNoZero({0xc0000c2f80, 0x15, 0x20}, {0x0?}, 0x0?, 0x1?)
        /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2074 +0x55
github.com/golang/protobuf/proto.(*marshalInfo).marshal(0xc0003725b0, {0xc0000c2f80, 0x5, 0x20}, {0xc001541468?}, 0x52?)
        /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:270 +0x3a3
github.com/golang/protobuf/proto.typeMarshaler.makeMessageSliceMarshaler.func6({0xc0000c2f80, 0x3, 0x20}, {0xc002233360?}, 0x30?, 0x20?)
        /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2264 +0x157
github.com/golang/protobuf/proto.(*marshalInfo).marshal(0xc000372460, {0xc0000c2f80, 0x0, 0x20}, {0x100c0008a44c0?}, 0xa8?)
 /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:270 +0x3a3
github.com/golang/protobuf/proto.(*InternalMessageInfo).Marshal(0xc0000c2f80?, {0xc0000c2f80, 0x0, 0x20}, {0x1b46de0, 0xc001301340}, 0x1?)
        /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:141 +0x6c
server/protocol.(*RankStatusReply).XXX_Marshal(0x1000000000001?, {0xc0000c2f80?, 0xc001301340?, 0x4103a5?}, 0x0?)
        /Volumes/Jenkins/Home/workspace/test/src/server/protocol/ank_svc.pb.go:2046 +0x2c
github.com/golang/protobuf/proto.(*Buffer).Marshal(0xc0026cc008, {0x1b46de0, 0xc001301340})
        /Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2742 +0x167
google.golang.org/grpc/encoding/proto.marshal({0x1684940?, 0xc001301340}, 0xc0026cc000)
        /Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/encoding/proto/proto.go:59 +0xac
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0x1684940, 0xc001301340})
        /Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/encoding/proto/proto.go:74 +0x6c
google.golang.org/grpc.encode({0x7f6fc189e000?, 0x2707de0?}, {0x1684940?, 0xc001301340?})
        /Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:543 +0x3e
google.golang.org/grpc.(*Server).sendResponse(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, {0x1684940?, 0xc001301340}, {0x0, 0x0}, 0x0?, {0x0, ...})
        /Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/server.go:837 +0x1b1
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, 0xc000885890, 0x26b6888, 0x0)
 /Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/server.go:1033 +0xed3
google.golang.org/grpc.(*Server).handleStream(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, 0x0)

From this panic stack, it seems this panic is caused by sendResponse. We are not clear you mentioned sendResponse should not be causing a panic. Could you please correct me if something is missing or misunderstood?

from grpc-go.

client -> gRPC gateway -> gRPC server

Is the client here also gRPC? Since I see a proxy gRPC gateway I suspect it might not be.

Could you please correct me if something is missing or misunderstood?

What I meant to say was that, sendResponse is an internal API provided by the server. And its unlikely that sendResponse has a bug that could potentially crash the server, but hasnt been caught by any of our tests or other users.

I would recommend upgrading the version of gRPC that is used, and see if this still occurs. If it does, we need to take a deeper look at the way the gRPC gateway is being setup.

from grpc-go.

client -> gRPC gateway -> gRPC server

Is the client here also gRPC? Since I see a proxy gRPC gateway I suspect it might not be.

NO, the HTTP is used on the client side.

What I meant to say was that, sendResponse is an internal API provided by the server. And its unlikely that sendResponse has a bug that could potentially crash the server, but hasnt been caught by any of our tests or other users.

Yes, I agree that one bug in sendResponse is unlikely, sorry for the misleading information before.

Originally, I thought the UnaryPanicHandler could catch this panic information and output some useful information about this panic. Unfortunately, it failed. I want to know whether there is a better way to handle this panic.

I would recommend upgrading the version of gRPC that is used, and see if this still occurs. If it does, we need to take a deeper look at the way the gRPC gateway is being setup.

Yes, we plan to upgrade the gRPC to the latest version on our server side.

from grpc-go.

Good read about Go philosophy around panic, recover - https://go.dev/blog/defer-panic-and-recover. Esp this part:

Recover is only useful inside deferred functions. During normal execution, a call to recover will return nil and have no other effect. If the current goroutine is panicking, a call to recover will capture the value given to panic and resume normal execution

from grpc-go.