Comments (5)
sendResponse
should not be causing a panic. I also noticed that a very old version of gRPC is being used ([email protected]). Could you also tell me in what situations are you facing the panic?
I need to check if there is best practices around handling panics this way using interceptors. I will check with the team and get back on that.
from grpc-go.
@arvindbr8 , Thank you very much for your response.
Here is the gRPC used on our side.
client -> gRPC gateway -> gRPC server
Unfortunately, this panic happened occasionally and we did NOT catch the real reason why this panic happened. Sorry for that no more information could be given to you except the panic information
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4822ef]
goroutine 1190261159 [running]:
unicode/utf8.ValidString({0x0?, 0xc001541398?})
/usr/local/Cellar/go/1.21.4/libexec/src/unicode/utf8/utf8.go:545 +0x4f
github.com/golang/protobuf/proto.appendUTF8StringValueNoZero({0xc0000c2f80, 0x15, 0x20}, {0x0?}, 0x0?, 0x1?)
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2074 +0x55
github.com/golang/protobuf/proto.(*marshalInfo).marshal(0xc0003725b0, {0xc0000c2f80, 0x5, 0x20}, {0xc001541468?}, 0x52?)
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:270 +0x3a3
github.com/golang/protobuf/proto.typeMarshaler.makeMessageSliceMarshaler.func6({0xc0000c2f80, 0x3, 0x20}, {0xc002233360?}, 0x30?, 0x20?)
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2264 +0x157
github.com/golang/protobuf/proto.(*marshalInfo).marshal(0xc000372460, {0xc0000c2f80, 0x0, 0x20}, {0x100c0008a44c0?}, 0xa8?)
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:270 +0x3a3
github.com/golang/protobuf/proto.(*InternalMessageInfo).Marshal(0xc0000c2f80?, {0xc0000c2f80, 0x0, 0x20}, {0x1b46de0, 0xc001301340}, 0x1?)
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:141 +0x6c
server/protocol.(*RankStatusReply).XXX_Marshal(0x1000000000001?, {0xc0000c2f80?, 0xc001301340?, 0x4103a5?}, 0x0?)
/Volumes/Jenkins/Home/workspace/test/src/server/protocol/ank_svc.pb.go:2046 +0x2c
github.com/golang/protobuf/proto.(*Buffer).Marshal(0xc0026cc008, {0x1b46de0, 0xc001301340})
/Users/jenkins/go/pkg/mod/github.com/golang/[email protected]/proto/table_marshal.go:2742 +0x167
google.golang.org/grpc/encoding/proto.marshal({0x1684940?, 0xc001301340}, 0xc0026cc000)
/Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/encoding/proto/proto.go:59 +0xac
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0x1684940, 0xc001301340})
/Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/encoding/proto/proto.go:74 +0x6c
google.golang.org/grpc.encode({0x7f6fc189e000?, 0x2707de0?}, {0x1684940?, 0xc001301340?})
/Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:543 +0x3e
google.golang.org/grpc.(*Server).sendResponse(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, {0x1684940?, 0xc001301340}, {0x0, 0x0}, 0x0?, {0x0, ...})
/Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/server.go:837 +0x1b1
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, 0xc000885890, 0x26b6888, 0x0)
/Users/jenkins/go/pkg/mod/google.golang.org/[email protected]/server.go:1033 +0xed3
google.golang.org/grpc.(*Server).handleStream(0xc0006ea300, {0x1b59040, 0xc00103e300}, 0xc0019cff00, 0x0)
From this panic stack, it seems this panic is caused by sendResponse
. We are not clear you mentioned sendResponse should not be causing a panic
. Could you please correct me if something is missing or misunderstood?
from grpc-go.
client -> gRPC gateway -> gRPC server
Is the client here also gRPC? Since I see a proxy gRPC gateway I suspect it might not be.
Could you please correct me if something is missing or misunderstood?
What I meant to say was that, sendResponse is an internal API provided by the server. And its unlikely that sendResponse has a bug that could potentially crash the server, but hasnt been caught by any of our tests or other users.
I would recommend upgrading the version of gRPC that is used, and see if this still occurs. If it does, we need to take a deeper look at the way the gRPC gateway is being setup.
from grpc-go.
client -> gRPC gateway -> gRPC server
Is the client here also gRPC? Since I see a proxy gRPC gateway I suspect it might not be.
NO, the HTTP is used on the client side.
What I meant to say was that, sendResponse is an internal API provided by the server. And its unlikely that sendResponse has a bug that could potentially crash the server, but hasnt been caught by any of our tests or other users.
Yes, I agree that one bug in sendResponse
is unlikely, sorry for the misleading information before.
Originally, I thought the UnaryPanicHandler
could catch this panic information and output some useful information about this panic. Unfortunately, it failed. I want to know whether there is a better way to handle this panic.
I would recommend upgrading the version of gRPC that is used, and see if this still occurs. If it does, we need to take a deeper look at the way the gRPC gateway is being setup.
Yes, we plan to upgrade the gRPC to the latest version on our server side.
from grpc-go.
Good read about Go philosophy around panic, recover - https://go.dev/blog/defer-panic-and-recover. Esp this part:
Recover is only useful inside deferred functions. During normal execution, a call to recover will return nil and have no other effect. If the current goroutine is panicking, a call to recover will capture the value given to panic and resume normal execution
from grpc-go.
Related Issues (20)
- Why is the service config passed as a JSON-String just to get converterted to a struct anyway? HOT 7
- Cardinality violations should use error code “unimplemented” HOT 4
- GitHub Action: branch protection checks are skipped, and also not blocking merges HOT 3
- grpc.NewClient with namedpipe on Windows throws resolverError HOT 2
- User agent becomes grpc-go/1.64. on server side of grpc gateway HOT 3
- xds: move functionality from `xds/internal` to `internal/xds`
- stubserver: add support to optionally pass in a `grpc.Server` or `xds.GRPCServer` HOT 2
- Github Action: Codecov action is broken and is failing silently HOT 1
- Upgrade to using math/rand/v2 to get perf enhancements HOT 2
- xds: tests shouldn't rely on the presence of an entry in the `authorities` field of the bootstrap configuration with an empty key
- Experimental API related to metadata HOT 4
- Linter rule for using context.Background() without a timeout in tests HOT 4
- gRPC is incompatible with tls.Listener HOT 2
- Closing connection takes up to 15 minutes. HOT 5
- Feature Request: expose handleRawConn or add ServeConn HOT 26
- Flaky test: TimerAndWatchStateOnErrorCallback HOT 4
- xds: bootstrap config is not emitted to logs in a human readable way
- Strongly-type request inside a Stream Server Interceptor HOT 2
- Proxy connection buffer necessary? HOT 1
- Why does grpc.NewClient silently ignore DialOptions? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grpc-go.