Comments (7)
Use pprof to find out where the majority of the memory is allocated.
from gnet.
@panjf2000 is this really ok?
nothing extra added and using iperf... memory is bytesliced gotten from internal mechanism... if i do gomemlimit, the whole thing becomes too slow. any fix for this?
basically, it seemed easy to crash with oom by iperfing the server if it's not guarded by firewall or something. i would prefer to hope u can have a fix for this or at least give us some suggestion on workarounds to ensure it's not using memory this way other than gomemlimit
package main
import (
"flag"
"fmt"
"log"
"net/http"
_ "net/http/pprof"
"github.com/panjf2000/gnet/v2"
)
type echoServer struct {
gnet.BuiltinEventEngine
eng gnet.Engine
addr string
multicore bool
}
func (es *echoServer) OnBoot(eng gnet.Engine) gnet.Action {
es.eng = eng
log.Printf("echo server with multi-core=%t is listening on %s\n", es.multicore, es.addr)
return gnet.None
}
func (es *echoServer) OnTraffic(c gnet.Conn) gnet.Action {
buf, _ := c.Next(-1)
c.Write(buf)
return gnet.None
}
func main() {
var port int
var multicore bool
var pprofPort int
// Example command: go run main.go --port 9000 --multicore=true --pprofPort=6060
flag.IntVar(&port, "port", 9000, "Set the port for the TCP server")
flag.BoolVar(&multicore, "multicore", false, "Enable multicore processing")
flag.IntVar(&pprofPort, "pprofPort", 6060, "Set the port for the pprof HTTP server")
flag.Parse()
// Start pprof server
go func() {
log.Printf("Starting pprof server on http://localhost:%d/debug/pprof/\n", pprofPort)
log.Println(http.ListenAndServe(fmt.Sprintf("localhost:%d", pprofPort), nil))
}()
// Setup and start TCP echo server
echo := &echoServer{addr: fmt.Sprintf("tcp://:%d", port), multicore: multicore}
log.Fatal(gnet.Run(echo, echo.addr, gnet.WithMulticore(multicore)))
}
from gnet.
@panjf2000 i can confirm this is a bug,
running the example tcp echo using
iperf -c 127.0.0.1 -p 8080 -t 3600
u can crash it.
with GOMEMLIMIT=6GiB ./echoserver
you get 3000 req/s
without GOMEMLIMIT, u get 100k req/s with iperf
from gnet.
I've never run into this problem when benchmarking, and I think it is expected to allocate plenty of memory when the server is overloaded during benchmark testing, this happens to all network frameworks. Have you tried WithEdgeTriggeredIO?
from gnet.
Besides, you need to prove that gnet
has this issue while other network frameworks don't before you call it a bug.
from gnet.
@panjf2000 , is this how WithEdgeTriggeredIO is used?
same, the memory is used til oom,
iperf -c 127.0.0.1 -p 9000 -t 3600
package main
import (
"flag"
"fmt"
"log"
"net/http"
_ "net/http/pprof"
"github.com/panjf2000/gnet/v2"
)
type echoServer struct {
gnet.BuiltinEventEngine
eng gnet.Engine
addr string
multicore bool
}
func (es *echoServer) OnBoot(eng gnet.Engine) gnet.Action {
es.eng = eng
log.Printf("echo server with multi-core=%t is listening on %s\n", es.multicore, es.addr)
return gnet.None
}
func (es *echoServer) OnTraffic(c gnet.Conn) gnet.Action {
buf, _ := c.Next(-1)
c.Write(buf)
return gnet.None
}
func main() {
var port int
var multicore bool
var pprofPort int
// Example command: go run main.go --port 9000 --multicore=true --pprofPort=6060
flag.IntVar(&port, "port", 9000, "Set the port for the TCP server")
flag.BoolVar(&multicore, "multicore", false, "Enable multicore processing")
flag.IntVar(&pprofPort, "pprofPort", 6060, "Set the port for the pprof HTTP server")
flag.Parse()
// Start pprof server
go func() {
log.Printf("Starting pprof server on http://localhost:%d/debug/pprof/\n", pprofPort)
log.Println(http.ListenAndServe(fmt.Sprintf("localhost:%d", pprofPort), nil))
}()
// Setup and start TCP echo server
echo := &echoServer{addr: fmt.Sprintf("tcp://:%d", port), multicore: multicore}
log.Fatal(gnet.Run(echo, echo.addr, gnet.WithMulticore(multicore), gnet.WithEdgeTriggeredIO(true)))
}
from gnet.
@panjf2000 sorry my bad, it's working as it should. thx.
from gnet.
Related Issues (20)
- [Bug]: Read returns io.ErrShortBuffer on closed connection HOT 7
- Special treatment for the first package HOT 8
- [Question]: How to shut down the started tcp server and turn off port listening? HOT 2
- [Question]: When using io.CopyBuffer(), do you need to call Flush to send the data? HOT 2
- [Bug]: Discard() panic HOT 17
- [Feature]: `io_uring` support HOT 6
- [Question]: there's gnet https server now, how to use gnet https client? HOT 1
- UDP FD handle is not released HOT 1
- [Question]: The redis server service implemented using gnet has too many read tcp i/o timeouts in the production environment. HOT 6
- [Question]: redhub upgraded to use gnet/v2 issues, seemed "stalled" HOT 1
- [Feature]: tls support HOT 5
- [Bug]: gnet.Conn.Peek crashes slice bounds out of range HOT 6
- [bug]: socket fd being reused during `OnClose()` lead to data corruption HOT 6
- [Question]: where is the example project? HOT 3
- [Question]: type engine struct { HOT 2
- [Question]: Does udp buffer support on-demand allocation? HOT 2
- [Question]: client blocks when dialing new connections HOT 19
- [Question]: How to set the client write timeout parameter? HOT 2
- [Question]: Communicate with netty HOT 2
- [Bug]: concurrent map write and read HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gnet.