Comments (8)
It's not entirely true that data will stay at the beginning of the buffer until the stream is closed, is it? If you reach a point where the bytes.Buffer
is completely emptied, its internal position tracker will be reset to point to the beginning of its buffer and its capacity will be reused (https://golang.org/src/bytes/buffer.go?s=8509:8559#L248).
But I agree that it does seem like even if you are constantly Read
ing (e.g. doing an io.Copy
call into a file) you might never actually reach that condition unless you happen to do just the right read with just the right buffer size, and thus the buffer would continue to grow. Is that what you see in practice? I ask as someone considering using this library.
from yamux.
@havoc-io Yes, Buffer has this property. However if you look at https://github.com/hashicorp/yamux/blob/master/stream.go#L107-L112 you will see that this branch will never be taken as Read()
will only be called if there is something in the buffer. But even if it would apply, as we can't guarantee that the reader is faster than the writer and that the buffer will ever be empty it would not be the best solution.
You can see the problem happens in the unit tests if you output the capacity of the buffer after TestSendData_Large
with fmt.Printf("Recv buf cap: %d\n", stream.recvBuf.Cap())
I would fix it by replacing the Buffer with a simple slice with a behavior similar to here: https://github.com/bradfitz/http2/blob/master/buffer.go
from yamux.
Ah, that's true. It's seems like a weird design decision on the part of bytes.Buffer
to have truncation triggered on the beginning of Read
rather than the end. Triggering at the end would at least potentially truncate in this case, though of course something like a ring buffer would make more sense. Thanks for clarifying!
from yamux.
The problem is the window update logic is broken. A Stream#Read
which does not empty the recvBuf
buffer in a single call will cause unbounded memory growth for large streams.
from yamux.
I believe this issue was solved by #53, so I think it can be closed. Using the following test (which mismatches write and read speeds) on darwin
/amd64
, I see memory growing extremely rapidly pre-#53 and stopping at around 5.0 MB post-#53. There's a bit of a slow asymptotic growth towards 5.0 MB upon starting the executable, perhaps due to the Go runtime/GC behavior, but it does seem to stop eventually.
package main
import (
"crypto/rand"
"io"
"io/ioutil"
"net"
"time"
"github.com/hashicorp/yamux"
)
type sleepReader struct {
reader io.Reader
}
func (r *sleepReader) Read(buffer []byte) (int, error) {
time.Sleep(10 * time.Millisecond)
return r.reader.Read(buffer)
}
func main() {
clientConn, serverConn := net.Pipe()
clientSession, _ := yamux.Client(clientConn, yamux.DefaultConfig())
serverSession, _ := yamux.Server(serverConn, yamux.DefaultConfig())
go func() {
c, _ := serverSession.Accept()
reader := &sleepReader{c}
io.Copy(ioutil.Discard, reader)
}()
c, _ := clientSession.Open()
io.Copy(c, rand.Reader)
}
from yamux.
I have a similar problem under production conditions like OP.
My memory is not growing 1Mb per 1Mb streamed data, but it is growing over time.
Here is a pprof dump (linux, amd64), that pointed me to this issue:
Showing nodes accounting for 9243.28kB, 100% of 9243.28kB total
Showing top 10 nodes out of 14
flat flat% sum% cum cum%
8731.23kB 94.46% 94.46% 8731.23kB 94.46% bytes.makeSlice
512.05kB 5.54% 100% 512.05kB 5.54% github.com/hashicorp/yamux.newStream
0 0% 100% 8731.23kB 94.46% bytes.(*Buffer).ReadFrom
0 0% 100% 8731.23kB 94.46% bytes.(*Buffer).grow
0 0% 100% 512.05kB 5.54% github.com/hashicorp/yamux.(*Session).Open
0 0% 100% 512.05kB 5.54% github.com/hashicorp/yamux.(*Session).OpenStream
0 0% 100% 8731.23kB 94.46% github.com/hashicorp/yamux.(*Session).handleStreamMessage
0 0% 100% 8731.23kB 94.46% github.com/hashicorp/yamux.(*Session).recv
0 0% 100% 8731.23kB 94.46% github.com/hashicorp/yamux.(*Session).recvLoop
0 0% 100% 8731.23kB 94.46% github.com/hashicorp/yamux.(*Stream).readData
The biggest jumps occur, when a new stream (OpenStream
) begins to stream. I was wondering, if the problem is on my side, that the code retains open streams, but I'm pretty sure it doesn't.
Is there anything else I could provide, apart from a test case, that reproduces the issue?
from yamux.
@pbedat What version of yamux are you using, could you retest with the latest. Any repro would be great as well!
from yamux.
@evanphx I was using github.com/hashicorp/yamux v0.0.0-20200609203250-aecfd211c9ce (Go 1.15)
I switched to smux right after that post, which served me well. I could try to reproduce this, but I don't think it would be worth it. A lot has changed in 1.5 years 😉
If you really really need a repro I will try my best.
from yamux.
Related Issues (20)
- TestPing failed on Windows HOT 1
- yamux: Invalid protocol version: 123
- Protocol Version Error HOT 1
- Should we close the session when streams receive window exceeded ? HOT 1
- Is Session.Accept Result Really Compatible With net.Conn? HOT 2
- What is Disconnect Flow
- [bug] TestGoAway failed
- Add link to who using yamux
- 请问,是否支持UDP协议?
- Download speeds have reduced and upload speeds gone up with the latest version of yamux
- `go test ./...` is failed HOT 1
- Config should take a Logger interface HOT 3
- SESSION关闭后r.Session().NumStreams()依然可以获取通道数?是不是有问题
- How can I change the "initialStreamWindow" gracefully? HOT 1
- Data lost when closing Session HOT 2
- Weird question HOT 2
- "context canceled" error when using with http.ReverseProxy
- at stream read/write wrap underlying error to returned error
- stream recvWindow decremented incorretly HOT 4
- Panic when sending large data HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yamux.