Comments (13)
adiroiban
commented on June 15, 2024
I am trying to reproduce it.
I am using something like this on Ubuntu...
$ trial -u -j 8 twisted.internet.test.test_tcp
It failed with
twisted.internet.test.test_tcp.TCPTransportTestsBuilder_GIReactorTests.test_writeSequenceWithUnicodeRaisesException
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/home/adi/chevah/twisted/src/twisted/internet/posixbase.py", line 487, in _doReadOrWrite
why = selectable.doWrite()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 617, in doConnect
self._connectDone()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 630, in _connectDone
self.protocol = self.connector.buildProtocol(self.getPeer())
File "/home/adi/chevah/twisted/src/twisted/internet/base.py", line 1291, in buildProtocol
return self.factory.buildProtocol(addr)
File "/home/adi/chevah/twisted/src/twisted/internet/protocol.py", line 131, in buildProtocol
assert self.protocol is not None
builtins.AssertionError:
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_EPollReactorTests.test_clientConnectionFailedStopsReactor
===============================================================================
[ERROR]
Traceback (most recent call last):
File "/home/adi/chevah/twisted/src/twisted/internet/asyncioreactor.py", line 138, in _readOrWrite
why = method()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 617, in doConnect
self._connectDone()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 630, in _connectDone
self.protocol = self.connector.buildProtocol(self.getPeer())
File "/home/adi/chevah/twisted/src/twisted/internet/base.py", line 1291, in buildProtocol
return self.factory.buildProtocol(addr)
File "/home/adi/chevah/twisted/src/twisted/internet/protocol.py", line 131, in buildProtocol
assert self.protocol is not None
builtins.AssertionError:
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_AsyncioSelectorReactorTests.test_clientConnectionFailedStopsReactor
-------------------------------------------------------------------------------
Ran 473 tests in 0.410s
FAILED (skips=87, errors=2, successes=384)
If I don't run in parallel... it doesn't look like it's failing.
from twisted.
adiroiban
commented on June 15, 2024
Another failure anter running in a loop for about 1 minute
[ERROR]
Traceback (most recent call last):
File "/home/adi/chevah/twisted/src/twisted/internet/selectreactor.py", line 148, in _doReadOrWrite
why = getattr(selectable, method)()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 617, in doConnect
self._connectDone()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 630, in _connectDone
self.protocol = self.connector.buildProtocol(self.getPeer())
File "/home/adi/chevah/twisted/src/twisted/internet/base.py", line 1291, in buildProtocol
return self.factory.buildProtocol(addr)
File "/home/adi/chevah/twisted/src/twisted/internet/protocol.py", line 131, in buildProtocol
assert self.protocol is not None
builtins.AssertionError:
twisted.internet.test.test_tcp.TCP6ClientTestsBuilder_SelectReactorTests.test_clientConnectionFailedStopsReactor
-------------------------------------------------------------------------------
Ran 473 tests in 0.810s
FAILED (skips=87, errors=1, successes=385)
from twisted.
glyph
commented on June 15, 2024
The issue here has to do with vagaries regarding the maximum outbound buffer size. It writes one big outgoing chunk, then assumes that this chunk is big enough to flood its own buffer. IF the OS were to actually consume that data somehow, and just put it into a big buffer, hanging is what we would logically see.
from twisted.
glyph
commented on June 15, 2024
I can actually get a failure relatively quickly with just trial -u twisted.internet.test.test_tcp.AbortConnectionTests_AsyncioSelectorReactorTests.test_fullWriteBuffer; it craps out a bit over 500 iterations for me.
from twisted.
adiroiban
commented on June 15, 2024
trial -u twisted.internet.test.test_tcp.AbortConnectionTests_AsyncioSelectorReactorTests.test_fullWriteBuffer
On Ubuntu Desktop 23.10 (default kernel setup) I am not able to get that test to fail.
It was running for more than 30.000 iteration, without a failure
from twisted.
adiroiban
commented on June 15, 2024
After the merge of #12160 i still get an error.... but it might be due to OS limitations
$ netstat -tan | grep TIME_WAIT | wc -l
21214
[ERROR]
Traceback (most recent call last):
File "/home/adi/chevah/twisted/src/twisted/internet/asyncioreactor.py", line 138, in _readOrWrite
why = method()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 617, in doConnect
self._connectDone()
File "/home/adi/chevah/twisted/src/twisted/internet/tcp.py", line 630, in _connectDone
self.protocol = self.connector.buildProtocol(self.getPeer())
File "/home/adi/chevah/twisted/src/twisted/internet/base.py", line 1291, in buildProtocol
return self.factory.buildProtocol(addr)
File "/home/adi/chevah/twisted/src/twisted/internet/protocol.py", line 131, in buildProtocol
assert self.protocol is not None
builtins.AssertionError:
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_AsyncioSelectorReactorTests.test_clientConnectionFailedStopsReactor
-------------------------------------------------------------------------------
Ran 473 tests in 0.419s
FAILED (skips=87, errors=1, successes=385)
Sometimes the error is something like this ...no sure if it's related
[FAIL]
Traceback (most recent call last):
File "/home/adi/chevah/twisted/src/twisted/internet/test/test_tcp.py", line 684, in test_addresses
self.fail(clientFactory.failReason.getTraceback())
twisted.trial.unittest.FailTest: Traceback (most recent call last):
Failure: twisted.internet.error.ConnectBindError: Couldn't bind: 98: Address already in use.
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_AsyncioSelectorReactorTests.test_addresses
-------------------------------------------------------------------------------
Ran 473 tests in 0.381s
from twisted.
glyph
commented on June 15, 2024
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_AsyncioSelectorReactorTests.test_addresses
I am pretty sure this one is an OS limitation.
from twisted.
glyph
commented on June 15, 2024
twisted.internet.test.test_tcp.TCP4ClientTestsBuilder_AsyncioSelectorReactorTests.test_clientConnectionFailedStopsReactor
I suspect this one is yet another, different test flake :)
from twisted.
glyph
commented on June 15, 2024
It was running for more than 30.000 iteration, without a failure
I have instrumented it and I can get it to fail locally, reliably, in the same way: the server connection doesn't get notified of its connection dropping.
…
And, oh, of course, this is a predictable failure mode, because the server does stopReading and never starts again, which means that various reactors will not be able to detect that it has lost its connection. I am surprised this doesn't happen more often.
from twisted.
glyph
commented on June 15, 2024
This might just be a bug in asyncio.
There are definitely some potential race conditions in the Twisted code here too, but when I fixed them and instrumented doRead and looked at when it was being called, I can see that when a second call to recv would provoke EWOULDBLOCK, asyncio's main loop does not wake up again to notify that the socket is disconnected; despite the fact that the socket reader is still registered as a reader with the asyncio loop, the loop stays asleep.
from twisted.
glyph
commented on June 15, 2024
Scratch that, not asyncio, I can reproduce it on the select reactor.
from twisted.
glyph
commented on June 15, 2024
It happens much more frequently if I lose focus on the terminal window where the test is running which makes me wonder if the OS is implicated too
from twisted.
glyph
commented on June 15, 2024
I think I'm going to stop debugging this for a while because this looks like a kernel bug where select (and kqueue, and CFRunLoop, and select and friends) will notify that a socket whose peer has closed has some work to do, recv() will give errno 35, and if you busy-loop polling the file descriptor, it will start giving errno 54 as expected after 5 seconds or so, but if you just select() on it it will not start appearing as readable at that transition. Building a test case to illustrate that the sockets layer is broken in this way would be an annoying amount of work, because we'd really want an absolutely minimal test case that doesn't involve any Twisted code, just sockets.
from twisted.
Related Issues (20)
- TwistedWeb ReverseProxyResource example has string not bytes as third argument
- Setup benchmarks using Codspeed.io HOT 6
- Speed up twisted.web server, part 2 of N
- Deprecate t.w.h.HTTPClient HOT 4
- t.w.h.HTTPClient parses Content-Length laxly HOT 1
- disttrial / `trial -j` should support debug prints
- `twisted.python.test.test_sendmsg.SendmsgTests.test_sendSubProcessFD` is flaky HOT 2
- Deprecation docs should use NEXT placeholders
- Improve defer.inlineCallbacks tests
- twisted.web generates deprecation warnings internally HOT 4
- Expand benchmarks to run on PyPy as well HOT 2
- ``twisted.python.code`` package proposal HOT 8
- Performance benchmarks for Deferreds HOT 1
- Instantiating multiprocessing.SimpleQueue() from an imported package fails unless SimpleQueue() is instantiated inside Twisted application first HOT 9
- http logger should respect the encoding of its log outputs HOT 5
- Update twisted.words.protocol.irc to use IRCv3 HOT 3
- IndexError due to empty args in OpenSSL.SSL.ZeroReturnError handling in Twisted HOT 17
- t.protocols.policies.LimitConnectionsByPeer
- Add support for Encrypt-then-MAC MAC algorithms HOT 1
- twisted.names.test.test_names.ServerDNSTests.test_zoneTransferConnectionFails fails when there is no network access
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from twisted.