benoitc / socketpool Goto Github PK

socketpool is simple but still need a documentation.

I frequently see the following error in logs when socketpool is used from inside a thread (via threading module). I'm unaware if the time module or socketpool are inherently un-thread-safe, nor can I tell if there are any negative effects of the issue. It is quite bizarre though:

The exception:

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/opt/webapps/filmbot/venv/local/lib/python2.7/site-packages/socketpool/backend_thread.py", line 50, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'sleep'

The related code:

    def run(self):
        self.running = True
        while True:
            time.sleep(self.delay)
            self.pool.murder_connections()

where somehow time has become None.

I see at the top of the backend_thread module that the author assigns a reference to time.sleep, yet never uses the reference and instead still accesses sleep from the time import. Not sure what the utility of that is, or if it's an omission, etc.

While packaging socketpool for openSUSE I found that tests are still only py2k.

This patch ports tests to py3k.

I just tried "pip install socketpool" in my python3.4.3 virtualenv and got a UnicodeError. Is there any dependency I should be aware of?

Socketpool is heavily tested using reskit but still some unitests should be written.

It seems there is only 1 test, but it is currently failing for me:

FAIL: test_size_on_isconnected_failure (test_pool_01.PoolTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/abuild/rpmbuild/BUILD/socketpool-0.5.2/tests/test_pool_01.py", line 27, in test_size_on_isconnected_failure
    self.assert_(pool.size == 0)
AssertionError: False is not true

unit test works well from socketpool, but restkit test crash with version 0.4.2

File "/Users/mlecarme/eggs/socketpool-0.4.1-py2.6.egg/socketpool/util.py", line 100, in is_connected
kq.control(events, 0)
OSError: [Errno 22] Invalid argument

in the geven backend there seem to be non-debuggable deadlocks,
running the same code in multiple threads instead of greenlets didn't deadlock

i will provide a complex test ad the end of the week
but um unable to imagine a simple one atm

It looks like when reaper calls murder_connections, the pool.size is not being decremented. This means the pool size will increase after a murder of idle connections, and a subsequent fetch of a connection creating a new conn. I tested with very low max_lifetime value.

I thought of something like this:

diff --git a/socketpool/pool.py b/socketpool/pool.py
index 070228e..fe222ad 100644
--- a/socketpool/pool.py
+++ b/socketpool/pool.py
@@ -52,6 +52,8 @@ class ConnectionPool(object):
             for priority, candidate in pool:
                 if not self.too_old(candidate):
                     pool.put((priority, candidate))
+                else:
+                    self.size -= 1

     def start_reaper(self):
         self._reaper = self.backend_mod.ConnectionReaper(self,

But I am concerned about decrementing without some type of synchronization primitive (mutex, etc) to guard against greentlets or threads stepping on that shared state variable. I also see a few increment/decrement operations elsewhere in the class. Are those assumed safe?

Heya,

Please consider adding support for pooling based on local ips, because there are scenarios where control is needed over which local ip is used for accessing remote services.

So you would be matching on local_Ip, host, and port.

This would break current behavior, so perhaps you could implement it to be configurable.

Keep up the good work!

Python 2.6.7 (r267:88850, Jul 31 2011, 19:30:54) 
>>> import socketpool
>>> socketpool.util.load_backend('gevent')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...snip/venv/lib/python2.6/site-packages/socketpool/util.py", line 60, in load_backend
    mod = import_module("socketpool.backend_%s" % backend_name)
  File "...snip/venv/lib/python2.6/site-packages/socketpool/util.py", line 43, in import_module
    return imp.load_source(name, name)
IOError: [Errno 2] No such file or directory

I get the same error when trying to create a pool.
It also doesn't work if I use the full path 'socketpool.backend_gevent'.

1. support async？
2. support auto kill bad connect

Hi,

A while ago, I had observed an issue with connection "leaking". In my use case, I have connections to two different endpoints. I used first endpoint for a call to initiate app and then repeated call the second endpoints to update data. What I observed is that the pool always create a new connection for the second endpoint instead reuse the one in pool. I looked at your code and seems it is related to following piece of code

            for priority, candidate in self.pool:
                ...
                matches = candidate.matches(**options)
                if not matches:
                    # let's put it back
                    self.pool.put((priority, candidate))
                ...

Here the code compares the candidate to the matching criteria and put the candidate back to the pool if it is a no-match. The problem is that because the queue is a PriorityQueue, the SAME candidate will be pulled out again for comparison instead of the next one in the queue. This can be demonstrated with following test

from socketpool.backend_thread import PriorityQueue

if name == 'main':
q = PriorityQueue()
q.put("one")
q.put("two")
qs = q.qsize()
for item in q:
print item
q.put(item)
qs -= 1
if qs <= 0: break

which prints

one
one

instead of

one
two

So I had to put in a brutal fix to make it work for my app. https://github.com/hsun/socketpool/commit/227648bceabc2519a39557fa8ea4606f0f25215c

I haven't looked at all your recent changes since my commit. But it seems the latest code is still using the same PriorityQueue so I will assume the same issue still exists. Would you mind taking a look at the code to see whether it is a real issue or I might have used the library in a wrong way?

-thanks

It seems to me that if a connection falls out of scope or is deleted (del) with no additional references to it, it would no longer be available in the pool (and possibly not cleanly release resources as the underlying socket may not have been closed?).

Would it make sense to pass the pool itself to the Connection as an argument upon creation, and provide the ability in a __del__ inside the Connection to check self._connected, and put itself back into the pool if it is indeed still connected?

A call to invalidate would set _connected to False, so in that case when reference count hits zero, it should be cleaned up.

Something like..

class TcpConnector(Connector):
    def __init__(self, host, port, pool, backend_mod):
        self._s = backend_mod.Socket(socket.AF_INET, socket.SOCK_STREAM)
        self._s.connect((host, port))
        self.host = host
        self.port = port
        self.backend_mod = backend_mod
        self._pool = pool
        self._connected = True
        self._life = time.time()

    def __del__():
        if self._connected == True:
            self._pool.release_connection(self)
        else:
            self._pool = None

    ## rest of class code..

Thoughts?

When installing socketpool 0.5.2 on a Linux box the documentation ends up in /usr/local/socketpool/ which is a tad too generic to my taste.

Out of the 177 modules that I have installed via pip, 175 of them puts their READMEs etc. in the /usr/local/lib/python2.7/${MODULE}/ directory. The remaining two — socketpool and restkit — put theirs in /usr/local/${MODULE}/.

Could you fix that, please?

Happy new year.

Sometimes, when I run Django admin commands on a project that uses socketpool, I see these errors just before the command exits:

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/vagrant/eggs/socketpool-0.5.3-py2.7.egg/socketpool/backend_thread.py", line 51, in run
  File "/vagrant/eggs/socketpool-0.5.3-py2.7.egg/socketpool/pool.py", line 90, in murder_connections
  File "/vagrant/eggs/socketpool-0.5.3-py2.7.egg/socketpool/pool.py", line 83, in too_old
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'time'

i have no idea what its doing there, from the look of it, it shouldn't even be there, but it is

Hi

on the opensuse build server I get the following error when running setup.py

[   47s] Traceback (most recent call last):
[   47s]   File "setup.py", line 26, in <module>
[   47s]     long_description = f.read()
[   47s]   File "/usr/lib64/python3.4/encodings/ascii.py", line 26, in decode
[   47s]     return codecs.ascii_decode(input, self.errors)[0]
[   47s] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3249: ordinal not in range(128)
[   47s] error: Bad exit status from /var/tmp/rpm-tmp.UcThvc (%build)

the package already has a patch for this (not sure by whom), which fixes the problem, just wanted to report this upstream

patch:

-with open(os.path.join(os.path.dirname(__file__), 'README.rst')) as f:
+with open(os.path.join(os.path.dirname(__file__), 'README.rst'),encoding='utf-8') as f:

Arun

Now it is second time, when I noticed, that after some time of being idle, django runserver process takes 100% of CPU. When it happened again I pressed Ctrl+C and got this traceback:

Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/home/sirex/.buildout/eggs/socketpool-0.3.0-py2.7.egg/socketpool/backend_thread.py", line 49, in run
  File "/home/sirex/.buildout/eggs/socketpool-0.3.0-py2.7.egg/socketpool/pool.py", line 51, in murder_connections
  File "/home/sirex/.buildout/eggs/socketpool-0.3.0-py2.7.egg/socketpool/pool.py", line 45, in too_old
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'time'

In my project I use couchdbkit, which uses restkit, and restkit uses socketpool.

Maybe you suspect, what can cause this load? Maybe some endless loop?

If this traceback does not tell anything, maybe you can recommend a way, how to debug such occasions?