linsomniac / python-memcached Goto Github PK
View Code? Open in Web Editor NEWA python memcached client library.
A python memcached client library.
I'd like to update the codebase to use Google's coding style for docstrings:
Args:
path (str): The path of the file to wrap
field_storage (FileStorage): The :class:`FileStorage` instance to wrap
temporary (bool): Whether or not to delete the file when the File
instance is destructed
Returns:
BufferedFileStorage: A buffered writable file descriptor
Doing so would enable it to work with Sphinx using the Napoleon extension, but did not want to put forth the work if it would not be accepted.
Would you be willing to accept such a pull-request if it is clean, doesn't break anything, etc? I could also add a docs dir with sphinx autodoc so a readthedocs.org site could be setup.
In Python Memcached 1.56 (w/ six-1.9.0 & Python 2.7.10)
doing
cache.set('foo', False)
cache.get('foo')
generates a ValueError: invalid literal for int() with base 10: 'False'
line 1235 memcache.py
When the method _val_to_store_info is called the path code that it is followed with a boolean is
elif isinstance(val, int):
flags |= Client._FLAG_INTEGER
val = str(val)
if six.PY3:
val = val.encode('ascii')
# force no attempt to compress this silly string.
min_compress_len = 0
Since the flags bits have been set to an integer, the boolean value is changed to a string ('False'). So in the method _recv_value the code tries to change that string to an int which fails .
In Python Memcached 1.54
the original code in _val_to_store_info is
elif isinstance(val, int):
flags |= Client._FLAG_INTEGER
val = "%d" % val
# force no attempt to compress this silly string.
min_compress_len = 0
the issue also affect when checking for long values
After checking a whole loada things, I've found that for some keys, python-memcached just won't get, or set them on my machine (homebrew'd 1.4.15, pip installed python-memcached 1.53 on Mac OSX 10.9). My first issue was a shot in the dark, not having a clear idea as to what was going on, but after more digging I now definitely now.
It all hinges around
def _get_server(self, key):
If we add two debug printout lines,
def _get_server(self, key):
if isinstance(key, tuple):
serverhash, key = key
else:
serverhash = serverHashFunction(key)
for i in range(Client._SERVER_RETRIES):
server = self.buckets[serverhash % len(self.buckets)]
if server.connect():
#print "(using server %s)" % server,
print 'got server {} for {}'.format(serverhash % len(self.buckets),
key)
return server, key
print 'server {} failed for {}'.format(serverhash % len(self.buckets),
key)
serverhash = serverHashFunction(str(serverhash) + str(i))
return None, None
Trying to get or set the key 'NFL::CAR_TB',
import memcache
mc_cl = memcache.Client('127.0.0.1')
mc_cl.set('NFL::CAR_TB', 1)
mc_cl.get('NFL::CAR_TB')
mc_cl.set('NFL::CAR_UB', 1)
mc_cl.get('NFL::CAR_UB')
results, on my machine with,
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 3 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 3 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 8 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820703)) failed for NFL::CAR_TB
server 5 (inet:.:11211 (dead until 1384820703)) failed for NFL::CAR_TB
got server 6 for NFL::CAR_UB
got server 6 for NFL::CAR_UB
If I alter _get_server to
def _get_server(self, key):
choices = range(len(self.buckets) - 1)
random.shuffle(choices)
if isinstance(key, tuple):
choice, key = key
else:
choice = choices.pop()
for _ in range(Client._SERVER_RETRIES):
server = self.buckets[choice]
if server.connect():
#print "(using server %s)" % server,
print 'got server {} for {}'.format(choice,
key)
return server, key
print 'server {} ({}) failed for {}'.format(choice, server, key)
choice = choices.pop()
return None, None
Then everything works much better,
got server 4 for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 7 (inet:.:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 0 (inet:1:11211 (dead until 1384820883)) failed for NFL::CAR_TB
server 2 (inet:7:11211 (dead until 1384820883)) failed for NFL::CAR_TB
got server 4 for NFL::CAR_TB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_UB
server 3 (inet:.:11211 (dead until 1384820883)) failed for NFL::CAR_UB
got server 4 for NFL::CAR_UB
server 1 (inet:2:11211 (dead until 1384820883)) failed for NFL::CAR_UB
got server 6 for NFL::CAR_UB
~~
multi_get and multi_set not getting / setting all keys
I was getting very inconsistent results when trying to set multiple keys in my python program,
import memcache
TWO_HOURS = 2 * 60 * 60
mc_cl = memcache.Client('127.0.0.1')
mapping = {...}
mc_cl.flush_all()
ret = mc_cl.set_multi(mapping=mapping, time=TWO_HOURS)
getret = mc_cl.get_multi(mapping.keys())
if len(mapping) != len(getret):
print 'not set\n\t{}'.format('\n\t'.join([str((k, mapping[k])) for k in
[a for a in mapping.keys()
if a not in getret.keys()]]))
And after analysing the raw memcache output, it seems like not all the keys are being set, and not all the keys are being requested thereafter. All things done and up to date on homebrew and pip, on Mac OSX 10.9.
Memcache output below and reading it states that only 101 keys were attempted to be written and read, whereas there were 228 items.
import re
gamesSet = []
gamesGet = []
with open('memout.log') as f:
for line in f.read().split('\n'):
match = re.match('^<21 set ([^ ]+) .*$', line)
if match is not None:
gamesSet.append(match.group(1))
continue
match = re.match('^<21 get (.*)$', line)
if match is not None:
gamesGet += match.group(1).split(' ')
continue
print len(mapping), mapping
print len(gamesSet), gamesSet
print len(gamesGet), gamesGet
That log is below.
<21 new auto-negotiating client connection
21: Client using the ascii protocol
<21 flush_all
>21 OK
<22 new auto-negotiating client connection
22: Client using the ascii protocol
<22 flush_all
>22 OK
<21 set NCAAF::BUFF_KENTST 1 7200 34
>21 STORED
<21 set NCAAF::ND_USC 1 7200 29
>21 STORED
~~
$ pip install python-memcached==1.50
Downloading/unpacking python-memcached==1.50
Running setup.py egg_info for package python-memcached
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/lakr/.venv/mtk_1.5/build/python-memcached/setup.py", line 9, in <module>
long_description=open("README.md").read(),
IOError: [Errno 2] No such file or directory: 'README.md'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/home/lakr/.venv/mtk_1.5/build/python-memcached/setup.py", line 9, in <module>
long_description=open("README.md").read(),
IOError: [Errno 2] No such file or directory: 'README.md'
----------------------------------------
Command python setup.py egg_info failed with error code 1 in /home/lakr/.venv/mtk_1.5/build/python-memcached
Storing complete log in /home/lakr/.pip/pip.log
Get this when installing via pypi. Surely related to 17fc525.
In the source code, I found the following comment:
mc.set("key", "1") # note that the key used for incr/decr must be
# a string.
mc.incr("key")
mc.decr("key")
However, I tested incr
using both integer and string. Both seem to work:
In [14]: cache.add('foo', 1)
Out[14]: True
In [15]: cache.incr('foo')
Out[15]: 2
In [16]: cache.get('foo')
Out[16]: 2
In [17]: cache.add('bar', '1')
Out[17]: True
In [18]: cache.incr('bar')
Out[18]: 2
In [19]: cache.get('bar')
Out[19]: '2'
So is the comment still relevant?
When set_multi decides that a value is too big to be stored, it still wants a 'STORED' from the server... the server is finally marked as dead.
Here is how to reproduce:
import memcache
c = memcache.Client(('localhost:11211', ))
assert not any(server.deaduntil for server in c.servers), 'servers are marked dead before set_multi!'
assert 'a' in c.set_multi({'a': 'a' * 1024 * 1025}), 'we were able to write more than 1Mb of data!'
assert not any(server.deaduntil for server in c.servers), 'servers are marked dead after set_multi!'
raises an AssertionError: servers are marked dead after set_multi!
but it shouldn't
It will be easier to track what happened between two versions if information will be grouped by version.
I could reformat the Changelog file and make a pull-request if you wish.
I recently ran into a situation where our build server did not have a running instance of memcached. I would have expected things to fail gracefully. (Clearly, there wouldn't be any caching happening. Each request would simply have to re-fetch any cached data.) But instead of failing gracefully, the caching code failed after the connect attempt to the memcached server failed.
It looks like once the server is marked dead there are some additional calls to cmemcache_hash that get made with strings as keys. To reproduce this, find a server without a memcached instance and try this:
python
Python 3.5.1 (default, Apr 6 2016, 16:48:42)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> cache = memcache.Client(['127.0.0.1:11211'], debug=1)
>>> cache.set("working?",1,1)
MemCached: MemCache: inet:127.0.0.1:11211: connect: [Errno 111] Connection refused. Marking dead.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 700, in set
return self._set("set", key, val, time, min_compress_len, noreply)
File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 983, in _set
server, key = self._get_server(key)
File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 413, in _get_server
serverhash = serverHashFunction(str(serverhash) + str(i))
File "/<snipped>/lib/python3.5/site-packages/memcache.py", line 65, in cmemcache_hash
(((binascii.crc32(key) & 0xffffffff)
**TypeError: a bytes-like object is required, not 'str'**
So, I temporarily "fixed" the issue with something like this:
--- /<snipped>/lib/python3.5/site-packages/memcache.py 2016-04-07 17:23:41.428028602 -0500
+++ /<snipped>/lib/python3.5/site-packages/memcache.py_fixed 2016-04-07 17:23:13.606003863 -0500
@@ -61,6 +61,9 @@
def cmemcache_hash(key):
+ if isinstance(key, six.text_type):
+ key = key.encode('utf-8')
+
return (
(((binascii.crc32(key) & 0xffffffff)
>> 16) & 0x7fff) or 1)
I'm not sure if this is the right approach to correct the issue or not. Hopefully we can get back to a point where python-memcached can fail gracefully when the underlying service is absent.
Thanks,
Matthew Lee
Following the docs on running tests and hitting an ImportError while running. Six is definitely installed, not sure if this is a venv issue, or something that's been brought up before?
(memcached)nick@localhost python-memcached (master) $ tox -e py27
py27 create: /Users/nick/Code/python-memcached/.tox/py27
py27 installdeps: -r/Users/nick/Code/python-memcached/requirements.txt, -r/Users/nick/Code/python-memcached/test-requirements.txt
py27 develop-inst: /Users/nick/Code/python-memcached
py27 installed: argparse==1.3.0,coverage==3.7.1,flake8==2.2.4,hacking==0.10.2,mccabe==0.2.1,nose==1.3.7,pbr==0.10.8,pep8==1.5.7,pyflakes==0.8.1,-e git+https://github.com/nicholasserra/python-memcached.git@664bd3e23fe500fbde4c70636e2d24c8fd2f35af#egg=python_memcached-master,six==1.9.0,stevedore==1.2.0,virtualenv==12.0.2,virtualenv-clone==0.2.5,virtualenvwrapper==4.3.1,wsgiref==0.1.2
py27 runtests: PYTHONHASHSEED='3267181802'
py27 runtests: commands[0] | nosetests
EE
======================================================================
ERROR: Failure: ImportError (No module named six)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/nick/Code/python-memcached/tests/test_memcache.py", line 5, in <module>
import six
ImportError: No module named six
======================================================================
ERROR: Failure: ImportError (No module named six)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/nick/Code/python-memcached/.tox/py27/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/nick/Code/python-memcached/tests/test_setmulti.py", line 17, in <module>
import memcache
File "/Users/nick/Code/python-memcached/memcache.py", line 60, in <module>
import six
ImportError: No module named six
----------------------------------------------------------------------
Ran 2 tests in 0.004s
FAILED (errors=2)
ERROR: InvocationError: '/Users/nick/Code/python-memcached/.tox/py27/bin/nosetests'
cache.get_multi(new_keys)
File "site-packages/memcache.py", line 930, in get_multi
line = server.readline()
File "site-packages/memcache.py", line 1129, in readline
raise _ConnectionDeadError()
It currently catches _Error
and socket.error
, but not _ConnectionDeadError
.
I noticed this while doing a restart of a memcached machine.
In [1]: import memcache
In [2]: mc = memcache.Client(['somenonexistenthost'])
In [3]: mc.get('qwe')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-bb201bdf51b9> in <module>()
----> 1 mc.get('qwe')
/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in get(self, key)
1087 @return: The value or None.
1088 '''
-> 1089 return self._get('get', key)
1090
1091 def gets(self, key):
/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in _get(self, cmd, key)
1032 if self.do_check_key:
1033 self.check_key(key)
-> 1034 server, key = self._get_server(key)
1035 if not server:
1036 return None
/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in _get_server(self, key)
411 # print("(using server %s)" % server,)
412 return server, key
--> 413 serverhash = serverHashFunction(str(serverhash) + str(i))
414 return None, None
415
/home/ei-grad/.local/lib/python3.5/site-packages/memcache.py in cmemcache_hash(key)
63 def cmemcache_hash(key):
64 return (
---> 65 (((binascii.crc32(key) & 0xffffffff)
66 >> 16) & 0x7fff) or 1)
67 serverHashFunction = cmemcache_hash
TypeError: a bytes-like object is required, not 'str'
In [4]: mc = memcache.Client(['localhost'])
In [6]: mc.set('qwe', 'it works')
Out[6]: True
In [7]: mc.get('qwe')
Out[7]: 'it works'
See also #70
I realize this will be controversial given how long the Python memcache API has been around. Relative to the memcached protocol, Python's API doesn't support CAS beyond very trivial use cases.
The Python API forces implementations to manage CAS ID's implicitly. This results in several problems:
Since an implementation doesn't know if a CAS ID will subsequently be needed, it must hold on to all ID's indefinitely. Granted, this is not much of a problem for applications with a fixed number of items involved in CAS. However there are use cases which can yield an unbounded number of such items, for example when implementing concurrent collections on memcache using an item per node. (I have https://github.com/google/memcache-collections which implements a deque this way.)
While the API does provide reset_cas(), it is of limited use given that the retained cache ID's are global state. This makes it very hard to combine different libraries employing CAS into one application since they will inevitably have different CAS ID lifetime requirements. Furthermore, the Python memcache implementation makes bugs related to CAS ID lifetime exceedingly hard to detect since it silently succeeds cas() calls for which a CAS ID couldn't be found.
aside: The current API limits CAS use to single reads via gets()-- it isn't possible to use get_multi() with CAS. For this, App Engine extended get_multi() with a for_cas parameter.
A very useful building block for distributed data structures is the MCAS operation, i.e. atomic CAS across multiple, independent addresses. It's possible in general to implement MCAS on top of the single CAS primitive as described in "Practical lock freedom", Keir Fraser, 2004 (http://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf). This would be interesting to implement on top of memcache, as it would allow atomic mutation of multiple items, even when they exist on different memcached servers in a cluster. However the lock free algorithm requires other processes to "help" when they find transactions in an intermediate state. To implement this on memcache, the CAS ID's would need to be stored, shared, and used among clients (which may be on different threads, processes, or machines). The memcached protocol supports this, but the Python API does not.
I don't have a proposal yet, but I'd like to investigate extending the Python memcache API to support explicit management of CAS ID's.
Hi there, I am a bit confused about the current latest release on PyPI (1.31) whereas I used to package 1.47 (https://pypi.python.org/pypi/python-memcached/1.47) for openSUSE. However, it seems like the contents of 1.31 is newer than 1.47. The issue here is that distributions already packaged and ship 1.47 (or 1.48) and downgrading would be a huge pain (involving RPM epoch). Also, when comparing both versions, 1.467has more than 100000 downloads compared to ~500 of 1.37. So I suspect we're not the only ones hitting this issue.
So first of all thank you for taking over maintainership of this module. But in order to solve the issue, could you please create a release with a more appropriate version number? Maybe a 2.0 is appropriate even ;-)
Hi,
I have the working implementation of ketama consistent hashing algorithm on top of your library. May I send you pull request ? Or you are keeping the library as minimal as possible ? Please let me know, I would love to share my implementation.
Warm Regards,
Haridas N.
It appears that delete will return nonzero even on failure (NOT_FOUND server response). This appears to violate the api description that says "@return: Nonzero on success." The memecached text protocol defines only 'DELETED' as "success", and 'NOT_FOUND' to indicate the server did not have the key at the time the delete was requested -- and records this in server stats as 'delete_misses'.
The fix is trivial (simply remove NOT_FOUND from the expect list for delete), and I was going to just open a pull req for it, but there appears to be some history behind the decision.
It appears this change was part of 81933bc and related to an old-bug (at least this is the bug referenced in the commit).
Is there historical/deeper meaning surrounding this change?
Is it a bug or just a kludge to maintain some pre-existing behavior (e.g. api stability)?
It seems that handler of _ConnectionDeadError is missed.
try:
server.send_cmd(cmd)
line = server.readline()
if line and line.strip() in expected: return 1
self.debuglog('%s expected %s, got: %s'
% (cmd, ' or '.join(expected), repr(line)))
except socket.error, msg:
if isinstance(msg, tuple): msg = msg[1]
server.mark_dead(msg)
return 0
Hi,
The latest release is now over a year old and doesn't include any of the python 3 compatibility. Can you release a new version please?
Hi,
how can I figure out if a host is alive or not without using expensive timestamp set/get checks?
Hi,
I am getting errors while trying to load python-memcached. With easy-install
Best match: python-memcached 1.53
Downloading https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.53.tar.gz#md5=89570d26e7e7b15caa668a6b2678bd3c
Processing python-memcached-1.53.tar.gz
Writing /var/folders/_b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/setup.cfg
Running python-memcached-1.53/setup.py -q bdist_egg --dist-dir /var/folders/b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/egg-dist-tmp-ibs24u
Traceback (most recent call last):
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/bin/easy_install", line 9, in
load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1992, in main
with_ei_usage(lambda:
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1979, in with_ei_usage
return f()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1996, in
distclass=DistributionWithoutHelpCommands, **kw
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 929, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/distutils/dist.py", line 948, in run_command
cmd_obj.run()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 380, in run
self.easy_install(spec, not self.no_deps)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 623, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 653, in install_item
dists = self.install_eggs(spec, download, tmpdir)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 849, in install_eggs
return self.build_and_install(setup_script, setup_base)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1130, in build_and_install
self.run_setup(setup_script, setup_base, args)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/command/easy_install.py", line 1115, in run_setup
run_setup(setup_script, args)
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 69, in run_setup
lambda: execfile(
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 120, in run
return func()
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/sandbox.py", line 71, in
{'file':setup_script, 'name':'main'}
File "/Users/burakk/BurakWorks/Web/VIRTUAL_ENVIRONMENTS/django_ozo/lib/python3.3/site-packages/setuptools/compat.py", line 92, in execfile
exec(compile(source, fn, 'exec'), globs, locs)
File "setup.py", line 4, in
File "/var/folders/_b/zp6hg4_s52l19z0z42kvzzt80000gn/T/easy_install-qat645/python-memcached-1.53/memcache.py", line 410
except socket.error, msg:
^
SyntaxError: invalid syntax
We ran into an issue on our live memcached servers where set_multi()
was raising an error like the following (this is test code, but it was basically the same traceback):
Traceback (most recent call last):
File "C:\dev\main\python\test\setmultibug.py", line 41, in <module>
main()
File "C:\dev\main\python\test\setmultibug.py", line 37, in main
bad_keys = client.set_multi(mapping)
File "C:\dev\main\tools\python\modules\memcache.py", line 735, in set_multi
line = server.readline()
File "C:\dev\main\tools\python\modules\memcache.py", line 1163, in readline
recv = self.socket.recv
AttributeError: 'NoneType' object has no attribute 'recv'
I'm not sure the root cause of the connection issue in our case, but it shouldn't cause an AttributeError -- it should mark the server dead and move along. What I believe is happening is the connection was being forcibly closed between readline()
calls when reading the per-key responses in set_multi()
.
readline()
actually handles this case with its if not data: ...
clause, but it does it by calling self.close_socket()
directly, which sets self.socket = None
.
Here's a test program that stubs out socket.socket
to pretend it closed the connection between recv() calls to repro the issue:
"""Repro memcache set_multi() AttributeError:
AttributeError: 'NoneType' object has no attribute 'recv'
"""
import socket
import memcache
RECV_CHUNKS = ['chunk1']
class FakeSocket(object):
def __init__(self, *args):
print 'FakeSocket{0!r}'.format(args)
self._recv_chunks = list(RECV_CHUNKS)
def connect(self, *args):
print 'FakeSocket.connect{0!r}'.format(args)
def sendall(self, *args):
print 'FakeSocket.sendall{0!r}'.format(args)
def recv(self, *args):
if self._recv_chunks:
data = self._recv_chunks.pop(0)
else:
data = ''
print 'FakeSocket.recv{0!r} -> {1!r}'.format(args, data)
return data
def close(self):
print 'FakeSocket.close()'
def main():
socket.socket = FakeSocket
client = memcache.Client(['memcached'], debug=True)
mapping = {'foo': 'FOO', 'bar': 'BAR'}
bad_keys = client.set_multi(mapping)
print 'set_multi({0!r}) -> {1!r}'.format(mapping, bad_keys)
if __name__ == '__main__':
main()
We've been using python-memcached v1.48 with Python 2.7.8 and Django 1.7.7.
I've just tried deploying python-memcached v1.54 and it increased average Memcached get response times from ~4.5ms to ~16ms, and sets from ~0.3ms to ~0.4ms (as reported by New Relic).
Before I try bisecting further, is this expected?
(If it helps, our version of six
is 1.9.0).
I receive the above exception when issuing a get request for a key "groovy-1" inserted by a Java client.
The key in question (inserted by a Java client) that causes a problem.
echo "get key groovy-1" | nc 127.0.0.1 11211
VALUE groovy-1 32 13
1377311483802
END
Does a flag value of "32" affect this python client's ability to read the value?
Thank you for looking into the matter.
The code that causes the Exception is as follows:
#!/usr/bin/env python
import memcache
mc = memcache.Client( [ '127.0.0.1:11211'] )
# access a non-existent key in memcache, should throw
# UnboundLocalError: local variable 'val' referenced before assignment
# At least that exception is thrown for me using python_memcached-1.48-py2.7.egg
# installed using easy_install on Aug 23rd 2013.
print "groovy-1 = %s" % mc.get( "groovy-1" )
This test (adapted from Django's test suite) passes on Python 2, but fails on Python 3:
diff --git a/tests/test_memcache.py b/tests/test_memcache.py
index 321dc7b..0415de9 100644
--- a/tests/test_memcache.py
+++ b/tests/test_memcache.py
@@ -137,6 +137,30 @@ class TestMemcache(unittest.TestCase):
value = self.mc.get(key)
self.assertEqual(value, 5)
+ def test_binary_string(self):
+ # Binary strings should be cacheable
+ from zlib import compress, decompress
+ value = 'value_to_be_compressed'
+ compressed_value = compress(value.encode())
+
+ # Test set
+ self.mc.set('binary1', compressed_value)
+ compressed_result = self.mc.get('binary1')
+ self.assertEqual(compressed_value, compressed_result)
+ self.assertEqual(value, decompress(compressed_result).decode())
+
+ # Test add
+ self.mc.add('binary1-add', compressed_value)
+ compressed_result = self.mc.get('binary1-add')
+ self.assertEqual(compressed_value, compressed_result)
+ self.assertEqual(value, decompress(compressed_result).decode())
+
+ # Test set_many
+ self.mc.set_multi({'binary1-set_many': compressed_value})
+ compressed_result = self.mc.get('binary1-set_many')
+ self.assertEqual(compressed_value, compressed_result)
+ self.assertEqual(value, decompress(compressed_result).decode())
+
def test_ignore_too_large_value(self):
# NOTE: "MemCached: while expecting[...]" is normal...
key = 'keyhere'
======================================================================
ERROR: test_binary_string (__main__.TestMemcache)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/test_memcache3.py", line 53, in test_binary_string
compressed_result = self.mc.get('binary1')
File "/home/tim/code/python-memcached/memcache.py", line 1093, in get
return self._get('get', key)
File "/home/tim/code/python-memcached/memcache.py", line 1077, in _get
return _unsafe_get()
File "/home/tim/code/python-memcached/memcache.py", line 1065, in _unsafe_get
value = self._recv_value(server, flags, rlen)
File "/home/tim/code/python-memcached/memcache.py", line 1235, in _recv_value
val = buf.decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte
I don't know if anyone's profiled where memcache.py is spending most of its time, but in my recent benchmarks, cProfile tells me it was spending far and away most of its time in check_key(). This is quite sad, as once you've tested your code, you're not going to be sending through bad keys, so these expensive checks are basically worthless in production.
The most expensive check is looping through all the chars in the key, calling ord(c), and ensuring each char is not a control character. This is the most expensive check, and also the least likely to actually happen in practice (in my experience it's very rare to build a key containing control chars).
I'm using a subclass of memcache.Client that overrides check_key() and speeds up my multi_get/multi_set benchmark by literally 30%. I still perform the most basic sanity checks on the key:
class Client(memcache.Client):
def check_key(self, key, key_extra_len=0):
assert key, "Can't use None or empty string as key"
assert isinstance(key, str), 'Keys must be str instances'
Even eliminating the for char in key:
loop gets you 80% of this speed improvement.
What about an option to the Client() constructor check_keys that bypasses what's now check_key(), or just performs the basic checks I show above? It could be check_keys=True by default to emulate current behaviour.
# some web code here
# keys = [key, key+'_queued']
File "./services/cache.py", line 33, in mget
results = self.get_multi(keys)
File "/var/www/heydayserver/env/local/lib/python2.7/site-packages/memcache.py", line 972, in get_multi
retvals[prefixed_to_orig_key[rkey]] = val # un-prefix returned key.
KeyError: '-34.9832:146.2685_queued'
I am not sure why this is happening though, all I use the memcache for is to store json strings with keys. It happens sporadically, around redeploy of a Flask server.
Hi,
I'm trying to use memcache on Heroku with "Memcached Cloud". The connection credentials I'm given include a username and a password, however it doesn't seem like there's anywhere to provide that information in the library. Is there any plan to support usernames/passwords in the library, or is that beyond the scope of this library?
Thanks!
Is it possible to make client automatically skip a server in case it went down?
We have a cluster of 4 machines. One of them is down which cause application servers to to pause (Default config) and fill up the queue.
I found failover config in PHP Client. Is there counterpart in this client? What's the best way to deal with this?
It would be useful to have git tags corresponding to pypi release tarballs:
https://pypi.python.org/packages/source/p/python-memcached/
I get this error under Python 3.4 using 2251916:
Error handling request
Traceback (most recent call last):
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 199, in get_response
response = middleware_method(request, response)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/core/middleware.py", line 192, in process_response
context = RequestContext(request)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/template/context.py", line 169, in __init__
self.update(processor(request))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/conf/context_processors.py", line 40, in settings
settings_dict = cache_get(cache_key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/utils/cache.py", line 48, in cache_get
packed = cache.get(_hashed_key(key))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 75, in get
val = self._cache.get(key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1002, in get
return self._get('get', key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 986, in _get
return _unsafe_get()
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 957, in _unsafe_get
server.send_cmd("%s %s" % (cmd, key))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1299, in send_cmd
self.socket.sendall(cmd + '\r\n')
TypeError: 'str' does not support the buffer interface
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/user/.virtualenvs/project/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 93, in handle
self.handle_request(listener, req, client, addr)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 134, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/wsgi.py", line 206, in __call__
response = self.get_response(request)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 203, in get_response
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/handlers/base.py", line 236, in handle_uncaught_exception
return callback(request, **param_dict)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/utils/decorators.py", line 99, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/core/views.py", line 220, in server_error
context = RequestContext(request, {"STATIC_URL": settings.STATIC_URL})
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/template/context.py", line 169, in __init__
self.update(processor(request))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/conf/context_processors.py", line 40, in settings
settings_dict = cache_get(cache_key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/mezzanine/utils/cache.py", line 48, in cache_get
packed = cache.get(_hashed_key(key))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 75, in get
val = self._cache.get(key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1002, in get
return self._get('get', key)
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 986, in _get
return _unsafe_get()
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 957, in _unsafe_get
server.send_cmd("%s %s" % (cmd, key))
File "/user/.virtualenvs/project/lib/python3.4/site-packages/memcache.py", line 1299, in send_cmd
self.socket.sendall(cmd + '\r\n')
TypeError: 'str' does not support the buffer interface
Is it possible to programmatically check if the connection is really working after firing a memcache.cas
request? Can I check the response code?
I could do a dumb:
memcache.set('Connection') == 'True'
if memcache.get('Connection') == 'True':
continue
However is there a better way to do that?
From time to time when getting some data from memcache i get this error:
ERR = local variable 'val' referenced before assignment
In case anyone about to start using python-memcached sees this.
This package is not actively maintained at the moment. For example with the latest release there is a 400% performance slowdown under Python 2.7 (#71) that would be fixed by the unmerged #86.
As such for now I'd strongly recommend using pylibmc instead, which in most cases is a drop-in replacement:
https://github.com/lericson/pylibmc
http://sendapatch.se/projects/pylibmc/misc.html#differences-from-python-memcached
Hi!
I ran code via 2to3 tool and it seems quite simple to port it. Maybe you have some thoughts about it, what Python versions might be supported?
Context: memcached + python-memcached + django-memcached
Error:
/usr/local/lib/python2.7/site-packages/memcache.py in _recv_value
1235 val = int(buf)
Local vars:
buf 'True'
self <memcache.Client object at 0x7f926d7590b8>
rlen 6
flags 2
server <memcache._Host object at 0x7f926d826190>
This error does not occur in v <= 1.54
Installing python-memcached over PyPI takes ages.
I suspect some url configuration on the PyPI package causes this.
Could you fix this issue?
Below is my console output for debugging this:
$ pypi-show-urls -v -p python-memcached
Download candidates for python-memcached
========================================
Candidates from https://pypi.python.org/simple/python-memcached/
----------------------------------------------------------------
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.51.tar.gz#md5=580603a82b7078642ed18ea000f825c5
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.48.tar.gz#md5=58f8c328304df6aca1f8b60170e98932
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.31.tar.gz#md5=e1de1fa7d504bb9ce0f9e295fd77adb0
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.47.tar.gz#md5=e4e9d65e5721a1bb01f8d657ddf3f03e
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.53.tar.gz#md5=89570d26e7e7b15caa668a6b2678bd3c
https://pypi.python.org/packages/source/p/python-memcached/python-memcached-1.52.tar.gz#md5=b80e2d024470e9baf693d016c9e96d6a
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.45.tar.gz
http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.46.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.44.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.43.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.38.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.41.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.42.tar.gz
http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.47.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.40.tar.gz
ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz
http://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-1.48.tar.gz
Candidates from http://www.tummy.com/Community/software/python-memcached/
-------------------------------------------------------------------------
ftp://ftp.tummy.com/pub/python-memcached/python-memcached-latest.tar.gz
Versions only available externally
----------------------------------
1.38
1.42
1.45
1.44
1.46
1.41
1.40
1.43
latest
$ pip install -vUI python-memcached
Downloading/unpacking python-memcached
Could not fetch URL ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz (from http://f.pypi.python.org/simple/python-memcached/): <urlopen error ftp error: [Errno ftp error] 550 Failed to change directory.>
Will skip URL ftp://ftp.tummy.com/pub/python-memcached/old-releases/python-memcached-latest.tar.gz when looking for download links for python-memcached
Using version 1.53 (newest of versions: 1.53, 1.52, 1.51, 1.48, 1.48, 1.47, 1.47, 1.46, 1.45, 1.44, 1.43, 1.42, 1.41, 1.40, 1.38, 1.31, latest, latest)
Using download cache from /Users/diederik/Library/Caches/pip-downloads/http%3A%2F%2Ff.pypi.python.org%2Fpackages%2Fsource%2Fp%2Fpython-memcached%2Fpython-memcached-1.53.tar.gz
Running setup.py egg_info for package python-memcached
running egg_info
creating pip-egg-info/python_memcached.egg-info
writing pip-egg-info/python_memcached.egg-info/PKG-INFO
writing top-level names to pip-egg-info/python_memcached.egg-info/top_level.txt
writing dependency_links to pip-egg-info/python_memcached.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.rst'
warning: no files found matching '*.txt'
warning: no files found matching 'MakeFile'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.gitignore' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'pip-egg-info/python_memcached.egg-info/SOURCES.txt'
Installing collected packages: python-memcached
Found existing installation: python-memcached 1.53
Uninstalling python-memcached:
Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.py
Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.pyc
Removing file or directory /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/python_memcached-1.53-py2.7.egg-info
Successfully uninstalled python-memcached
Running setup.py install for python-memcached
running install
running build
running build_py
creating build
creating build/lib
copying memcache.py -> build/lib
running install_lib
copying build/lib/memcache.py -> /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages
byte-compiling /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/memcache.py to memcache.pyc
running install_egg_info
running egg_info
writing python_memcached.egg-info/PKG-INFO
writing top-level names to python_memcached.egg-info/top_level.txt
writing dependency_links to python_memcached.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'python_memcached.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.rst'
warning: no files found matching '*.txt'
warning: no files found matching 'MakeFile'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '.gitignore' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'python_memcached.egg-info/SOURCES.txt'
Copying python_memcached.egg-info to /Users/diederik/Sites/virtualenvs/demo-env/lib/python2.7/site-packages/python_memcached-1.53-py2.7.egg-info
running install_scripts
writing list of installed files to '/var/folders/24/5y_gshzd7vg9f5mv_hpxf1y40000gn/T/pip-JNRJb5-record/install-record.txt'
Successfully installed python-memcached
Cleaning up...
Removing temporary dir /Users/diederik/Sites/virtualenvs/demo-env/build...
Hi,
I just noticed that some of my TravisCI tests started failing. They depend on python-memcached==1.5.3
, and the failure looks like this:
ERROR: test_cache_is_working (bulk_sms.tests.test_sending.MultiprocSendingTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/<projectname>/bulk_sms/tests/test_sending.py", line 330, in test_cache_is_working
self.assertTrue(cache.get('CACHETEST', default=False),
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/django/core/cache/backends/memcached.py", line 82, in get
val = self._cache.get(key)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1089, in get
return self._get('get', key)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1073, in _get
return _unsafe_get()
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1061, in _unsafe_get
value = self._recv_value(server, flags, rlen)
File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/memcache.py", line 1235, in _recv_value
val = int(buf)
ValueError: invalid literal for int() with base 10: 'True'
When I look at this page, the source package has different code than the wheel package:
Specifically, line 93 of memcache.py inside the wheel package says this:
__version__ = "1.55"
Would it be possible to upload a new wheel package with 1.53 code in it? Thanks for your work modernizing this package!
hi,
Im working on linux 12.04 64 bits
and i getting the following error:
File "build/bdist.linux-x86_64/egg/memcache.py", line 704, in set
return self._set("set", key, val, time, min_compress_len, noreply)
File "build/bdist.linux-x86_64/egg/memcache.py", line 1024, in _set
return _unsafe_set()
File "build/bdist.linux-x86_64/egg/memcache.py", line 998, in _unsafe_set
store_info = self._val_to_store_info(val, min_compress_len)
File "build/bdist.linux-x86_64/egg/memcache.py", line 962, in _val_to_store_info
pickler.dump(val)
File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "/usr/lib/python2.7/pickle.py", line 615, in _batch_appends
save(x)
File "/usr/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled
when i run tests i get this :
py27 develop-inst-nodeps: /var/www/boorar.git
py27 installed: The directory '/home/ricardo/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,coverage==4.0.3,flake8==2.2.4,hacking==0.10.2,mccabe==0.2.1,nose==1.3.7,pbr==1.8.1,pep8==1.5.7,pyflakes==0.8.1,-e git+https://github.com/linsomniac/python-memcached.git@37f55ca4ad94ca4ade30d6be28e1facb79ac3182#egg=python_memcached-master,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='4142489713'
py27 runtests: commands[0] | nosetests
...MemCached: while expecting 'DELETED', got unexpected response 'NOT_FOUND'
MemCached: while expecting 'DELETED', got unexpected response 'NOT_FOUND'
....MemCached: while expecting 'STORED', got unexpected response 'SERVER_ERROR object too large for cache'
......MemCached: MemCache: inet:127.0.0.1:11211: test. Marking dead.
.....MemCached: MemCache: inet:memcached:11211: connection closed in readline(). Marking dead.
MemCached: MemCache: inet:memcached:11211 (dead until 1450219647): connection closed in readline(). Marking dead.
.
----------------------------------------------------------------------
Ran 19 tests in 0.261s
OK
py27 runtests: commands[1] | python -c import memcache; memcache._doctest()
Doctests: TestResults(failed=0, attempted=19)
_______________________________________________ summary ________________________________________________
py27: commands succeeded
congratulations :)
and this is my code:
import traceback
from memcache import Client
mc = Client(servers=['localhost:8000'])
try:
# key = "0_acf2c7795a8d7fc8c6b321ef9e3103960a0d699b6e3df12b428bebff"
# value = "0"
return mc.set(key, value, time, min_compress_len, noreply)
except Exception, e:
traceback.print_exc()
can you help me to figure out what is happening?
value was a result from psycopg2, i converted to json and worked:
import json
value = json.loads(json.dumps(value))
...
Right now, failed operations in python-memcached
either return 0
or None
. For example, get
returns None
if it cannot connect to the server, but set
returns 0
.
It is more pythonic to raise exceptions when an unexpected condition has occurred.
See for example how redis-py raises a ConnectionError
exception on invoking get
when the server is down.
We got a bunch of MemcachedKeyCharacterError exceptions from inside our django app. We've been unable to reproduce the problem. Debugging these would be a lot easier if the exception included the offending key. Something like the following should do the trick:
--- /home/roy/deploy/current/python/lib/python2.7/site-packages/memcache.py 2013-10-10 16:43:47.484586967 -0400
+++ /tmp/memcache.py 2013-11-02 11:41:36.316624255 -0400
@@ -1059,7 +1059,7 @@
% self.server_max_key_length)
if not valid_key_chars_re.match(key):
raise Client.MemcachedKeyCharacterError(
- "Control characters not allowed")
+ "Control characters not allowed (key = %r)" % key)
class _Host(object):
Bisected to 04f1c78
Failing test case (adapted from Django's test suite):
diff --git a/tests/test_memcache.py b/tests/test_memcache.py
index 321dc7b..03147d6 100644
--- a/tests/test_memcache.py
+++ b/tests/test_memcache.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
from __future__ import print_function
import unittest
@@ -137,6 +138,14 @@ class TestMemcache(unittest.TestCase):
value = self.mc.get(key)
self.assertEqual(value, 5)
+ def test_unicode_value(self):
+ key = 'key'
+ value = six.u('Iñtërnâtiônàlizætiøn2')
+
+ self.mc.set(key, value)
+ cached_value = self.mc.get(key)
+ self.assertEqual(value, cached_value)
+
def test_ignore_too_large_value(self):
# NOTE: "MemCached: while expecting[...]" is normal...
key = 'keyhere'
======================================================================
FAIL: test_unicode_value (__main__.TestMemcache)
----------------------------------------------------------------------
Traceback (most recent call last):
File "tests/test_memcache.py", line 147, in test_unicode_value
self.assertEqual(value, cached_value)
AssertionError: u'I\xc3\xb1t\xc3\xabrn\xc3\xa2ti\xc3\xb4n\xc3\xa0liz\xc3\xa6ti\xc3\xb8n2' != 'I\xc3\x83\xc2\xb1t\xc3\x83\xc2\xabrn\xc3\x83\xc2\xa2ti\xc3\x83\xc2\xb4n\xc3\x83\xc2\xa0liz\xc3\x83\xc2\xa6ti\xc3\x83\xc2\xb8n2'
Apparently, changing the value of server_max_value_length in the memcache module does NOT change the value inherited by the clients.
From looking at the source, I can not for the life of me figure out why!
This example was run on Ubuntu 13.04, and I've also observed it on a remote machine running Ubuntu 12.04 (on Amazon cloud).
Python 2.7.4 (default, Apr 19 2013, 18:28:01)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import memcache
>>> memcache.SERVER_MAX_VALUE_LENGTH
1048576
>>> memcache.SERVER_MAX_VALUE_LENGTH *= 10
>>> memcache.SERVER_MAX_VALUE_LENGTH
10485760
>>> c = memcache.Client(['localhost'])
>>> c.server_max_value_length
1048576
>>> memcache.SERVER_MAX_VALUE_LENGTH
10485760
here is the traceback:
Traceback (most recent call last):
File "setup.py", line 8, in
version=get_module_constant('memcache', 'version'),
File "/home/ubuntu/.myvenv/lib/python3.6/site-packages/setuptools/depends.py", line 164, in get_module_constant
return extract_constant(code, symbol, default)
File "/home/ubuntu/.myvenv/lib/python3.6/site-packages/setuptools/depends.py", line 195, in extract_constant
const = code.co_consts[arg]
IndexError: tuple index out of range
From the CHANGELOG and commit messages, I gather that a 1.54 release is imminent.
People may be interested in it because of the presumed ability to set SERVER_MAX_VALUE_LENGTH at import time.
version 1.53
Error from memcached: Failed to write, and not due to blocking: Connection reset by peer
data = zip(range(1000000,1200000), range(1, 200000))
set_multi_results = set_multi(dict(data))
get_many_results = get_multi(range(1000000, 1200000))
set_multi_results
contains all of the keys, because the server died.
get_many_results
will contain some arbitrary fraction of keys that were actually set, testing on my system was about 30,000.
Looks like a lot of work has been done since 1.53 — what's the status on a new release?
If memcached isn't running Python3 gets the following error on set(): TypeError: 'str' does not support the buffer interface. Should this be handled better?
We're hitting this in python-keystoneclient test code that checks if memcached is available. See: https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/tests/unit/test_auth_token_middleware.py#L78
Here's the difference between python 2 and 3 behavior:
ubuntu@w1:$ sudo service memcached stop$ python
ubuntu@w1:
Python 2.7.10 (default, Jun 1 2015, 16:21:46)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import memcache
c = memcache.Client(['localhost:11211'])
c.set('ping', 'pong', time=1)
0
quit()
ubuntu@w1:~$ python3
Python 3.4.3+ (default, Jun 2 2015, 14:09:35)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
import memcache
c = memcache.Client(['localhost:11211'])
c.set('ping', 'pong', time=1)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3/dist-packages/memcache.py", line 700, in set
return self._set("set", key, val, time, min_compress_len, noreply)
File "/usr/lib/python3/dist-packages/memcache.py", line 983, in _set
server, key = self._get_server(key)
File "/usr/lib/python3/dist-packages/memcache.py", line 413, in _get_server
serverhash = serverHashFunction(str(serverhash) + str(i))
File "/usr/lib/python3/dist-packages/memcache.py", line 65, in cmemcache_hash
(((binascii.crc32(key) & 0xffffffff)
TypeError: 'str' does not support the buffer interface
quit()
We found a problem in versions 1.56 and 1.57. The code below (memcache.py:399) sometimes (maybe always) store str into serverhash variable and cmemcache_hash/serverHashFunction function is failing because of that. The version 1.54 is ok.
def _get_server(self, key):
if isinstance(key, tuple):
serverhash, key = key
else:
serverhash = serverHashFunction(key)
...
Traceback (most recent call last):
File "/srv/venv/lib/python3.4/site-packages/django/core/handlers/base.py", line 132, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/srv/venv/lib/python3.4/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/srv/venv/lib/python3.4/site-packages/django/views/generic/base.py", line 71, in view
return self.dispatch(request, *args, **kwargs)
File "/srv/venv/lib/python3.4/site-packages/newrelic-2.54.0.41/newrelic/hooks/component_djangorestframework.py", line 8, in _nr_wrapper_APIView_dispatch_
return wrapped(*args, **kwargs)
File "/srv/venv/lib/python3.4/site-packages/rest_framework/views.py", line 456, in dispatch
response = self.handle_exception(exc)
File "/srv/venv/lib/python3.4/site-packages/rest_framework/views.py", line 453, in dispatch
response = handler(request, *args, **kwargs)
File "/srv/app/api/views/base.py", line 163, in post
return self.data_valid(serializer.validated_data)
File "/srv/app/api/views/get_suppliers_list.py", line 50, in data_valid
suppliers = cache.get(key)
File "/srv/venv/lib/python3.4/site-packages/django/core/cache/backends/memcached.py", line 85, in get
val = self._cache.get(key)
File "/srv/venv/lib/python3.4/site-packages/newrelic-2.54.0.41/newrelic/api/datastore_trace.py", line 43, in _nr_datastore_trace_wrapper_
return wrapped(*args, **kwargs)
File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 1092, in get
return self._get('get', key)
File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 1037, in _get
server, key = self._get_server(key)
File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 416, in _get_server
serverhash = serverHashFunction(str(serverhash) + str(i))
File "/srv/venv/lib/python3.4/site-packages/memcache.py", line 66, in cmemcache_hash
(((binascii.crc32(key) & 0xffffffff)
TypeError: 'str' does not support the buffer interface
The code below shows this behaviour:
import memcache
mc = memcache.Client(['127.0.0.1:11211'], debug=0)
keys = ["", "userA", "userB"]
prefix = "settings_"
mc.delete_multi(keys, key_prefix=prefix)
In my case I use empty string to signify the default for a collection of keys. Memcache will store it fine but python-memcache incorrectly reports I'm using a key of None. I think this is the cause:
In _map_and_prefix_keys() we check validity of the key before adding the prefix: https://github.com/linsomniac/python-memcached/blob/master/memcache.py#L725
For a variety of reasons that are ultimately unimportant, I have not been able to find any attention for maintaining this package. I've recently had an offer from Morgan Fainberg to take over maintenance of the package. That proposal includes importing it into the OpenStack CI infrastructure for testing and development. That would remove it from Github, as I understand it, and use the review.openstack.org gerrit instance, and releasing it on pypi.
If anyone has an opinion on this, please comment here.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.