jamesls / fakeredis Goto Github PK

Code example:

        redis = FakeStrictRedis()
        redis.set("key", "value", ex=1)
        redis.delete("key")
        redis.set("key", "value")
        sleep(2)
        print redis.get("key")

Should return "value", but return None.
Version 0.4.2

It would be nice if fakeredis supported LUA scripts like redis does. LUA bindings for python are available so it might not be that hard - never used them though.

Why would it be useful? Well, chances are good that the usage of lua scripts means there is some more complex logic involved that cannot be easily done with simply redis commands. Chances are good that this logic should be well-tested and thus having lua support in fakeredis would be helpful.

I think I've discovered a discrepancy between redis's watch statement implementation and fakeredis's mocking of that implementation.
Code that runs fine on actual redis throws a watch error every time when using fakeredis.

I reduced the code down into the minimal example, below.

key = 'testkey'
fake_redis = fakeredis.FakeStrictRedis()
redis = RedisProxy() # however you connect to your redis
with fake_redis.pipeline(transaction=True) as pl:
    pl.watch(key)
    pl.hset(key, 'blah1', "blah2")

    pl.execute()

if you use redis.pipeline, this code works fine, if you use fake_redis.pipeline, you get the following

In [8]:     pl.execute()
---------------------------------------------------------------------------
WatchError                                Traceback (most recent call last)
<ipython-input-8-622a4d39c733> in <module>()
----> 1 pl.execute()

/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/fakeredis-0.5.1-py2.7.egg/fakeredis.pyc in execute(self)
   1247                     'Watched key%s %s changed' % (
   1248                         '' if len(mismatches) == 1 else
-> 1249                         's', ', '.join(k for (k, _, _) in mismatches)))
   1250         ret = [getattr(self.owner, name)(*args, **kwargs)
   1251                for name, args, kwargs in self.commands]

WatchError: Watched key testkey changed

A watch shouldn't throw an error if it's modified by the same connection that set the watch - it should only error if it's modified by another connection, to prevent race conditions.

Problem:

The code assumes that min and max range query parameters will be numerically comparable values.

e.g. https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L888

redis-py passes on the pieces as arguments to redis pools, as can be shown at: https://github.com/andymccurdy/redis-py/blob/26c56b9d816c9d1cc1393c04b04b4f1d688f7353/redis/client.py#L1696

This allows for the redis query parameters -inf and +inf to be passed transparently, for queries that expect a range.

The implementation should fill-in the ranges in case those strings are passed.

More on the parameters: http://redis.io/commands/zrangebyscore

In redis-py, StrictRedis accepts a decode_responses argument, set to False by default, but when set to True, it will decode all bytes values using its charset argument, default to utf-8.

When my team uses Redis, we set this argument to True. So we expect strings (we use python 3), not bytes.
But fakeredis always returns bytes. It encodes everything in bytes but never decodes.

So in our tests we have inconsistent behavior if we swap between a real redis server and fakeredis.

Viewing the code of fakeredis it's not so easy to support it because not everything goes through a single method, as does redis-py with read_response.

Do you think it's possible to have this feature? Would you accept a pull-request on this subject?

In the contrived example below, there is a type difference for the returned value in
some circumstances. hset/hget with fake returns a long whereas real returns a string.

mimicking the return type of real redis is probably desired here.

Basically, whatever is shipped off into real redis returns as a string. However, only some content that goes into fake redis returns as a string. Some return as the native object type without pickling.

--code--
import redis
import fakeredis

real = redis.StrictRedis()
fake = fakeredis.FakeStrictRedis()

fake.set("blah", 0L)
real.set("blah", 0L)
print "fake:", type(fake.get("blah"))
print "real:", type(real.get("blah"))

fake.hset("blah1", "key", 0L)
real.hset("blah1", "key", 0L)
print "fake:", type(fake.hget("blah1", "key"))
print "real:", type(real.hget("blah1", "key"))

--output--
fake: <type 'str'>
real: <type 'str'>
fake: <type 'long'>
real: <type 'str'>

Hi, it may just be my machine but the current release on PyPi doesn't install correctly:

$ pip install fakeredis
Downloading/unpacking fakeredis
  Running setup.py egg_info for package fakeredis
    Traceback (most recent call last):
      File "<string>", line 14, in <module>
      File "/Users/foo/dev/something/venv/build/fakeredis/setup.py", line 11, in <module>
        'README.rst')).read(),
    IOError: [Errno 2] No such file or directory: '/Users/foo/something/venv/build/fakeredis/README.rst'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 14, in <module>

  File "/Users/foo/dev/something/venv/build/fakeredis/setup.py", line 11, in <module>

    'README.rst')).read(),

However the current master does:

$ pip install git+git://github.com/jamesls/fakeredis.git
Downloading/unpacking git+git://github.com/jamesls/fakeredis.git
  Cloning git://github.com/jamesls/fakeredis.git to /var/folders/m_/xrysd28j79s9mpdc180tsnzc0000gn/T/pip-4q3Ooz-build
  Running setup.py egg_info for package from git+git://github.com/jamesls/fakeredis.git

Requirement already satisfied (use --upgrade to upgrade): redis in ./venv/lib/python2.7/site-packages (from fakeredis==0.1.1)
Installing collected packages: fakeredis
  Running setup.py install for fakeredis

Successfully installed fakeredis
Cleaning up...

anychance you could update pypi?

Maybe flushall() should be mentioned in the readme, as it's quite essential for using FakeRedis in unit tests.

Example:

    def setUp(self):
        # setup fake redis for testing
        self.r = fakeredis.FakeStrictRedis()

    def tearDown(self):
        # clear data in fake redis
        self.r.flushall()

I'd like to add support for redis lexicographical ordering functions, which are currently missing.

redis-py supports python3.2 and python3.3, so fakeredis should support those versions as well.

I noticed that hmset seems to mutate the hash that is passed in for storage. It looks like everything is being cast to a string on the input dict itself, instead of on a copy.

In [1]: conn = fakeredis.FakeStrictRedis()

In [2]: to_store = {'key': [123, 456]}

In [3]: conn.hmset('fake-key', to_store)
Out[3]: True

In [4]: to_store
Out[4]: {'key': '[123, 456]'}

This differs from the behavior of actual redis, which does not mutate the input dictionary

In [5]: conn = get_redis_connection()  # wired up to real redis

In [6]: to_store = {'key': [123, 456]}

In [7]: conn.hmset('fake-key', to_store)
Out[7]: True

In [8]: to_store
Out[8]: {'key': [123, 456]}

As you might imagine, this resulted in some very confusing test results.

I think the problem here is that the ranges are treated categorically as numbers instead of preprocessing (see: https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L888) which causes a string / int comparison in the case of using exclusive bound notation.

Here is redis-py's transaction()

As you can see, it takes keyword arguments: def transaction(self, func, *watches, **kwargs)
fakeredis' transaction() does not expect keyword arguments: def transaction(self, func, *keys)

Naturally this results in a TypeError when trying to use keyword arguments.

The three current kwargs are shard_hint=None, value_from_callable=False, and watch_delay=None. The fakeredis implementation, however, does not support any of these.

According to the redis documentation:

If key already exists and is a string, this command appends the value at the end of the string. If key does not exist it is created and set as an empty string, so APPEND will be similar to SET in this special case.

If a non-existent key is appended to fakeredis it returns a KeyError.

StrictRedis' ttl and pttl methods return -1 when expire is not set and -2 when key does not exist (documentation of python redis, documentation of redis), but FakeStrictRedis' methods return None instead.

In [1]: r = redis.StrictRedis(db=0)

In [2]: r.ttl("foo")
Out[2]: -2

In [3]: f = fakeredis.FakeStrictRedis()

In [4]: f.ttl("foo")
Out[4]: None

Apparently it's just adding the first key/value of **kwargs to the ZSet...

Any reason?

Also not returning the number of items added...

While trying to mock a list, I found that trying to return the type returns None.

import fakeredis
redis=fakeredis.FakeStrictRedis()
redis.lpush('bar', 2)
2
print redis.type('bar')
None

There is an issue with how keys are stored in _StrKeyDict#_ex_key and _StrKeyDict#_dict. Consider the following case

import fakeredis
from time import sleep
r = fakeredis.FakeStrictRedis()


class Key(object):
    def __init__(self, k):
        self.k = k

    def __str__(self):
        return str(self.k)


r.set(Key("a"), "value", 1)
r.set(Key("a"), "value", 10)

sleep(2)

print r.get(Key("a")) # None instead of value

Redis returns the results of mget as a list of strings. In fakeredis, if you use incr and then mget, you get a list of integers, not strings.

Here's the bug in action (note the last item is a list containing an integer):

>>> import fakeredis
>>> cache = fakeredis.FakeStrictRedis()
>>> cache.incr('a', 1)
1
>>> cache.get('a')
'1'
>>> cache.incr('a', 1)
2
>>> cache.get('a')
'2'
>>> cache.mget(['a'])
[2]

Here's how it works against my redis instance:

>>> from django.core.cache import get_cache
>>> 
>>> redis_cache = get_cache('persistent').raw_client
>>> redis_cache.incr('a',1)
1
>>> redis_cache.get('a')
'1'
>>> redis_cache.incr('a',1)
2
>>> redis_cache.get('a')
'2'
>>> redis_cache.mget(['a'])
['2']

It looks like incr always transforms the item into an integer, but get works around it by calling to_bytes on the value before returning it. Perhaps the solution is to do the same on each item in mget.

When commands allow a variable number of arguments, would it be possible to implement something similar to what redis-py uses, where it allows you to pass multiple keys to a command either via a list, or as an *args list.

See: https://github.com/andymccurdy/redis-py/blob/master/redis/client.py#L23

Examples:

hmget('key', 'hashkey1', 'hashkey1')
hmget('key', ['hashkey1', 'hashkey1'])

Comparison:

https://github.com/andymccurdy/redis-py/blob/master/redis/client.py#L1409
https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L620

I use the variable length lists at the moment, but I'd have to convert them in order to seamlessly use fakeredis. I'm happy to do so, but I thought I'd put in a feature request anyway!

Thanks for the great package!

ZRANGE and ZREVRANGE both break score ties by ordering alphabetically on keys, but ZREVRANGE should do so in reverse order

Real Redis: ResponseError: wrong number of arguments for 'mget' command
Fake redis: no issues

It looks like RQ doesn't work with fakeredis. This is probably a concern of RQ, but thought I'd ask here to see what you think.

The problem is that RQ does something different when the connection is Redis vs StrictRedis, and uses isinstance to differentiate.

FakeRedis and FakeStrictRedis inherit from object directly, so this obviously won't work. Since all the methods are there, would it be possible to inherit from Redis and StrictRedis instead? All the behavior would still be masked, but now these classes can be dropped in wherever the superclasses are explicitly expected.

Sure, using isinstance isn't very Pythonic, but doing this would allow the fake classes to fit in more real-world cases. Mock objects do this too: isinstance(Mock(int), int) # => True.

If you set a key to a value in a hash (thereby provisioning it in Fakeredis' internal storage), but then access it as if it were a simple key, fakeredis responds by stringifying an internal object and passing it back, where it should raise a TypeError.

Steps to reproduce:

1. fakeredis.hset('hash', 'key', 'value')
2. fakeredis.get('hash') produces '<fakeredis._StrKeyDict object at 0x10ad47d90>'

How Redis handles this:

1. HSET blab bleebs 5
2. HGET blab bleebs produces 5
3. GET blab produces (error) WRONGTYPE Operation against a key holding the wrong kind of value

https://pypi.python.org/pypi/lupa is a bridge from python to lua.
It can be used to provide the same environment as redis provides so that lua scripts will be able to be tested without using a real redis instance.
What do you think?

Currently fakeredis only looks for libc when searching for an implementation of strtod. On Windows, libc is not available and this function is found in msvcrt.lib.

The fix is to check for both libraries and use whichever is present.

In [21]: f=fakeredis.FakeRedis()

In [22]: f.keys()
Out[22]: []

In [23]: f.keys('qwer*')

TypeError Traceback (most recent call last)
in ()
----> 1 f.keys('qwer*')

TypeError: keys() takes exactly 1 argument (2 given)

The latest version of redis-py has a "score_cast_func" parameter on the sorted set functions that accept a withscores parameter.

As fakeredis doesn't have a matching parameter, attempts to use with such calls fail

fakeredis.FakeStrictRedis doesn't implement redis-py's .lock() method. Are you open to a pull request that uses the standard lib's threading.Lock to fake the same behavior?

(Of course, I'll follow the contribution guidelines :)

In some cases, people may use redis.Redis.execute_command directly

The test_sort_alpha fails with the latest redis 2.6 RC:

ResponseError: One or more scores can't be converted into double

When using StrictRedis as a base class, and when using a pipeline to commit commands, in the subclassed pipeline uses the original StrictRedis implementation of the command, not the subclassed command.

Here is an implementation of a pop operation using both a subclassed StrictRedis and FakeStrictRedis.

class PrintRedis(redis.StrictRedis):
    def get(self, key):
        printme = super(PrintRedis, self).get(key)
        print 'PrintRedis: {}'.format(printme)
        return printme

    def pop(self, key):
        pipe = self.pipeline()
        printme, _ = pipe.get(key).delete(key).execute()
        print 'PrintRedis: {}'.format(printme)
        return printme

class FakePrintRedis(fakeredis.FakeStrictRedis):
    def get(self, key):
        printme = super(FakePrintRedis, self).get(key)
        print 'FakePrintRedis: {}'.format(printme)
        return printme

    def pop(self, key):
        pipe = self.pipeline()
        printme, _ = pipe.get(key).delete(key).execute()
        print 'FakePrintRedis: {}'.format(printme)
        return printme

Ideally, these both would operate the same, since fakeredis is a fake. However, that's not the case--

Print Redis Test

>>> p = PrintRedis()
>>> p.set('printredis', 'testdata')
True
>>> p.get('printredis')
PrintRedis get: testdata
'testdata'
>>> p.pop('printredis')
PrintRedis pop: testdata
'testdata'
>>> p.get('printredis')
PrintRedis get: None

Fake Print Redis Test

>>> f = FakePrintRedis()
>>> f.set('fakeprintredis', 'testdata')
True
>>> f.get('fakeprintredis')
FakePrintRedis get: testdata
'testdata'
>>> f.pop('fakeprintredis')
FakePrintRedis get: testdata
FakePrintRedis pop: testdata
'testdata'
>>> f.get('fakeprintredis')
FakePrintRedis get: None

As you can see, the fake print redis prints to stdout twice-- the .get call in the pipeline is being called from the subclassed FakePrintRedis version of the function.

Incidentally, my use case for this is in testing--

class TestRedisPrint(unittest.TestCase):
    def test_redis_pop(self):
        patcher = patch.object(PrintRedis, '__bases__',
                               (fakeredis.FakeStrictRedis,))
        with patcher:
            patcher.is_local = True
            redis_set('pop_testkey', 'pop_testvalue')
            with self.assertPrinted("PrintRedis pop: pop_testvalue"):
                testvalue = redis_pop('pop_testkey')

Which is a failing test.

On the other hand, I kind of prefer the pipeline suing the subclassed function... but since this is the fake, we should make it match the real one.

pyredis allows for passing in a string of arguments specifying an unlimited number of key/value pairs. I can also pass in **kwargs as a mapping. Fakeredis forces me to pass a dict so the two APIs differ causing testing to guide my implementation.

Relevant code:

https://github.com/jamesls/fakeredis/blob/master/fakeredis.py#L356

Relevant documentation:

https://github.com/jamesls/fakeredis#unimplemented-commands

There's a discrepancy here. Would it be better to implement the randomkey method, or just add it to the list of unimplemented commands?

redis-py always returns hash values as bytes, even when they are valid floating point numbers. fakeredis is consistent with that when you set the value with HSET:

In [1]: import redis, fakeredis

In [2]: real = redis.StrictRedis()
In [3]: real.hset('h', 'x', 1.5)
Out[3]: 1
In [4]: real.hgetall('h')
Out[4]: {b'x': b'1.5'}

In [5]: fake = fakeredis.FakeStrictRedis()
In [6]: fake.hset('h', 'x', 1.5)
Out[6]: 1
In [7]: fake.hgetall('h')
Out[7]: {b'x': b'1.5'}

However, it breaks when you use HINCRBYFLOAT:

# Standard behavior
In [8]: real.hincrbyfloat('h2', 'x', 1.5)
Out[8]: 1.5
In [9]: real.hgetall('h2')
Out[9]: {b'x': b'1.5'}

# Incompatible behavior
In [10]: fake.hincrbyfloat('h2', 'x', 1.5)
Out[10]: 1.5
In [11]: fake.hgetall('h2')
Out[11]: {b'x': 1.5}
In [12]: fake.hget('h2', 'x')
Out[12]: 1.5

HINCRBY has the same problem, it changes the type to int:

In [13]: real.hincrby('h3', 'x', 15)
Out[13]: 15
In [14]: real.hgetall('h3')
Out[14]: {b'x': b'15'}

In [15]: fake.hincrby('h3', 'x', 15)
Out[15]: 15
In [16]: fake.hgetall('h3')
Out[16]: {b'x': 15}
In [17]: fake.hget('h3', 'x')
Out[17]: 15

I think it would be useful to mock the errors thrown when you are unable to connect to the redis server. It appears that redis-py throws a ConnectionError with these args ('Error 61 connecting to localhost:6379. Connection refused.',) when it cannot reach the server.

Hi,

Great library, thanks.

I noticed that smembers returns None instead of an empty list sometimes which is different to what redis does. I haven't looked at other similar functions.

Cheers,

Hadley

Hi,

It seems that some arguments (time and value) order are inverted for the setex function.
I got an error when using fakeredis version 0.7.0

Using redis-py:

import redis
# ...
r.setex('mykey', 'myvalue', 10)

Using fakeredis version 0.7.0

import fakeredis
r = fakeredis.FakeStrictRedis()
r.setex('mykey', 10, 'myvalue')

Can actually be overcome by naming the args:

import redis
r.setex(name='mykey', value='myvalue', time=10)

But it would be cool if it can have the same order for the args.
What do you think?

Regards.

When using a tuple as a key, redis will use the string form of the tuple. Fakeredis returns the key as the original tuple.

Example:
import redis/fakeredis
import time
client = redis.StrictRedis()
client.zadd('test_set', time.time(), (1, 2, 3))
result = client.zrange('test_set', 0, 0)

result returns tuple in array [(1, 2, 3)] in fakeredis, but string in array ['(1, 2, 3)'] in Redis

Here's what happens when I try to use 'get' on a key that I've used to store a list:

>>> from redis_cache import get_redis_connection
>>> client = get_redis_connection('persistent')
>>> client.rpush('foo', 1)
1L
>>> client.get('foo')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 551, in get
    return self.execute_command('GET', name)
  File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 361, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/client.py", line 371, in parse_response
    response = connection.read_response()
  File "/home/noah/envs/adioso/lib/python2.6/site-packages/redis/connection.py", line 311, in read_response
    raise response
ResponseError: WRONGTYPE Operation against a key holding the wrong kind of value

Using fakeredis, I don't get any error:

>>> import fakeredis
>>> client = fakeredis.FakeStrictRedis()
>>> client.rpush('foo', 1)
1
>>> client.get('foo')
"['1']"

I think the fix to this might involve tracking the type of item at each key, which could get involved. Is this of interest to fakeredis?

Example test that fails:

def test_sort_with_set(self):
    self.redis.sadd('foo', '3')
    self.redis.sadd('foo', '1')
    self.redis.sadd('foo', '2')
    self.assertEqual(self.redis.sort('foo'), ['1', '2', '3'])

SADD, HDEL, SREM, ZREM, ZADD, LPUSH, and RPUSH

This appears simple to add. If I created a pull request, would you be interested in merging it in?