redisgears / redisgears-py Goto Github PK

View Code? Open in Web Editor NEW

27.0 7.0 11.0 74 KB

RedisGears python client

Home Page: https://redisgears.io/

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

redis redisgears python-client map-reduce stream-processing

redisgears-py's Introduction

redisgears-py

RedisGears python client (support python3 only!)

Example: Using the Python Client:

from gearsclient import GearsRemoteBuilder as GearsBuilder
from gearsclient import execute
import redis

conn = redis.Redis(host='localhost', port=6379)

# count for each genre how many times it appears

res = GearsBuilder('KeysOnlyReader', r=conn).\
	  map(lambda x:execute('hget', x, 'genres')).\
	  filter(lambda x:x != '\\N').\
	  flatmap(lambda x: x.split(',')).\
	  map(lambda x: x.strip()).\
	  countby().\
	  run()


for r in res[0]:
	print('%-15s: %d' % (r['key'], r['value']))

Installing

pip install git+https://github.com/RedisGears/redisgears-py.git

Notice that the library also need to be installed in RedisGears virtual env.

Developing

Create a virtualenv to manage your python dependencies, and ensure it's active. virtualenv -v venv
Install pypoetry to manage your dependencies. pip install poetry
Install dependencies. poetry install

tox runs all tests as its default target. Running tox by itself will run unit tests. Ensure you have a running redis, with the module loaded.

redisgears-py's People

Contributors

Stargazers

Watchers

Forkers

andresrinivasan manueljgarciar banker gavindmello satheeshcdo filipecosta90 rrguez pierremasson webclinic017 chayim edgepathai

redisgears-py's Issues

Known Vulnerabilities link references redisjson-py

In the readme, the "Known Vulnerabilities" link currently references https://snyk.io/test/github/RedisJSON/redisjson-py. Actually, not sure if https://snyk.io/test/github/RedisJSON/redisjson-py or https://snyk.io/test/github/RedisGears/redisgears-py exists because both pages display "Unable to test GitHub repository" message.

redisgears-py/README.md

Line 7 in 8fb6605

 [![Known Vulnerabilities](https://snyk.io/test/github/RedisJSON/redisjson-py/badge.svg?targetFile=pyproject.toml)](https://snyk.io/test/github/RedisJSON/redisjson-py?targetFile=pyproject.toml) 

Missing **kargs into run()

Describe the bug
StreamReader don't seem to take into account 'fromId' parameter.

To reproduce
Redis

xadd mystream * field value0
"1649425213053-0"
xadd mystream * field value1
"1649425219031-0"
xadd mystream * field value2
"1649425223070-0"

from gearsclient import GearsRemoteBuilder as GearsBuilder
import redis

conn = redis.Redis(host="localhost", port=6379)

GearsBuilder('StreamReader', r=conn).run('mystream', fromId='1649425219031')

([{'key': 'mystream', 'id': '1649425213053-0', 'value': {'field': 'value0'}}, {'key': 'mystream', 'id': '1649425219031-0', 'value': {'field': 'value1'}}, {'key': 'mystream', 'id': '1649425223070-0', 'value': {'field': 'value2'}}], [])

Excepted Behaviour

([{'key': 'mystream', 'id': '1649425223070-0', 'value': {'field': 'value2'}}], [])

Possible Fix

Add **kargs into self.pipe.run(arg, False, collect, **kargs) (line 290)

def run(self, arg=None, collect=True, **kargs):
    self.map(lambda x: cloudpickle.dumps(x))
    self.pipe.run(arg, False, collect, **kargs)
    selfBytes = cloudpickle.dumps(self.pipe)
    serverCode = '''
    import cloudpickle
    p = cloudpickle.loads(%s)
    p.createAndRun(GB)
    ''' % selfBytes
    results = self.r.execute_command('RG.PYEXECUTE', serverCode, *self.requirements)
    res, errs = results
    res = [cloudpickle.loads(record) for record in res]
    return res, errs

Using hget in map function, return redundant values

code

from gearsclient import GearsRemoteBuilder as GB
from gearsclient import execute
import redis

conn = redis.Redis(host='127.0.0.1', port=6464, db=0)
conn.hset('student', 'name', 'osborn')

res = GB(reader='KeysOnlyReader', r=conn).\
    map(lambda x: execute('hget', 'student', 'name')).\
    run()

print(res)

output

i expect to get one value which is 'osborn', but this code returns 5 'osborn', i don't know why this happen and how to fix it.

setup.py and GH releases are missmatched

checking setup.py we see 0.1.
checking GH releases we see 1.0.

Is Redis Gears execution atomic?

If i have a python gears file for writing 1 million records into Redis.

Does the main thread of Redis block until the python file completes execution?

redis gears transaction command

I see the official document says that redis gears support transaction, so i give a try. Here's my code:

def watch1(x):
    execute('watch', 'k1')
    execute('multi')
    v1 = execute('get', 'k1')
    if v1 is None:
        v1 = 0
    else:
        v1 = int(v1)
    execute('set', 'k1', v1 + 1)
    execute('exec')

GB(reader='ShardsIDReader', r=conn_gears). \
    map(watch1). \
    run()

this code reports mistake about execute('exec') this line, so i wonder what's the right command for transaction.

redisgears-py should support > python 3.7

ERROR: Package 'gearsclient' requires a different Python: 3.8.10 not in '<3.8,>=3.6'

Report [redis.exceptions.ConnectionError: Connection closed by server.] when use redisgears-py

My code

from gearsclient import GearsRemoteBuilder as GB
from gearsclient import execute
import redis

conn_gears = redis.Redis(host='10.73.3.56', port=6379)

def hset_test(x):
    return execute('hgetall', 'gamblewin')

GB(reader='ShardsIDReader', r=conn_gears). \
    map(hset_test). \
    run()

Mistake

Traceback (most recent call last):
  File "C:/Users/ganosboo/RedisProject/umd-redis-python/test/gearstest.py", line 13, in <module>
    map(hset_test). \
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\gearsclient\redisgears_builder.py", line 297, in run
    results = self.r.execute_command('RG.PYEXECUTE', serverCode, *self.requirements)
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\client.py", line 915, in parse_response
    response = connection.read_response()
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\connection.py", line 256, in readline
    self._read_from_socket()
  File "C:\ProgramData\Continuum\Anaconda\lib\site-packages\redis\connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server

If use python redis client, there will no mistakes, such as conn_gears.hgetall('gamblewin').

No matching distribution found for redisgears

redisgears-py/gearsclient/redisgears_builder.py

Line 314 in 5e0efb2

from redisgears import log as redisLog

redisgears-py/gearsclient/redisgears_builder.py

Line 318 in 5e0efb2

from redisgears import config_get as redisConfigGet

redisgears-py/gearsclient/redisgears_builder.py

Line 323 in 5e0efb2

from redisgears import executeCommand as redisExecute

redisgears-py/gearsclient/redisgears_builder.py

Line 327 in 5e0efb2

from redisgears import getMyHashTag as redisHashtag

redisgears-py/gearsclient/redisgears_builder.py

Line 332 in 5e0efb2

from redisgears import atomicCtx as redisAtomic

Override reply from redis-gears

Hi, I am trying to use RedisGears-Py for implementation of read through cache.

import redis
from gearsclient import GearsRemoteBuilder as GearsBuilder
from gearsclient import execute

def fetch_data(r):
    key = r['key']
    value = 1000
    execute('SET', key, value)  
    # return execute('GET', key)  
    return value

# res = GearsBuilder().foreach(fetch_data).register(tigger='GET', mode="sync", onRegistered=fetch_data)

# Register the RedisGears function
def register_read_through_cache():
    gb = GearsBuilder('CommandReader')
    gb.filter(lambda x: x['command'].lower() == 'get')
    gb.foreach(fetch_data)
    gb.register(trigger="get")

# Register the function when the script runs
register_read_through_cache()
# print(res)

But, on the first hit its always None
I want to get the result in the first get itself, how do i override this.

RG.PYEXECUTE not working

this exception is rise through this code

from gearsclient import GearsRemoteBuilder as GB
GB().run()

Some questions about gears in cluster envrionment

Since for now, i don't have Redisgears cluster environment, can not conduct experiments, so may need to ask these questions.

Dose gears support put key into a specific slot? such as exec('set', '{slot1}key1', '1')?
When execute gears function, it seems gears, not like lua, doesn't require all keys to be operated in the same slot. But what if i only want to execute functions on keys in one slot, but gears will execute on all shards, isn't that a waste of computing resource?

If i make gears function atomic, and it runs on all shards in cluster, so does that mean whole redis cluster will be blocked?

`code() takes at most 15 arguments` when running any command.

Getting an error no matter what I run or register through gearsclient.

Even noop code like:

gb = GearsBuilder('KeysOnlyReader', r=conn)
gb.run()

returns

Traceback (most recent call last):
  File "<input>", line 1, in <module>
    gb3.run()
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/gears-5k_d3sYO-py3.8/lib/python3.8/site-packages/gearsclient/redisgears_builder.py", line 314, in run
    ''' % selfBytes
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/gears-5k_d3sYO-py3.8/lib/python3.8/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/gears-5k_d3sYO-py3.8/lib/python3.8/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/Users/<user>/Library/Caches/pypoetry/virtualenvs/gears-5k_d3sYO-py3.8/lib/python3.8/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: ['Traceback (most recent call last):\n', '  File "<string>", line 3, in <module>\n', 'TypeError: code() takes at most 15 arguments (16 given)\n']

conn.execute_command("RG.PYEXECUTE", "GB().run()"). works fine.

Running on python 3.8 on Mac OS 10.15.5 connecting to current RedisGears docker image. I did manually change .dumps() in the library to use pickle protocol 4 since python 3.8 defaults to protocol 5 which doesn't exist in the RedisGear docker image using python 3.7.
eg: selfBytes = cloudpickle.dumps(self.pipe, protocol=4)

redisgears / redisgears-py Goto Github PK

redisgears-py's Introduction

redisgears-py

Example: Using the Python Client:

Installing

Developing

redisgears-py's People

Contributors

Stargazers

Watchers

Forkers

redisgears-py's Issues

Recommend Projects

Recommend Topics

Recommend Org