Streaming of multi-bulk replies about asyncio-redis HOT 4 CLOSED

Weird. Thanks for telling me. This is surely an issue.

from asyncio-redis.

Just to be clear, I have not confirmed this behavior. I was trying to wrap my head around asyncio in general and this library in particular, wondering how this code would run, and this is my assumption after looking through the code.

Presumably the Redis protocol doesn't support retrieving values from a "get" query without consuming the full response from the previous query, so I thought maybe the library would use a second connection while streaming a multi-bulk response, but I concluded it wasn't, so I just wanted to bring it up as a topic.

I'm not sure what the right behavior is.

from asyncio-redis.

(ignore this comment, it was not really a reply to your question.)

Apparently, there's an issue that if the multi bulk reply will contain 500,000 items, we already create that many futures. Better would be to use a queue for that. We should be able to support at least streaming per IP packet. We don't have to wait before all the IP packets arrive, before we return a SetReply.

Further, in the case of sets, redis doesn't seem to support streaming that well. Try to run smembers from redis-cli for such a huge set. It doesn't stream like keys does. I'm not yet sure about this.

from asyncio-redis.

Oh, your are right, these calls will be pipelined. This means that the get() will be blocking untill the whole smembers answer has been received. It's best to use a connection pool for this.

Connection pooling docs:
http://asyncio-redis.readthedocs.org/en/latest/#connection-pooling

Some example of how the streaming works:

https://github.com/jonathanslenders/asyncio-redis/blob/64dda366793cc5652b40a2af7868863f8c81f9e5/examples/streaming-multi-bulk/test.py

from asyncio-redis.