Using set for a large number gets slower as we move from 100,000 to 1 million keys about rediscala HOT 21 OPEN

Yes i noticed it during my tests, at some point the scale is exponential (bad).
I suspected the thread scheduler to be the limitation.
Or the way Future.sequence works.

If you can isolate a test that scale linearly up to 1M of futures, I would be interested to see it.

By replacing akka-io with another java.nio library (xnio) I was able to pass the 1M req (at the speed of around 500k req/s)

from rediscala.

@etaty thanks for your response. I'm trying to isolate this now. Can you tell me how you replaced akka-io with xnio ? I've also asked help from the Akka folks. I'll keep you posted.

from rediscala.

Well it was an experiment, no something ready for prod and integrated with rediscala, it was also a year ago.

from rediscala.

@etaty - the Akka team thinks its the lack of backpressure in the Akka client. That is the client is being flooded with so many requests that it cannot handle them. How many RedisClient you can really create. I was thinking about creating 10 client and giving 100K operations to each of these. What do you think about this setup ?

from rediscala.

Yes it is much more fair
Do you have a link to the conversation with akka team ? (is it public ?)

from rediscala.

Sure.
https://groups.google.com/forum/#!topic/akka-user/NrSkEwMrS3s

I think this use case even though it not that common needs to be addressed in one way or the other. Hopefully we can find a good resolution to this issue.

Thanks for responding quickly.

from rediscala.

@etaty - I've put an update to this on the Akka thread in case you want to have a look at it.
https://groups.google.com/forum/#!topic/akka-user/NrSkEwMrS3s

from rediscala.

You Could use a "DummyRedisClient"
I mean just send Promise to an actor, and the actor complete the Promise.
You could use that base to compare when you add an akka-stream in front of the actor and without it.
Also look at the GC

from rediscala.

Also another piece to look at, is akka-io the buffers and osx buffers.
Because maybe once they are all full they might start to be slow.

from rediscala.

@etaty thanks for reverting back.

How is sending to the DummyRedisClient different from sending it to the RedisClient ?
The GC pressure is high (at least for young generation (Edge space) ). I gave the process a max of 12G of heap space but it didn't make a big difference.

As I've mentioned in the Akka thread, when i bump the number of RedisClients to 100 and give each of these actors 10K values to put in Redis, it is significantly and noticeable faster. Its is not clear to me exactly why.

Looks like akka-io (TCP) actor doesn't have an internal buffer.
Please see http://doc.akka.io/docs/akka/snapshot/scala/io-tcp.html
"The basic model of the TCP connection actor is that it has no internal buffering (i.e. it can only process one write at a time, meaning it can buffer one write until it has been passed on to the O/S kernel in full). Congestion needs to be handled at the user level, for both writes and reads."

I'm going to look at OSX buffers now.

If you look at the network IO graph. The amount of writes to the TCP socket goes up till the first read on the socket. As soon as the first read is done the write speeds drops drastically.

from rediscala.

I did post an update on the Akka thread. Looks like I was not applying the back pressure correctly.
Now I can get much better performance compared to my initial version. I need to tweak it a little bit more to get better performance.I'll measure it and report it once I've a better version.

Here is how the read/writes look like after applying back pressure.

from rediscala.

Hey thanks for your work.
Could you publish the code in gist (or a repo if you prefer), so other can help you ?
In the first graph, the write of 24MB/s is "normal" because rediscala is designed to batch write between 2 socket writes. But after it is strange. Maybe if you can put in parallel CPU, MEM and GC.
Are you using visualvm ?

from rediscala.

@etaty - thank you for responding. I like Rediscala and would like to figure out a way around this limitation.

I've create a gist here.
https://gist.github.com/soumyasd/ac5b1d5f2ec3af21ace8

I've also updated some screenshots and analysis on the Akka thread in case you want to take a look at that.

I use YourKit for the profiling my JVM.

from rediscala.

@etaty - is the TCP NO_DELAY setting for Rediscala configurable ? I think it's waiting for each Future to complete before sending the next one.

from rediscala.

@etaty - is there a way to batch multiple Redis Operations using the RedisClient ?

from rediscala.

Here is another chart for 1 million messages of 10K each. So support for batch mode will greatly increase the throughput and also reduce the latency. Let me know what you think.

from rediscala.

You can do a batch with a Transaction message

from rediscala.

@etaty Thanks. I tried with transactions (batching) and it looks like I can get a better throughput of course at the cost of latency.

Here is the code
https://gist.github.com/soumyasd/ac5b1d5f2ec3af21ace8#file-redisstreamclient-scala-L52-L60

I think a flag in Rediscala to control the TCP NO_DELAY will be useful.

If you can think of any other optimization please let me know.

Also, is there support for Redis protocol over UDP ?

from rediscala.

I am using the Nagle algo in rediscala (so i am waiting for the ack of a socket write before sending the next write). It limits the number of messages flowing into the akka-io-worker (and so the number of system call). That's why rediscala is much faster.
http://en.wikipedia.org/wiki/Nagle's_algorithm
It might also be the cause of the "halting" with 1M, because the worker receive 1message with a huge number of operations.

No redis works only with tcp

from rediscala.

Okay. So is there is no more optimization you can think of ?

What message sizes did you use in the following -
"By replacing akka-io with another java.nio library (xnio) I was able to pass the 1M req (at the speed of around 500k req/s)"

from rediscala.

just get or ping (small message)

from rediscala.