Massive slowdown between 0.9.4 and 1.2.x on Solaris about restkit HOT 22 CLOSED

Though I don't reproduce it.

Main change compared to the 0.9.4 is that you have to specifically define the pool. What's your code ? Which python version ?

from restkit.

Unfortunately this a closed system so can't be more specific other than in the test case we are calling Views one with map/reduce and the other with map via couchdbkit. Calling couchdb directly is instant, calling via couchdbkit/restkit is not - unless we use the older versions.

from restkit.

Did you set a pool ?

from restkit.

I can see how one sets the pool in restkit, but couchdbkit doesn't appear to have a way to use it? Or am I missing something in the docs?

from restkit.

You can pass a resource object to the Server actually. There is a new version that should have been out yesterday that will make it easier during the week.

from restkit.

Thanks very much for the swift responses. I'll keep an eye out.

from restkit.

I just want to add my own bump to this ticket. Writing documents in couchdbkit - which goes through restkit - it slow to the point of making couchdbkit unusable. I ended up writing my own couch wrapper before finding this ticket.

Using curl or my simple wrapper around httplib, I can create 100 documents in 0.65 seconds. Using couchdbkit, creating 100 documents takes 4.6 seconds. I understand that you can use a connection pool based on the above conversation, but I don't see any mention of using a connection pool in the couchdbkit documentation. When all of your code examples show a normal instantiation without any connection pool being passed in, you have to expect that the vast majority of users will not be using one. It would be really nice if restkit "prioritized for the common case" and used httplib or something that is reasonably speedy when no connection pool is passed in.

from restkit.

What is your current version of restkit and couchdbit ? Which python ? Any code to share that would help to reproduce your problem? I actually can't. Also what is your version of couchdb.

from restkit.

Majority of users should use a connection pool if they don't want to put their system offline. You obviously want to reuse connections when it's possible.

from restkit.

I'm using Python 2.6.5
restkit 3.2.0
couchdbkit 0.5.4
I'm not using straight-up couchdb, but rather BigCouch. However, I am only using it on a single node for the testing that I am doing. I'm not sure what version of BigCouch I'm using. I did a git clone very recently.

I have some performance testing code that I've written that I would gladly share. Should I just send it to the email address you have listed on your GitHub profile page?

from restkit.

A quick test:

from couchdbkit import Server

s = Server()
db = s.get_or_create_db("testdb")
def test():
    db.save_doc({"a": 1})


if __name__ == "__main__":
    from timeit import Timer
    t = Timer("test()", "from __main__ import test")
    print t.timeit(number=100)
    print "nb docs: %s" % len(db)
    s.delete_db("testdb")

When I run the script:

(couchdbkit)enki:couchdbkit benoitc$ python test.py
0.321156978607
nb docs: 100

from restkit.

On Sat, Mar 19, 2011 at 8:00 PM, macie-korte
[email protected]
wrote:

I'm using Python 2.6.5
restkit 3.2.0
couchdbkit 0.5.4
I'm not using straight-up couchdb, but rather BigCouch.  However, I am only using it on a single node for the testing that I am doing.  I'm not sure what version of BigCouch I'm using.  I did a git clone very recently.

I have some performance testing code that I've written that I would gladly share.  Should I just send it to the email address you have listed on your GitHub profile page?

Reply to this email directly or view it on GitHub:
#5 (comment)

Send me it on my mail, thanks.

• benoît

from restkit.

I've did more tests and it seems that most of the time is lost in body read. fixing that.

from restkit.

Some tests with your scripts based on debug branch from restkkit and couchdbkit:

http://friendpaste.com/24rthY9DtXef7Y9y2sBBRj

Something on linux, make body reading slower. I'm investing that.

from restkit.

I'm glad to hear you could reproduce the problem under Linux. I never would have guessed the problem was based on the operating system. Good luck finding the problem! If it would help, I could run the test on another operating system such as Windows to see if it falls into the slow or fast category.

from restkit.

isolated the problem to the socket.recv function in http parser. So it wasn't due to restkit somehow.

Apparently adding the nodelay socket option to couchdb http do the trick:

on truck that's easy, add to you local.ini:

[httpd]
socket_options = [{nodelay, true}]

let me know if it works for you.

from restkit.

Actually, I couldn't get the speed-up to work when I added that option into couch. I tried it for both bigcouch and couchdb. However, I DID get the speed-up to occur with a slight tweak to your restkit/client.py module:

try:
  sck = socket.socket(af, socktype, proto)
  sck.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)

  if self.timeout is not None:
    sck.settimeout(self.timeout)

  sck.connect(sa)

I just added the "sck.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)" line at line #184 of the connect() function. Now couchdb access is running at the same speed as my comparison code.

from restkit.

the same, on the gentoo I just installed. I wonder now what is the best way to put it as an option.

from restkit.

TCP_NODELAY could kill the bandwidth when you stream big body (multiple writes). Rather than relying on it I changed the code so it try to send body + header in one block rather than doing 2 sent. Applied in 883bb5f . Tested on linux and mac.

Let me know if it works for you

from restkit.

note: on linux for local connection, it's better to use no_delay option to true in couchdb. Advantage is less interersting when you go for big body.

from restkit.

last head of restkit provides me on linux:

$ python couch_api_speed_test.py
couchdbkit: storing 100 results took this much time: 0:00:00.694517
couchdbkit: retrieving 100 results took this much time: 0:00:00.478437
fastcouch_httplib_example: storing 100 results took this much time: 0:00:00.781407
fastcouch_httplib_example: retrieving 100 results took this much time: 0:00:00.379531

and osx:

couchdbkit: storing 100 results took this much time: 0:00:00.306992
couchdbkit: retrieving 100 results took this much time: 0:00:00.185265
fastcouch_httplib_example: storing 100 results took this much time: 0:00:00.544357
fastcouch_httplib_example: retrieving 100 results took this much time: 0:00:02.360847

The first one is only possible if you set no_delay to true in couchdb.

from restkit.

last patch seems to fix it. closing issue.

from restkit.