Randomized Delay about colly HOT 7 CLOSED

That sounds pretty reasonable. I'll take a look.

from colly.

@jodizzle this would be a handy addition. Do you want to work on it?

As a workaround you can add an OnRequest callback with random sleeps:

c.OnRequest(func(_ *colly.Request) {
    time.Sleep(time.Duration(rand.Randint(5)+1) * time.Second)
})

from colly.

@asciimoo Ah, I hadn't thought of the OnRequest approach. Thanks.

And sure, I can try working on it. Do you have any thoughts about what the best implementation would be?

from colly.

I like the idea of extending LimitRule with a RandomDelay time.Duration attribute which is 0 by default. If its value isn't 0, a random duration between 0 and RandomDelay could be added to LimitRule.Delay on every request. This solution doesn't break the current API and it's intuitive for me. What do you think?

from colly.

Not sure if its okay, but I got interested in the code base so I just implemented the enhancement.

@jodizzle let me know if you still want to fix it and I can decline my pull request.

#67

from colly.

@jlr52 Fine by me -- I was being slow. Your approach seems similar to what I was thinking anyway. I'll take a closer look.

from colly.

implemented in #67 - thanks @jlr52

from colly.