[Question] How to benchmark the execution? about batched-fn HOT 6 CLOSED

Thanks for the explanation!

from batched-fn.

Hi, you might want to start by putting timing code in the handler closure, e.g. right here:

That will tell you how it takes to process each batch at a given batch size.

In general (for deep learning models, at least) I would set max_batch_size as large as you can without running out of memory, and max_delay small relative to the time it takes to process a batch.

from batched-fn.

Hi, thanks for the suggestion. That helps! I have one more question, how about channel_cap?

from batched-fn.

When you set channel_cap, batched_fn! will internally use a bounded flume channel instead of an unbounded one. As a result, calls to your batched fn might return Error::Full.

One reason you might want to use this feature is to return 503 errors when your server is getting too many requests at once. In particular, set channel_cap to some number greater than max_batch_size and then catch Error::Full in your server code, and return a 503 error when that happens.

from batched-fn.

I realize the documentation about channel_cap has been lacking, so here's this: #19

from batched-fn.

You're welcome!

from batched-fn.