[Question] How to make `max_batch_size` configurable? about batched-fn HOT 3 OPEN

So, the reason configuration options need to be static is because the batched_fn! macro expands to define a BatchedFn instance and worker thread statically. These things need to be defined statically because batched_fn! is meant to be dropped into a function/method, and if they weren't declared static then they would be recreated every time the function/method is called.

So there is probably no way we could allow non-static expressions to be passed into batched_fn!. However, it would be possible to change the value of certain configuration options (like max_batch_size) on-the-fly. We'd just need to provide a Channel to communicate changes from the main/calling thread to the worker thread.

from batched-fn.

Ah, I see what the issue is. Thanks for bringing that up. #21 should solve this, but requires a small breaking change:

Instead of:

let handler = batched_fn! { ... };
return handler(input).await;

You would have to change the last line to:

return handler.evaluate_in_batch(input).await;

from batched-fn.

Ugh, actually that won't work. There might be another way around this, but I'll have to put some more thought into it. I'll get back to you.

from batched-fn.