feat: Support multi-route in mosec about mosec HOT 5 CLOSED

Possible solution:

• add a Router() on the Python side
• append the related workers to the Router
• register the Router with the related endpoint (by default will use /inference)

Problems:

• if a worker belongs to multiple routers, how to reuse it?

from mosec.

\assign

from mosec.

if a worker belongs to multiple routers, how to reuse it?

Perhaps we need to register the route info at rust side, so at different stages of different routes, the same worker can be used for overlapping tasks?

from mosec.

if a worker belongs to multiple routers, how to reuse it?

Perhaps we need to register the route info at rust side, so at different stages of different routes, the same worker can be used for overlapping tasks?

That's necessary for the controller to know the route info.

@cutecutecat This might be difficult to be done in one shot. You can do it step by step. Maybe the 1st target is to support the customized HTTP endpoint instead of the default /inference with the routing concept. Multi-route can be the next one.

from mosec.

if a worker belongs to multiple routers, how to reuse it?

Perhaps we need to register the route info at rust side, so at different stages of different routes, the same worker can be used for overlapping tasks?

That's necessary for the controller to know the route info.

@cutecutecat This might be difficult to be done in one shot. You can do it step by step. Maybe the 1st target is to support the customized HTTP endpoint instead of the default /inference with the routing concept. Multi-route can be the next one.

It might be ok to deploy a simple multi-route Server, but Worker reuse is a bit hard, and need to design at next step.

I think some Graph algorithms is required to calculating shard Workers.

from mosec.