[Performance Optimization] Multiple channels when getting shuffle data in client side about incubator-uniffle HOT 6 CLOSED

@zuston The current implementation limit the number of connection, because don't want too many connection established between client and shuffle server.
We also plan to improve this by using netty instead of Grpc, not only about throughput but also reduce unnecessary serialization in Grpc.

from incubator-uniffle.

Glad to hear this. From the flame graph, due to extra memory-copy, it cost too much time in shuffle server side.

If using the netty to directly manipulate shuffle data bytebuf in off-heap, it may get better performance.

Besides, do we need to w/r shuffle data to files using nio to match with the netty? Please let me know whether i'm wrong.

from incubator-uniffle.

@zuston The current implementation limit the number of connection, because don't want too many connection established between client and shuffle server. We also plan to improve this by using netty instead of Grpc, not only about throughput but also reduce unnecessary serialization in Grpc.

Could we put this into the 0.7 version plan?

from incubator-uniffle.

@zuston The current implementation limit the number of connection, because don't want too many connection established between client and shuffle server. We also plan to improve this by using netty instead of Grpc, not only about throughput but also reduce unnecessary serialization in Grpc.

Could we put this into the 0.7 version plan?

Yes, I think so

from incubator-uniffle.

So do we have a roadmap of version-release/features in github? @jerqi

from incubator-uniffle.

So do we have a roadmap of version-release/features in github? @jerqi

Not yet. 0.6 version's main feature is to support K8S.

from incubator-uniffle.