[Question] Can PagedKVCache support different size of kvcache in different layers? about mlc-llm HOT 4 OPEN

The current version does not support it yet and it might be hard to modify for it.

from mlc-llm.

hi, @Hzfengsy , so if I want to deploy my pruned model, now the only solution is use nn.KVCache to replace the PagedKVCache as the old commit did? Or if I can create more than one PagedKVCache and use different PagedKVCache in different layers? Could you give me some suggestion?
thank you for reply.

from mlc-llm.

KV cache is a common interface, the solution right now would be to create a difference instance of kv cache implementation of the same interfaceand replace it

from mlc-llm.

@BenchuYee hi, I want to adjust the KVCache for more flexible usage, which old commit did you use to build the nn.KVCache model? and BTW, do you guys observe obvious performance drop using nn.KVCache instead of PagedKVCache (no batch requirest considered)?

from mlc-llm.