Add prefix cache aware routing #641
Conversation
varungup90
changed the title
varungup90
changed the title
| end = len(unMatchedTokens) | ||
| } | ||
| chunk := unMatchedTokens[i:end] | ||
| prefixHash := xxhash.Sum64(IntArrayToByteArray(chunk)) |
There was a problem hiding this comment.
here. it just consider the current block size? just like to confirm this is not 100% same as vLLM's solution right? their 1st block hash is part of 2nd hash input
There was a problem hiding this comment.
Yes, hash only for current block. No link list kind of behavior that vllm has. For our usecase we do not need that link list behavior.
There was a problem hiding this comment.
if that case, we need to do up to O(n) calculations? n=number of blocks
There was a problem hiding this comment.
Yes. If you are referring to that link list behavior could prevent O(n) calculations then it wont be the case. Total computations stays the same.
There was a problem hiding this comment.
it's a little bit different. LinkedList you can do binary search etc for optimization. In this way, we can only do O(n).
There was a problem hiding this comment.
Let me look into this, but for our usecase we need to evaluate all blocks to ensure a 50%+ hit rate.
* Add prefix cache aware routing * end to end stiching * fix lint errors * nit * add integ test for prefix caching * add constants * address review comments * add prefix cache eviction * add unit test for prefix cache eviction
* Add prefix cache aware routing * end to end stiching * fix lint errors * nit * add integ test for prefix caching * add constants * address review comments * add prefix cache eviction * add unit test for prefix cache eviction
No description provided.