Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory costs and time-to-first-token by up to 8x for multi-turn AI applications.

Abstract: This paper proposes a Web cache replacement algorithm that considers object size and usage in its design. The algorithm is characterized by a parameter k, which is used as a criterion to ...

Abstract: The least recently used (LRU) algorithm is one of the page replacement algorithms used in the swap mechanism of the Linux kernel. The LRU algorithm has evolved through various modifications ...