IEEE

Bauhaus: Restructuring Vector Database for LLM Retrieval on CXL-Based Tiered Memory

Abstract: Retrieval-augmented generation pipelines store large volumes of embedding vectors in vector databases for semantic search. In Compute Express Link (CXL)-based tiered memory systems, ...
IEEE

Efficient KV Cache Spillover Management on Memory-Constrained GPU for LLM Inference

Abstract: The rapid growth of model parameters presents a significant challenge when deploying large generative models on GPU. Existing LLM runtime memory management solutions tend to maximize batch ...