LMCache enables AMD GPUs to overflow KV cache onto a multi-tier memory hierarchy spanning DRAM, SSD and remote storage

Tensormesh, the company pioneering caching-accelerated inference optimization for enterprise AI, today announced a collaboration with AMD through which Tensormesh KV cache solution and AMD virtual memory offering will work together to allow more models to be served on fewer GPUs while retaining high KV cache hit rates and throughput even with oversubscribed high-bandwidth memory (HBM). Tensormesh is working with AMD, leveraging its GPU technology and AMD Live Context Virtualization components, and is tested using Dell servers with 8x AMD/ATI accelerators (MI355) GPUs and Dell storage. Tensormesh integrated LMCache coordinates KV cache management.

For users, running more models on the same set of GPUs means lower costs. LMCache users can now reuse the infrastructure they have built to virtually expand their GPUs' capacity. In addition, customers can reuse KV cache chunks stored for short-term memory virtualization later for prefix or non-prefix KV cache matching, maximizing system efficiency.

The Results

This new approach is far more efficient than building GPUs with more memory, which has led to the industry’s current memory supply crisis and increased the number of GPUs. Enterprises running AI over large document sets can now have a better experience and a lower bill. Testing showed:

• Near-instant responses at scale: On a 300 GB document set (Kimi-K2.6), time-to-first-token dropped from 3.4 seconds to under half a second, a nearly 7x improvement, thanks to reusing cached KV data from DRAM and NFS instead of recomputing it from scratch.
• No slowdown as usage grows: Output throughput held steady at ~48 tokens per second regardless of workload size, while unoptimized inference throughput fell by nearly 40% under the same load.
• Twice the model density: The same hardware doubled the model density.

“This powerful new collaboration builds on AMD’s recent strategic investment in Tensormesh and expands the capabilities of AMD GPUs,” explained Junchen Jiang, Tensormesh CEO and LMCache co-creator. “Together, we’re greatly enhancing the memory that inference engines can access for model weights and KV cache, using all of the memory resources on each node.”

“We recognize Tensormesh and LMCache as KV cache management leaders,” said Anush Elangovan, AMD’s vice president of AI software. “And we’re thrilled to announce AMD’s breakthrough in virtual GPU memory management, amplified by LMCache and Tensormesh.”

The companies are sharing performance results from their collaboration at this week’s AMD Advancing AI 2026, a month before the public release of the LMCache open-source code.

Additional Resources:

• Fixing AI’s Most Expensive Problem with Junchen Jiang, Tensormesh CEO
• LMCache: Build The Foundation of AI Memory Tensor with KV Cache Infrastructure
• Persistent KV Cache: Own Your Context Caching Lifecycle

About Tensormesh

Tensormesh is the leader in caching-accelerated inference optimization for enterprise AI. Founded by faculty, PhD researchers and alumni from the University of Chicago, UC Berkeley, and Carnegie Mellon, and led by Junchen Jiang, University of Chicago faculty member and co-creator of LMCache, Tensormesh builds on years of academic research in distributed systems and AI infrastructure. The company has raised $24.5 million in total funding and is backed by Valley Capital Partners, NVentures, AMD Ventures, CoreWeave, and Laude Ventures.

View source version on businesswire.com: https://www.businesswire.com/news/home/20260723669125/en/

“This powerful new collaboration builds on AMD’s recent investment in Tensormesh & expands the capabilities of AMD GPUs. We’re enhancing the memory that inference engines can access for model weights and KV cache, using memory resources on each node.”