By: XConn Technologies
As large language models (LLMs) and generative AI workloads continue to grow in complexity, memory is quickly becoming the new bottleneck. While GPUs are unmatched in parallel compute performance, their onboard memory capacity remains limited. Modern AI workloads, especially LLM inference with heavy KV cache usage, routinely exceed 80 to 120 GB per GPU, leading to high latency and costly data movement across systems.
At the CXL Pavilion (Booth #817) during Supercomputing 2025 (SC25), XConn Technologies will showcase a live demo illustrating how CXL memory pooling can shatter this bottleneck and unlock a new class of scalable, memory-centric AI architectures.
Currently, in a multi-GPU inference setup, data must traverse a long and inefficient path:
GPU → DRAM → NIC → Storage Server → NIC → DRAM → GPU
Each hop adding overhead, latency, and energy costs. When serving large-scale LLMs such as OPT-6.7B or GPT variants, even small inefficiencies in prefill/decode KV cache management multiply into seconds of delay and wasted compute cycles. This results in:
- Longer Time to First Token (TTFT)
- Low GPU utilization
- High data-movement energy cost
CXL provides memory-semantic access with latency in the 200–500 ns range, compared to ~100 μs for NVMe technology and >10 ms for storage-based memory sharing. This latency improvement enables truly dynamic, fine-grained sharing of memory resources across compute nodes.
XConn’s CXL switch introduces a shared CXL memory pool that acts as a direct-access, low-latency extension of GPU memory. Instead of routing data through network interfaces and storage servers, GPUs (or their host CPUs) can perform direct reads/writes to the CXL memory pool with CUDA-compatible semantics, eliminating redundant copies and thick software stacks.
Demo Overview
The demo will show two servers, each with an NVIDIA H100 GPU (80GB), run the OPT-6.7B model with 64 prompts per request and 512 tokens per prompt to showcase the performance benefits of CXL memory pooling. By disaggregating the workload between the prefill and decode stages, the setup demonstrates how shared CXL memory can accelerate AI inferencing.
Compared to RDMA-based sharing, the CXL memory pool achieved 3.8 times speedup compared with 200G RDMA, 6.5 times speedup compared with 100G RDMA, and a dramatic reduction in TTFT and improved bandwidth efficiency. With this, the demonstration highlights how AI workloads benefit from dynamic, scalable memory expansion up to 100 TiB per cluster, cost-effective resource utilization, energy-efficient data access with minimal CPU involvement, and integration with AI frameworks like NVIDIA Dynamo and KV Block Manager
A key enabler of this architecture is XConn’s next-generation technology, the Ultra IO Transformer, which allows PCIe GPUs, most of which don’t natively support CXL, to directly access the CXL memory pool through XConn’s hybrid switch, maintaining low-latency, high-bandwidth communication.
Real World Applications
CXL memory pooling is moving beyond the lab to deliver tangible benefits across real-world data center and AI environments. By enabling flexible, shared access to large pools of memory, it helps organizations accelerate data-intensive workloads, improve resource utilization, and reduce total cost of ownership. Real world applications of CXL memory pooling include:
- AI Inference & KV Cache Scaling: CXL memory augments GPU VRAM for KV cache storage, accelerating token decoding and reducing TTFT for LLM serving.
- Scientific & HPC Workloads: Projects like PNNL Crete use CXL pools for high-throughput memory sharing across compute nodes.
- Cloud Databases: Large in-memory databases integrate CXL memory pools to enable a high-performance database buffer pool with flexible scaling.
The memory wall has long been the limiting factor for AI scalability. Deploying a CXL memory pool creates a new tier of high-speed, disaggregated memory, reshaping how we build and deploy AI infrastructure.
XConn’s demo proves that CXL architectures are not just theoretical. CXL systems are ready to power real-world LLM inference today, bringing higher performance, lower latency, and scalable memory capacity, with lower TCO for AI at any scale.
Check out the demo at the CXL Pavilion (Booth #817) at SC25 from November 18-20. We hope to see you there!