Introduction
With the ever-increasing compute and memory bandwidth demands on data center servers, system implementors continuously strive to find novel techniques to provide higher performance for a wide range of application workloads while maintaining or even decreasing the cost of new infrastructure. The fundamental challenge of traditional servers results from being locked to specific memory types and having limited DRAM memory channels that prevent the flexibility needed to implement these techniques and optimize cost and performance metrics. Let’s look at how Compute Express Link’s CXL.memory sub-protocol can be used to create CXL™ memory controllers to:
- Expand memory bandwidth and/or capacity beyond what the host processor’s native DDR memory channels allow
- Enable host processors to have media-independence and support different memory types
- Decrease solutions costs by right-sizing memory capacity for targeted application workloads
Compute Express Link utilizes the PCI Express® (PCIe®) 5.0 physical layer infrastructure and the PCIe alternate protocol to address the demanding needs of high-performance computational workloads. The CXL.io sub-protocol provides similar functionality as traditional PCIe, including discovery, configuration, register access and interrupts. While the CXL.cache sub-protocol provides a CXL device access to the host processor’s memory, CXL.memory, or CXL.mem for short, provides the host processor low latency access to the CXL device’s attached memory. CXL.mem maintains a unified, coherent memory space between the host processor’s memory and any memory on the attached CXL device.
Benefits of Serial-Attached Memory
Systems implementors have traditionally looked for more parallel DDR interfaces to be supported on the host processor to address the increase in compute memory bandwidth and capacity requirements. However, with DDR4 and DDR5 having 380 pins per channel, there is a limit to the number of DDR interfaces that can be feasibly supported in a CPU, GPU or SoC package. In the figure below, we see how a CXL device utilizing CXL.mem, a “CXL memory controller”, can be used to expand memory beyond the host processor’s limited DDR memory channels while ensuring a seamless user experience through coherency.
The memory bandwidth requirements for many applications have increased faster than the available memory bandwidth due to the limited number of DDR interfaces available per CPU socket. This has resulted in bandwidth constraints and increased memory latency per CPU core. Compared to a DDR5 interface with 380 pins providing 32GB/s bandwidth, a CXL memory controller can provide the same or higher bandwidth utilizing x8 or x16 CXL lanes to provide 32GB/s or 64GB/s bandwidth, respectively. CXL serial-attached memory can be used to alleviate the bandwidth constraints of today’s solutions.
Traditional server architectures utilizing parallel memory solutions are not well-equipped to provide the flexibility to address the varying memory and bandwidth requirements of different application workloads. Today’s host processors (CPUs, GPUs and SoCs) are locked to specific memory interface types. CXL enables memory media-independence by leveraging the PCIe lanes available on the host processor to connect to a wide variety of memory interfaces. CXL memory controllers can be designed to support different memory types, such as DDR4, DDR5 or even persistent memory or storage class memory to address varying application workload and cost requirements.
Parallel memory DIMMs also pose a challenge as they do not provide fine capacity granularity. Servers may have more capacity than is required for the application resulting in higher costs due to unutilized memory capacity. The flexibility provided by CXL memory controllers open the options to deliver finer capacity granularity than traditional DIMMs, allowing system implementors to right-size the memory capacity to fit their application and reduce costs. Furthermore, novel CXL implementations may enable new form-factors, including the promise of both CXL memory modules and memory drives.
Summary
CXL memory controllers utilizing CXL.memory sub-protocol will provide low-latency serial-attached memory expansion with a unified, coherent memory space between the host processor’s memory and any memory on the attached CXL device. Serial-attached memory solutions using Compute Express Link have many benefits compared to parallel interfaces and can be used to replace or augment parallel memory solutions to:
- Expand memory bandwidth and/or capacity beyond what the host processor’s native DDR memory channels allow
- Enable host processors to have media-independence and support different memory types
- Decrease solutions costs by right-sizing memory capacity for targeted application workloads
For more information on CXL technology, visit our Resource Library for white papers, presentations and webinars.