CXL® Memory Form-Factor Comparison: Examination of Form-Factors for this Growing Standard

Introduction

Computer systems have used direct attached memory modules in the form of DIMMs for decades. These modules attach directly to the host motherboard and to the CPU using a parallel bus. Expanding the number of modules a system can utilize involves adding memory controllers and pins to the CPU which quickly becomes prohibitive.

As modern applications such as AI & Machine Learning, image processing, in-memory databases and real time analytics consume more memory than ever before, a new solution is needed to keep up with the demand for low latency, high bandwidth memory.

Compute Express Link® (CXL®) is a high-bandwidth, low latency, CPU-to-Device interconnect standard. It builds on existing PCI Express® (PCIe®) infrastructure, leveraging the PCIe physical and electrical interface while adding additional transfer protocols. Memory expansion, memory pooling, including sharing, and system composability are usage models enabled by CXL. CXL enables support for three types of devices:

• Type 1: Smart NICs & Accelerators

• Type 2: Accelerators with Cache

• Type 3: Memory

Due to the growing demand for high density and cost effective memory solutions, industry efforts are initially focusing on memory expansion with Type 3 Memory modules. These modules can be built in several different emerging form factors as described in this article. They include EDSFF (Enterprise and Datacenter Standard Form Factor) and AIC (Add-In-Card) form factors.

EDSFF Form Factors

EDSFF is maintained by the Storage Networking Industry Association (SNIA) and defines a family of standardized form factors with several advantages over the current industry standard SSD form factors. EDSFF supports a number of different PCIe lane configurations (x4, x8, and x16) as well as several different physical form factors, all of which are suitable for CXL based designs as well.

EDSFF E1.S

Figure 1 – left to right: E1.S 5.9mm (courtesy of SMART Modular Systems); E1.S Symmetric Enclosure (courtesy of Intel); E1.S (courtesy of KIOXIA); E1.S 5.9mm (courtesy of Micron); E1.S 15mm (courtesy of Micron)

Figure 2 – Supermicro® Server with 32 E1.L fitted in 1U chassis

EDSFF E3.S and E3.L

Figure 3 – EDSFF Modules

Figure 4 – SMART Modular Technologies E3.S CXL Memory Module

Add in Card (AIC) Form Factors

The AIC form factor, also referred to as CEM (Card Electromechanical), is the longest standing PCIe form factor which supports a wide variety of devices from graphics accelerators to network cards. The standard may be downloaded from the PCI-SIG website. AICs support different connector sizes and lane counts (x1, x2, x4, x8, and x16) and sizes with full and half height (referred to as “low profile”) as well as full and half-length. This form factor allows CXL AICs that use conventional DRAM memory modules to be installed making for good value memory expansion.

Figure 5 – SMART Modular 8-DIMM Add-In-Card

Figure 6 – SMART Modular 4-DIMM Add-In-Card

Form Factor Comparison

In general, EDSFF devices have lower capacity and lower power than AIC devices, but fit in smaller form factors suitable for front loading on servers. AICs, by contrast, have higher capacities and power, but need to be installed internally in the system and take up more space.

Conclusion

There are several form factors available for CXL devices that leverage the PCIe ecosystem and framework. CXL modules and cards may be added to existing computer shelves, towers, and blades with no mechanical or electrical re-design requirement. Providing a reasonable comprise between footprint and capacity allows for a smooth integration of CXL devices into contemporary computer system designs.