By: Narasimha Babu GVL, Scientist, Synopsys
Threat Mitigation in CXL
Data, Data, and Data everywhere! Data processing is no longer localized but distributed. In the use case, “Software as a Service (SaaS)”, cloud-based computing models place increasingly demanding needs for data security with the ability to set up, control and execute within Trusted Execution Environments.
The CXL stack has 2 independent paths, CXL.io (for traditional PCI Express® (PCIe®) technology related semantics) and CXL.cachemem for coherency and memory semantics. This blog will focus on the security features of CXL.cachemem.
CXL.cachemem defines a two-level framework for security as shown in figure 1 below.
- Transport Security is a bottom layer wire level Integrity and Data Encryption (IDE) (more hardware-based) in CXL 2.0 and the focus of this writeup.
- An application-level (more software-based) entity for confidential computing with Trusted Execution Environments in CXL 3.0 and later versions of the specification for Type 3 systems with direct connection between hosts and devices.
CXL.cachemem two-level framework for security (Figure 1)
The specification is crafted not to add latency overhead when secure features are used in the system. Details of the overall framework are presented below.
- CXL builds on Distributed Management Task Force (DMTF) specified features like Security Protocol and Data Model (SPDM) and Component Measurement Authentication (CMA) and well-known Cryptographic algorithms like AES-GCM (Advanced Encryption Standard with Galois Counter Mode) and MAC (Message Authentication Code).
- Leverages PCIe-defined Data Object Exchange (DOE) and Vendor Defined Message (VDM) for transport of protocol information.
The table below provides a top-level overview of different specification elements, which play a role in the CXL.cache and CXL.mem security protocol.
|
Specification forum |
Area of interest |
Intent and Usage |
| DMTF | Security Protocol and Data Model (SPDM) | CXL.cachemem IDE Key Management |
| DMTF | MCTP | Transfer of SPDM Key management messages over the link |
| PCI-SIG | DOE | Transfer of SPDM Key management messages over the PCIe link |
| NIST | Crypto algorithms | AES-GCM with 256-bit key encryption |
| CXL | IDE semantics | Skid / Containment mode / PCRC
Trusted Execution Environments Security Protocol (TSP) |
Security Feature Evolution in the CXL Specification
IDE was introduced in CXL 2.0 to provide confidentiality, integrity, and replay protection for the flits sent and received on the CXL link. The feature operates on a flit-level granularity and uses 256-bit keys for encryption.
Message Authentication Code (MAC) gives an additional layer of security where a certain MAC value is sent for a fixed set of flits based on the Aggregation Mode. Encrypted PCRC (a CRC-32 calculated on flit plaintext content) mechanism was introduced to provide robustness against hard and soft faults that are internal to the encryption and decryption engines. Encrypted PCRC is integrated into the MAC check mechanism and does not consume additional link bandwidth.
The figure below summarizes the key specification features of CXL IDE – transport-level security.
CXL IDE – Transport level security (Figure 2)
The key differences between IDE in CXL 2.0 and CXL 3.0 are the following:
- Aggregation Mode, which represents the number of flits in an epoch features different values for both containment and skid modes in 68B vs 256B modes.
- In CXL 2.0, MAC was sent as part of protocol flit in 68B mode in an H6 slot. In CXL 3.0, in 256B mode, IDE.MAC is a control flit.
- Key Refresh Time Capability2 and Truncation Transmit Delay Capability. Rx Min TruncationTransmitDelay2 were introduced in CXL 3.0 to give individual controls in 68B and 256B modes.
- More IDE Errors were defined in CXL 3.0.
Design Considerations for IDE in CXL Hosts & Devices
Hosts and devices that implement IDE should make use of the AES-GCM, as defined in NIST* Special Publication 800-38D.
To establish an IDE flow between two entities, the system software or firmware entity needs to do the following:
- Host Software sets up independent SPDM secure sessions with Upstream Port.
- Host Software sends a CXL_QUERY to the Upstream Port and compares the CXL_QUERY_RESP it receives against the CXL IDE Capability structure associated with this port. It exists with an error if there is a mismatch.
- Configure CXL.cachemem IDE Rx/Tx Keys, Initialization Vector (IV) registers.
- Program additional registers like Key Refresh Time Capability/Control Registers which indicate the number of IDE.IDLE flits the Receiver and Transmitter respectively need to block processing of protocol flits after IDE.START.
- Program Truncation Transmit Delay Capability/Control Registers which indicate the number of IDE.IDLE flits the Receiver and Transmitter respectively need to block processing of protocol flits after IDE.TMAC.
- Optionally program CXL IDE Control register to disable PCRC generation which is enabled by default.
- Enable CXL.cachemem IDE using CXL_IDE_KM (Ley Management) messages.
IDE Design Verification
Security features in CXL technology span across multiple domains/specifications, resulting in hybrid designs consisting of hardware and software components. This also means the verification space is significant. Figure 3 provides an overview of the scope and goals to be achieved for systems to interoperate smoothly. The figure illustrates the various aspects of the CXL protocol that need to be verified at different levels including sub-system, SoC, and system / platform.
IDE Verification (Figure 3)
As shown in the above figure, multiple vectors closely interact to drive the verification space with different goals. From a Design Verification perspective, block level to System / Platform level with only Hardware (RTL) or a combination of Hardware plus Software as a dominant focus coming into play driving the verification scope and goals.
Additionally, at the physical layer, links could be bifurcated (e.g. 2 links with x8 lanes) where different device types become important. There are possibilities of certain devices having IDE enabled versus some without it in the system.
Hence, the discovery of IDE capabilities is a necessity across all the devices in all possible bifurcation modes. The configuration register accesses related to the IDE capabilities need to be tested. The configuration accesses might be implemented in such a way that a sideband network might exist between the controllers, which might need verification as well.
Secure modes. which include skid and containment modes, need to be tested along with PCRC.
One crucial aspect for Transaction Integrity is that it needs to be maintained even in the presence of errors, IDE verification with the introduction of Poison and Viral errors during traffic flow is key. Also, system behavior when negative scenarios are introduced needs to be checked. Incorrect keys on either side of the link should be checked and ensured that transaction integrity is not maintained in such a case.
Similarly, Downstream Port Containment (DPC) is a feature wherein, for Type 3 devices, error isolation could be achieved. In the case of multiple Type 3 devices, if there is an error such as timeout or link down, the port in question could be isolated and accesses to the rest of the devices could be completed, thereby ensuring error containment. DPC needs to be verified with IDE enabled such that after the port with the error gets restored, IDE could be reconfigured and ensured that secure accesses are still possible over the link. Reconfiguration of unaffected ports would not be needed and must be verified.
The error flows highlight the Reliability, Availability, and Serviceability (RAS) aspects of the system.
At the SoC level, the crossing of protocol features is essential to ensure that CXL.cachemem traffic with IDE is enabled, works well with PCIe traffic, and is crossed with reset and power management flows. Traffic flows with IDE enabled is done with CXL components going through different types of reset flows such as cold reset, warm reset, and hot reset.
CXL 2.0 introduced a Memory Interleaving feature with multiple Interleave ways and granularity. When this feature is enabled, secure accesses need to be checked by varying Interleave granularity and Interleave ways. Since multiple Type 3 devices are part of an Interleaved configuration, IDE needs to be enabled across multiple devices and verified.
Performance impact and analysis is quintessential for several traffic profiles enabling IDE. Special interest is with the containment mode, wherein performance impact is to be expected and assess the level of impact.
Also, at the platform level, interoperability testing is needed to ensure that link partners can effectively communicate with encryption enabled. Interoperability testing should be done with different vendors in the market. CXL.cachemem performance enhancement features such as Multi Data Header (MDH) support are features to be checked when IDE is enabled.
Key Switching is a mechanism where keys can be programmed during traffic flow without quiescence. This needs to be tested across the link to ensure the smooth flow of traffic with proper encryption ensured despite keys being modified.
TSP semantics for confidential computing adds an additional dimension. TSP flows can be set up with Transport Security (IDE enabled or disabled). The CXL transaction layer defined additional fields related to TSP in M2S and S2M requests and responses. When the TSP is set up and the state machine is set to CONFIG_LOCKED state, these additional fields are utilized. The behavior of the DUT for proper operation depending on the TE state needs to be verified.
Key Asks from Verification Components
For effective pre-silicon verification and validation, and to gain confidence with interoperability and first-time silicon success, the verification components play a key role. This section lists key asks but is not limited to the components.
The verification components must support the following:
- Skid and Containment mode.
- MAC disable mode with just pure encryption/decryption would assist debugging.
- PCRC enablement and disablement.
- Configuration mechanisms for keys, key refresh, and truncation timer.
- Support the truncated MAC (TMAC).
- Key Switch support.
- Tracker files with as much debug information as possible. Some include configuration information, pre-encrypted and post-encrypted data, mid-simulation resets if any.
- Callbacks to create integrity error scenarios.
- Functional coverage model to capture IDE-related information.
Reference documents related to this topic:
From Compute Link Express Consortium
From Distributed Management Task Force (DMTF)
From National Institute of Standards and Technology (NIST) Special Publication
Acknowledgements:
I would like to express my thanks and gratitude to CXL Consortium members Kashyap Sharath, Bangalore Siddalingadevaru Reshma, and Elza Wong for assisting this blog write up with their valuable contributions on the content and review feedback. I also extend my gratitude for the feedback and guidance provided by Synopsys team.