The “Introduction to the Compute Express Link® (CXL®) Fabric Manager” webinar explored the CXL Fabric Manager (FM) and Multi-Logical Device (MLD) management and detailed the Component Command Interface (CCI) transport protocols, background operations and categories of requests. If you missed the live webinar, the recording is available on BrightTALK and YouTube. The presentation is also available for download on the CXL Consortium’s Resources webpage.
We received great questions from the audience during the live Q&A. Below is a recap of the questions and answers discussed in the webinar.
Q: Can the Sanitize command operate on securely locked memory?
The Sanitize command will only successfully execute on unlocked memory. Issuing Sanitize to a device whose memory is locked will result in the “Invalid Security State” return code.
Q: In the diagram below, the host is shown talking to the CCI mailbox from the switch port. The FM can also talk to the CCI using the System Management (SM) bus. Why does the host driver need to talk to the CCI? Is it to demand some actions from the FM? How will the FM know if there is any request from the host using the CCI mailbox?
The host access to a switch CCI and FM access to a switch CCI are not for the purposes of a host communicating with an FM. The CCIs operate almost entirely independently from one another and have no visibility into each other’s state. It is expected that any host to FM interaction is going to be application specific and falls outside of the scope of CXL. This is typically handled through the system orchestrator. If the host had information relevant to the actions the FM was executing, the information would be passed to and processed by the system orchestrator. The communication path as defined today is Host A and Host B communicate through the system orchestrator. The Host A and Host B access to the CCI is for things such as reading out error logs in the CXL switch – less “system level configuration and deployment” and more “low-level error handling and detection”. However, there is no strict requirement. The diagram below is a representative example.
Q: Does the FM need to communicate to devices using CXL.io? Could you give some examples of use cases? Does the FM need to send any CXL.io to host in any case for error reporting?
The Mailbox CCI is present in PCI Express® (PCIe®) memory-mapped IO (MMIO) space. From a transport perspective the host is sending CXL.io transactions to interact with the mailbox CCI and, as stated in the spec, one possible FM deployment is SW running in a host, which would be an example of an FM using CXL.io to communicate with devices.
There is no CXL.io path defined from the FM to the host. There is no requirement for FM and host interaction, so no path has been defined.
Q: How does the switch know if the switch is an MLD port to report it to the FM?
During link negotiation, both ends of the link advertise their capabilities. These include the link speeds they support, the flit modes they support and whether they support an MLD. It is not mandatory for a CXL switch to support an MLD port. The CXL switch is responsible for applying the LD-ID (Logical Device Identifier) tag on transactions passing across an MLD port from the host. The host doesn’t understand what LD-ID from an MLD has been assigned to it, so its .io and .mem transactions leave the LD-ID field blank. It is the switch’s responsibility to tag the correct LD-ID on each transaction. This is an optional feature of a switch, and this capability is advertised during Alternate Protocol Negotiation. Both ends of a link will advertise that they are MLD capable, so the switch will know a link up time whether a port has negotiated in MLD mode. This is how the switch discovers an MLD is present to report that to the FM.
Q: Are background commands allowed to return data in the mailbox payload. If so, how do they share the output payload with the regular command output payload running in parallel?
Background commands are not allowed to support output parameters. The structure of a command set that includes background commands is different from the standard input parameters and output parameters command. The command is broken up into an “initiate process” command, “check status” command, and “get output data” command. They are not allowed to return output payloads. There would be issues with background commands overriding the output commands of non-background commands.
Q: You recommended only one CCI to support background command in a single device. But the CEL says background command per opcode regardless which CCI it is using. How will this work?
In the CXL switch below, there are three CCIs presented. The Transfer Firmware command for updating the CXL switch firmware is a background command. The recommendation is for only one CCI to support the Transfer Firmware command, for example. The Command Effects Log entry that host A and host B pull from their mailbox CCIs will not include the Transfer Firmware opcode. Whereas the MCTP based CCI will support Transfer Firmware in its Command Effects Log when the FM reads it.
Q: You mentioned that the host will boot later after a switch in FM is up. Does this mean that the FM will discover all devices before the host comes up?
In general, yes. In a large datacenter with a lot of appliances and hosts, it is going to change. You might have some hosts up with resources already deployed, or a host goes down and resources are assigned to it before it boots up. That is going to depend on the process of system composition defined for a datacenter. There is no hard requirement, this is more of a representative example.
Q: The slides show Gigabit Ethernet (GbE) as an FM physical interface, is there an industry consensus building around GbE or are there other interfaces being widely considered?
Everything above the CCI to FM interface falls outside of the scope of CXL. CXL is not specifying that GbE is used as the transport to an orchestrator. The orchestrator is not a CXL defined term; these are all just possible implementations. There are no restrictions or requirements on the FM’s capability and its interaction model with a higher level body of logic like a system orchestrator.
Stay tuned for future webinar presentations from the CXL Consortium. Subscribe to the CXL Consortium YouTube channel and CXL Consortium BrightTALK channel to be notified about upcoming webinars.
Follow us on Twitter and LinkedIn for more updates! For additional information about CXL, please visit our website.
Compute Express Link® and CXL® Consortium are trademarks of the Compute Express Link Consortium.