Compute Express LinkTM (CXLTM)
Specification
October 2020
Revision 2.0
Evaluation Copy
LEGAL NOTICE FOR THIS PUBLICLY-AVAILABLE SPECIFICATION FROM COMPUTE EXPRESS LINK CONSORTIUM, INC.
© 2019-2020 COMPUTE EXPRESS LINK CONSORTIUM, INC. ALL RIGHTS RESERVED.
This CXL Specification Revision 1.1 (this “CXL Specification” or this “document”) is owned by and is proprietary to Compute Express Link
Consortium, Inc., a Delaware nonprofit corporation (sometimes referred to as “CXL” or the “CXL Consortium” or the “Company”) and/or its
successors and assigns.
NOTICE TO USERS WHO ARE MEMBERS OF THE CXL CONSORTIUM:
If you are a Member of the CXL Consortium (sometimes referred to as a “CXL Member”), and even if you have received this publicly-available version
of this CXL Specification after agreeing to CXL Consortium’s Evaluation Copy Agreement (a copy of which is available
https://www.computeexpresslink.org/download-the-specification, each such CXL Member must also be in compliance with all of the following CXL
Consortium documents, policies and/or procedures (collectively, the “CXL Governing Documents”) in order for such CXL Member’s use and/or
implementation of this CXL Specification to receive and enjoy all of the rights, benefits, privileges and protections of CXL Consortium membership: (i)
CXL Consortium’s Intellectual Property Policy; (ii) CXL Consortium’s Bylaws; (iii) any and all other CXL Consortium policies and procedures; and (iv)
the CXL Member’s Participation Agreement.
NOTICE TO NON-MEMBERS OF THE CXL CONSORTIUM:
If you are not a CXL Member and have received this publicly-available version of this CXL Specification, your use of this document is subject to your
compliance with, and is limited by, all of the terms and conditions of the CXL Consortium’s Evaluation Copy Agreement (a copy of which is available at
https://www.computeexpresslink.org/download-the-specification).
In addition to the restrictions set forth in the CXL Consortium’s Evaluation Copy Agreement, any references or citations to this document must
acknowledge the Compute Express Link Consortium, Inc.’s sole and exclusive copyright ownership of this CXL Specification. The proper copyright
citation or reference is as follows: “© 2019-2020 COMPUTE EXPRESS LINK CONSORTIUM, INC. ALL RIGHTS RESERVED.” When making
any such citation or reference to this document you are not permitted to revise, alter, modify, make any derivatives of, or otherwise amend the referenced
portion of this document in any way without the prior express written permission of the Compute Express Link Consortium, Inc.
Except for the limited rights explicitly given to a non-CXL Member pursuant to the explicit provisions of the CXL Consortium’s Evaluation Copy
Agreement which governs the publicly-available version of this CXL Specification, nothing contained in this CXL Specification shall be deemed as
granting (either expressly or impliedly) to any party that is not a CXL Member: (ii) any kind of license to implement or use this CXL Specification or any
portion or content described or contained therein, or any kind of license in or to any other intellectual property owned or controlled by the CXL
Consortium, including without limitation any trademarks of the CXL Consortium.; or (ii) any benefits and/or rights as a CXL Member under any CXL
Governing Documents.
LEGAL DISCLAIMERS FOR ALL PARTIES:
THIS DOCUMENT AND ALL SPECIFICATIONS AND/OR OTHER CONTENT PROVIDED HEREIN IS PROVIDED ON AN “AS IS” BASIS. TO
THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, COMPUTE EXPRESS LINK CONSORTIUM, INC. (ALONG WITH THE
CONTRIBUTORS TO THIS DOCUMENT) HEREBY DISCLAIM ALL REPRESENTATIONS, WARRANTIES AND/OR COVENANTS, EITHER
EXPRESS OR IMPLIED, STATUTORY OR AT COMMON LAW, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, VALIDITY, AND/OR NON-INFRINGEMENT.
In the event this CXL Specification makes any references (including without limitation any incorporation by reference) to another standard’s setting
organization’s or any other party’s (“Third Party”) content or work, including without limitation any specifications or standards of any such Third Party
(“Third Party Specification”), you are hereby notified that your use or implementation of any Third Party Specification: (i) is not governed by any of
the CXL Governing Documents; (ii) may require your use of a Third Party’s patents, copyrights or other intellectual property rights, which in turn may
require you to independently obtain a license or other consent from that Third Party in order to have full rights to implement or use that Third Party
Specification; and/or (iii) may be governed by the intellectual property policy or other policies or procedures of the Third Party which owns the Third
Party Specification. Any trademarks or service marks of any Third Party which may be referenced in this CXL Specification is owned by the respective
owner of such marks.
NOTICE TO ALL PARTIES REGARDING THE PCI-SIG UNIQUE VALUE PROVIDED IN THIS CXL SPECIFICATION:
NOTICE TO USERS: THE UNIQUE VALUE THAT IS PROVIDED IN THIS CXL SPECIFICATION IS FOR USE IN VENDOR DEFINED
MESSAGE FIELDS, DESIGNATED VENDOR SPECIFIC EXTENDED CAPABILITIES, AND ALTERNATE PROTOCOL NEGOTIATION ONLY
AND MAY NOT BE USED IN ANY OTHER MANNER, AND A USER OF THE UNIQUE VALUE MAY NOT USE THE UNIQUE VALUE IN A
MANNER THAT (A) ALTERS, MODIFIES, HARMS OR DAMAGES THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-
SIG ECOSYSTEM OR ANY PORTION THEREOF, OR (B) COULD OR WOULD REASONABLY BE DETERMINED TO ALTER, MODIFY,
HARM OR DAMAGE THE TECHNICAL FUNCTIONING, SAFETY OR SECURITY OF THE PCI-SIG ECOSYSTEM OR ANY PORTION
THEREOF (FOR PURPOSES OF THIS NOTICE, “PCI-SIG ECOSYSTEM” MEANS THE PCI-SIG SPECIFICATIONS, MEMBERS OF PCI-SIG
AND THEIR ASSOCIATED PRODUCTS AND SERVICES THAT INCORPORATE ALL OR A PORTION OF A PCI-SIG SPECIFICATION AND
EXTENDS TO THOSE PRODUCTS AND SERVICES INTERFACING WITH PCI-SIG MEMBER PRODUCTS AND SERVICES).
C6124.0003 BN/FCURCI 39895222v1
- 1 -
Evaluation Copy
Contents
Contents
1.0
2.0
3.0
2.3
2.4
Introduction ............................................................................................................................................................................................27
Audience ....................................................................................................................................................................................27
1.1
Terminology / Acronyms ....................................................................................................................................................27
1.2
1.3
Reference Documents..........................................................................................................................................................30
Motivation and Overview....................................................................................................................................................30
1.4
Compute Express Link.......................................................................................................................................30
1.4.1
1.4.2
Flex Bus....................................................................................................................................................................32
Flex Bus Link Features .........................................................................................................................................................35
1.5
Flex Bus Layering Overview...............................................................................................................................................35
1.6
1.7
Document Scope....................................................................................................................................................................37
Compute Express Link System Architecture...........................................................................................................................39
Type 1 CXL Device.................................................................................................................................................................39
2.1
2.2
Type 2 CXL Device.................................................................................................................................................................40
Bias Based Coherency Model .........................................................................................................................41
2.2.1
Host Bias ...........................................................................................................................................42
2.2.1.1
Device Bias .......................................................................................................................................42
2.2.1.2
Mode Management.......................................................................................................................43
2.2.1.3
2.2.1.4
Software Assisted Bias Mode Management.......................................................................43
Hardware Autonomous Bias Mode Management............................................................43
2.2.1.5
Type 3 CXL Device.................................................................................................................................................................44
Multi Logical Device ..............................................................................................................................................................44
2.4.1
LD-ID for CXL.io and CXL.mem......................................................................................................................45
LD-ID for CXL.mem.......................................................................................................................45
2.4.1.1
2.4.1.2
LD-ID for CXL.io .............................................................................................................................45
2.4.2
Pooled Memory Device Configuration Registers ...................................................................................45
2.5
CXL Device Scaling ................................................................................................................................................................47
Compute Express Link Transaction Layer................................................................................................................................48
CXL.io...........................................................................................................................................................................................48
3.1
CXL.io Endpoint....................................................................................................................................................49
3.1.1
CXL Power Management VDM Format .......................................................................................................50
3.1.2
3.1.2.1
Credit and PM Initialization.......................................................................................................54
3.1.3
CXL Error VDM Format ......................................................................................................................................55
3.1.4
Optional PCIe Features Required for CXL.................................................................................................56
3.1.5
Error Propagation................................................................................................................................................56
3.1.6
Memory Type Indication on ATS...................................................................................................................56
3.1.7
Deferrable Writes.................................................................................................................................................57
CXL.cache ..................................................................................................................................................................................58
Overview..................................................................................................................................................................58
3.2.1
CXL.cache Channel Description.....................................................................................................................59
3.2.2
3.2.2.1
Channel Ordering..........................................................................................................................59
3.2.2.2
Channel Crediting .........................................................................................................................59
CXL.cache Wire Description ............................................................................................................................60
D2H Request ...................................................................................................................................60
3.2.3.1
D2H Response................................................................................................................................61
3.2.3.2
3.2.3.3
D2H Data...........................................................................................................................................61
H2D Request ...................................................................................................................................62
3.2.3.4
H2D Response................................................................................................................................62
3.2.3.5
3.2.3.6
H2D Data...........................................................................................................................................63
CXL.cache Transaction Description .............................................................................................................63
3.2.4.1
Device to Host Requests ............................................................................................................63
Device to Host Response ...........................................................................................................74
3.2.4.2
3.2
3.2.3
3.2.4
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
3
Evaluation Copy
Contents
3.3
3.4
3.5
3.2.5
Host to Device Requests ............................................................................................................76
3.2.4.3
3.2.4.4
Host to Device Response ...........................................................................................................77
Cacheability Details and Request Restrictions........................................................................................79
3.2.5.1
GO-M Responses...........................................................................................................................79
Device/Host Snoop-GO-Data Assumptions ......................................................................79
3.2.5.2
Device/Host Snoop/WritePull Assumptions .....................................................................79
3.2.5.3
Snoop Responses and Data Transfer on CXL.cache Evicts.........................................80
3.2.5.4
Multiple Snoops to the Same Address ................................................................................80
3.2.5.5
3.2.5.6
Multiple Reads to the Same Cache Line..............................................................................80
Multiple Evicts to the Same Cache Line ..............................................................................80
3.2.5.7
Multiple Write Requests to the Same Cache Line...........................................................80
3.2.5.8
Normal Global Observation (GO)............................................................................................80
3.2.5.9
3.2.5.10
Relaxed Global Observation (FastGO)..................................................................................81
3.2.5.11
Evict to Device-Attached Memory .........................................................................................81
3.2.5.12 Memory Type on CXL.cache.....................................................................................................81
3.2.5.13 General Assumptions ..................................................................................................................81
3.2.5.14
Buried Cache State Rules...........................................................................................................82
CXL.mem....................................................................................................................................................................................83
3.3.1
Introduction ...........................................................................................................................................................83
QoS Telemetry for Memory.............................................................................................................................84
3.3.2
QoS Telemetry Overview...........................................................................................................84
3.3.2.1
Reference Model for Host Support of QoS Telemetry..................................................85
3.3.2.2
3.3.2.3
Memory Device Support for QoS Telemetry.....................................................................86
3.3.3
M2S Request (Req)..............................................................................................................................................94
3.3.4
M2S Request with Data (RwD)........................................................................................................................97
3.3.5
S2M No Data Response (NDR)........................................................................................................................98
3.3.6
S2M Data Response (DRS) ...............................................................................................................................99
3.3.7
Forward Progress and Ordering Rules .................................................................................................... 100
Transaction Ordering Summary ...................................................................................................................................101
Transaction Flows to Device-Attached Memory.................................................................................................... 103
Flows for Type 1 and Type 2 Devices ...................................................................................................... 103
3.5.1
Notes and Assumptions ..........................................................................................................103
3.5.1.1
3.5.1.2
Requests from Host...................................................................................................................104
3.5.1.3
Requests from Device in Host and Device Bias............................................................. 109
Type 2 and Type 3 Memory Flows ............................................................................................................ 112
3.5.2.1
Speculative Memory Read...................................................................................................... 112
Flows for Type 3 Devices .................................................................................................................................................113
3.6
Compute Express Link Link Layers...........................................................................................................................................115
CXL.io Link Layer .................................................................................................................................................................115
4.1
CXL.mem and CXL.cache Common Link Layer....................................................................................................... 117
4.2
4.2.1
Introduction ........................................................................................................................................................117
High-Level CXL.cache/CXL.mem Flit Overview ...................................................................................119
4.2.2
Slot Format Definition .................................................................................................................................... 124
4.2.3
H2D and M2S Formats............................................................................................................. 125
4.2.3.1
4.2.3.2
D2H and S2M Formats............................................................................................................. 131
Link Layer Registers.........................................................................................................................................139
Flit Packing Rules.............................................................................................................................................. 139
Link Layer Control Flit..................................................................................................................................... 141
Link Layer Initialization...................................................................................................................................145
CXL.cache/CXL.mem Link Layer Retry..................................................................................................... 146
LLR Variables................................................................................................................................147
4.2.8.1
LLCRD Forcing ............................................................................................................................. 149
4.2.8.2
LLR Control Flits .........................................................................................................................151
4.2.8.3
RETRY Framing Sequences.................................................................................................... 151
4.2.8.4
4.2.8.5
LLR State Machines ...................................................................................................................152
Interaction with Physical Layer Reinitialization............................................................. 156
4.2.8.6
4.2.4
4.2.5
4.2.6
4.2.7
4.2.8
3.5.2
4.0
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
4
Evaluation Copy
Contents
5.0
6.0
7.0
5.2
6.3
7.2
5.1.3
4.2.9
4.2.8.7
CXL.cache/CXL.mem Flit CRC ............................................................................................... 157
Poison and Viral ................................................................................................................................................158
4.2.9.1
Viral ..................................................................................................................................................158
Compute Express Link ARB/MUX.............................................................................................................................................. 160
Virtual LSM States............................................................................................................................................................... 161
5.1
Additional Rules for Local vLSM Transitions ........................................................................................ 164
5.1.1
Rules for Virtual LSM State Transitions Across Link..........................................................................164
5.1.2
5.1.2.1
General Rules............................................................................................................................... 164
Entry to Active Exchange Protocol ..................................................................................... 164
5.1.2.2
Status Synchronization Protocol.........................................................................................165
5.1.2.3
State Request ALMP..................................................................................................................167
5.1.2.4
5.1.2.5
State Status ALMP ..................................................................................................................... 169
5.1.2.6
Unexpected ALMPs ...................................................................................................................171
Applications of the vLSM State Transition Rules................................................................................ 172
Initial Link Training .................................................................................................................... 172
5.1.3.1
Status Exchange Snapshot Example..................................................................................175
5.1.3.2
5.1.3.3
L1 Abort Example....................................................................................................................... 176
ARB/MUX Link Management Packets.........................................................................................................................177
5.2.1
ARB/MUX Bypass Feature............................................................................................................................. 178
5.3
Arbitration and Data Multiplexing/Demultiplexing .............................................................................................. 179
Flex Bus Physical Layer ..................................................................................................................................................................180
Overview .................................................................................................................................................................................180
6.1
Flex Bus.CXL Framing and Packet Layout.................................................................................................................181
6.2
Ordered Set Blocks and Data Blocks........................................................................................................181
6.2.1
Protocol ID[15:0]............................................................................................................................................... 182
6.2.2
6.2.3
x16 Packet Layout ............................................................................................................................................ 183
x8 Packet Layout............................................................................................................................................... 184
6.2.4
x4 Packet Layout............................................................................................................................................... 187
6.2.5
x2 Packet Layout............................................................................................................................................... 187
6.2.6
x1 Packet Layout............................................................................................................................................... 187
6.2.7
6.2.8
Special Case: CXL.io -- When a TLP Ends on a Flit Boundary........................................................ 187
6.2.9
Framing Errors ...................................................................................................................................................188
Link Training..........................................................................................................................................................................190
PCIe vs Flex Bus.CXL Mode Selection...................................................................................................... 190
6.3.1
6.3.1.1
Hardware Autonomous Mode Negotiation..................................................................... 190
CXL 2.0 Versus CXL 1.1 Negotiation ..................................................................................194
6.3.1.2
Flex Bus.CXL Negotiation with Maximum Supported Link
6.3.1.3
Speed of 8GT/s or 16GT/s ..................................................................................................... 196
Link Width Degradation and Speed Downgrade..........................................................197
Recovery.Idle and Config.Idle Transitions to L0 .................................................................................................... 197
L1 Abort Scenario ............................................................................................................................................................... 197
Exit from Recovery.............................................................................................................................................................. 197
Retimers and Low Latency Mode .................................................................................................................................197
6.7.1
SKP Ordered Set Frequency and L1/Recovery Entry........................................................................ 198
Switching...............................................................................................................................................................................................201
Overview .................................................................................................................................................................................201
7.1
Single VCS Switch............................................................................................................................................. 201
7.1.1
7.1.2
Multiple VCS Switch ........................................................................................................................................202
7.1.3
Multiple VCS Switch with MLD Ports........................................................................................................203
Switch Configuration and Composition..................................................................................................................... 204
CXL Switch Initialization Options............................................................................................................... 204
7.2.1
Static Initialization ..................................................................................................................... 204
7.2.1.1
7.2.1.2
Fabric Manager Boots First .................................................................................................... 205
Fabric Manager and Host Boot Simultaneously............................................................207
7.2.1.3
6.4
6.5
6.6
6.7
6.3.1.4
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
5
Evaluation Copy
Contents
7.3
7.4
7.5
7.6
7.2.4
7.3.2
7.3.3
7.2.2
7.2.3
Sideband Signal Operation ..........................................................................................................................208
Binding and Unbinding...................................................................................................................................209
7.2.3.1
Binding and Unbinding of a Single Logical Device Port ............................................209
7.2.3.2
Binding and Unbinding of a Pooled Device....................................................................211
PPB and vPPB Behavior for MLD Ports ...................................................................................................214
MLD Type 1 Configuration Space Header ....................................................................... 215
7.2.4.1
MLD PCI-Compatible Configuration Registers .............................................................. 215
7.2.4.2
MLD PCI Express Capability Structure .............................................................................. 215
7.2.4.3
7.2.4.4
MLD PPB Secondary PCI Express Capability Structure.............................................. 218
MLD Physical Layer 16.0 GT/s Extended Capability ...................................................219
7.2.4.5
MLD Physical Layer 32.0 GT/s Extended Capability ...................................................219
7.2.4.6
7.2.4.7
MLD Lane Margining at the Receiver Extended Capability....................................... 220
MLD ACS Extended Capability .................................................................................................................... 220
7.2.5
MLD PCIe Extended Capabilities................................................................................................................220
7.2.6
MLD Advanced Error Reporting Extended Capability....................................................................... 220
7.2.7
7.2.8
MLD DPC Extended Capability.................................................................................................................... 222
CXL.io, CXL.cache/CXL.mem Decode and Forwarding........................................................................................ 222
7.3.1
CXL.io..................................................................................................................................................................... 222
CXL.io Decode.............................................................................................................................. 222
7.3.1.1
7.3.1.2
CXL 1.1 Support..........................................................................................................................223
CXL.cache............................................................................................................................................................. 223
CXL.mem .............................................................................................................................................................. 223
7.3.3.1
CXL.mem Request Decode..................................................................................................... 223
CXL.mem Response Decode .................................................................................................224
7.3.3.2
7.3.3.3
QoS Message Aggregation..................................................................................................... 224
7.3.4
FM Owned PPB CXL Handling..................................................................................................................... 224
CXL Switch PM...................................................................................................................................................................... 224
CXL Switch ASPM L1 ....................................................................................................................................... 224
7.4.1
CXL Switch PCI-PM and L2 ...........................................................................................................................224
7.4.2
7.4.3
CXL Switch Message Management............................................................................................................ 224
CXL Switch RAS.................................................................................................................................................................... 226
Fabric Manager Application Programming Interface...........................................................................................226
CXL Fabric Management................................................................................................................................226
7.6.1
Fabric Management Model...........................................................................................................................227
7.6.2
FM Command Transport Protocol............................................................................................................. 228
7.6.3
7.6.4
CXL Switch Management............................................................................................................................... 229
Initial Configuration...................................................................................................................229
7.6.4.1
Dynamic Configuration............................................................................................................ 229
7.6.4.2
7.6.4.3
MLD Port Management............................................................................................................ 229
MLD Component Management...................................................................................................................230
Management Requirements for System Operations.........................................................................230
Initial System Discovery ..........................................................................................................230
7.6.6.1
CXL Switch Discovery............................................................................................................... 231
7.6.6.2
MLD and Switch MLD Port Management .........................................................................231
7.6.6.3
Event Notifications .................................................................................................................... 231
7.6.6.4
7.6.6.5
Binding Ports and LDs on a Switch..................................................................................... 231
Unbinding Ports and LDs on a Switch............................................................................... 232
7.6.6.6
Hot-Add and Managed Hot-Removal of Devices .........................................................232
7.6.6.7
7.6.6.8
Surprise Removal of Devices.................................................................................................233
Fabric Management Application Programming Interface ............................................................... 233
7.6.7.1
Switch Event Notifications Command Set....................................................................... 234
Virtual Switch Command Set ................................................................................................ 240
7.6.7.2
Unbind vPPB (Opcode 5202h) ............................................................................................. 242
7.6.7.3
MLD Port Command Set..........................................................................................................243
7.6.7.4
7.6.7.5
MLD Component Command Set..........................................................................................246
Fabric Management Event Records ..........................................................................................................252
Physical Switch Event Records............................................................................................. 252
7.6.8.1
7.6.5
7.6.6
7.6.7
7.6.8
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
6
Evaluation Copy
Contents
8.0
8.1.11.1
8.2
8.1.4
8.1.5
7.6.8.2
7.6.8.3
Virtual CXL Switch Event Records....................................................................................... 253
MLD Port Event Records .........................................................................................................254
Control and Status Registers....................................................................................................................................................... 256
Configuration Space Registers ...................................................................................................................................... 257
8.1
PCI Express Designated Vendor-Specific Extended Capability (DVSEC) ID Assignment...257
8.1.1
CXL Data Object Exchange (DOE) Type Assignment.........................................................................258
8.1.2
PCIe DVSEC for CXL Device..........................................................................................................................258
8.1.3
8.1.3.1
DVSEC CXL Capability (Offset 0Ah)....................................................................................260
DVSEC CXL Control (Offset 0Ch).........................................................................................261
8.1.3.2
DVSEC CXL Status (Offset 0Eh)............................................................................................ 262
8.1.3.3
DVSEC CXL Control2 (Offset 10h) ...................................................................................... 262
8.1.3.4
DVSEC CXL Status2 (Offset 12h).........................................................................................262
8.1.3.5
8.1.3.6
DVSEC CXL Lock (Offset 14h)............................................................................................... 263
DVSEC CXL Capability2 (Offset 16h) .................................................................................263
8.1.3.7
8.1.3.8
DVSEC CXL Range registers...................................................................................................263
Non-CXL Function Map DVSEC ..................................................................................................................268
Non-CXL Function Map Register 0 (Offset 0Ch) ...........................................................269
8.1.4.1
8.1.4.2
Non-CXL Function Map Register 1 (Offset 10h)............................................................269
Non-CXL Function Map Register 2 (Offset 14h)............................................................270
8.1.4.3
Non-CXL Function Map Register 3(Offset 18h)............................................................. 270
8.1.4.4
Non-CXL Function Map Register 4 (Offset 1Ch) ...........................................................270
8.1.4.5
Non-CXL Function Map Register 5 (Offset 20h)............................................................270
8.1.4.6
8.1.4.7
Non-CXL Function Map Register 6 (Offset 24h)............................................................271
8.1.4.8
Non-CXL Function Map Register 7(Offset 28h)............................................................. 271
CXL 2.0 Extensions DVSEC for Ports........................................................................................................271
CXL Port Extension Status (Offset 0Ah) ...........................................................................272
8.1.5.1
8.1.5.2
Port Control Extensions (Offset 0Ch) ................................................................................ 273
Alternate Bus Base (Offset 0E........................................................................................... h)274
8.1.5.3
Alternate Bus Limit (Offset 0Fh) ..........................................................................................274
8.1.5.4
Alternate Memory Base (Offset 10h) .................................................................................274
8.1.5.5
Alternate Memory Limit (Offset 12h).................................................................................274
8.1.5.6
8.1.5.7
Alternate Prefetchable Memory Base (Offset 14h)...................................................... 275
Alternate Prefetchable Memory Limit (Offset 16h) ..................................................... 275
8.1.5.8
8.1.5.9
Alternate Memory Prefetchable Base High (Offset 18h)...........................................275
8.1.5.10 Alternate Prefetchable Memory Limit High (Offset 1Ch) ..........................................275
CXL RCRB Base (Offset 20h)..................................................................................................276
8.1.5.11
8.1.5.12
CXL RCRB Base High (Offset 24h) ....................................................................................... 276
GPF DVSEC for CXL Port................................................................................................................................276
GPF Phase 1 Control (Offset 0Ch)....................................................................................... 277
8.1.6.1
8.1.6.2
GPF Phase 2 Control (Offset 0Eh)....................................................................................... 277
GPF DVSEC for CXL Device...........................................................................................................................278
8.1.7.1
GPF Phase 2 Duration (Offset 0Ah) ....................................................................................279
8.1.7.2
GPF Phase 2 Power (Offset 0Ch) .........................................................................................279
PCIe DVSEC for Flex Bus Port...................................................................................................................... 279
Register Locator DVSEC.................................................................................................................................279
Register Offset Low (Offset Varies)..................................................................................... 281
8.1.9.1
8.1.9.2
Register Offset High (Offset Varies)....................................................................................281
8.1.10 MLD DVSEC .........................................................................................................................................................281
8.1.10.1 Number of LD Supported (Offset 0Ah)............................................................................. 282
8.1.10.2
FLR LD-ID Hot Reset Vector (Offset 0Ch).........................................................................282
8.1.11 Table Access DOE............................................................................................................................................. 282
Read Entry..................................................................................................................................... 283
8.1.12 Memory Device Configuration Space Layout ....................................................................................... 284
8.1.12.1
PCI Header - Class Code Register (Offset 09h).............................................................. 284
8.1.12.2 Memory Device PCIe Capabilities and Extended Capabilities.................................284
Memory Mapped Registers ............................................................................................................................................. 284
CXL 1.1 Upstream and Downstream Port Registers ..........................................................................286
8.2.1
8.1.8
8.1.9
8.1.6
8.1.7
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
7
Evaluation Copy
Contents
8.2.6
8.2.7
8.2.8
8.2.2
8.2.3
8.2.4
8.2.5
CXL 1.1 Downstream Port RCRB..........................................................................................286
8.2.1.1
CXL 1.1 Upstream Port RCRB................................................................................................ 288
8.2.1.2
8.2.1.3
Flex Bus Port DVSEC.................................................................................................................290
CXL 1.1 Upstream and Downstream Port Subsystem Component Registers ........................ 293
CXL 2.0 Component Registers .................................................................................................................... 294
Component Register Layout and Definition..........................................................................................294
CXL.cache and CXL.mem Registers...........................................................................................................294
CXL Capability Header Register (Offset 0x0)..................................................................296
8.2.5.1
8.2.5.2
CXL RAS Capability Header (Offset: Varies)....................................................................297
CXL Security Capability Header (Offset: Varies)............................................................297
8.2.5.3
CXL Link Capability Header (Offset:Varies)..................................................................... 297
8.2.5.4
CXL HDM Decoder Capability Header (Offset: Varies)................................................ 297
8.2.5.5
CXL Extended Security Capability Header (Offset: Varies)....................................... 298
8.2.5.6
8.2.5.7
CXL IDE Capability Header (Offset: Varies)...................................................................... 298
CXL Snoop Filter Capability Header (Offset: Varies)...................................................298
8.2.5.8
CXL RAS Capability Structure ............................................................................................... 298
8.2.5.9
CXL Security Capability Structure....................................................................................... 302
8.2.5.10
CXL Link Capability Structure ............................................................................................... 303
8.2.5.11
8.2.5.12
CXL HDM Decoder Capability Structure...........................................................................307
CXL Extended Security Capability Structure..................................................................319
8.2.5.13
CXL IDE Capability Structure.................................................................................................320
8.2.5.14
8.2.5.15
CXL Snoop Filter Capability Structure .............................................................................. 322
CXL ARB/MUX Registers ................................................................................................................................323
ARB/MUX Arbitration Control Register for CXL.io (Offset 0x180) ........................ 323
8.2.6.1
ARB/MUX Arbitration Control Register for CXL.cache and CXL.mem (Offset
8.2.6.2
0x1C0)............................................................................................................................................. 323
BAR Virtualization ACL Register Block..................................................................................................... 323
8.2.7.1
BAR Virtualization ACL Size Register (Offset 00h)....................................................... 324
CXL Device Register Interface ..................................................................................................................... 325
CXL Device Capabilities Array Register (Offset 00h)...................................................326
8.2.8.1
CXL Device Capability Header Register (Offset Varies).............................................. 326
8.2.8.2
Device Status Registers (Offset Varies)............................................................................. 327
8.2.8.3
8.2.8.4
Mailbox Registers (Offset Varies).........................................................................................327
8.2.8.5
Memory Device Registers ....................................................................................................... 333
CXL Device Command Interface.................................................................................................................335
Events.............................................................................................................................................. 336
8.2.9.1
Firmware Update........................................................................................................................347
8.2.9.2
8.2.9.3
Timestamp .................................................................................................................................... 351
Logs..................................................................................................................................................352
8.2.9.4
Memory Device Commands...................................................................................................355
8.2.9.5
8.2.9.6
FM API Commands .................................................................................................................... 386
Reset, Initialization, Configuration and Manageability ..................................................................................................388
Compute Express Link Boot and Reset Overview .................................................................................................388
9.1
General ..................................................................................................................................................................388
9.1.1
Comparing CXL and PCIe Behavior...........................................................................................................389
9.1.2
9.1.2.1
Switch Behavior ..........................................................................................................................389
Compute Express Link Device Boot Flow .................................................................................................................391
Compute Express Link System Reset Entry Flow..................................................................................................391
Compute Express Link Device Sleep State Entry Flow ....................................................................................... 392
Function Level Reset (FLR) .............................................................................................................................................. 393
Cache Management............................................................................................................................................................ 394
CXL Reset................................................................................................................................................................................394
Effect on the Contents of the Volatile HDM..........................................................................................396
9.7.1
9.7.2
Software Actions............................................................................................................................................... 396
Global Persistent Flush (GPF) ........................................................................................................................................397
9.8.1
Host and Switch Responsibilities............................................................................................................... 397
Device Responsibilities...................................................................................................................................398
9.8.2
9.2
9.3
9.4
9.5
9.6
9.7
8.2.9
9.0
9.8
October 26, 2020
Revision 2.0, Version 1.0
Compute Express Link Specification
8
Evaluation Copy