第1页 / 共1842页
第2页 / 共1842页
第3页 / 共1842页
第4页 / 共1842页
第5页 / 共1842页
第6页 / 共1842页
第7页 / 共1842页
第8页 / 共1842页
IB Specification Vol 1-Release-1.3-2015-03-03.pdf
Table of Contents
Chapter 1: Introduction 66
Chapter 2: Glossary 76
Chapter 3: Architectural Overview 93
Chapter 4: Addressing 150
Chapter 5: Data Packet Format 159
Chapter 6: Physical Layer Interface 174
Chapter 7: Link Layer 178
Chapter 8: Network Layer 234
Chapter 9: Transport Layer 242
Chapter 10: Software Transport Interface 449
Chapter 11: Software Transport Verbs 572
Chapter 12: Communication Management 696
Chapter 13: Management Model 761
Chapter 14: Subnet Management 846
Chapter 15: Subnet Administration 959
Chapter 16: General Services 1019
Chapter 17: Channel Adapters 1117
Chapter 18: Switches 1139
Chapter 19: Routers 1158
Chapter 20: Volume 1 Compliance Summary 1171
Annex A1: I/O Infrastructure 1222
Annex A2: Console Service Protocol 1241
Annex A3: Application Specific Identifiers 1274
Annex A4: Sockets Direct Protocol (SDP) 1296
Annex A5: Booting Annex 1371
Annex A6: Boot Information Service 1503
Annex A7: Configuration Management 1536
Annex A8: Device Management 1607
Annex A9: Verb Extensions Annex 1729
Annex A10: Congestion Control 1730
Annex A11: RDMA IP CM Service 1781
Annex A12: Support for iSCSI Extensions for RDMA (iSER) 1790
Annex A13: Quality of Service (QoS) 1796
Annex A14: Extended Reliable Connected (XRC) Transport Service 1819
Annex A15: Hierarchy Information 1820
List of Figures
List of Tables
Chapter 1: Introduction
1.1 Acknowledgments
1.2 InfiniBand Conceptual Overview
1.2.1 The Problem
1.2.2 Features
1.2.3 Benefits
1.3 Scope
1.4 Document Organization
1.4.1 Series of Volumes
1.4.2 Volume 1 Organization
1.5 Document Conventions
1.5.1 Byte Ordering
1.5.2 Numeric Values
1.6 Disclaimer
Chapter 2: Glossary
Chapter 3: Architectural Overview
3.1 Architecture Scope
3.1.1 Topologies & Components
3.2 Communication
3.2.1 Queuing
3.2.2 Connections
3.3 Communications Stack
3.4 IBA Components
3.4.1 Links & Repeaters
3.4.2 Channel Adapters
3.4.3 Switches
3.4.4 Routers
3.4.5 Management Components
3.4.5.1 Subnet Managers
3.4.5.2 Subnet Management Agents
3.4.5.3 General Service Agents
3.5 IBA Features
3.5.1 Queue Pairs
3.5.2 Types of Service
3.5.3 Keys
3.5.4 Virtual Memory Addresses
3.5.5 Protection Domains
3.5.6 Partitions
3.5.7 Virtual Lanes
3.5.8 Quality of Service
3.5.8.1 Service Level
3.5.8.2 SL to VL mapping
3.5.8.3 Partitions
3.5.9 Injection Rate Control
3.5.10 Addressing
3.5.11 Multicast
3.5.11.1 Multicast Example
3.5.11.2 Group Management
3.5.11.2.1 Multicast Group Create
3.5.11.2.2 Multicast Group Join
3.5.11.2.3 Multicast Group Leave
3.5.11.2.4 Multicast Group Delete
3.5.11.3 Multicast Prune
3.5.12 Verbs
3.6 Channel & Memory Semantics
3.6.1 Communication Interface
3.6.2 IBA Transport Services
3.7 IBA Layered Architecture
3.7.1 Physical Layer
3.7.2 Link Layer
3.7.3 Network Layer
3.7.4 Transport Layer
3.7.5 Upper Layer Protocols
3.7.5.1 Subnet Management
3.7.5.2 General Services
3.8 IBA Transaction Flow
3.9 IBA Management Infrastructure
3.9.1 Management Datagrams
3.9.2 Management Methods
3.9.2.1 Gets & Sets
3.9.2.2 Traps and Notices
3.9.2.3 Sends
3.9.2.4 Reports
3.9.3 Management Interfaces
3.9.4 Subnet Management Interface
3.9.4.1 Fabric Initialization
3.9.4.2 Directed Routes
3.9.5 General Service Interface
3.9.5.1 Redirection
3.10 I/O Operation
Chapter 4: Addressing
4.1 Terminology And Concepts
4.1.1 GID Usage and Properties
4.1.2 Channel Adapter, Switch, and Router Addressing Rules
4.1.3 Local Identifiers
Chapter 5: Data Packet Format
5.1 Packet Types
5.2 Data Packet Format
5.2.1 Local Route Header (LRH) - 8 Bytes
5.2.2 Global Route Header (GRH) - 40 Bytes
5.2.3 Base Transport Header (BTH) - 12 Bytes
5.2.4 Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes
5.2.5 Datagram Extended Transport Header (DETH) - 8 Bytes
5.2.6 RDMA Extended Transport Header (RETH) - 16 Bytes
5.2.7 Atomic Extended Transport Header (AtomicETH) - 28 Bytes
5.2.8 XRC Extended Transport Header (XRCETH)
5.2.9 ACK Extended Transport Header (AETH) - 4 Bytes
5.2.10 Atomic ACK Extended Transport Header (AtomicAckETH) - 8 Bytes
5.2.11 Immediate Data Extended Transport Header (ImmDt) - 4 Bytes
5.2.12 INVALIDATE EXTENDED TRANSPORT HEADER (IETH) - 4 BYTES
5.2.13 Payload
5.2.14 Invariant CRC
5.2.15 Variant CRC
5.3 Raw Packet Format
5.4 Packet Examples
Chapter 6: Physical Layer Interface
6.1 Overview
6.2 Services provided by the Physical Layer.
6.3 Interface between physical and Link Layers.
6.3.1 Interface between physical receive and link receive.
6.3.1.1 Phy_link - Physical Link Status
6.3.1.2 L_Init_Train - Link Initiate Retraining
6.3.1.3 rcv_stream - Receive Stream
6.3.2 Interface between physical Transmit and link Transmit.
6.3.2.1 Xmit_stream - Transmit Stream
6.3.2.2 Xmit_Ready - Physical Transmitter Ready
Chapter 7: Link Layer
7.1 Overview
7.1.1 State Machine Conventions
7.2 Link States
7.2.1 LinkDown State
7.2.2 LinkInitialize State
7.2.3 LinkArm State
7.2.4 LinkActive State
7.2.5 LinkActDefer State
7.2.6 Management State Change Commands
7.2.7 State Machine Terms
7.3 Packet Receiver States
7.4 Data Packet Check
7.5 Link Packet Check
7.6 Virtual Lanes Mechanisms
7.6.1 VL identification
7.6.2 Number of VLs supported
7.6.3 Special VLs
7.6.4 Buffering and Flow Control For Data VLs
7.6.5 Service Level
7.6.6 VL Mapping Within a Subnet
7.6.7 Initialization and Configuration
7.6.8 VL Scheduling and Flow Control For VL15 and Flow Control Packets
7.6.9 VL Arbitration and Prioritization
7.6.9.1 VL Arbitration When Only One Data VL Is Implemented
7.6.9.2 VL Arbitration When Multiple Data VL s Are Implemented
7.6.9.2.1 Arbitration Rules Between VL15, Link Control and Data VL Packets
7.6.9.2.2 Arbitration Rules for Data VL Packets
7.6.9.2.3 Arbitration Rules Between High and Low Priority Components
7.6.9.2.4 Arbitration Rules Within the High and Low components
7.7 Local Route Header
7.7.1 Virtual Lane (VL) - 4 bits
7.7.2 Link Version (LVer) - 4 bits
7.7.3 Service Level (SL) - 4 bits
7.7.4 Reserve - 2 bits
7.7.5 Link Next Header (LNH) - 2 bits
7.7.6 Destination Local Identifier (DLID) - 16 bits
7.7.7 Reserve - 5 bits
7.7.8 Packet Length (PktLen) - 11 bits
7.7.9 Source Local Identifier (SLID) - 16 bits
7.8 CRCs
7.8.1 Invariant CRC (ICRC) - 4 Bytes
7.8.2 Variant CRC (VCRC) - 2 Bytes
7.8.3 Link Packet CRC (LPCRC) - 2 Bytes
7.8.4 CRC Calculation Samples
7.8.4.1 ICRC Generator
7.8.4.2 VCRC Generator
7.8.4.3 Sample Packets
7.8.4.3.1 Local Packet Example
7.8.4.3.2 Global Packet Example
7.8.4.3.3 Link Packet Example
7.9 Flow Control
7.9.1 Introduction
7.9.2 Flow Control Blocks
7.9.3 Relationship to Virtual Lanes
7.9.4 Flow Control Packet
7.9.4.1 Flow Control Packet Fields
7.9.4.1.1 Operand (Op) - 4 Bits
7.9.4.1.2 Flow Control Total Blocks Sent (FCTBS) - 12 Bits
7.9.4.1.3 Flow Control Credit Limit (FCCL) -12 Bits
7.9.4.1.4 Virtual Lane (VL) - 4 Bits
7.9.4.1.5 Link Packet Cyclic Redundancy Check (LPCRC) - 16 Bits
7.9.4.2 Calculation of FCTBS
7.9.4.3 Calculation of FCCL
7.9.4.4 Transmission of Packets
7.10 IBA and Raw Packet Multicast
7.10.1 Overview
7.10.2 IBA Unreliable Multicast Operational Rules
7.10.3 Raw Packet Multicast
7.10.3.1 Raw Multicast Operational Rules
7.10.4 Group Management
7.11 Subnet Multipathing
7.11.1 Multipathing Requirements on end node
7.12 Error detection and handling
7.12.1 Error Detection
7.12.2 Error Recovery Procedures
7.12.3 Error Notification
Chapter 8: Network Layer
8.1 Overview
8.2 Packet Routing
8.2.1 Overview
8.2.2 Global Fabric Characteristics
8.2.2.1 Inheritance of Subnet Requirements
8.2.2.2 Packet Errors and Error Detection
8.2.2.3 Service Levels
8.2.3 Support for Multiple Global Paths
8.2.4 Global Multicast
8.3 Global Route Header
8.3.1 IP Version (IPVer) - 4 bits
8.3.2 Traffic Class (TClass) - 8 bits
8.3.3 Flow Label (FlowLabel) - 20 bits
8.3.4 Payload Length (PayLen) - 16 bits
8.3.5 Next Header (NxtHdr) - 8 bits
8.3.6 Hop Limit (HopLmt) - 8 bits
8.3.7 Source Global Identifier (SGID) - 128 bits
8.3.8 Destination Global Identifier (DGID) - 128 bits
8.4 Global Route Header Usage
8.4.1 Global Route Header Generation
8.4.2 Global Route Header Modification
8.4.3 Global Route Header Verification
Chapter 9: Transport Layer
9.1 Overview
9.2 Base Transport Header
9.2.1 Operation Code (OpCode)
9.2.2 Reserved Transport Function OpCodes
9.2.3 Solicited Event (SE) - 1 bit
9.2.4 MigReq (M) - 1 Bit
9.2.5 Pad Count (PadCnt) - 2 bits
9.2.6 Transport Header Version (TVer) - 4 bits
9.2.7 Partition Key (P_Key) - 16 bits
9.2.8 Destination QP (DestQP) - 24 bits
9.2.9 FECN/Res1 (F/Res1) - 1 bit
9.2.10 BECN/Res1 (B/Res1) - 1 bit
9.2.11 Reserve 6(Resv6) - 6 bits
9.2.12 AckReq (A) - 1 Bit
9.2.13 Reserved 7 (resv7) - 7 bits
9.2.14 Packet Sequence Number (PSN) - 24 bits
9.3 Extended Transport Headers
9.3.1 Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes
9.3.1.1 Reserve - 8 bits
9.3.1.2 End-to-End (EE) Context - 24 bits
9.3.2 Datagram Extended Transport Header (DETH) - 8 Bytes
9.3.2.1 Q_Key - 32 bits
9.3.2.2 Reserve - 8 bits
9.3.2.3 Source QP (SrcQP) - 24 bits
9.3.3 RDMA Extended Transport Header (RETH) - 16 Bytes
9.3.3.1 Virtual Address (VA) - 64 bits
9.3.3.2 R_Key - 32 bits
9.3.3.3 DMA Length (DMAlen) - 32 bits
9.3.4 ATOMIC Extended Transport Header (AtomicETH) - 28 Bytes
9.3.4.1 Virtual Address (VA) - 64 bits
9.3.4.2 R_Key - 32 bits
9.3.4.3 Swap (Add) Data (SwapDt) - 64 bits
9.3.4.4 Compare Data (CmpDt) - 64 bits
9.3.5 ACK Extended Transport Header (AETH) - 4 Bytes
9.3.5.1 Syndrome
9.3.5.2 Message Sequence Number (MSN)
9.3.5.3 ATOMIC Acknowledge Extended Transport Header (AtomicAckETH) - 8 Bytes
9.3.5.4 Original Remote Data (OrigRemDt) - 64 bits
9.3.6 Immediate Extended Transport Header (ImmDt) - 4 Bytes
9.3.7 Invalidate Extended Transport Header (IETH) - 4 Bytes
9.3.7.1 R_Key - 32 bits
9.3.8 XRC Extended Transport Header (XRCETH)
9.3.8.1 Reserved - 8 bits
9.3.8.2 XRCSRQ - 24 bits
9.4 Transport Functions
9.4.1 SEND Operation
9.4.1.1 Send With Invalidate
9.4.1.1.1 Invalidate Operation Ordering
9.4.1.1.2 Responder - R_Key Validation
9.4.1.1.3 R_Key Validation for Remote Memory Invalidate
9.4.2 RESYNC Operation
9.4.3 RDMA WRITE Operation
9.4.4 RDMA READ Operation
9.4.5 ATOMIC Operations
9.4.5.1 Atomicity Guarantees
9.4.5.2 ATOMIC Acknowledgment Generation and Ordering Rules
9.4.5.3 Error Behavior
9.4.6 Reserved and Manufacturer Defined Transport Function OpCodes
9.5 Transaction Ordering
9.6 Packet Transport Header Validation
9.6.1 Validating Header Fields
9.6.1.1 BTH Checks
9.6.1.1.1 BTH:TVer Validation
9.6.1.1.2 BTH:Destination QP, OpCode Check
9.6.1.1.3 BTH:P_Key
9.6.1.2 GRH Checks
9.6.1.2.1 GRH:Next Header
9.6.1.2.2 GRH:IPVers
9.6.1.2.3 GRH:SGID, GRH:DGID
9.6.1.3 XRC Checks
9.6.1.4 RDETH Checks
9.6.1.4.1 RDETH:EE Context
9.6.1.5 DETH Checks
9.6.1.5.1 DETH:Q_Key
9.6.1.6 LRH Checks
9.6.1.6.1 LRH:SLID, LRH:DLID
9.6.1.6.2 IBA Unreliable Multicast Checks
9.7 Reliable Service
9.7.1 Packet Sequence Numbers (PSN)
9.7.1.1 PSN Model for Reliable Service
9.7.2 ACK/NAK Protocol
9.7.3 Requester: Generating Request Packets
9.7.3.1 Requester Side - Generating PSN
9.7.3.2 Requester - Special Rules for Reliable Datagram
9.7.3.2.1 RDD Checking
9.7.3.2.2 RESYNC Generation
9.7.3.3 Requester - Generating Opcodes
9.7.3.4 Requester - Generating Payloads
9.7.4 Responder: Receiving Inbound Request Packets
9.7.4.1 Responder - Inbound Packet Validation
9.7.4.1.1 Responder - Special Rules for Reliable Datagram Checking
9.7.4.1.2 Responder - PSN Verification
9.7.4.1.3 Responder - OpCode Sequence Check
9.7.4.1.4 Responder OpCode Validation
9.7.4.1.5 Responder R_Key Validation
9.7.4.1.6 Responder - Length Validation
9.7.4.1.7 Responder Local Operation Validation
9.7.5 Responder: Generating Responses
9.7.5.1 Responder Side Behavior
9.7.5.1.1 Generating PSNs for Acknowledge Messages
9.7.5.1.2 Coalesced Acknowledge Messages
9.7.5.1.3 Acknowledging RDMA READ Requests
9.7.5.1.4 Acknowledging Duplicate Requests
9.7.5.1.5 Generating NAKs
9.7.5.1.6 Acknowledge Message Scheduling
9.7.5.1.7 Response Formats
9.7.5.1.8 Response Format for SEND, RESYNC or RDMA WRITE Requests
9.7.5.1.9 RDMA READ Responses
9.7.5.2 AETH Format
9.7.5.2.1 End-to-End Flow Control Credit Field
9.7.5.2.2 NAK Codes
9.7.5.2.3 PSN Sequence Error
9.7.5.2.4 Remote Access Error
9.7.5.2.5 Invalid Request
9.7.5.2.6 Remote Operational Error
9.7.5.2.7 Invalid RD Request
9.7.5.2.8 RNR NAK
9.7.6 Requester: Receiving Responses
9.7.6.1 Validating Inbound Response Packets
9.7.6.1.1 PSNs for Retried Requests
9.7.6.1.2 Requester Response to a NAK Message
9.7.6.1.3 Detecting Lost Acknowledge Messages and Timeouts
9.7.6.1.4 Duplicate Acknowledgements
9.7.7 Reliable Connections
9.7.7.1 Generating MSN Value
9.7.7.1.1 Requester Behavior On Receiving a New MSN
9.7.7.2 End-to-End (Message Level) Flow Control
9.7.7.2.1 Transferring Credits from Responder to Requester
9.7.7.2.2 Negotiating Connections: Initial Credits
9.7.7.2.3 Responder Algorithm for Calculating Credits
9.7.7.2.4 Requester Behavior
9.7.7.2.5 Requester Behavior - Limited Send WQEs
9.7.8 Reliable Datagram
9.7.8.1 Reliable datagram Characteristics
9.7.8.2 Example RD Operations
9.7.8.2.1 Example Outbound Request
9.7.8.2.2 Example Inbound Request
9.7.8.2.3 Example Outbound Acknowledge
9.7.8.2.4 Example Inbound Acknowledge
9.7.8.3 Reliable Datagram Operations
9.7.8.3.1 SEND and RDMA WRITE with Immediate Data processing
9.7.8.3.2 RDMA READ processing
9.7.8.3.3 Atomics processing
9.7.8.4 Ordering Rules
9.7.8.5 Handling QP errors - RESYNC
9.7.8.6 Responder Generation of MSN
9.7.8.6.1 Requester Behavior On Receiving a New MSN
9.7.9 XRC
9.7.9.1 Introduction
9.7.9.2 XRC Packet Format Rules
9.8 Unreliable Service
9.8.1 Validating and Executing Requests
9.8.2 Unreliable Connections
9.8.2.1 Requester Behavior
9.8.2.1.1 Requester - Generating PSN
9.8.2.1.2 Requester - Generating Opcodes
9.8.2.1.3 Requester - Generating Payloads
9.8.2.1.4 Completing a Message Send or RDMA WRITE
9.8.2.2 Responder Behavior
9.8.2.2.1 Responder - Validating the PSN
9.8.2.2.2 Responder - OpCode Sequence Check
9.8.2.2.3 Responder OpCode Validation
9.8.2.2.4 Responder Remote Access Validation
9.8.2.2.5 Responder - Length Validation
9.8.2.2.6 Responder - Local Operation Validation
9.8.2.2.7 Completing a Message Receive
9.8.3 Unreliable Datagrams
9.8.3.1 Requester Behavior
9.8.3.1.1 Generating PSN
9.8.3.1.2 Completing a Message Send
9.8.3.2 Responder Behavior
9.8.3.2.1 Responder - Validating the PSN
9.8.3.2.2 Responder - Length Validation
9.8.3.2.3 Responder OpCode Validation
9.8.3.2.4 Responder - Local Operation Validation
9.8.3.2.5 Completing a Message Receive
9.8.4 Raw datagrams
9.8.4.1 Raw Datagram Packet Size
9.9 Error detection and handling
9.9.1 Reporting Errors to the Verbs Layer
9.9.2 Requester Side Error Behavior
9.9.2.1 Requester Side Error Detection - Locally Detected Errors
9.9.2.1.1 Requester Error Retry Counters
9.9.2.2 Requester Side Error Detection - Remotely Detected Errors
9.9.2.3 Summary - Requester Side Error Behavior
9.9.2.4 Requester Side Error Response
9.9.2.4.1 Requester Class A Fault Behavior
9.9.2.4.2 Requester Class B Fault Behavior
9.9.2.4.3 Requester Class C Fault Behavior
9.9.2.4.4 Requester Class D Fault Behavior
9.9.2.4.5 Requester Class E Fault Behavior
9.9.2.4.6 Requester Class F Fault Behavior
9.9.3 Responder Side Behavior
9.9.3.1 Responder Side Error Response
9.9.3.1.1 Responder Class A Fault Behavior
9.9.3.1.2 Responder Class B Fault Behavior
9.9.3.1.3 Responder Class C Fault Behavior
9.9.3.1.4 Responder Class D Fault Behavior
9.9.3.1.5 Responder Class D1 Fault Behavior
9.9.3.1.6 Responder Class E Fault Behavior
9.9.3.1.7 Responder Class F Fault Behavior
9.9.3.1.8 Responder Class G Fault Behavior
9.9.3.1.9 Responder Class H Fault Behavior
9.9.3.1.10 Responder Class J Fault Behavior
9.9.3.1.11 Responder Class K Fault Behavior
9.9.3.1.12 Responder Class L Fault Behavior
9.9.3.1.13 Responder Class M Fault Behavior
9.10 Header and Data Field Source
9.10.1 Field source when generating packets
9.10.2 Transport Connection Parameters
9.10.3 Packet Header and Data Field Validation
9.11 Static Rate Control
9.11.1 Static rate control for Heterogeneous Links
Chapter 10: Software Transport Interface
10.1 Overview
10.1.1 Introduction
10.2 Managing HCA Resources
10.2.1 HCA
10.2.1.1 Opening an HCA
10.2.1.2 HCA Attributes
10.2.1.3 Modifying HCA Attributes
10.2.1.4 Closing an HCA
10.2.2 Addressing
10.2.2.1 Source Addressing
10.2.2.2 Destination Addressing
10.2.2.3 Loopback
10.2.3 Protection Domains
10.2.3.1 Allocating a Protection Domain
10.2.3.2 Deallocating a Protection Domain
10.2.4 Queue Pairs
10.2.4.1 Creating a Queue Pair
10.2.4.2 Queue Pair Attributes
10.2.4.3 Modifying Queue Pair Attributes
10.2.4.4 Destroying a Queue Pair
10.2.4.5 Special QPs
10.2.5 Q_Keys
10.2.6 Completion Queues
10.2.6.1 Creating a Completion Queue
10.2.6.2 Completion Queue Attributes
10.2.6.3 Modifying Completion Queue Attributes
10.2.6.4 Destroying a Completion Queue
10.2.7 End-to-End Contexts
10.2.7.1 Creating an EE Context
10.2.7.2 EE Context Attributes
10.2.7.3 Modifying EE Context Attributes
10.2.7.4 Destroying an EE Context
10.2.8 Reliable Datagram Domains
10.2.8.1 Allocating A Reliable Datagram Domain
10.2.8.2 Deallocating A Reliable Datagram Domain
10.2.9 Shared Receive Queue
10.2.9.1 Motivation for supporting SRQ
10.2.9.2 Shared Receive Queue Creation
10.2.9.3 Shared Receive Queue Modification
10.2.9.4 Shared Receive Queue Destruction
10.2.9.5 SRQ States
10.2.10 XRC Software Transport Interface
10.2.11 InfiniBand Header Data and Sources
10.3 Resource States
10.3.1 Queue Pair and EE Context States
10.3.1.1 Reset
10.3.1.2 Initialized (Init)
10.3.1.3 Ready to Receive (RTR)
10.3.1.4 Ready to Send (RTS)
10.3.1.5 Send Queue Drain (SQD)
10.3.1.6 Send Queue Error (SQEr)
10.3.1.7 Error
10.4 Automatic Path Migration
10.4.1 Path Migration State Diagram
10.4.1.1 Migrated
10.4.1.2 Rearm
10.4.1.3 Armed
10.5 Multicast Services
10.5.1 Multicast Groups and Multicast Message Reception
10.5.1.1 IBA Unreliable Multicast Reception
10.5.1.2 Raw Packet Multicast Reception
10.5.2 Multicast Work Requests
10.5.2.1 IBA Unreliable Multicast Work Requests
10.5.2.2 Raw Packet Multicast Work Requests
10.5.3 Multicast Destination Establishment
10.6 Memory Management
10.6.1 Overview
10.6.2 Memory Registration
10.6.2.1 Memory Regions
10.6.2.2 Allocation of Memory Registration Resources
10.6.2.3 Memory Region Types
10.6.3 Access to Registered Memory
10.6.3.1 Local Access to Registered Memory
10.6.3.2 Remote Access to Registered Memory
10.6.3.3 Local Access Keys
10.6.3.4 Remote Access Keys
10.6.3.5 Protection Domains
10.6.3.6 Scope of Access
10.6.3.7 Fast Registration
10.6.3.8 Multiple Registration of Memory regions
10.6.4 Addressing Memory
10.6.4.1 Virtual Addresses (“Pointers”)
10.6.4.2 Virtual to physical translations
10.6.4.3 Registration of virtually addressed regions
10.6.4.3.1 Registration of Zero Based Virtual Address (ZBVA) Memory Regions
10.6.4.3.2 Reserved L_Key
10.6.4.3.3 Byte Alignment and Length of Memory Regions
10.6.4.3.4 Registered Memory Residency
10.6.4.4 Registration of physically addressed regions
10.6.4.4.1 Physical Buffer lists
10.6.4.5 Memory Region Error Checking
10.6.4.5.1 Error Checking of Local Accesses to Memory Regions
10.6.4.5.2 Error Checking of Remote Accesses to Memory Regions
10.6.5 Invalidation of Memory Regions
10.6.5.1 Invalidation Ordering
10.6.6 Deregistration of regions
10.6.7 Memory Access Control
10.6.7.1 Local Access Control
10.6.7.2 Remote Access Control
10.6.7.2.1 Remote Access Directly With Memory Regions
10.6.7.2.2 Remote Access Through Memory Windows
10.6.7.2.3 Type 1 Memory Windows
10.6.7.2.4 Type 2 Memory Windows
10.6.7.2.5 Rebinding or Deallocating Active Windows
10.6.7.2.6 Deregistering Regions with Bound Windows
10.6.7.2.7 Error Checking at Window Bind Time
10.6.7.2.8 Error Checking at Window Access Time
10.6.7.2.9 Error Checking at Type 2 Memory Window Invalidate Time
10.7 Work Requests
10.7.1 Creating Work Requests
10.7.2 Work Request Types
10.7.2.1 Send/Receive
10.7.2.2 RDMA
10.7.2.3 Atomic Operations
10.7.2.4 Bind Memory Windows
10.7.2.5 Local Invalidate
10.7.2.6 Fast Register Physical MR
10.7.3 Work Request Contents
10.7.3.1 Signaled Completions
10.7.3.2 Scatter/Gather
10.8 Work Request Processing Model
10.8.1 Overview
10.8.2 Submitting Work Requests to a Work Queue
10.8.2.1 Submitting A List of Work Requests
10.8.3 Work Request Processing
10.8.3.1 Reliable Datagram Ordering Rules
10.8.3.2 Shared Receive Queue Ordering Rules
10.8.3.3 Send Queue Ordering Rules
10.8.4 Completion Processing
10.8.5 Returning Completed Work Requests
10.8.5.1 Freed Resource Count
10.8.5.2 Completion Queue Errors
10.8.6 Unsignaled Completions
10.8.7 Asynchronous Completion Notification
10.9 Partitioning
10.9.1 Introduction
10.9.1.1 Limited and Full Membership
10.9.1.2 Special P_Keys
10.9.1.3 Operation Across Subnets
10.9.2 The Partition Key Table (P_Key Table)
10.9.3 Partition Key Matching
10.9.4 Bad P_Key Trap and P_Key Violations Counter (Optional)
10.9.5 CI Partition Support
10.9.5.1 EE Context (Reliable Datagram) Support
10.9.5.2 Partition Key Changes
10.9.6 TCA Partition Support
10.9.7 Fabric Partition Support
10.9.8 Partition Enforcement on Management Queue Pairs
10.9.9 Related Enforcement of Management Message Checking
10.10 Error Handling Semantics and Mechanisms
10.10.1 Error Types
10.10.2 Error Handling Mechanisms
10.10.2.1 Immediate Errors
10.10.2.2 Completion Errors
10.10.2.3 Asynchronous Errors
10.10.3 Effects of Errors on QP Service Types
10.10.3.1 Reliable Connection QPs:
10.10.3.2 Reliable Datagram QPs:
10.10.3.3 Unreliable Connected QPs:
10.10.3.4 Unreliable Datagram QPs:
10.10.3.5 Raw QPs:
10.10.3.6 Extended Reliable Connected
10.10.4 Effects of Transport Layer Errors
Chapter 11: Software Transport Verbs
11.1 Verbs Introduction and Overview
11.1.1 Verb Extensions
11.1.2 Verb Classes
11.1.2.1 Mandatory vs. Optional Verbs
11.1.2.2 Mandatory vs. Optional Verb Functionality
11.1.2.3 Consumer Accessibility
11.2 Transport Resource Management
11.2.1 HCA
11.2.1.1 Open HCA
11.2.1.2 Query HCA
11.2.1.3 Modify HCA Attributes
11.2.1.4 Close HCA
11.2.1.5 Allocate Protection Domain
11.2.1.6 Deallocate Protection Domain
11.2.1.7 Allocate Reliable Datagram Domain
11.2.1.8 Deallocate Reliable Datagram Domain
11.2.1.9 Allocate XRC Domain
11.2.1.10 Deallocate XRC Domain
11.2.2 Address Management Verbs
11.2.2.1 Create Address Handle
11.2.2.2 Modify Address Handle
11.2.2.3 Query Address Handle
11.2.2.4 Destroy Address Handle
11.2.3 Shared Receive Queue
11.2.3.1 Create Shared Receive Queue
11.2.3.2 Query Shared Receive Queue
11.2.3.3 Modify Shared Receive Queue
11.2.3.4 Destroy Shared Receive Queue
11.2.4 XRC SRQs
11.2.4.1 Create XRC Shared Receive Queue
11.2.4.2 Query XRC Shared Receive Queue
11.2.4.3 Modify XRC Shared Receive Queue
11.2.4.4 Destroy XRC Shared Receive Queue
11.2.5 Queue Pair
11.2.5.1 Create Queue Pair
11.2.5.2 Modify Queue Pair
11.2.5.3 Query Queue Pair
11.2.5.4 Destroy Queue Pair
11.2.6 Get Special QP
11.2.7 XRC Target QP
11.2.7.1 Create XRC Target QP
11.2.7.2 Modify XRC Target Queue Pair
11.2.7.3 Query XRC Target Queue Pair
11.2.7.4 Destroy XRC Target Queue Pair
11.2.8 Completion Queue
11.2.8.1 Create Completion Queue
11.2.8.2 Query Completion Queue
11.2.8.3 Resize Completion Queue
11.2.8.4 Destroy Completion Queue
11.2.9 EE Context
11.2.9.1 Create EE Context
11.2.9.2 Modify EE Context Attributes
11.2.9.3 Query EE Context
11.2.9.4 Destroy EE Context
11.2.10 Memory Management
11.2.10.1 Allocate L_Key
11.2.10.2 Register Memory Region
11.2.10.3 Register Physical Memory Region
11.2.10.4 Query Memory Region
11.2.10.5 Deregister Memory Region
11.2.10.6 Reregister Memory Region
11.2.10.7 Reregister Physical Memory Region
11.2.10.8 Register Shared Memory Region
11.2.10.9 Allocate Memory Window
11.2.10.10 Query Memory Window
11.2.10.11 Bind Memory Window
11.2.10.12 Deallocate Memory Window
11.3 Multicast
11.3.1 Attach QP to Multicast Group
11.3.2 Detach QP from Multicast Group
11.4 Work Request Processing
11.4.1 Queue Pair Operations
11.4.1.1 Post Send Request
11.4.1.2 Post Receive Request
11.4.2 Completion Queue Operations
11.4.2.1 Poll for Completion
11.4.2.2 Request Completion Notification
11.5 Event Handling
11.5.1 Set Completion Event Handler
11.5.2 Set Asynchronous Event Handler
11.6 Result Types
11.6.1 Immediate Return Results
11.6.2 Completion Return Status
11.6.3 Asynchronous Events
11.6.3.1 Affiliated Asynchronous Events
11.6.3.2 Affiliated Asynchronous Errors
11.6.3.3 Unaffiliated Asynchronous Events
11.6.3.4 Unaffiliated Asynchronous Errors
11.6.4 Verb Extension Summary
11.6.5 XRC Summary
Chapter 12: Communication Management
12.1 Overview
12.2 Establishment
12.2.1 Quiet Time
12.3 Automatic Path Migration
12.4 Release
12.4.1 Stale Connection
12.5 Service Types
12.5.1 Supported Protocols
12.5.2 Connected Services
12.5.3 Unreliable Datagram Service
12.5.4 Reliable Datagram
12.5.5 Extended Reliable Connected Service
12.6 Communication Management Messages
12.6.1 Required Messages
12.6.2 Conditionally Required Messages
12.6.3 Optional Messages
12.6.4 Message Usage
12.6.5 REQ - Request for Communication
12.6.6 MRA - Message Receipt Acknowledgment
12.6.7 REJ - Reject
12.6.7.1 Example REJ message
12.6.7.2 Rejection Reason
12.6.8 REP - Reply to Request for Communication
12.6.9 RTU - Ready To Use
12.6.10 DREQ - Request for communication Release (Disconnection REQuest)
12.6.11 DREP - Reply to Request for communication Release
12.7 Message Field Details
12.7.1 Local Communication ID
12.7.2 Remote Communication ID
12.7.3 ServiceID
12.7.4 Remote CM Response Timeout
12.7.5 Local CM Response Timeout
12.7.6 Transport Service Type
12.7.7 Subnet Local
12.7.8 This Section Has Been Deleted
12.7.9 Local CA GUID
12.7.10 Local Port GID
12.7.11 Local Port LID
12.7.12 Local QPN
12.7.13 Local Q_Key
12.7.14 Local EECN
12.7.15 Remote EECN
12.7.16 Service Level
12.7.17 Traffic Class
12.7.18 Flow Label
12.7.19 Hop Limit
12.7.20 Primary Remote Port GID
12.7.21 Primary Remote Port LID
12.7.22 Alternate Remote Port GID
12.7.23 Alternate Remote Port LID
12.7.24 Partition Key
12.7.25 Packet Rate
12.7.26 End-to-End Flow Control
12.7.27 Max CM Retries
12.7.28 Path Packet Payload MTU
12.7.29 Responder Resources
12.7.30 Initiator Depth
12.7.31 Starting PSN
12.7.32 Service Timeout
12.7.33 Target ACK Delay
12.7.34 Local ACK Timeout
12.7.35 PrivateData
12.7.36 Failover Accepted
12.7.37 Remote QPN/EECN
12.7.38 Retry Count
12.7.39 RNR Retry Count
12.7.40 SAP Supported
12.7.41 Extended Transport Type
12.8 Alternate Path Management
12.8.1 LAP - Load Alternate Path
12.8.2 APR - Alternate Path Response
12.8.2.1 AP Status
12.8.3 SAP - Suggest Alternate Path
12.8.4 SPR - Suggest Path Response
12.8.4.1 Suggest AP Status
12.9 State Transition Diagrams For Communication Establishment and Release
12.9.1 Diagram Description
12.9.2 Invalid State Input Handling
12.9.3 timeouts
12.9.4 State Diagram Notes
12.9.5 Communication Establishment and Release - Active
12.9.6 Communication Establishment - Passive
12.9.7 State and Transition Definitions
12.9.7.1 Active States
12.9.7.2 Passive States
12.9.8 State Details
12.9.8.1 Timeout
12.9.8.2 RTU Timeout
12.9.8.3 Established
12.9.8.3.1 REQ Received / REP Received
12.9.8.4 TimeWait
12.9.8.5 Message Receipt Acknowledgment (MRA)
12.9.8.6 Timeouts and Retries
12.9.8.7 REJ Retry
12.9.8.8 REJ Sent
12.9.8.9 REP Sent / MRA(REP) Received
12.9.9 Connection State
12.10 Communication Establishment Ladder Diagrams
12.10.1 Active Client to Passive Server - Both Client and Server Accept Communication
12.10.2 Active Client to Passive Server - Server Rejects Communication
12.10.3 Active Client to Passive Server - Client Rejects Communication
12.10.4 Peer to Peer - Both Accept Communication
12.10.5 Active Peer to Active Peer - Passive Rejects Communication
12.10.6 Active Peer to Active Peer - Active Rejects Communication
12.10.7 Active Client to Passive Server with Redirector - All Accept Communication
12.10.8 Communication Release
12.10.8.1 Disconnect Request
12.11 Service ID Resolution Protocol
12.11.1 SIDR_REQ - Service ID Resolution Request
12.11.1.1 RequestID
12.11.1.2 Partition Key
12.11.1.3 Service ID
12.11.1.4 Private Data
12.11.2 SIDR_REP - Service ID Resolution Response
12.11.2.1 Status
12.11.2.2 QPN
12.11.2.3 Q_Key
12.11.3 Path Information
Chapter 13: Management Model
13.1 Introduction
13.2 Assumptions, and Scope
13.2.1 Assumptions
13.2.2 Scope
13.3 Managers, Agents, and Interfaces
13.3.1 Introduction
13.3.2 Required Managers and Agents
13.4 Management Datagrams
13.4.1 Conventions
13.4.2 Management Datagram Format
13.4.3 Management Datagram Fields
13.4.4 Management Classes
13.4.5 Management Class Methods
13.4.6 Management Messaging
13.4.6.1 Methods and Message Sequencing
13.4.6.1.1 Get()/GetResp()
13.4.6.1.2 Set()/GetResp()
13.4.6.1.3 Send()
13.4.6.1.4 Trap()
13.4.6.1.5 TrapRepress()
13.4.6.1.6 Report()/ReportResp()
13.4.6.2 Timers and Timeouts
13.4.6.2.1 PortInfo:SubnetTimeout
13.4.6.2.2 RespTimeValue
13.4.6.3 Timeout/Timer Usage
13.4.6.4 TransactionID usage
13.4.7 Status Field
13.4.8 Management Class Attributes
13.4.8.1 ClassPortInfo
13.4.8.2 Notice
13.4.8.3 InformInfo
13.4.9 Traps
13.4.10 Notice Queue
13.4.11 Event Forwarding
13.5 MAD Processing
13.5.1 MAD Interfaces
13.5.1.1 Processing Subnet Management Packets (SMPs)
13.5.1.2 Processing General Services Management Packets (GMPs)
13.5.2 GSI Redirection
13.5.3 MAD Validation
13.5.3.1 MAD Validation for Subnet Management MADs
13.5.3.2 Mad Validation for Subnet Administration and General Services
13.5.3.2.1 MAD Validation at the GSI
13.5.3.2.2 MAD Validation at the SA and GSAs
13.5.3.3 Consolidated MAD Validation Flow Diagrams
13.5.4 Response Generation and Reversible Paths
13.5.4.1 Reversible Paths
13.5.4.2 Common Response Actions
13.5.4.3 Constructing a Response Without a GRH
13.5.4.4 Constructing a Response With a GRH
13.5.4.5 Responses to MADs
13.6 Reliable Multi-Packet Transaction Protocol
13.6.1 Management Class Use of RMPP
13.6.2 RMPP Packet Formats
13.6.2.1 RMPP Header
13.6.2.2 Status Codes
13.6.2.3 DATA Packet
13.6.2.4 ACK Packet
13.6.2.5 ABORT and STOP Packets
13.6.3 Timeouts
13.6.3.1 Response Timeout (Resp)
13.6.3.2 Total Transaction Timeout (Ttime)
13.6.4 Ladder Diagram (Example)
13.6.5 Flow Diagrams
13.6.5.1 Context State Variables
13.6.5.2 Context & Dispatching
13.6.5.3 Common Termination Flow
13.6.5.4 Receiver Flow Diagram
13.6.5.5 Sender Main Flow Diagram
13.6.5.6 Direction Switch
13.6.6 Startup Scenarios
13.6.6.1 Receiver-Initiated Transfer
13.6.6.2 Sender-Initiated Transfer
13.6.6.3 Sender-Initiated Double-Sided Transfer
Chapter 14: Subnet Management
14.1 Subnet Management Model
14.2 Subnet Management Class
14.2.1 Datagram Formats and Use
14.2.1.1 SMP Data Format - LID Routed
14.2.1.2 SMP Data Format - Directed Route
14.2.2 SMPs and Directed Route Algorithm
14.2.2.1 Outgoing Directed Route SMP Initialization
14.2.2.2 Outgoing Directed Route SMP handling by SMI
14.2.2.3 Returning Directed Route SMP Initialization
14.2.2.4 Returning Directed Route SMP handling by SMI
14.2.3 Methods
14.2.4 Management Key
14.2.4.1 Levels of Protection
14.2.4.2 Lease Period
14.2.4.3 Notes on Expected Usage
14.2.4.4 Update Procedure
14.2.4.5 Initialization
14.2.4.6 SMI
14.2.5 Attributes
14.2.5.1 Notices and Traps
14.2.5.2 NodeDescription
14.2.5.3 NodeInfo
14.2.5.4 SwitchInfo
14.2.5.5 GUIDInfo
14.2.5.6 PortInfo
14.2.5.6.1 Interpretation of DiagCode
14.2.5.7 P_KeyTable
14.2.5.8 SLtoVLMappingTable
14.2.5.9 VLArbitrationTable
14.2.5.10 LinearForwardingTable
14.2.5.11 RandomForwardingTable
14.2.5.12 MulticastForwardingTable
14.2.5.13 SMInfo
14.2.5.14 VendorDiag
14.2.5.15 LedInfo
14.2.5.16 LinkSpeedWidthPairsTable
14.2.5.17 VendorSpecificMadsTable
14.2.5.18 CableInfo
14.2.5.19 PortInfoExtended
14.2.6 Subnet Management MAD Status
14.2.6.1 Status Precedence
14.2.6.2 SMP Version Not Supported (status_field[4:2] = 0x1)
14.2.6.3 SMP Method Not Supported (status_field[4:2] = 0x2)
14.2.6.4 SMP Method/Attribute Combination Not Supported (status_field[4:2] = 0x3)
14.2.6.5 SMP AttributeModifier Errors (status_field[4:2] = 0x7)
14.2.6.6 SMP Attribute Component Errors (status_field[4:2] = 0x7)
14.3 Subnet Management Agent
14.3.1 SubnGet()
14.3.2 SubnSet()
14.3.3 SubnGetResp()
14.3.4 SubnTrap()
14.3.5 SubnTrapRepress()
14.3.6 Port State Change
14.3.7 P_Key Mismatch on Switch External Ports
14.3.8 Transport Key Mismatch
14.3.9 M_Key mismatch
14.3.10 Link Layer Errors
14.3.11 Change CapabilityMask
14.3.12 Change SystemImageGUID
14.3.13 Other Local Changes
14.3.14 SMSupportsExtendedLinkSpeeds
14.3.14.1 PortInfo
14.3.14.2 LinkSpeedWidthPairsTable
14.4 Subnet Manager
14.4.1 SM State Machine
14.4.1.1 Control Packets
14.4.1.2 Discovering State
14.4.1.3 Standby State
14.4.1.4 Not-Active State
14.4.1.5 Master State
14.4.2 Subnet Discovery Actions
14.4.3 Initialization Actions
14.4.4 Node Reinitialization
14.4.5 Port State Transitions
14.4.6 Subnet Sweeping
14.4.7 Authentication
14.4.8 SM Disable Mechanism
14.4.9 In and Out of Service Traps
14.4.10 Multicast Group Create/Delete Traps
14.4.11 Client Reregistration
14.4.12 UnPath/RePath Traps
Chapter 15: Subnet Administration
15.1 Introduction and Overview
15.1.1 SA Function
15.1.2 Relationship Between SA and the SM
15.1.3 Overview
15.2 SA MADs
15.2.1 SA MAD Format
15.2.1.1 SA Header
15.2.1.2 SA Header Fields
15.2.1.3 SA-Specific Optional Capabilities
15.2.2 Summary of Methods
15.2.3 Subnet Administration Status Values
15.2.4 Attributes and Attribute Tables
15.2.4.1 Embedded Attributes
15.2.4.2 Record Identifier (RID) Fields
15.2.4.3 Tables
15.2.5 Attributes
15.2.5.1 Summary of Attributes
15.2.5.2 NodeRecord
15.2.5.3 PortInfoRecord
15.2.5.4 SLtoVLMappingTableRecord
15.2.5.5 SwitchInfoRecord
15.2.5.6 LinearForwardingTableRecord
15.2.5.7 RandomForwardingTableRecord
15.2.5.8 MulticastForwardingTableRecord
15.2.5.9 VLArbitrationTableRecord
15.2.5.10 SMInfoRecord
15.2.5.11 P_KeyTableRecord
15.2.5.12 InformInfoRecord
15.2.5.13 LinkRecord
15.2.5.14 ServiceRecord
15.2.5.14.1 ServiceP_Key
15.2.5.14.2 ServiceKey
15.2.5.14.3 ServiceLease
15.2.5.14.4 ServiceData
15.2.5.15 ServiceAssociationRecord
15.2.5.16 PathRecord
15.2.5.17 MCMemberRecord
15.2.5.17.1 Group Membership
15.2.5.17.2 Creating a Multicast Group
15.2.5.17.3 Joining a Multicast Group
15.2.5.17.4 Leaving & Deleting a Multicast Group
15.2.5.17.5 Querying a Multicast Group
15.2.5.17.6 P_Keys in Multicast Group Operations
15.2.5.18 GuidInfoRecord
15.2.5.19 TraceRecord
15.2.5.20 MultiPathRecord
15.2.5.21 LinkSpeedWidthPairsTableRecord
15.2.5.22 CableInfoRecord
15.2.5.23 PortInfoExtendedRecord
15.3 Reliable Multi-Packet Transaction Protocol
15.4 Operations
15.4.1 Restrictions on Access
15.4.1.1 Access Restrictions For PathRecords
15.4.1.2 Access Restrictions For Other Attributes
15.4.2 Locating Subnet Administration
15.4.3 Event Forwarding Subsystem
15.4.4 Administration Query Subsystem
15.4.5 SubnAdmGetTable() / SubnAdmGetTableResp()
15.4.6 SubnAdmGet() / SubnAdmGetResp(): Get an Attribute
15.4.7 SubnAdmSet(): Set an Attribute
15.4.8 SubnAdmDelete(): Delete an Attribute
15.4.9 SubnAdmGetTraceTable(): Trace a Path
15.4.10 SubnAdmGetMulti() / SubnAdmGetMultiResp(): Send & Receive Multiple Packets
Chapter 16: General Services
16.1 Performance Management
16.1.1 MAD Format
16.1.1.1 Status Field
16.1.2 Methods
16.1.3 Mandatory Attributes
16.1.3.1 ClassPortInfo
16.1.3.2 PortSamplesControl
16.1.3.3 CounterSelect Values
16.1.3.4 PortSamplesResult
16.1.3.5 PortCounters
16.1.3.6 Typical Performance Attribute Use Model
16.1.4 Optional Attributes
16.1.4.1 PortRcvErrorDetails
16.1.4.2 PortXmitDiscardDetails
16.1.4.3 PortOpRcvCounters
16.1.4.4 PortFlowCtlCounters
16.1.4.5 PortVLOpPackets
16.1.4.6 PortVLOpData
16.1.4.7 PortVLXmitFlowCtlUpdateErrors
16.1.4.8 PortVLXmitWaitCounters
16.1.4.9 SwPortVLCongestion
16.1.4.10 PortSamplesResultExtended
16.1.4.11 PortCountersExtended
16.1.4.12 PortExtendedSpeedsCounters
16.1.5 Performance Management Status
16.1.5.1 Mandatory PM Attribute Status
16.1.5.1.1 PM AttributeModifier Errors (Status[4:2] = 0x7)
16.1.5.1.2 PM Attribute Component Errors (Status[4:2] = 0x7)
16.1.5.2 Optional PM Attribute Status
16.1.5.2.1 PM AttributeModifier Errors (Status[4:2] = 0x7)
16.1.5.2.2 PM Attribute Component Errors (Status[4:2] = 0x7)
16.2 Baseboard Management
16.2.1 MAD Format
16.2.1.1 Status Field
16.2.2 Methods
16.2.3 Attributes
16.2.3.1 ClassPortInfo
16.2.3.2 Notice
16.2.3.3 BKeyInfo
16.2.3.4 IB-ML Attributes
16.2.4 B_Key General Use
16.2.4.1 B_Key Assumptions
16.2.4.2 B_Key Protection Scope
16.2.4.3 B_Key Operation
16.2.4.4 B_Key Initialization
16.2.4.5 B_Key Recovery
16.2.4.6 Levels of Protection
16.3 Device Management
16.3.1 MAD Format
16.3.1.1 Status Field
16.3.2 Methods
16.3.3 Attributes
16.3.3.1 ClassPortInfo
16.3.3.2 Notice
16.3.3.3 IOUnitInfo
16.3.3.4 IOControllerProfile
16.3.3.5 ServiceEntries
16.3.3.6 DiagnosticTimeout
16.3.3.7 PrepareToTest
16.3.3.8 TestDeviceOnce
16.3.3.9 TestDeviceLoop
16.3.3.10 DiagCode
16.3.4 Device Diagnostic Framework
16.3.4.1 Behaviors
16.4 SNMP Tunneling
16.4.1 MAD Format
16.4.1.1 Status Field
16.4.2 Methods
16.4.3 Attributes
16.4.3.1 ClassPortInfo
16.4.3.2 Obsolete Section
16.4.3.3 PduInfo
16.4.4 Operations
16.4.4.1 SNMP Targets for Beyond the InfiniBand Endnode
16.4.4.2 Trap Event Subscription
16.5 Vendor-specific
16.5.1 MAD Format
16.5.2 Status Field
16.5.3 Methods
16.5.4 Attributes
16.5.4.1 ClassPortInfo
16.6 Application-specific
16.6.1 MAD Format
16.6.1.1 Status Field
16.6.2 Methods
16.6.3 Attributes
16.6.3.1 ClassPortInfo
16.7 Communication Management
16.7.1 MAD Format
16.7.1.1 Status Field
16.7.2 Methods
16.7.3 Attributes
16.7.3.1 ClassPortInfo
Chapter 17: Channel Adapters
17.1 Overview
17.2 Common Functional Requirements
17.2.1 Multiple Ports per Channel Adapter
17.2.1.1 Topologies Supported With Multi-Ported Channel Adapters
17.2.1.2 Association of QPs with Ports
17.2.1.3 Port Attributes and Functions
17.2.1.4 Switching Packets through Multiple Ports
17.2.2 Channel Adapter Attributes
17.2.3 Deadlock Prevention
17.2.4 Checking Incoming Packets
17.2.5 Non-Volatile State
17.2.6 Static Rate Control
17.2.7 Management Messages
17.2.7.1 Subnet Management
17.2.7.2 General Services
17.2.8 Automatic Path Migration
17.2.8.1 Automatic Path Migration Protocol
17.2.8.1.1 Initialization
17.2.8.1.2 Migration Request
17.2.8.1.3 Migration Response
17.2.8.1.4 Re-enabling Migration
17.3 Host Channel Adapter
17.3.1 Loopback
17.4 Target Channel Adapter
17.4.1 Contrast to a Host Channel Adapter
17.4.1.1 Memory Protection
17.4.2 Device Administration
17.4.3 Fabric Loopback
Chapter 18: Switches
18.1 Overview
18.1.1 Switch Port 0
18.2 Detailed Functional Requirements
18.2.1 Attributes
18.2.2 Initialization
18.2.3 Configuration
18.2.4 Packet Relay Requirements
18.2.4.1 Switch Ports
18.2.4.2 Receiver Queuing
18.2.4.2.1 Inbound P_Key Enforcement
18.2.4.3 Packet Relay
18.2.4.3.1 Linear Forwarding Table Requirements
18.2.4.3.2 Random Forwarding Table Requirements
18.2.4.3.3 Required Multicast Relay
18.2.4.3.4 Optional Multicast Relay
18.2.4.4 Transmitter Queuing
18.2.4.4.1 Outbound P_Key Enforcement
18.2.4.5 Packet Transmission
18.2.5 Error Handling
18.2.5.1 Switch Ports
18.2.5.2 Receiver Queuing
18.2.5.3 Packet Relay
18.2.5.4 Transmitter Queueing
18.2.5.5 Packet Transmission
18.2.6 Subnet Management Agent Requirements
Chapter 19: Routers
19.1 Overview
19.2 Detailed functional requirements
19.2.1 Attributes
19.2.2 Initialization
19.2.3 Configuration
19.2.4 Packet Relay Model
19.2.4.1 Path Selection
19.2.4.2 Router Ports
19.2.4.3 Receiver Queuing
19.2.4.3.1 Inbound P_Key Enforcement
19.2.4.4 Packet Relay
19.2.4.5 Transmitter Queuing
19.2.4.5.1 Outbound P_Key Enforcement
19.2.4.6 Packet Transmission
19.2.5 Error Handling
19.2.5.1 Router Ports Errors
19.2.5.2 Receiver Queuing Errors
19.2.5.3 Packet Relay Errors
19.2.5.4 Transmitter Queueing Errors
19.2.5.5 Packet Transmission Errors
19.2.6 Subnet Management Agent Requirements
Chapter 20: Volume 1 Compliance Summary
20.1 Compliance Definition
20.1.1 Product Application
20.2 Volume 1 Compliance Categories
20.2.1 Volume 1 Compliance Qualifiers
20.2.1.1 Claiming Support for Optional Features
20.2.1.2 Compliance Statements with Multiple Qualifiers
20.2.2 Compliance Statement Lists
20.2.2.1 Hypertext Links
20.2.2.2 Compliance Statement Labels
20.2.2.3 Compliance Statement Titles
20.2.3 Common Requirements
20.3 HCA-CI Compliance Category
20.4 TCA Compliance Category
20.5 Switch Compliance Category
20.6 Router Compliance Category
20.7 Subnet Manager Compliance Category
20.8 Subnet Administration Compliance Category
20.9 Communication Manager Compliance Category
20.10 Performance Manager Compliance Category
20.11 Vendor-Defined Manager Compliance Category
20.12 Optional Management Agent Compliance Category
20.13 Common Port Requirements
20.14 Common MAD Requirements
Annex A1: I/O Infrastructure
A1.1 Introduction
A1.1.1 Purpose
A1.1.2 Glossary
A1.2 Principles of I/O
A1.2.1 I/O Operation Overview
A1.2.2 Managed I/O Units
A1.2.3 ROM Repository
A1.2.4 I/O Device Drivers
A1.2.4.1 Matching an I/O Controller with an I/O Device Driver
A1.2.4.1.1 Creating Compatibility Strings for an I/O Controller
A1.2.4.1.2 Comparing Compatibility Strings
A1.2.4.2 Using an I/O Controller
A1.2.5 I/O Attachment
A1.2.5.1 Direct attachment
A1.2.5.2 Fabric Attachment
A1.2.5.3 Power Management
A1.3 I/O Management
A1.3.1 I/O Device Resolution
A1.3.1.1 Resolving A Path
A1.3.1.2 Persistent Information
A1.3.1.3 Configuration Changes
A1.3.1.3.1 Identifiers Supporting Configuration Changes
A1.3.1.3.2 Node Replacement
A1.3.1.3.3 IOC Replacement
A1.3.2 Retry-Backoff Policy
A1.4 Impact of Partitions on I/O
A1.4.1 I/O Units and Partitions
A1.4.2 Hosts and I/O Partitions
A1.4.2.1 Query for Path
A1.4.2.2 Query for Service
A1.4.2.3 Query for List of I/O Units
A1.5 Storage I/O
A1.5.1 IB Storage Concepts
A1.5.2 Protocol Specific Fields
A1.5.3 Storage Protocols
Annex A2: Console Service Protocol
A2.1 Introduction
A2.1.1 Glossary
A2.1.2 Compliance
A2.1.3 Overview
A2.1.4 Goals
A2.2 The IB Console Abstraction
A2.2.1 Console IO Controllers
A2.2.2 Console Server Processes
A2.3 Console Service Protocol
A2.3.1 Error Reporting
A2.3.2 Console Device Enumeration
A2.3.3 Capability Query
A2.3.4 Session Establishment
A2.3.5 Normal Operation
A2.3.5.1 ASCII Text Streams
A2.3.5.2 UTF-8 Text Streams
A2.3.5.3 HTTP Console Support
A2.3.6 Session Handoff and Maintenance
A2.3.7 Connection Maintenance Messages
A2.3.8 Service Connection and Session Termination
A2.4 Compliance Summary
A2.4.1 CSP Client Compliance Category
A2.4.2 CSP Server Compliance Category
Annex A3: Application Specific Identifiers
A3.1 Introduction
A3.1.1 Glossary
A3.1.2 Compliance
A3.2 Service ID
A3.2.1 Goals and Scope
A3.2.2 Principles of Service ID Usage
A3.2.2.1 Background
A3.2.2.2 Considerations
A3.2.2.3 Assigning Service IDs
A3.2.3 Service ID Structure
A3.2.3.1 IBTA Assigned Service IDs
A3.2.3.1.1 Null Service ID
A3.2.3.1.2 ROM Repository
A3.2.3.1.3 Console Process
A3.2.3.1.4 Service IDs for Testing
A3.2.3.1.5 Sockets Direct Protocol
A3.2.3.2 IETF Service IDs
A3.2.3.3 Local OS Administered Service IDs
A3.2.3.3.1 Port Association
A3.2.3.3.2 Coherency Requirements
A3.2.3.4 Externally Administrated Service IDs
A3.2.4 Resolving Service Names
A3.2.4.1 Service Advertisement
A3.2.4.2 Multicast Query
A3.2.4.3 Alternatives
A3.3 I/O Controller Identification
A3.3.1 Vendor Information
A3.3.2 Generic Information
A3.3.3 IBTA Protocols
A3.3.4 Other Protocols
A3.4 Service Names
A3.4.1 IBTA Service Names
A3.4.2 I/O Service Records
A3.4.3 ServiceRecord Attribute
A3.5 Management Class Codes
A3.6 Queue Keys
A3.7 Compliance Summary
A3.7.1 Service ID Administration
A3.7.2 Service Application
A3.7.3 Managed I/O Unit
Annex A4: Sockets Direct Protocol (SDP)
A4.1 Introduction
A4.1.1 Architectural Goals
A4.1.2 Overview of the Byte-Stream Protocol
A4.2 Glossary
A4.3 SDP message Formats
A4.3.1 Base Sockets Direct Header (BSDH)
A4.3.1.1 Message identifier (MID)
A4.3.1.2 Flags
A4.3.1.3 Buffers (Bufs)
A4.3.1.4 Length (Len)
A4.3.1.5 Message Sequence Number (MSeq)
A4.3.1.6 Message Sequence Number Acknowledgement (MSeqAck)
A4.3.2 Connection Management Messages
A4.3.2.1 Hello Message (HH)
A4.3.2.1.1 Major Protocol Version Number (MajV) - 4 Bits
A4.3.2.1.2 Minor Protocol Version Number (MinV) - 4 Bits
A4.3.2.1.3 IP Version (IPV) - 4 Bits
A4.3.2.1.4 Maximum Advertisements (MaxAdverts) - 8 Bits
A4.3.2.1.5 Desired Remote Receive Size (DesRemRcvSz) - 32 Bits
A4.3.2.1.6 Local Receive Size (LocalRcvSz) - 32 Bits
A4.3.2.1.7 LocalPort - 16 Bits
A4.3.2.1.8 Internet Protocol Address (SrcIP, DstIP) - 128 Bits
A4.3.2.1.9 Rsvd
A4.3.2.1.10 Capabilities (Cap) - 4 bits
A4.3.2.1.11 Extended MaxAdverts (ExtMaxAdverts) - 16 bits
A4.3.2.2 HelloAck Message (HAH)
A4.3.2.2.1 Major Protocol Version Number (MajV) - 4 Bits
A4.3.2.2.2 Minor Protocol Version Number (MinV) - 4 Bits
A4.3.2.2.3 Extended MaxAdverts (ExtMaxAdverts) - 16 bits
A4.3.2.2.4 Actual Receive Size (ActRcvSz) - 32 Bits
A4.3.2.2.5 Rsvd
A4.3.2.2.6 Capabilities (Cap) - 4 bits
A4.3.2.3 DisConn Message
A4.3.2.4 AbortConn Message
A4.3.3 Data Transfer and Flow Control Messages
A4.3.3.1 Data Message
A4.3.3.2 SrcAvail Message (SrcAH)
A4.3.3.2.1 Length (Len) - 32 bits
A4.3.3.2.2 Virtual Address (VA) - 64 bits
A4.3.3.2.3 R_Key - 32 bits
A4.3.3.3 SinkAvail Message (SinkAH)
A4.3.3.3.1 Length (Len) - 32 bits
A4.3.3.3.2 Virtual Address (VA) - 64 bits
A4.3.3.3.3 R_Key - 32 bits
A4.3.3.3.4 NonDiscards - 32 bits
A4.3.3.4 RDMA Messages
A4.3.3.5 SendSm Message
A4.3.3.6 RdmaWrCompl Message (RWCH)
A4.3.3.6.1 Length (Len) - 32 Bits
A4.3.3.7 RdmaRdCompl Message (RRCH)
A4.3.3.7.1 Length (Len) - 32 Bits
A4.3.3.8 ModeChange Message (MCH)
A4.3.3.8.1 S - 1 bit
A4.3.3.8.2 Mode - 3 bits
A4.3.3.8.3 Rsvd - 28 Bits
A4.3.3.9 SrcAvailCancel Message
A4.3.3.10 SinkAvailCancel Message
A4.3.3.11 SinkCancelAck Message
A4.3.4 Private Buffer Resizing Messages
A4.3.4.1 ChRcvBuf Message (CRBH)
A4.3.4.1.1 Desired Size (DesSz) - 32 bits
A4.3.4.2 ChRcvBufAck Message (CRBAH)
A4.3.4.2.1 Actual Size (ActSz) - 32 bits
A4.3.5 Socket Duplication Messages
A4.3.5.1 SuspComm Message
A4.3.5.1.1 Service ID - 64 Bits
A4.3.5.2 SuspCommAck Message
A4.4 Address Resolution
A4.5 Connection Management
A4.5.1 Connection Setup
A4.5.1.1 InfiniBand Reliable Connection Setup
A4.5.1.2 Aborting Connection Setup
A4.5.2 Automatic Path Migration
A4.5.2.1 Determining Alternate Paths
A4.5.2.2 Example Alternate Path Selection Procedure
A4.5.2.3 Configuring Alternate Paths
A4.5.3 Connection Teardown
A4.5.3.1 Graceful Close
A4.5.3.2 Abortive Close
A4.6 Data Transfer Mechanisms
A4.6.1 Bcopy
A4.6.2 Read Zcopy
A4.6.3 Write Zcopy
A4.6.4 Transaction Mechanism
A4.6.5 Miscellaneous Data Transfer Issues
A4.6.5.1 Detecting Stale SinkAvail Advertisements
A4.6.5.2 Mechanisms For Forcing Bcopy
A4.6.5.2.1 Data Sink Forcing Bcopy
A4.6.5.2.2 Data Source Forcing Bcopy
A4.6.5.3 Processing Out-Of-Band Data
A4.6.5.4 SrcAvail Revocation
A4.6.5.5 SinkAvail Revocation
A4.6.5.6 Buffering ULP Payload
A4.7 Private Buffer Management
A4.7.1 SDP Message Ordering
A4.7.2 Send Credit Calculation
A4.7.3 Initialization of Send Credit
A4.7.4 Gratuitous Update Of The Remote Peer’s Send Credit
A4.7.5 Use of Send Credits
A4.7.6 Receive Buffer Resizing
A4.7.6.1 Conflict Resolution
A4.7.6.2 Flow Control Issues During Resizing
A4.8 SDP Modes
A4.8.1 Buffered Mode
A4.8.2 Combined Mode
A4.8.3 Pipelined Mode
A4.9 SDP Mode Transitions
A4.9.1 Transition From Combined Mode to Buffered Mode
A4.9.2 Transition From Buffered Mode to Combined Mode
A4.9.3 Transition From Combined Mode to Pipelined Mode
A4.9.4 Transition From Pipelined Mode to Combined Mode
A4.9.5 State Mode Transition Summary
A4.10 Socket Duplication
A4.10.1 Implementing Socket Duplication
A4.10.1.1 Socket Duplication Procedure
A4.10.1.2 Conflict Resolution
A4.10.2 HCA Managed Failover
A4.11 InfiniBand Transport Layer Issues
A4.11.1 InfiniBand Message Requirements
A4.11.2 Solicited Events
A4.11.3 Keepalive Messages
A4.12 SDP Compliance Category
Annex A5: Booting Annex
A5.1 Introduction
A5.1.1 Purpose
A5.1.2 Glossary
A5.1.3 Overview
A5.1.4 Console
A5.1.5 Storage Boot Method
A5.1.6 Network Boot Method
A5.1.7 Boot Environment
A5.1.8 Managing the Behavior of a Booting Platform
A5.1.8.1 Boot Resolution Methods
A5.1.8.2 ROM Repository
A5.1.8.3 Boot Environment Extension
A5.1.8.4 Proprietary Driver Load
A5.1.9 Subnet Initialization
A5.1.10 Boot/Reboot
A5.2 BootManager
A5.2.1 General Operation
A5.2.2 Detecting New Boot Platforms
A5.2.3 Multiple Boot Managers
A5.2.4 Protecting the BtM_Key
A5.2.5 Event Reporting
A5.2.6 SA Advertisement
A5.3 BootAgent
A5.4 MAD Format
A5.4.1 Boot Management MAD Status
A5.4.2 MAD BtM_Key
A5.5 Boot Management Methods and Attributes
A5.6 Boot Management Attribute Definitions
A5.6.1 ClassPortInfo
A5.6.2 BtM_KeyInfo
A5.6.2.1 BtM_Key General Use
A5.6.2.2 BtM_Key Assumptions
A5.6.2.3 BtM_Key Operations
A5.6.2.4 BtM_Key Initialization
A5.6.2.5 BtM_Key Recovery
A5.6.2.6 Lease Period
A5.6.3 PlatformBootInfo Attribute
A5.6.3.1 BootPlatformUUID
A5.6.3.2 PlatformInfo
A5.6.3.3 Booting Platform Capability
A5.6.3.3.1 Extended Boot Environment
A5.6.3.3.2 Proprietary Driver Load
A5.6.3.3.3 Protocols Supported
A5.6.3.3.4 Boot Resolution Methods Supported
A5.6.3.3.5 Update Locator Records Supported
A5.6.3.4 Boot Record Locator Sources
A5.6.3.5 Record Count
A5.6.3.6 Deleting Persistent Records
A5.6.3.7 Status Components
A5.6.4 PortBootInfo Attribute
A5.6.4.1 BisPortPriority
A5.6.4.2 RomPortPriority
A5.6.4.3 ConsolePortPriority
A5.6.4.4 IocPortPriority
A5.6.4.5 NetworkBootPortPriority
A5.6.4.6 Time-outs
A5.6.4.6.1 InitTimeout and BisTimeout
A5.6.4.6.2 EndNodeTimeout
A5.6.5 Persistent Locator Records
A5.6.5.1 Device-Service
A5.6.5.2 IocGUID-SID
A5.6.5.3 PortGID
A5.6.5.4 Protocol
A5.6.5.5 RecordFunction
A5.6.6 Node Reboot
A5.6.6.1 NodeReboot Attribute
A5.6.6.2 Reboot Time Line
A5.6.7 Traps and Notice Queues
A5.6.7.1 Notice Attribute
A5.6.7.1.1 KeyViolation Notice
A5.6.7.1.2 ChangeReport Notice
A5.6.7.1.3 StatusReport Notice
A5.6.7.1.4 Heartbeat Notice
A5.6.7.2 TrapRepress
A5.6.7.3 Trap Subscription / Reporting
A5.6.7.3.1 Subscription Integrity
A5.6.7.3.2 Subscription Timeout
A5.6.7.3.3 Heartbeat
A5.6.8 InformInfo Attribute
A5.7 Platforms Use of BIS
A5.7.1 BIS Usage Overview
A5.7.2 PlatformBootInfo Source
A5.7.3 PortBootInfo Source
A5.7.4 Determining to use a BIS
A5.7.5 Finding a BIS
A5.7.6 Selecting A BIS
A5.7.7 Prioritizing Multiple BISs
A5.7.8 Other Considerations
A5.7.8.1 Reliable Multi-Packet Protocol
A5.7.8.2 Port GID to LID Resolution
A5.7.8.3 Reporting Failures
A5.8 IB Network Booting
A5.9 Retry Backoff
A5.10 IB Boot Process - Summary
A5.11 AdditionalInfo
A5.11.1 SRP
A5.11.2 Console
A5.12 ROM Repository
A5.12.1 Introduction
A5.12.2 Overview of IOC Boot Driver Download
A5.12.3 ROM Repository Model
A5.12.4 Identifying a ROM Repository
A5.12.5 ROM Repository Access Methods
A5.12.6 ROM Repository Messages
A5.12.7 Reading ROM Repository Information
A5.12.8 IMAGE Descriptor
A5.12.9 Reading an Image Descriptor
A5.12.10 Reading an image
A5.12.11 Adding and Updating an Image
A5.12.11.1 Initiating an Image Add Operation
A5.12.11.2 Initiating an Image Update
A5.12.11.3 Writing Descriptor and Image Data
A5.12.11.4 Deleting an Image
A5.13 Compliance Summary
A5.13.1 Booting Specification Compliance Categories
A5.13.2 BootAgent (BtA) Compliance Category
A5.13.3 BootManager (BtM) Compliance Category
A5.13.4 Boot Platform (BtPlatform) Compliance Category
A5.13.5 ROM Repository Compliance Category
Annex A6: Boot Information Service
A6.1 Introduction
A6.1.1 Glossary
A6.1.2 Compliance
A6.2 BIS Overview
A6.2.1 BIS Operational Model
A6.2.2 Relationship with other Management Classes
A6.2.2.1 Boot Management
A6.2.2.2 Subnet Administration
A6.2.2.3 Subnet Management
A6.2.2.4 Device Management
A6.2.3 Characteristics
A6.3 BIS Class Specification
A6.3.1 Registration
A6.3.2 BIS Query Operation
A6.3.3 BIS Data Formats
A6.3.3.1 Reserved Fields
A6.3.3.2 BIS Status Values
A6.3.4 BIS Methods
A6.3.4.1 Common Methods
A6.3.4.2 Query Methods
A6.3.4.2.1 Multi-Packet Transaction
A6.3.4.2.2 Keep Alive Packets
A6.3.4.3 Lost Messages
A6.3.5 Attributes
A6.3.5.1 ClassPortInfo
A6.3.5.2 BootQueryInfo Attribute
A6.3.5.2.1 BootInfoRequested
A6.3.5.2.2 BootPlatformUUID
A6.3.5.2.3 PortGUID
A6.3.5.2.4 BootSupport
A6.3.5.2.5 PlatformInfo
A6.3.5.3 PlatformBootInfo Attribute
A6.3.5.4 PortBootInfo Attribute
A6.3.5.5 RomRepositoryLocatorRecord Attribute
A6.3.5.6 ConsoleLocatorRecord Attribute
A6.3.5.7 OsLocatorRecord Attribute
A6.3.5.8 Protocol Field
A6.4 Booting using Boot Information Records
A6.4.1 Overview
A6.4.2 General Operation
A6.5 Compliance Summary
Annex A7: Configuration Management
A7.1 Introduction
A7.1.1 Glossary
A7.1.2 Compliance
A7.2 Overview
A7.2.1 Objective
A7.2.2 Usage Model
A7.2.3 Configuration Management Application
A7.2.3.1 Passive Management
A7.2.3.2 Active Management
A7.2.3.3 Multiple Managers
A7.3 Configuration Management Operational Model
A7.4 Configuration Management Characteristics
A7.4.1 Configuration Domain
A7.4.2 Partition Usage
A7.4.3 SA usage
A7.4.4 Manager Interaction
A7.5 Configuration Management Operation
A7.5.1 Interaction with Subnet Manager
A7.5.2 Initialization and SA Registration
A7.5.3 Coherency between Configuration Managers
A7.5.4 Active vs. Passive Configuration Management
A7.5.5 Device Manager Operation
A7.5.6 Protecting the Manager_Key
A7.5.7 I/O Unit Trap Forwarding
A7.5.7.1 Configuring IOUs for Traps
A7.5.7.2 Trap Subscription / Reporting
A7.5.7.3 Subscription Integrity
A7.5.7.4 Subscription Timeout
A7.5.7.5 Heartbeat
A7.5.8 Graceful Hot Removal
A7.5.9 Diagnostics
A7.5.9.1 Diagnostic Framework
A7.5.9.2 Version 1 Diagnostics
A7.5.9.3 7.5.9.3 Diagnostics under Passive Management
A7.6 DevAdm Class Definition
A7.6.1 Operation
A7.6.1.1 Query
A7.6.1.2 Event Notification Subsystem
A7.6.1.2.1 Event Subscription
A7.6.1.2.2 Event Reporting
A7.6.1.2.3 Subscription Integrity
A7.6.1.2.4 Subscription Timeout:
A7.6.1.2.5 Configuration Change Notification
A7.6.1.2.6 IOC On-line Notification
A7.6.1.2.7 IOC Off-line Notification
A7.6.1.2.8 Resource Allocation Change Notification
A7.6.1.2.9 Removal Requested Notification
A7.6.1.2.10 Diagnostic Notification
A7.6.1.2.11 Reset Notification
A7.6.1.2.12 Heartbeat
A7.6.2 DevAdm Message Format
A7.6.2.1 Reserved Fields
A7.6.2.2 DevAdm Status Values
A7.6.2.3 Methods
A7.6.2.4 RMPP Header
A7.6.2.5 RequesterID
A7.6.2.6 Component Mask
A7.6.2.7 Lost Messages
A7.6.3 Attributes
A7.6.3.1 ClassPortInfo
A7.6.3.2 Notice
A7.6.3.2.1 Configuration Change
A7.6.3.2.2 IOC On-line
A7.6.3.2.3 IOC Off-line
A7.6.3.2.4 Resource Allocation Change
A7.6.3.2.5 Heartbeat
A7.6.3.3 InformInfo
A7.6.3.4 LogIn
A7.6.3.5 S_KeyInfo
A7.6.3.6 C_KeyInfo
A7.6.3.7 RemovalReq
A7.6.3.8 DiagNotice
A7.6.3.9 ResetNotice
A7.7 Compliance
A7.7.1 Compliance Categories
A7.7.2 Configuration Manager Compliance Summary
A7.7.3 I/O Client Compliance Summary
A7.7.4 Common Management Requirements
Annex A8: Device Management
A8.1 Introduction
A8.1.1 Glossary
A8.1.2 Compliance
A8.1.3 Goals and Objectives
A8.2 Overview
A8.2.1 Usage Model
A8.2.1.1 Device Manager (DM)
A8.2.1.2 I/O Resource Manager
A8.2.1.3 I/O Client
A8.2.1.4 I/O Management Application
A8.2.2 I/O Unit Model
A8.2.3 Device Management Model
A8.2.3.1 Authority
A8.2.3.2 Device Information
A8.2.3.3 Device Assignment
A8.2.3.3.1 Resource Allocation
A8.2.3.3.2 QP Allocation
A8.2.3.3.3 Shared Pools
A8.2.3.3.4 Client Identification and Default Pools
A8.2.3.3.5 Passive Management
A8.2.3.4 Physical Management
A8.2.3.4.1 Modular Subassemblies
A8.2.3.4.2 Graceful Hot removal
A8.2.3.4.3 I/O Module Examples
A8.2.3.5 Device Diagnostics
A8.2.4 Levels of Access
A8.3 Device Mgt MAD Specification
A8.3.1 MAD Format
A8.3.1.1 Class Version
A8.3.1.1.1 Backward Compatibility
A8.3.1.1.2 Backward Compatibility Level
A8.3.1.1.3 Backward Compatibility Requirements
A8.3.1.2 Status Field
A8.3.1.3 RMPP Header
A8.3.1.4 Access_Key and KeyType
A8.3.1.4.1 Manager_Key
A8.3.1.4.2 Supervisor_Key
A8.3.1.4.3 Client_Key
A8.3.1.4.4 DiagToken
A8.3.1.5 Component Mask
A8.3.2 Methods
A8.3.3 Attributes
A8.3.3.1 ClassPortInfo
A8.3.3.2 Notice
A8.3.3.2.1 MgrKey Violation Notice
A8.3.3.2.2 SupvKey Violation Notice
A8.3.3.2.3 Client Violation Notice
A8.3.3.2.4 DiagToken Violation Notice
A8.3.3.2.5 Heartbeat Notice
A8.3.3.2.6 StatusReport Notice
A8.3.3.2.7 IOC Change Notice
A8.3.3.2.8 ServiceRecord Change Notice
A8.3.3.2.9 Slot Status Change Notice
A8.3.3.2.10 IOM Removal Notice
A8.3.3.2.11 DiagSessionState Notice
A8.3.3.2.12 Diag Session Violation Notice
A8.3.3.3 InformInfo
A8.3.3.4 DA Info
A8.3.3.5 IOUnitInfo
A8.3.3.6 IOControllerProfile
A8.3.3.7 ServiceRecord
A8.3.3.7.1 CM Validatation
A8.3.3.7.2 SharedUD
A8.3.3.8 ProtocolRecord
A8.3.3.9 SlotControlStatus
A8.3.3.10 Reset
A8.3.3.11 ProductInfo
A8.3.3.11.1 IocGUID
A8.3.3.11.2 ProductData
A8.3.3.12 KeyInfo
A8.3.3.12.1 Manager_Key General Use
A8.3.3.12.2 Manager_Key Assumptions
A8.3.3.12.3 Manager_Key Check
A8.3.3.12.4 Manager_Key Initialization
A8.3.3.12.5 Manager_Key Recovery
A8.3.3.12.6 Lease Period
A8.3.3.13 IouResourceInfo
A8.3.3.13.1 Table Sizes
A8.3.3.13.2 QP Resources
A8.3.3.13.3 Update Lock
A8.3.3.13.4 MaxClientPriority
A8.3.3.14 PlatformPoolRecord
A8.3.3.14.1 Supervisor Key
A8.3.3.14.2 Action Component
A8.3.3.14.3 Manager Update
A8.3.3.14.4 Non-Volitile
A8.3.3.14.5 PlatformQoS
A8.3.3.14.6 Platform QPmin, QPmax, and QPminTarget
A8.3.3.14.7 PlatformPriorityMin & PlatformPriorityMax
A8.3.3.14.8 Service Object List
A8.3.3.15 ClientPoolRecord
A8.3.3.15.1 Client Key
A8.3.3.15.2 Supervisor Key
A8.3.3.15.3 Action Component
A8.3.3.15.4 Manager Update Lock
A8.3.3.15.5 Non-Volitile
A8.3.3.15.6 LockedRecord
A8.3.3.15.7 ClientQoS
A8.3.3.15.8 Client QPmin, QPmax, and QPminTarget
A8.3.3.15.9 ClientPriority
A8.3.3.15.10 Service Object List
A8.3.3.16 KeyChange
A8.3.3.17 DiagSession
A8.3.3.18 DiagnosticTimeout
A8.3.3.19 TestDeviceOnce
A8.3.3.20 TestDeviceLoop
A8.3.3.21 DiagCode
A8.4 Resource Allocation Framework
A8.4.1 QP Allocation
A8.4.2 Filtering Information
A8.4.3 Restricting Access
A8.4.4 Consuming QPs
A8.5 Device Diagnostic Framework
A8.5.1 Behaviors
A8.5.2 Preparing for Diagnostic Tests
A8.5.3 Invoking Diagnostic Tests
A8.6 IOC Graceful Hot Removal
A8.6.1 Required Hot Plug Facilities
A8.6.2 Operation
A8.6.3 STATE DIAGRAM
A8.6.4 I/O Module Indicators
A8.6.4.1 LED Blink Rate Definitions
A8.6.4.2 LED Color
A8.7 Device Manager
A8.8 I/O Unit Implementation
A8.9 Compliance
A8.9.1 Compliance Categories
A8.9.2 Device Management Agent Compliance Summary
A8.9.3 Common Management Requirements
Annex A9: Verb Extensions Annex
A9.1 Introduction
A9.1.1 Overview
Annex A10: Congestion Control
A10.1 Congestion Control in InfiniBand Networks
A10.1.1 Glossary
A10.1.2 Congestion Overview
A10.1.3 Congestion control Summary
A10.1.3.1 Current Performance metrics
A10.1.3.2 Operation with CC Unaware switches and Channel Adapters
A10.2 Congestion Control Mechanism
A10.2.1 Switch Behavior
A10.2.1.1 Congestion Detection
A10.2.1.1.1 Root vs. Victim
A10.2.1.2 Congestion Marking
A10.2.1.3 Congestion Log
A10.2.1.4 Switch Performance counters for congestion
A10.2.1.5 Switch Credit Starvation
A10.2.2 CA Behavior
A10.2.2.1 Injection rate control
A10.2.2.1.1 CCT Entry Format
A10.2.2.1.2 Rate Decrease
A10.2.2.1.3 Rate Increase
A10.2.2.2 CA Congestion Threshold Event Notification Log
A10.2.2.3 CA Performance Counters
A10.3 Packet Formats
A10.3.1 BTH: FECN and BECN locations
A10.3.2 Congestion Notification Packet (CNP) format
A10.4 Congestion Control Management
A10.4.1 Congestion Control MAD Format
A10.4.1.1 CC_Key
A10.4.1.1.1 CC_Key Assumptions
A10.4.1.1.2 CC_Key Protection Scope
A10.4.1.1.3 CC_Key Operation
A10.4.1.1.4 CC_Key Initialization
A10.4.1.1.5 CC_Key Recovery
A10.4.1.1.6 Levels of Protection
A10.4.1.2 Congestion Control Log Data
A10.4.1.3 CCMgt Data
A10.4.2 Methods
A10.4.3 Attributes
A10.4.3.1 ClassPortInfo
A10.4.3.2 Traps and Notices
A10.4.3.3 CongestionInfo
A10.4.3.4 CongestionKeyInfo
A10.4.3.5 CongestionLog
A10.4.3.6 SwitchCongestionSetting
A10.4.3.7 SwitchPortCongestionSetting
A10.4.3.8 CACongestionSetting
A10.4.3.8.1 Port_Control
A10.4.3.8.2 Control_Map
A10.4.3.8.3 CACongestionEntryList
A10.4.3.8.4 CACongestionEntry
A10.4.3.9 CongestionControlTable
A10.4.3.9.1 CCTI_Limit
A10.4.3.9.2 CCTI_Entry_List
A10.4.3.9.3 CongestionControlTableEntry
A10.4.3.10 TimeStamp
A10.5 Congestion Management Performance Counters
A10.5.1 PortSamplesControl
A10.5.2 Counter Select Values
A10.5.3 Optional Performance Management Attributes
A10.5.4 PortRcvConCtrl
A10.5.5 PortSLRcvFECN
A10.5.6 PortSLRcvBECN
A10.5.7 PortXmitConCtrl
A10.5.8 PortVLXmitTimeCong
A10.6 Optional PM Attribute status
A10.6.1 PM AttributeModifier Errors (Status[4:2] = 0x7)
A10.6.2 PM Attribute Component Errors (Status[4:2] = 0x7)
A10.7 Compliance Summary
A10.7.1 CCMgt Switch Compliance Category
A10.7.2 CCMgt CA Compliance Category
Annex A11: RDMA IP CM Service
A11.1 Introduction
A11.2 Glossary
A11.3 Convention
A11.4 IP Connection Establishment
A11.4.1 Major Version (4 bits)
A11.4.2 Minor Version (4 bits)
A11.4.3 IP version (4 bits)
A11.4.4 Res (4 bits)
A11.4.5 Source Port (16 bits)
A11.4.6 Source & Destination IP Addresses (128 bits)
A11.4.7 IP Service ID Resolution
A11.5 IP Protocol Ports Mapping into IBTA Service IDs
A11.6 Error Handling
A11.6.1 SIDR Error Handling
A11.7 REJ Reasons
A11.7.1 SIDR Additional Information
A11.8 Compliancy Summary
Annex A12: Support for iSCSI Extensions for RDMA (iSER)
A12.1 Introduction
A12.2 Glossary
A12.3 Convention
A12.4 iSER Connection Establishment
A12.5 iSER Extension for Handling Send with Invalidate
A12.6 iSER Extension for Handling Virtual Address
A12.7 Compliance Summary
Annex A13: Quality of Service (QoS)
A13.1 Introduction
A13.2 Glossary
A13.3 Overview
A13.3.1 Objectives
A13.3.2 Passive vs Active QoS Management
A13.3.3 QoS Management Model
A13.3.4 Congestion Management vs QoS Management
A13.3.5 Congestion Control Manager and QoS Manager Coexistence
A13.4 VL Requirements
A13.5 Subnet Administration
A13.5.1 SA-Specific Optional Capabilities
A13.5.2 Subnet Administration Status Values
A13.5.3 QoS Extended Attributes
A13.5.3.1 PathRecord
A13.5.3.1.1 General
A13.5.3.1.2 ServiceID8MSB and ServiceID56LSB
A13.5.3.1.3 QoSClass
A13.5.3.2 MultiPathRecord
A13.5.3.2.1 General
A13.5.3.2.2 QoSClass
A13.5.3.2.3 ServiceID8MSB and ServiceID56LSB
A13.6 Performance Counters
A13.6.1 New counters for Active QoS
A13.6.2 QoS Performance Counter Requirements
A13.6.3 PortSamplesControl
A13.6.4 Counter Select Values
A13.6.5 Optional Performance Management Attributes
A13.6.6 PortXmitDataSL
A13.6.7 PortRcvDataSL
A13.6.8 PortXmitDataSLExt
A13.6.9 PortRcvDataSLExt
A13.7 Optional PM Attribute status
A13.7.1 PM AttributeModifier Errors (Status[4:2] = 0x7)
A13.7.2 PM Attribute Component Errors (Status[4:2] = 0x7)
A13.8 Compliance
Annex A14: Extended Reliable Connected (XRC) Transport Service
A14.1 Introduction
Annex A15: Hierarchy Information
A15.1 Introduction
A15.2 Glossary
A15.3 Overview
A15.4 Mode of Operation
A15.4.1 Storing and Updating Hierarchical Information
A15.4.2 Fabric Discovery
A15.4.3 Decoding Hierarchical Information
A15.5 Subnet Management Extensions
A15.5.1 Hierarchy Information Capability
A15.5.2 Hierarchy Information OtherLocalChanges bit
A15.5.3 HierarchyInfo Attribute
A15.5.3.1 HierarchyInfo Query
A15.5.3.2 Setting HierarchyInfo
A15.5.4 Error handling for HierarchyInfo Attribute
A15.5.4.1 HierarchyInfo METHOD/ATTRIBUTE COMBINATION NOT SUPPORTED (STATUS_FIELD[4:2] = 0X3)
A15.5.4.2 HierarchyInfo AttributeModifier Errors (status_field[4:2] = 0x7)
A15.5.4.3 HierarchyInfo ATTRIBUTE Component Errors (status_field[4:2] = 0x7)
A15.6 Subnet Administration Extensions
A15.6.1 HierarchyInfo Support
A15.6.2 HierarchyInfoRecord Attribute
A15.7 Hierarchy Template File
A15.7.1 Key Schema Design Considerations
A15.7.2 Schema Elements
A15.7.2.1 Bits Element
A15.7.2.2 Schema Data Types
A15.7.2.2.1 Integer Element
A15.7.2.2.2 Enum Element
A15.7.2.3 Representation Element
A15.7.2.4 Record Element
A15.7.3 RecordSelector, RecordCode and ApplyToLevels Elements
A15.7.4 Formal Template File XML Schema
A15.7.5 A Generic Physical Hierarchy Template
A15.7.6 A Generic Port Hierarchy Template
A15.8 Compliance
InfiniBandTM Architecture
Specification Volume 1
Release 1.3
March 3rd 2015
Final
Copyright © 2015 by InfiniBandSM Trade Association.
All rights reserved.
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
March 3rd 2015
FINAL
Table 1 Revision History
Revision
Release Date
1.0
1.0.a
1.1
1.2
1.2.1
1.3
9/26/2000
Release 1.0
6/19/2001
11/06/2002
9/7/2004
Release 1.0 augmented with errata material. Updates only
correct errors - no additional features have been added.
Release 1.0.a augmented with additional features. Revised
SA and CM Class with new version.
Release 1.1 augmented with additional features (added
annexes A7 through A10). Incorporated errata.
11/30/2004
Release 1.2 augmented with additional features (added
annex A11-A13).
3/3/2015
Release 1.3 - added Annexes and integrated XRC into the
main body. Updated Chapters include new Change Bars
LEGAL DISCLAIMER
This IBTA specification provided “AS IS” and without
any warranty of any kind, including, without limita-
tion, any express or implied warranty of non-infringe-
ment, merchantability or fitness for a particular
purpose.
In no event shall IBTA or any member of IBTA be liable
for any direct, indirect, special, exemplary, punitive,
or consequential damages, including, without limita-
tion, lost profits, even if advised of the possibility of
such damages.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 2
Proprietary and Confidential
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
TABLE OF CONTENTS
March 3rd 2015
FINAL
Chapter 1: Introduction...........................................................................................................66
Acknowledgments..................................................................................................................... 66
1.1
1.2
InfiniBand Conceptual Overview............................................................................................... 69
The Problem ........................................................................................................................... 69
1.2.1
1.2.2
Features.................................................................................................................................. 70
1.2.3
Benefits................................................................................................................................... 70
Scope........................................................................................................................................ 71
Document Organization ............................................................................................................ 72
1.4.1
Series of Volumes................................................................................................................... 72
1.4.2
Volume 1 Organization............................................................................................................ 73
Document Conventions ............................................................................................................ 73
1.5.1
Byte Ordering.......................................................................................................................... 73
1.5.2
Numeric Values....................................................................................................................... 74
Disclaimer ................................................................................................................................. 75
1.3
1.4
1.5
1.6
Chapter 2: Glossary ................................................................................................................76
3.2
3.3
3.4
Chapter 3: Architectural Overview.........................................................................................93
3.1
Architecture Scope ................................................................................................................... 94
3.1.1
Topologies & Components...................................................................................................... 95
Communication......................................................................................................................... 97
3.2.1
Queuing .................................................................................................................................. 97
3.2.2
Connections.......................................................................................................................... 101
Communications Stack ........................................................................................................... 101
IBA Components..................................................................................................................... 103
3.4.1
Links & Repeaters................................................................................................................. 103
Channel Adapters ................................................................................................................. 103
3.4.2
Switches ............................................................................................................................... 105
3.4.3
3.4.4
Routers ................................................................................................................................. 106
Management Components.................................................................................................... 107
3.4.5
3.4.5.1 Subnet Managers.................................................................................................. 107
3.4.5.2 Subnet Management Agents................................................................................. 107
3.4.5.3 General Service Agents ........................................................................................ 108
IBA Features........................................................................................................................... 108
Queue Pairs.......................................................................................................................... 108
3.5.1
3.5.2
Types of Service ................................................................................................................... 109
Keys.......................................................................................................................................111
3.5.3
3.5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 3
Proprietary and Confidential
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
3.5.9
3.5.10
3.5.11
3.5.4
3.5.5
3.5.6
3.5.7
3.5.8
March 3rd 2015
FINAL
Virtual Memory Addresses.................................................................................................... 113
Protection Domains............................................................................................................... 113
Partitions............................................................................................................................... 113
Virtual Lanes......................................................................................................................... 114
Quality of Service.................................................................................................................. 115
3.5.8.1 Service Level......................................................................................................... 116
3.5.8.2 SL to VL mapping.................................................................................................. 116
3.5.8.3 Partitions ............................................................................................................... 116
Injection Rate Control ........................................................................................................... 116
Addressing............................................................................................................................ 117
Multicast................................................................................................................................ 119
3.5.11.1 Multicast Example ................................................................................................. 120
3.5.11.2 Group Management .............................................................................................. 121
3.5.11.3 Multicast Prune...................................................................................................... 124
3.5.12
Verbs..................................................................................................................................... 124
Channel & Memory Semantics ............................................................................................... 124
3.6.1
Communication Interface...................................................................................................... 125
3.6.2
IBA Transport Services ......................................................................................................... 127
IBA Layered Architecture........................................................................................................ 131
Physical Layer....................................................................................................................... 132
3.7.1
3.7.2
Link Layer ............................................................................................................................. 133
Network Layer....................................................................................................................... 134
3.7.3
Transport Layer..................................................................................................................... 135
3.7.4
3.7.5
Upper Layer Protocols .......................................................................................................... 137
3.7.5.1 Subnet Management............................................................................................. 137
3.7.5.2 General Services................................................................................................... 138
IBA Transaction Flow.............................................................................................................. 139
IBA Management Infrastructure.............................................................................................. 140
Management Datagrams ...................................................................................................... 143
3.9.1
Management Methods .......................................................................................................... 144
3.9.2
3.9.2.1 Gets & Sets ........................................................................................................... 144
3.9.2.2
Traps and Notices ................................................................................................. 144
3.9.2.3 Sends .................................................................................................................... 145
3.9.2.4 Reports.................................................................................................................. 145
Management Interfaces ........................................................................................................ 145
Subnet Management Interface.............................................................................................. 145
3.9.4.1
Fabric Initialization................................................................................................. 146
3.9.4.2 Directed Routes..................................................................................................... 146
General Service Interface ..................................................................................................... 146
3.9.5.1 Redirection ............................................................................................................ 147
3.6
3.7
3.8
3.9
3.9.3
3.9.4
3.9.5
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 4
Proprietary and Confidential
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
3.10
March 3rd 2015
FINAL
I/O Operation .......................................................................................................................... 147
Chapter 4: Addressing..........................................................................................................150
4.1
Terminology And Concepts..................................................................................................... 151
GID Usage and Properties.................................................................................................... 152
4.1.1
Channel Adapter, Switch, and Router Addressing Rules...................................................... 156
4.1.2
4.1.3
Local Identifiers..................................................................................................................... 156
Chapter 5: Data Packet Format ............................................................................................159
Packet Types .......................................................................................................................... 159
5.1
5.2
Data Packet Format................................................................................................................ 160
Local Route Header (LRH) - 8 Bytes .................................................................................... 163
5.2.1
Global Route Header (GRH) - 40 Bytes................................................................................ 163
5.2.2
5.2.3
Base Transport Header (BTH) - 12 Bytes............................................................................. 164
Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes.................................... 166
5.2.4
Datagram Extended Transport Header (DETH) - 8 Bytes.................................................... 166
5.2.5
RDMA Extended Transport Header (RETH) - 16 Bytes........................................................ 167
5.2.6
5.2.7
Atomic Extended Transport Header (AtomicETH) - 28 Bytes............................................... 168
XRC Extended Transport Header (XRCETH)....................................................................... 168
5.2.8
ACK Extended Transport Header (AETH) - 4 Bytes ............................................................. 169
5.2.9
5.2.10
Atomic ACK Extended Transport Header (AtomicAckETH) - 8 Bytes .................................. 169
Immediate Data Extended Transport Header (ImmDt) - 4 Bytes .......................................... 170
5.2.11
INVALIDATE EXTENDED TRANSPORT HEADER (IETH) - 4 BYTES ................................ 170
5.2.12
5.2.13
Payload................................................................................................................................. 170
Invariant CRC ....................................................................................................................... 171
5.2.14
5.2.15
Variant CRC .......................................................................................................................... 171
Raw Packet Format ................................................................................................................ 171
Packet Examples .................................................................................................................... 172
5.3
5.4
Chapter 6: Physical Layer Interface.....................................................................................174
6.1
Overview................................................................................................................................. 174
Services provided by the Physical Layer. ............................................................................... 174
6.2
Interface between physical and Link Layers........................................................................... 175
6.3
6.3.1
Interface between physical receive and link receive............................................................. 175
6.3.1.1 Phy_link - Physical Link Status.............................................................................. 175
L_Init_Train - Link Initiate Retraining..................................................................... 175
6.3.1.2
6.3.1.3
rcv_stream - Receive Stream................................................................................ 176
Interface between physical Transmit and link Transmit. ....................................................... 176
6.3.2.1 Xmit_stream - Transmit Stream............................................................................. 176
6.3.2.2 Xmit_Ready - Physical Transmitter Ready............................................................ 176
6.3.2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 5
Proprietary and Confidential
7.2
7.3
7.4
7.5
7.6
InfiniBandTM Architecture Release 1.3
March 3rd 2015
VOLUME 1 - GENERAL SPECIFICATIONS
FINAL
Chapter 7: Link Layer............................................................................................................178
Overview................................................................................................................................. 178
7.1
7.1.1
State Machine Conventions .................................................................................................. 178
Link States .............................................................................................................................. 179
LinkDown State..................................................................................................................... 179
7.2.1
LinkInitialize State ................................................................................................................. 179
7.2.2
7.2.3
LinkArm State........................................................................................................................ 179
LinkActive State .................................................................................................................... 180
7.2.4
LinkActDefer State ................................................................................................................ 180
7.2.5
7.2.6
Management State Change Commands............................................................................... 180
7.2.7
State Machine Terms ............................................................................................................ 181
Packet Receiver States........................................................................................................... 183
Data Packet Check ................................................................................................................. 186
Link Packet Check .................................................................................................................. 189
Virtual Lanes Mechanisms...................................................................................................... 191
VL identification .................................................................................................................... 193
7.6.1
7.6.2
Number of VLs supported..................................................................................................... 193
Special VLs ........................................................................................................................... 193
7.6.3
Buffering and Flow Control For Data VLs ............................................................................. 194
7.6.4
7.6.5
Service Level ........................................................................................................................ 196
VL Mapping Within a Subnet ................................................................................................ 197
7.6.6
Initialization and Configuration.............................................................................................. 199
7.6.7
VL Scheduling and Flow Control For VL15 and Flow Control Packets................................. 200
7.6.8
7.6.9
VL Arbitration and Prioritization ............................................................................................ 200
7.6.9.1 VL Arbitration When Only One Data VL Is Implemented ...................................... 200
7.6.9.2 VL Arbitration When Multiple Data VL s Are Implemented.................................... 200
Local Route Header................................................................................................................ 204
Virtual Lane (VL) - 4 bits ....................................................................................................... 204
7.7.1
Link Version (LVer) - 4 bits .................................................................................................... 205
7.7.2
7.7.3
Service Level (SL) - 4 bits..................................................................................................... 205
Reserve - 2 bits..................................................................................................................... 205
7.7.4
Link Next Header (LNH) - 2 bits............................................................................................ 205
7.7.5
Destination Local Identifier (DLID) - 16 bits .......................................................................... 205
7.7.6
7.7.7
Reserve - 5 bits..................................................................................................................... 206
Packet Length (PktLen) - 11 bits........................................................................................... 206
7.7.8
7.7.9
Source Local Identifier (SLID) - 16 bits................................................................................. 207
CRCs ...................................................................................................................................... 207
Invariant CRC (ICRC) - 4 Bytes............................................................................................ 207
7.8.1
Variant CRC (VCRC) - 2 Bytes ............................................................................................. 209
7.8.2
7.8.3
Link Packet CRC (LPCRC) - 2 Bytes.................................................................................... 209
CRC Calculation Samples .................................................................................................... 210
7.8.4
7.7
7.8
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 6
Proprietary and Confidential
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
March 3rd 2015
FINAL
7.9
7.10
7.11
7.12
7.8.4.1
ICRC Generator .................................................................................................... 210
7.8.4.2 VCRC Generator................................................................................................... 211
7.8.4.3 Sample Packets .................................................................................................... 212
Flow Control............................................................................................................................ 221
Introduction ........................................................................................................................... 221
7.9.1
Flow Control Blocks .............................................................................................................. 222
7.9.2
Relationship to Virtual Lanes ................................................................................................ 222
7.9.3
7.9.4
Flow Control Packet.............................................................................................................. 222
7.9.4.1
Flow Control Packet Fields.................................................................................... 223
7.9.4.2 Calculation of FCTBS............................................................................................ 223
7.9.4.3 Calculation of FCCL .............................................................................................. 223
Transmission of Packets ....................................................................................... 224
7.9.4.4
IBA and Raw Packet Multicast................................................................................................ 225
7.10.1
Overview............................................................................................................................... 225
IBA Unreliable Multicast Operational Rules.......................................................................... 226
7.10.2
Raw Packet Multicast............................................................................................................ 229
7.10.3
7.10.3.1 Raw Multicast Operational Rules .......................................................................... 229
7.10.4
Group Management.............................................................................................................. 231
Subnet Multipathing ................................................................................................................ 231
7.11.1
Multipathing Requirements on end node .............................................................................. 231
Error detection and handling................................................................................................... 231
Error Detection...................................................................................................................... 231
7.12.1
Error Recovery Procedures .................................................................................................. 233
7.12.2
7.12.3
Error Notification ................................................................................................................... 233
Chapter 8: Network Layer.....................................................................................................234
Overview................................................................................................................................. 234
8.1
8.2
Packet Routing ....................................................................................................................... 234
Overview.............................................................................................................................. 234
8.2.1
Global Fabric Characteristics................................................................................................ 235
8.2.2
8.2.2.1
Inheritance of Subnet Requirements..................................................................... 235
8.2.2.2 Packet Errors and Error Detection ........................................................................ 235
8.2.2.3 Service Levels....................................................................................................... 235
8.2.3
Support for Multiple Global Paths ......................................................................................... 235
8.2.4
Global Multicast .................................................................................................................... 237
Global Route Header .............................................................................................................. 237
IP Version (IPVer) - 4 bits...................................................................................................... 237
8.3.1
8.3.2
Traffic Class (TClass) - 8 bits................................................................................................ 237
Flow Label (FlowLabel) - 20 bits........................................................................................... 238
8.3.3
Payload Length (PayLen) - 16 bits........................................................................................ 238
8.3.4
8.3.5
Next Header (NxtHdr) - 8 bits ............................................................................................... 238
Hop Limit (HopLmt) - 8 bits................................................................................................... 238
8.3.6
8.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 7
Proprietary and Confidential
InfiniBandTM Architecture Release 1.3
VOLUME 1 - GENERAL SPECIFICATIONS
March 3rd 2015
FINAL
8.4
Source Global Identifier (SGID) - 128 bits ............................................................................ 238
8.3.7
8.3.8
Destination Global Identifier (DGID) - 128 bits...................................................................... 238
Global Route Header Usage...................................................................................................238
8.4.1
Global Route Header Generation ......................................................................................... 238
Global Route Header Modification........................................................................................ 240
8.4.2
8.4.3
Global Route Header Verification.......................................................................................... 241
Chapter 9: Transport Layer ..................................................................................................242
Overview................................................................................................................................. 242
9.1
Base Transport Header........................................................................................................... 245
9.2
9.2.1
Operation Code (OpCode).................................................................................................... 246
Reserved Transport Function OpCodes ............................................................................... 248
9.2.2
Solicited Event (SE) - 1 bit.................................................................................................... 249
9.2.3
9.2.4
MigReq (M) - 1 Bit................................................................................................................. 249
Pad Count (PadCnt) - 2 bits.................................................................................................. 249
9.2.5
Transport Header Version (TVer) - 4 bits .............................................................................. 250
9.2.6
Partition Key (P_Key) - 16 bits.............................................................................................. 250
9.2.7
9.2.8
Destination QP (DestQP) - 24 bits........................................................................................ 250
FECN/Res1 (F/Res1) - 1 bit.................................................................................................. 250
9.2.9
BECN/Res1 (B/Res1) - 1 bit ................................................................................................. 250
9.2.10
9.2.11
Reserve 6(Resv6) - 6 bits ..................................................................................................... 250
AckReq (A) - 1 Bit ................................................................................................................. 250
9.2.12
Reserved 7 (resv7) - 7 bits.................................................................................................... 251
9.2.13
9.2.14
Packet Sequence Number (PSN) - 24 bits ........................................................................... 251
Extended Transport Headers..................................................................................................251
Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes.................................... 251
9.3.1
9.3.1.1 Reserve - 8 bits ..................................................................................................... 251
9.3.1.2
End-to-End (EE) Context - 24 bits........................................................................ 251
Datagram Extended Transport Header (DETH) - 8 Bytes..................................................... 252
9.3.2.1 Q_Key - 32 bits...................................................................................................... 252
9.3.2.2 Reserve - 8 bits ..................................................................................................... 252
9.3.2.3 Source QP (SrcQP) - 24 bits ................................................................................. 252
RDMA Extended Transport Header (RETH) - 16 Bytes........................................................ 252
9.3.3.1 Virtual Address (VA) - 64 bits ................................................................................ 252
9.3.3.2 R_Key - 32 bits...................................................................................................... 252
9.3.3.3 DMA Length (DMAlen) - 32 bits ............................................................................ 253
ATOMIC Extended Transport Header (AtomicETH) - 28 Bytes ............................................ 254
9.3.4.1 Virtual Address (VA) - 64 bits ................................................................................ 254
9.3.4.2 R_Key - 32 bits...................................................................................................... 254
9.3.4.3 Swap (Add) Data (SwapDt) - 64 bits ..................................................................... 254
9.3.4.4 Compare Data (CmpDt) - 64 bits........................................................................... 254
ACK Extended Transport Header (AETH) - 4 Bytes ............................................................. 254
9.3.4
9.3.5
9.3
9.3.2
9.3.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
InfiniBandSM Trade Association
Page 8
Proprietary and Confidential
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
