ThinkSystem Mellanox ConnectX-6 HDR/200GbE QSFP56 1-port PCIe 4 VPI Adapter

The adapters have the following technical specifications.

Form factor

• Single-slot low-profile main adapter (6.6-inch x 2.71 in.)
• Single-slot low-profile auxilary adapter

PCI Express Interface

• Supports PCIe 4.0 or PCIe 3.0
• In PCIe 4.0 servers, the ConnectX-6 adapter is used by itself to connect 16 PCIe lanes. For two-socket servers, the adapter can be used with the Aux adapter to enable the Socket Direct feature.
• In PCIe 3.0 servers, the ConnectX-6 adapter is used with the Aux adapter to connect 32 PCIe lanes. For two-socket servers, the adapter can be used with the Aux adapter to enable the Socket Direct feature.
• PCIe Atomic
• TLP (Transaction Layer Packet) Processing Hints (TPH)
• PCIe switch Downstream Port Containment (DPC) enablement for PCIe hot-plug
• Advanced Error Reporting (AER)
• Access Control Service (ACS) for peer-to-peer secure communication
• Process Address Space ID (PASID) Address Translation Services (ATS)
• IBM CAPIv2 (Coherent Accelerator Processor Interface)
• Support for MSI/MSI-X mechanisms

Connectivity

• One QSFP56 port
• Supports passive copper cables with ESD protection
• Powered connectors for optical and active cable support

InfiniBand

• Supports interoperability with InfiniBand switches (up to HDR, as 4 lanes of 50Gb/s data rate)
• Total connectivity is 200 Gb/s:
• One port adapter supports a single 200 Gb/s link
• HDR / HDR100 / EDR / FDR / QDR / DDR / SDR
• IBTA Specification 1.3 compliant
• RDMA, Send/Receive semantics
• Hardware-based congestion control
• Atomic operations
• 16 million I/O channels
• 256 to 4Kbyte MTU, 2Gbyte messages
• 8 virtual lanes + VL15

Ethernet (requires firmware 20.28.1002 or later)

• Support interoperability with Ethernet switches (up to 200GbE, as 4 lanes of 50Gb/s data rate)
• Total connectivity is 200 Gb/s:
• One port adapter supports a single 200 Gb/s link
• Supports 200 GbE / 100GbE / 50GbE / 40GbE / 25GbE / 10GbE / 1GbE
• IEEE 802.3bj, 802.3bm 100 Gigabit Ethernet
• IEEE 802.3by, Ethernet Consortium 25, 50 Gigabit Ethernet, supporting all FEC modes
• IEEE 802.3ba 40 Gigabit Ethernet
• IEEE 802.3ae 10 Gigabit Ethernet
• IEEE 802.3az Energy Efficient Ethernet
• IEEE 802.3ap based auto-negotiation and KR startup
• IEEE 802.3ad, 802.1AX Link Aggregation
• IEEE 802.1Q, 802.1P VLAN tags and priority
• IEEE 802.1Qau (QCN) – Congestion Notification
• IEEE 802.1Qaz (ETS)
• IEEE 802.1Qbb (PFC)
• IEEE 802.1Qbg
• IEEE 1588v2
• Jumbo frame support (9.6KB)
• IPv4 (RFQ 791)
• IPv6 (RFC 2460)

Enhanced Features

• Hardware-based reliable transport
• Collective operations offloads
• Vector collective operations offloads
• PeerDirect RDMA (GPUDirect) communication acceleration
• 64/66 encoding
• Enhanced Atomic operations
• Advanced memory mapping support, allowing user mode registration and remapping of memory (UMR)
• Extended Reliable Connected transport (XRC)
• Dynamically Connected transport (DCT)
• On demand paging (ODP)
• MPI Tag Matching
• Rendezvous protocol offload
• Out-of-order RDMA supporting Adaptive Routing
• Burst buffer offload
• In-Network Memory registration-free RDMA memory access

CPU Offloads

• RDMA over Converged Ethernet (RoCE)
• TCP/UDP/IP stateless offload
• LSO, LRO, checksum offload
• RSS (also on encapsulated packet), TSS, HDS, VLAN and MPLS tag insertion/stripping, Receive flow steering
• Data Plane Development Kit (DPDK) for kernel bypass applications
• Open VSwitch (OVS) offload using ASAP2
• Flexible match-action flow tables
• Tunneling encapsulation / de-capsulation
• Intelligent interrupt coalescence
• Header rewrite supporting hardware offload of NAT router

Storage Offloads

• Block-level encryption: XTS-AES 256/512 bit key
• NVMe over Fabric offloads for target machine
• Erasure Coding offload - offloading Reed-Solomon calculations
• T10 DIF - signature handover operation at wire speed, for ingress and egress traffic
• Storage Protocols: SRP, iSER, NFS RDMA, SMB Direct, NVMe-oF

Overlay Networks

• RoCE over overlay networks
• Stateless offloads for overlay network tunneling protocols
• Hardware offload of encapsulation and decapsulation of VXLAN, NVGRE, and GENEVE overlay networks

Hardware-Based I/O Virtualization

• Single Root IOV
• Address translation and protection
• VMware NetQueue support
• SR-IOV: Up to 512 Virtual Functions
• SR-IOV: Up to 16 Physical Functions per host
• Virtualization hierarchies (network partitioning, NPAR)
• Virtualizing Physical Functions on a physical port
• SR-IOV on every Physical Function
• Configurable and user-programmable QoS
• Guaranteed QoS for VMs

HPC Software Libraries

• HPC-X, OpenMPI, MVAPICH, MPICH, OpenSHMEM, PGAS and varied commercial packages

Management and Control

• NC-SI, MCTP over SMBus and MCTP over PCIe - BMC interface
• PLDM for Monitor and Control DSP0248
• PLDM for Firmware Update DSP0267
• SDN management interface for managing the eSwitch
• I2C interface for device control and configuration
• General Purpose I/O pins
• SPI interface to Flash
• JTAG IEEE 1149.1 and IEEE 1149.6

Remote Boot

• Remote boot over InfiniBand
• Remote boot over Ethernet
• Remote boot over iSCSI
• Unified Extensible Firmware Interface (UEFI)
• Pre-execution Environment (PXE)