RDMA allows the network card to bypass the CPU and directly read and write the peer memory. It is the basis for high-performance storage and AI training. However, it is extremely sensitive to packet loss. A slight packet loss will cause throughput to drop off a cliff. RoCEv2 therefore must turn Ethernet into a lossless network. This article explains its configuration logic and common pitfalls.

Why is packet loss fatal to RDMA?

RoCEv2 is based on Ethernet transmission, but relies on a costly retransmission mechanism to recover after packet loss. A single packet loss often triggers the rollback of batches of packets, and the throughput may directly drop by an order of magnitude. Traditional networks allow for occasional packet loss, but the requirement for lossless networks is to suppress traffic hop by hop before congestion causes queue overflow, reducing the drop probability to close to zero.

Division of labor between PFC and ECN

  • ECN: Marks packets when the queue is close to the water level. The receiver returns a congestion signal and the sender slows down first. This is an early warning.
  • PFC: Pause upstream transmission according to priority to prevent buffer overflow and discarding of this hop, which is the last resort.
  • Sequence suggestion: first set the ECN water level, then turn on PFC at the dedicated priority level, and finally check the PFC Watchdog anti-deadlock

Configuration points and precautions

PFC is only enabled on the priority level that carries RDMA. Enabling all priority levels can easily cause pause frames to spread hop by hop to form a storm. Setting the ECN water level too low will frequently limit the speed. Generally, start from the manufacturer's default value and then perform stress testing and fine-tuning. Choose a network card model that supports hardware RDMA and multi-queue QoS, for examplePCI-EX8 25G dual optical port fiber optic network card (Intel XXV710 chip);Available on backbone sidePCI-EX16 100G single optical port fiber optic network card (Intel E810 chip)Access Leaf.

There are no parameters that can be set once and for all for lossless network tuning, and it requires repeated stress testing at low business peaks. EB-LINK's high-speed network cards and modules can be matched according to the switch code. For details, please call 0755-83179002.