RoCE runs RDMA directly on the Ethernet, with low latency and close to zero host CPU usage. It is a popular choice for all-flash storage and high-performance computing, but the premise is that the network is lossless. Once packet loss occurs, the throughput will plummet.

Prerequisite: PFC and ECN must be opened together

  • PFC priority flow control is responsible for preventing packets from congested ports from being discarded. It is necessary to designate an independent lossless queue for storage traffic.
  • ECN explicit congestion notification allows the sender to slow down in advance and reduce the chain reaction of pause frames.
  • The parameters of the switch and the network card must be consistent end-to-end. If one node is omitted, the problem will be exposed during the stress test.

Network card and speed: confirm support first before discussing firmware

The budget allows for suggestions to go directly to 100G, e.g.PCI-EX16 100G dual optical port fiber optic network card (Intel Intel E810 chip), before enabling, confirm the supported RDMA mode with the switch and check the firmware driver version. Priority is given to interconnection within the cabinet100G DAC high speed cable, both cost and power consumption are low; cross-building links are replaced with optical modules and jumpers, and the 100G optical fiber interconnection between buildings in a large data center in East China is upgraded (100G optical fiber interconnection upgrade between buildings in a large data center in East China) is implemented according to this idea.

Online verification: all three types of data must be run

First use the perftest package to run the throughput and delay baseline, and then observe whether the trigger frequency of the PFC and ECN counters at both ends is normal. Finally, increase the pressure step by step, keep an eye on the delay quantile and jitter, and readjust the rollback parameters if any abnormalities are found. Do not go online while sick.

EB-LINK's 25G/100G network cards and interconnection cables are tested one by one before shipment. For storage network solutions, please call 0755-83179002 for consultation.