After replacing the 10 Gigabit network card, the throughput did not increase. In the monitoring, one CPU core was fully loaded and the others were idle. This situation is probably caused by interrupts and multi-queue configuration. This article gives a path from positioning to tuning.

First confirm whether it is a single-core bottleneck

Run mpstat -P ALL 1 when the traffic is compressed. If the %soft of one core is close to full and the value of other cores is almost 0, it means that the packets are processed by a single queue. Check /proc/interrupts again, the network card interrupt line is named according to ixgbe, i40e, ice plus queue name, and observe whether the count is concentrated on a few cores. Confirm the bottleneck before changing the configuration to avoid indiscriminate parameter adjustment.

Queue and interrupt balancing

  • ethtool -l checks the queue upper limit, ethtool -L adjusts combined to match the number of available cores
  • In general scenarios, irqbalance is automatically dispersed; for NUMA sensitive services, it is recommended to manually bind the interrupt affinity to the local core of the node where the network card is located.
  • ethtool -X View the RSS hash distribution. If the count difference between each queue is too large, it means that the traffic itself is concentrated in a small number of flows.

Further fine-tuning aggregation and topology

ethtool -c and -C adjust interrupt aggregation to appropriately increase throughput at the expense of delay. For delay-sensitive services, do not sacrifice response for running scores. The network card should be inserted into the PCIe slot of the same NUMA node as the CPU. Accessing memory across nodes will eat up the benefits of multiple queues.PCI-EX16 100G dual optical port fiber optic network card (Intel Intel E810 chip)This type of high-speed card is especially obvious.100G optical fiber interconnection upgrade between buildings in a large data center in East ChinaThe slots are also fixed according to this principle during implementation.

After the outage is rectified, evaluate whether the link capacity needs to be upgraded. Parameters vary greatly depending on the release version. You can call EB-LINK technical support at 0755-83179002 to check the tuning list according to the network card model and system environment.