第346篇：数据中心 PFC 死锁导致 RoCE 业务中断

关键词：PFC、RoCEv2、死锁、DCBN、ECN、华为CloudEngine

一、案例背景

某AI计算中心部署了RoCEv2（RDMA over Converged Ethernet）网络承载GPU训练集群的东西向通信。在一次模型训练任务中，整个集群突然出现大规模丢包，训练任务停滞超过30分钟。经排查，根因为PFC（Priority Flow Control）死锁。

网络架构： - Leaf层：8台CE6865（支持RoCEv2、PFC、ECN） - Spine层：4台CE6865 - GPU服务器：64台（每台8卡A800，双网卡RoCE） - 网络速率：200Gbps（Leaf-Spine） - 协议栈：RoCEv2 over Ethernet

故障现象： 1. 64台GPU服务器中的40台同时出现RDMA通信中断 2. 故障发生时无明显流量高峰 3. 重启相关交换机端口后恢复正常 4. 故障在后续训练任务中重复出现

二、需求分析与方案设计

2.1 PFC死锁原理

PFC（Priority Flow Control）是IEEE 802.1Qbb标准定义的基于优先级的流控机制，用于在无损网络上实现RoCEv2。PFC死锁是RoCE网络中最严重的故障之一。

PFC死锁产生的条件： 1. 多个交换机之间形成环形PFC依赖 2. 每个交换机的队列都被上游的PFC Pause帧阻塞 3. 没有任何一个交换机能够自主清空队列

2.1.1 PFC死锁的四种类型

2.1.2 DCQCN（Data Center Quantized Congestion Notification）机制

DCQCN是RoCEv2的拥塞控制算法，由PFC（接收端流控）+ ECN（拥塞通知）+ CNP（拥塞通知报文）组成闭环反馈。

DCQCN关键参数：

2.2 RoCE网络可靠性保障机制

2.3 排错流程

三、技术细节与实施过程

3.1 PFC状态检查

# 检查PFC统计
<CE6865-Leaf-01> display qos pfc statistics
PFC Statistics:
  Interface          Priority  Pause Sent   Pause Received  Deadlock
  100GE1/0/1         3         15,234       45,678          2        <-- 死锁2次
  100GE1/0/2         3         12,345       38,901          1
  100GE1/0/3         3         8,765        25,432          0
  100GE1/0/4         3         6,543        18,765          0

# 检查PFC配置
<CE6865-Leaf-01> display qos pfc configuration
PFC Configuration:
  Interface          Priority  Enable  Xon Threshold  Xoff Threshold
  100GE1/0/1         3         Yes     10000          20000
  100GE1/0/2         3         Yes     10000          20000
  100GE1/0/3         3         Yes     10000          20000
  100GE1/0/4         3         Yes     10000          20000

3.2 死锁检测与分析

# 查看PFC死锁详情
<CE6865-Leaf-01> display qos pfc deadlock
PFC Deadlock Information:
  Total Deadlocks: 2
  Latest Deadlock:
    Time: 2024-03-15 14:23:45
    Interface: 100GE1/0/1
    Priority: 3
    Duration: 120 seconds
    Recovery: Auto-recover triggered
    Deadlock Ring:
      100GE1/0/1 → CE6865-Leaf-02 100GE1/0/3
      100GE1/0/3 → CE6865-Leaf-03 100GE1/0/1
      100GE1/0/1 → CE6865-Leaf-04 100GE1/0/2
      100GE1/0/2 → CE6865-Leaf-01 100GE1/0/1

死锁环分析：Leaf-01→Leaf-02→Leaf-03→Leaf-04→Leaf-01形成环形PFC依赖。

3.3 根因分析

根因： 1. 4台Leaf交换机通过Spine层互联，当某台Leaf的上行队列拥塞时发送PFC Pause 2. 上游Leaf收到Pause后停止发送，导致其下游队列也拥塞并发送Pause 3. 形成环路后，所有涉及的交换机都在等待对方清空队列，无人能动

触发条件：某GPU节点突发大流量（梯度同步），导致单一优先级队列拥塞。

四、实战配置/案例分析

4.1 PFC死锁恢复配置

# 启用PFC死锁自动恢复
<CE6865-Leaf-01> system-view
[CE6865-Leaf-01] qos
[CE6865-Leaf-01-qos] pfc
[CE6865-Leaf-01-qos-pfc] deadlock-detection enable
[CE6865-Leaf-01-qos-pfc] deadlock-recovery enable
[CE6865-Leaf-01-qos-pfc] deadlock-recovery method flush  # 死锁恢复方式：冲刷队列
[CE6865-Leaf-01-qos-pfc] deadlock-detection interval 5   # 检测间隔5秒
[CE6865-Leaf-01-qos-pfc] quit
[CE6865-Leaf-01-qos] quit

4.2 ECN优化（预防PFC死锁）

# 配置ECN参数
[CE6865-Leaf-01] qos
[CE6865-Leaf-01-qos] ecn
[CE6865-Leaf-01-qos-ecn] ecn-threshold low 60 high 80  # ECN阈值60%-80%
[CE6865-Leaf-01-qos-ecn] ecn-mode dynamic              # 动态ECN
[CE6865-Leaf-01-qos-ecn] quit
[CE6865-Leaf-01-qos] quit

# 在接口上应用ECN
[CE6865-Leaf-01] interface 100GE 1/0/1
[CE6865-Leaf-01-100GE1/0/1] qos queue 3 wred enable     # 启用WRED
[CE6865-Leaf-01-100GE1/0/1] qos queue 3 wred ecn enable # 启用ECN
[CE6865-Leaf-01-100GE1/0/1] quit

4.3 PFC阈值调优

# 调整PFC Xon/Xoff阈值
[CE6865-Leaf-01] interface 100GE 1/0/1
[CE6865-Leaf-01-100GE1/0/1] qos pfc priority 3
[CE6865-Leaf-01-100GE1/0/1-qos-pfc-3] xon-threshold 15000   # Xon阈值（队列水位下降到15000byte恢复发送）
[CE6865-Leaf-01-100GE1/0/1-qos-pfc-3] xoff-threshold 25000  # Xoff阈值（队列水位上升到25000byte发送Pause）
[CE6865-Leaf-01-100GE1/0/1-qos-pfc-3] quit
[CE6865-Leaf-01-100GE1/0/1] quit

4.3.1 PFC/ECN/DCQCN完整配置模板（华为CloudEngine）

以下为CloudEngine交换机RoCE无损网络的完整配置模板，涵盖全局QoS、PFC、ECN、DCQCN和接口级配置。

# =====================================================
# 华为 CloudEngine RoCEv2 无损网络完整配置模板
# 适用: CE6865/CE8860 等支持RoCEv2的交换机
# =====================================================

# ---- 1. 全局QoS模式 ----
qos queue mode per-queue

# ---- 2. DSCP到Priority映射 ----
qos
 dscp-to-exp-map 0x22 3    # CS3 → Priority 3 (RoCE数据)
 dscp-to-exp-map 0x2E 6    # CS6 → Priority 6 (CNP控制报文)
quit

# ---- 3. PFC全局配置（死锁检测+自动恢复） ----
qos
 pfc
  deadlock-detection enable
  deadlock-recovery enable
  deadlock-recovery method flush
  deadlock-detection interval 5
  deadlock-recovery threshold 3
 quit
quit

# ---- 4. ECN全局配置 ----
qos
 ecn
  ecn-mode dynamic
  ecn-threshold low 60 high 80
  ecn-mark-probability 10
 quit
quit

# ---- 5. 接口级PFC+ECN（上行+Leaf间） ----
interface Range 100GE 1/0/1 to 100GE 1/0/8
 qos pfc priority 3
  xon-threshold 15000
  xoff-threshold 25000
 quit
 qos queue 3 wred enable
 qos queue 3 wred ecn enable
 qos dscp-transmit queue 6 dscp 0x2E
quit

# ---- 6. 服务器侧接口 ----
interface Range 100GE 1/0/9 to 100GE 1/0/40
 qos pfc priority 3
  xon-threshold 10000
  xoff-threshold 20000
 quit
 qos queue 3 wred enable
 qos queue 3 wred ecn enable
quit

# ---- 7. DCBX自动协商 ----
dcbbx enable
interface Range 100GE 1/0/1 to 100GE 1/0/40
 dcbbx pfc role initiator
 dcbbx app-priority 0x22 3
 dcbbx app-priority 0x2E 6
 dcbbx will-age role controller
quit

# ---- 8. 队列调度权重 ----
qos
 scheduler
  queue 3 weight 800       # P3权重最高(RoCE)
  queue 6 weight 200       # P6较高(CNP)
  queue 0 weight 100
  queue 1 weight 100
  queue 2 weight 100
  queue 4 weight 100
  queue 5 weight 100
  queue 7 weight 100
 quit
quit

4.3.2 RoCE网络性能监控脚本

#!/usr/bin/env python3
"""
roce_monitor.py — RoCE网络性能监控（采集PFC/ECN、检测死锁、生成报告）
依赖: pip install netmiko pandas
"""
import argparse
import csv
import re
import os
from datetime import datetime
from pathlib import Path
from netmiko import ConnectHandler
import pandas as pd


class RoceMonitor:
    """RoCE网络性能监控器"""

    def __init__(self, device: dict):
        self.device = device
        self.conn = None
        self.hostname = ""

    def connect(self):
        self.conn = ConnectHandler(**self.device)
        self.hostname = self.conn.find_prompt().replace(">", "").replace("#", "")

    def collect_pfc_stats(self) -> list[dict]:
        """采集PFC暂停帧统计"""
        output = self.conn.send_command("display qos pfc statistics")
        records = []
        for line in output.strip().split("\n")[2:]:
            parts = line.split()
            if len(parts) >= 5:
                records.append({
                    "hostname": self.hostname,
                    "interface": parts[0],
                    "priority": parts[1],
                    "pause_sent": int(parts[2].replace(",", "")),
                    "pause_received": int(parts[3].replace(",", "")),
                    "deadlock_count": int(parts[4]),
                    "timestamp": datetime.now().isoformat()
                })
        return records

    def collect_ecn_stats(self) -> dict:
        """采集ECN统计"""
        output = self.conn.send_command("display qos ecn statistics")
        marked = re.findall(r"ECN Marked:\s*(\d+)", output)
        total = re.findall(r"Total Packets:\s*(\d+)", output)
        ecn_marked = int(marked[0]) if marked else 0
        total_pkts = int(total[0]) if total else 0
        return {
            "hostname": self.hostname,
            "ecn_marked": ecn_marked,
            "total_packets": total_pkts,
            "ecn_mark_rate": ecn_marked / total_pkts * 100 if total_pkts > 0 else 0
        }

    def collect_deadlock_info(self) -> list[dict]:
        """采集死锁信息"""
        output = self.conn.send_command("display qos pfc deadlock")
        deadlocks = []
        for block in output.split("Deadlock")[1:]:
            time_m = re.search(r"Time:\s*(.+)", block)
            iface_m = re.search(r"Interface:\s*(.+)", block)
            dur_m = re.search(r"Duration:\s*(\d+)", block)
            deadlocks.append({
                "hostname": self.hostname,
                "time": time_m.group(1).strip() if time_m else "N/A",
                "interface": iface_m.group(1).strip() if iface_m else "N/A",
                "duration_sec": int(dur_m.group(1)) if dur_m else 0
            })
        return deadlocks

    def collect_all(self) -> dict:
        """采集所有监控数据"""
        self.connect()
        data = {
            "pfc_stats": self.collect_pfc_stats(),
            "ecn_stats": self.collect_ecn_stats(),
            "deadlock_info": self.collect_deadlock_info()
        }
        self.conn.disconnect()
        return data

    @staticmethod
    def detect_anomalies(all_data: list[dict]) -> list[dict]:
        """检测异常"""
        anomalies = []
        for data in all_data:
            hostname = data["pfc_stats"][0]["hostname"] if data["pfc_stats"] else "?"
            for stat in data["pfc_stats"]:
                if stat["pause_received"] > 10000:
                    anomalies.append({
                        "type": "PFC_HIGH_PAUSE", "severity": "WARNING",
                        "hostname": stat["hostname"], "interface": stat["interface"],
                        "detail": f"PFC暂停帧: {stat['pause_received']}"
                    })
                if stat["deadlock_count"] > 0:
                    anomalies.append({
                        "type": "PFC_DEADLOCK", "severity": "CRITICAL",
                        "hostname": stat["hostname"], "interface": stat["interface"],
                        "detail": f"死锁次数: {stat['deadlock_count']}"
                    })
            ecn = data["ecn_stats"]
            if ecn.get("ecn_mark_rate", 0) > 5:
                anomalies.append({
                    "type": "ECN_HIGH_RATE", "severity": "WARNING",
                    "hostname": hostname,
                    "detail": f"ECN标记率: {ecn['ecn_mark_rate']:.2f}%"
                })
            for dl in data["deadlock_info"]:
                anomalies.append({
                    "type": "DEADLOCK_HISTORY", "severity": "CRITICAL",
                    "hostname": dl["hostname"],
                    "detail": f"死锁 {dl['time']}, 持续{dl['duration_sec']}s"
                })
        return anomalies

    @staticmethod
    def generate_report(all_data, anomalies, output_dir):
        """生成监控报告"""
        Path(output_dir).mkdir(parents=True, exist_ok=True)
        ts = datetime.now().strftime("%Y%m%d_%H%M%S")

        # 导出CSV
        pfc_rows = [s for d in all_data for s in d["pfc_stats"]]
        if pfc_rows:
            pd.DataFrame(pfc_rows).to_csv(f"{output_dir}/pfc_stats_{ts}.csv", index=False)

        # 异常报告
        with open(f"{output_dir}/anomalies_{ts}.txt", "w") as f:
            f.write(f"RoCE Monitor Report - {datetime.now()}\n{'='*60}\n")
            f.write(f"Total anomalies: {len(anomalies)}\n\n")
            for a in anomalies:
                sev = "CRITICAL" if a["severity"] == "CRITICAL" else "WARNING"
                f.write(f"[{sev}] {a['type']} @ {a['hostname']}\n")
                if "interface" in a:
                    f.write(f"  Interface: {a['interface']}\n")
                f.write(f"  {a['detail']}\n\n")


def main():
    parser = argparse.ArgumentParser(description="RoCE网络性能监控")
    parser.add_argument("--hosts", required=True)
    parser.add_argument("--output", default="roce_reports")
    args = parser.parse_args()

    devices = []
    with open(args.hosts) as f:
        for row in csv.DictReader(f):
            devices.append({
                "device_type": row["device_type"], "host": row["ip"],
                "username": row["username"], "password": row["password"]
            })

    all_data = []
    for dev in devices:
        m = RoceMonitor(dev)
        try:
            data = m.collect_all()
            all_data.append(data)
            print(f"[OK] {m.hostname}")
        except Exception as e:
            print(f"[FAIL] {dev['host']}: {e}")

    anomalies = RoceMonitor.detect_anomalies(all_data)
    RoceMonitor.generate_report(all_data, anomalies, args.output)
    print(f"\n报告: {args.output}/  异常: {len(anomalies)}")


if __name__ == "__main__":
    main()

监控频率与告警阈值：

4.4 DCBX自动协商配置

DCBX（Data Center Bridge eXchange）协议自动协商PFC/ECN/PG参数，避免手动配置的误差。

# 启用DCBX
[CE6865-Leaf-01] dcbbx enable
[CE6865-Leaf-01] interface 100GE 1/0/1
[CE6865-Leaf-01-100GE1/0/1] dcbbx pfc role initiator       # 主动协商PFC参数
[CE6865-Leaf-01-100GE1/0/1] dcbbx app-priority 0x22 3       # DSCP 0x22映射到Priority 3
[CE6865-Leaf-01-100GE1/0/1] dcbbx app-priority 0x23 3       # DSCP 0x23映射到Priority 3
[CE6865-Leaf-01-100GE1/0/1] dcbbx will-age role controller  # 控制老化时间
[CE6865-Leaf-01-100GE1/0/1] quit

# 验证DCBX协商结果
<CE6865-Leaf-01> display dcbbx interface 100GE 1/0/1
DCBX Information:
  Interface: 100GE1/0/1
  Role: Initiator
  PFC Priority: 3
  PFC Xon/Xoff: 15000/25000
  App Priority Table:
    DSCP 0x22 → Priority 3
    DSCP 0x23 → Priority 3

4.5 死锁恢复方式对比

4.6 RoCEv2 vs InfiniBand 对比

4.7 PFC排错实战清单

4.9 优化效果

4.10 RoCEv2无损网络设计最佳实践

4.11 PFC死锁排查Mermaid流程图

五、经验总结

附录：RoCEv2网络部署检查清单

在实际部署RoCEv2网络时，按照此清单逐项检查，可大幅降低故障率。

思考

1. PFC和ECN可以同时启用吗？两者如何协作避免丢包？
2. 如果RoCE网络中某些交换机不支持PFC死锁检测，如何设计网络架构避免死锁？
3. RoCEv2和InfiniBand在无损传输机制上有何本质区别？在AI训练场景下如何选择？

下篇预告：第347篇：Telemetry 发现现网隐藏拥塞点