第335篇：MPLS VPN 跨域互通故障排查实录

关键词：MPLS VPN、跨域互通、故障排查、路由泄露、MP-iBGP、RT属性、华为NE系列

一、案例背景

某运营商省级MPLS VPN网络由三个域组成：骨干域（Provider Domain）和两个客户域（Customer Domain A和B）。近期，某银行客户（属于Customer Domain A）反映其无法访问位于Customer Domain B的支付清算中心。

网络拓扑概况： - 骨干域：AS 65000，约30台NE40E/NE8000 PE设备 - 客户域A：AS 65001，8台PE设备，银行客户 - 客户域B：AS 65002，4台PE设备，支付清算中心

故障现象： - 银行客户CE设备无法ping通支付清算中心CE设备的LoopBack地址 - 银行客户CE设备可以正常访问同域内的其他网段 - 支付清算中心侧可以正常访问其域内资源

二、需求分析与方案设计

2.1 跨域互通架构分析

2.2 跨域互通原理

MPLS VPN跨域互通涉及三个关键技术点：

2.3 排查思路

三、技术细节与实施过程

3.1 第一阶段：PE-A侧路由检查

首先登录PE-A1（银行侧ASBR），检查VPNv4路由表：

# 查看VRF路由表
<PE-A1> display vpn-instance routing-table vpn-instance BANK_A
Total Number of Routes: 48

# 发现只有域A内部的路由，没有域B的路由
Network          NextHop         Pre   Cost     Label  Interface
10.1.0.0/24      192.168.1.2     170   0        3001   GE1/0/1
10.1.1.0/24      192.168.1.2     170   0        3002   GE1/0/1
10.1.2.0/24      192.168.2.2     170   0        3003   GE1/0/2
# ... 共48条域内路由

# 检查MP-iBGP邻居
<PE-A1> display bgp vpnv4 all peer vpn-instance BANK_A
Total Number of peers: 2
Peers: 2    Prefixed received: 48

# 检查路由泄露策略
<PE-A1> display vpn-instance vpn-instance BANK_A
 VPN-Instance Name   : BANK_A
 RD                  : 65001:100
 Route-Target        : 65001:100
 Route-Target        : 65000:100  <-- 导给骨干域的RT

发现问题：PE-A1的VRF中缺少域B的路由。检查发现VRF的RT导出策略中确实包含了骨干域的RT（65000:100），但导入策略中缺少域B的RT。

3.2 第二阶段：路由泄露策略分析

# 骨干域PE-P1侧检查
<PE-P1> display vpn-instance vpn-instance BANK_CROSS
 VPN-Instance Name   : BANK_CROSS
 RD                  : 65000:100
 Route-Target        : 65000:100   <-- 从域A导入
 Route-Target        : 65001:100   <-- 从域A导入
 # 缺少 65002:100（域B的RT）

# PE-P2侧检查（域B方向ASBR）
<PE-P2> display vpn-instance vpn-instance BANK_CROSS
 VPN-Instance Name   : BANK_CROSS
 RD                  : 65000:100
 Route-Target        : 65000:100
 Route-Target        : 65002:100   <-- 从域B导入
 # 缺少 65001:100（域A的RT）

根因确认：骨干域两侧的ASBR（PE-P1和PE-P2）的VRF配置中，RT导入策略不完整。PE-P1缺少域B的RT导入，PE-P2缺少域A的RT导入，导致路由无法跨域传递。

3.3 第三阶段：Label栈验证

修复RT配置后，进一步验证Label栈是否正确：

# PE-A1侧检查Label
<PE-A1> display mpls vpn-instance BANK_A routing-table 10.2.0.0
Vpn-Instance   : BANK_A
Network        : 10.2.0.0/24
NextHop        : 10.0.1.5 (PE-P1)
Label Info     :
  Inner Label  : 3050
  Outer Label  : 16001 (到PE-P1的LDP Label)
  Tunnel       : TE-Tunnel 0 (穿越隧道)

# PE-P1侧检查Label
<PE-P1> display mpls vpn-instance BANK_CROSS routing-table 10.2.0.0
Vpn-Instance   : BANK_CROSS
Network        : 10.2.0.0/24
NextHop        : 10.0.2.5 (PE-B1)
Label Info     :
  Inner Label  : 3050
  Outer Label  : 16002 (到PE-B1的LDP Label)

Label栈结构：

CE-A  →  [Inner:3050][Outer:16001]  →  PE-P1  →  [Inner:3050][Outer:16002]  →  PE-B1  →  [Inner:3050]  →  CE-B
         PE-A1封装                         剥Outer,换Outer                      剥Inner和Outer

3.4 修复方案

# PE-P1（域A方向ASBR）补充域B的RT导入
<PE-P1> system-view
[PE-P1] vpn-instance BANK_CROSS
[PE-P1-vpn-BANK_CROSS] route-target import 65002:100
[PE-P1-vpn-BANK_CROSS] quit

# PE-P2（域B方向ASBR）补充域A的RT导入
<PE-P2> system-view
[PE-P2] vpn-instance BANK_CROSS
[PE-P2-vpn-BANK_CROSS] route-target import 65001:100
[PE-P2-vpn-BANK_CROSS] quit

修复后，等待BGP路由收敛（约30秒），验证跨域互通：

# 从银行CE ping 支付清算中心CE
<CE-A> ping -a 10.1.0.1 10.2.0.1
PING 10.2.0.1: 56 data bytes, press CTRL_C to break
Reply from 10.2.0.1: bytes=56 Sequence=1 ttl=60 time=15 ms
Reply from 10.2.0.1: bytes=56 Sequence=2 ttl=60 time=12 ms
Reply from 10.2.0.1: bytes=56 Sequence=3 ttl=60 time=14 ms
--- 10.2.0.1 ping statistics ---
3 packet(s) transmitted, 3 packet(s) received, 0.00% packet loss

四、实战配置/案例分析

4.1 跨域互通完整配置模板

# ========== 客户域A PE配置 ==========
# PE-A1
[PE-A1] vpn-instance BANK_A
[PE-A1-vpn-BANK_A] rd 65001:100
[PE-A1-vpn-BANK_A] route-target export 65001:100
[PE-A1-vpn-BANK_A] route-target export 65000:100    # 导出到骨干域
[PE-A1-vpn-BANK_A] route-target import 65001:100
[PE-A1-vpn-BANK_A] route-target import 65000:100    # 从骨干域导入
[PE-A1-vpn-BANK_A] quit

# ========== 骨干域配置 ==========
# PE-P1（连接域A侧）
[PE-P1] vpn-instance BANK_CROSS
[PE-P1-vpn-BANK_CROSS] rd 65000:100
[PE-P1-vpn-BANK_CROSS] route-target export 65000:100
[PE-P1-vpn-BANK_CROSS] route-target export 65002:100    # 导出给域B方向
[PE-P1-vpn-BANK_CROSS] route-target import 65001:100    # 导入域A路由
[PE-P1-vpn-BANK_CROSS] route-target import 65002:100    # 导入域B路由
[PE-P1-vpn-BANK_CROSS] quit

# PE-P2（连接域B侧）
[PE-P2] vpn-instance BANK_CROSS
[PE-P2-vpn-BANK_CROSS] rd 65000:100
[PE-P2-vpn-BANK_CROSS] route-target export 65000:100
[PE-P2-vpn-BANK_CROSS] route-target export 65001:100    # 导出给域A方向
[PE-P2-vpn-BANK_CROSS] route-target import 65001:100    # 导入域A路由
[PE-P2-vpn-BANK_CROSS] route-target import 65002:100    # 导入域B路由
[PE-P2-vpn-BANK_CROSS] quit

# ========== 客户域B PE配置 ==========
# PE-B1
[PE-B1] vpn-instance PAYMENT_B
[PE-B1-vpn-PAYMENT_B] rd 65002:100
[PE-B1-vpn-PAYMENT_B] route-target export 65002:100
[PE-B1-vpn-PAYMENT_B] route-target export 65000:100
[PE-B1-vpn-PAYMENT_B] route-target import 65002:100
[PE-B1-vpn-PAYMENT_B] route-target import 65000:100
[PE-B1-vpn-PAYMENT_B] quit

4.1.1 MPLS L3VPN跨域三种方案对比

方案选择建议： - 2个域互联 → 方案1（Route Leaking），配置最简 - 3-5个域互联 → 方案2（Transit VRF），骨干域P路由器透明 - 5个以上域或运营商网络 → 方案3（BGP MPLS-iP），扩展性最佳

4.1.2 不同AS PE的完整配置模板

# =====================================================
# 跨域场景3：不同AS间的VPN路由传递（方案3 BGP MPLS-iP）
# AS 65001(域A) ←eBGP→ AS 65000(骨干) ←eBGP→ AS 65002(域B)
# =====================================================

# ---- 域A的ASBR（PE-A1, AS 65001）----
[PE-A1] bgp 65001
 router-id 10.1.1.1
 # 域内MP-iBGP
 peer 10.1.1.2 as-number 65001       # 域A内其他PE
 # 跨域eBGP到骨干域ASBR
 peer 10.0.1.1 as-number 65000       # PE-P1
 peer 10.0.1.1 connect-interface LoopBack 0
 peer 10.0.1.1 ebgp-max-hop 3
 peer 10.0.1.1 next-hop-local         # 保留Next-Hop
 # VRF
 vpn-instance BANK_A
  rd 65001:100
  route-target export 65001:100
  route-target import 65001:100
  # 向骨干域导出的RT
  route-target export 65000:100
  route-target import 65000:100
  # 跨域导入域B路由
  route-target import 65002:100
  peer 10.1.1.2 enable
  peer 10.0.1.1 enable
 quit
quit

# ---- 骨干域ASBR（PE-P1, AS 65000）----
[PE-P1] bgp 65000
 router-id 10.0.1.1
 # 域内MP-iBGP
 peer 10.0.1.2 as-number 65000       # PE-P2
 peer 10.0.1.2 connect-interface LoopBack 0
 # 域A方向eBGP
 peer 10.1.1.1 as-number 65001
 peer 10.1.1.1 connect-interface LoopBack 0
 peer 10.1.1.1 ebgp-max-hop 3
 peer 10.1.1.1 next-hop-local
 # 域B方向eBGP
 peer 10.2.1.1 as-number 65002
 peer 10.2.1.1 connect-interface LoopBack 0
 peer 10.2.1.1 ebgp-max-hop 3
 peer 10.2.1.1 next-hop-local
 # Transit VRF（承载跨域路由）
 vpn-instance BANK_TRANSIT
  rd 65000:100
  route-target export 65000:100
  route-target import 65000:100
  # 导入域A和域B的路由
  route-target import 65001:100
  route-target import 65002:100
  # 导出给对端域
  route-target export 65001:100
  route-target export 65002:100
  peer 10.0.1.2 enable                # 域内
  peer 10.1.1.1 enable                # 域A
  peer 10.2.1.1 enable                # 域B
 quit
quit

# ---- 域B的ASBR（PE-B1, AS 65002）----
[PE-B1] bgp 65002
 router-id 10.2.1.1
 peer 10.2.1.2 as-number 65002       # 域B内其他PE
 peer 10.0.1.1 as-number 65000       # PE-P1
 peer 10.0.1.1 connect-interface LoopBack 0
 peer 10.0.1.1 ebgp-max-hop 3
 peer 10.0.1.1 next-hop-local
 vpn-instance PAYMENT_B
  rd 65002:100
  route-target export 65002:100
  route-target import 65002:100
  route-target export 65000:100
  route-target import 65000:100
  route-target import 65001:100      # 导入域A路由
  peer 10.2.1.2 enable
  peer 10.0.1.1 enable
 quit
quit

跨域AS Path处理策略：

4.2 案例分析：AS Path环路检测导致的二次故障

修复RT配置后，银行部分分支机构的CE仍无法访问支付清算中心。进一步排查发现：

现象：某些PE-A设备上报BGP告警"AS Path Loop Detected"。

原因分析： - 域A使用AS 65001，域B使用AS 65002，骨干域使用AS 65000。 - 当域B的路由经过骨干域传递到域A时，AS Path中包含65000。 - 部分PE-A设备上配置了错误的as-path replace策略，将本地AS号65001错误地添加到了从骨干域传来的路由AS Path中。 - 当这些路由再经骨干域传递回域B时，域B的PE检测到AS Path中的65001（虽然65001不是域B的AS号），但由于某些PE也运行AS 65001（多归属场景），触发环路检测。

解决方案：

# 在骨干域ASBR上配置AS Path替换
[PE-P1] route-policy BANK_A_TO_CROSS import
[PE-P1-route-policy-BANK_A_TO_CROSS] if-match as-path-filter 1
[PE-P1-route-policy-BANK_A_TO_CROSS] apply as-path replace 65000
[PE-P1-route-policy-BANK_A_TO_CROSS] quit

# AS Path Filter（匹配域A的AS号）
[PE-P1] ip as-path-filter 1 _65001_

4.3 跨域互通排障命令速查

4.4 跨域互通自动化验证脚本

#!/usr/bin/env python3
# mplscross_verify.py - MPLS VPN跨域互通自动验证
import paramiko
import re
import json
import csv
import sys
from datetime import datetime
from pathlib import Path


class MplsVpnVerifier:
    """MPLS VPN跨域互通自动验证器"""

    def __init__(self, device_ip: str, username: str, password: str):
        self.device_ip = device_ip
        self.username = username
        self.password = password
        self.ssh = None
        self.hostname = ""

    def connect(self):
        self.ssh = paramiko.SSHClient()
        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        self.ssh.connect(self.device_ip, username=self.username, password=self.password)
        # 获取主机名
        stdin, stdout, stderr = self.ssh.exec_command("display this | include ^sysname")
        self.hostname = stdout.read().decode().split(":")[-1].strip()

    def exec(self, cmd: str) -> str:
        stdin, stdout, stderr = self.ssh.exec_command(cmd)
        return stdout.read().decode()

    def get_vrf_info(self, vrf_name: str) -> dict:
        """获取VRF的RD和RT信息"""
        output = self.exec(f"display vpn-instance vpn-instance {vrf_name}")
        rd = re.search(r"RD\s*:\s*(.+)", output)
        export_rt = re.findall(r"Route-Target\s*:\s*export\s*(.+)", output)
        import_rt = re.findall(r"Route-Target\s*:\s*import\s*(.+)", output)
        return {
            "vrf_name": vrf_name,
            "rd": rd.group(1).strip() if rd else "N/A",
            "export_rt": [rt.strip() for rt in export_rt],
            "import_rt": [rt.strip() for rt in import_rt]
        }

    def get_vrf_routes(self, vrf_name: str) -> list[dict]:
        """获取VRF路由表"""
        output = self.exec(f"display vpn-instance routing-table vpn-instance {vrf_name}")
        routes = []
        for line in output.strip().split("\n")[2:]:
            parts = line.split()
            if len(parts) >= 4:
                routes.append({
                    "network": parts[0],
                    "nexthop": parts[1],
                    "preference": parts[2],
                    "cost": parts[3] if len(parts) > 3 else "0"
                })
        return routes

    def check_bgp_peers(self, vrf_name: str = "") -> list[dict]:
        """检查BGP VPNv4 Peer状态"""
        cmd = f"display bgp vpnv4 all peer{' vpn-instance ' + vrf_name if vrf_name else ''}"
        output = self.exec(cmd)
        peers = []
        for line in output.strip().split("\n"):
            match = re.search(r"Peer\s+([\d.]+).*State\s+(\w+)", line)
            if match:
                peers.append({
                    "peer_ip": match.group(1),
                    "state": match.group(2)
                })
        return peers

    def check_label_forwarding(self, vrf_name: str, prefix: str) -> dict:
        """检查Label转发"""
        output = self.exec(f"display mpls vpn-instance {vrf_name} routing-table {prefix}")
        inner_label = re.search(r"Inner Label\s*:\s*(\d+)", output)
        outer_label = re.search(r"Outer Label\s*:\s*(\d+)", output)
        nexthop = re.search(r"NextHop\s*:\s*([\d.]+)", output)
        return {
            "prefix": prefix,
            "inner_label": inner_label.group(1) if inner_label else "N/A",
            "outer_label": outer_label.group(1) if outer_label else "N/A",
            "nexthop": nexthop.group(1) if nexthop else "N/A"
        }

    def verify_cross_domain(self, vrf_name: str, expected_prefixes: list[str],
                             expected_import_rt: list[str]) -> list[dict]:
        """完整的跨域验证"""
        results = []

        # 1. 检查VRF的RT Import配置
        vrf_info = self.get_vrf_info(vrf_name)
        missing_rt = []
        for rt in expected_import_rt:
            if rt not in vrf_info["import_rt"]:
                missing_rt.append(rt)
        if missing_rt:
            results.append({
                "check": "RT Import配置",
                "status": "FAIL",
                "detail": f"缺失Import RT: {', '.join(missing_rt)}"
            })
        else:
            results.append({"check": "RT Import配置", "status": "PASS",
                           "detail": f"Import RT: {', '.join(vrf_info['import_rt'])}"})

        # 2. 检查跨域路由是否存在
        routes = self.get_vrf_routes(vrf_name)
        route_map = {r["network"]: r for r in routes}
        for prefix in expected_prefixes:
            if prefix in route_map:
                r = route_map[prefix]
                results.append({
                    "check": f"路由 {prefix}",
                    "status": "PASS",
                    "detail": f"NextHop={r['nexthop']}, Pref={r['preference']}"
                })
            else:
                results.append({
                    "check": f"路由 {prefix}",
                    "status": "FAIL",
                    "detail": f"跨域路由 {prefix} 在 VRF {vrf_name} 中缺失！"
                })

        # 3. 检查BGP Peer状态
        peers = self.check_bgp_peers(vrf_name)
        for peer in peers:
            if peer["state"] == "Established":
                results.append({"check": f"BGP Peer {peer['peer_ip']}",
                               "status": "PASS", "detail": "Established"})
            else:
                results.append({"check": f"BGP Peer {peer['peer_ip']}",
                               "status": "FAIL", "detail": f"State={peer['state']}"})

        self.ssh.close()
        return results


def run_verification(devices_file: str, expected_cfg: str):
    """批量执行跨域验证"""
    # 读取设备列表
    devices = []
    with open(devices_file) as f:
        for row in csv.DictReader(f):
            devices.append(row)

    # 读取期望配置
    expected = json.loads(Path(expected_cfg).read_text())

    all_results = []
    for dev in devices:
        print(f"\n验证设备: {dev['ip']}...")
        verifier = MplsVpnVerifier(dev["ip"], dev["username"], dev["password"])
        try:
            verifier.connect()
            for vrf_check in expected["vrf_checks"]:
                results = verifier.verify_cross_domain(
                    vrf_check["vrf_name"],
                    vrf_check["expected_prefixes"],
                    vrf_check["expected_import_rt"]
                )
                all_results.extend([{
                    "device": dev["ip"],
                    "hostname": verifier.hostname,
                    "vrf": r["check"],
                    "status": r["status"],
                    "detail": r["detail"]
                } for r in results])
        except Exception as e:
            print(f"  连接失败: {e}")

    # 输出报告
    ts = datetime.now().strftime("%Y%m%d_%H%M%S")
    report_file = f"cross_domain_verify_{ts}.csv"
    with open(report_file, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["device", "hostname", "vrf", "status", "detail"])
        writer.writeheader()
        writer.writerows(all_results)

    fails = [r for r in all_results if r["status"] == "FAIL"]
    print(f"\n{'='*60}")
    print(f"验证完成: {len(all_results)}项, 通过{len(all_results)-len(fails)}, 失败{len(fails)}")
    print(f"报告: {report_file}")
    for f_item in fails:
        print(f"  🔴 {f_item['device']}({f_item['hostname']}): {f_item['detail']}")


if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument("--devices", default="devices.csv")
    parser.add_argument("--expected", default="expected_config.json")
    args = parser.parse_args()
    run_verification(args.devices, args.expected)

expected_config.json 示例：

{
  "vrf_checks": [
    {
      "vrf_name": "BANK_A",
      "expected_prefixes": ["10.2.0.0/24", "10.2.1.0/24", "10.2.2.0/24"],
      "expected_import_rt": ["65000:100", "65002:100"]
    },
    {
      "vrf_name": "BANK_CROSS",
      "expected_prefixes": ["10.1.0.0/24", "10.2.0.0/24"],
      "expected_import_rt": ["65001:100", "65002:100"]
    }
  ]
}

4.5 跨域互通排障流程图

4.5.1 跨域故障排查扩展流程图

跨域排查Checklist（逐跳）：

五、经验总结

思考

1. 如果客户域A和域B使用相同的AS号（如都是AS 65001），跨域路由泄露会面临什么问题？如何解决？
2. MPLS VPN跨域互通中，如果骨干域使用SR-TE隧道替代LDP，Label栈结构会发生什么变化？
3. 在多域MPLS网络中，如果需要对不同客户使用不同的跨域策略，如何在不修改VRF配置的前提下实现灵活的路由控制？

下篇预告：第336篇：BGP 路由黑洞导致业务中断分析