Chapter 10. MPLS Layer 3 VPN (L3VPN)
The following topics are covered in this chapter:
MPLS Layer 3 VPN (L3VPN) Overview
MPLS Layer 3 VPN Configuration
Virtual Routing and Forwarding (VRF) Instances
Route Targets (RTs) and Route Distinguishers (RDs)
Troubleshooting MPLS L3VPN
A virtual private network (VPN) provides connectivity to private networks over a public network. VPNs operate by tunneling, encrypting the payload, or both. With VPN tunneling, packets destined between private networks are encapsulated and assigned new packet headers that allow the packet to traverse the public network. Tunnels are a form of overlay routing because they exist on top of an existing network, known as the underlay network or a transport network.
The new packet headers provide a method of forwarding the packet across the public network without exposing the private network’s original packet headers. This allows the packet to be forwarded between the two endpoints without requiring any routers from extracting information from the payload (original packet headers and data). After the packet reaches the remote endpoint, the VPN tunnel headers are decapsulated (that is, removed). The endpoint checks the original headers and then forwards the packet out the appropriate interface to the private network.
MPLS VPNs
ISPs use Multiprotocol Label Switching (MPLS) to provide a scalable peer-to-peer architecture that provides a dynamic method of tunneling for packets to transit from provider edge (PE) router to PE router without looking into the original packet’s contents.
With traditional routing, a router receives a packet and checks the header for the destination IP address. It then locates the longest matching route in the forwarding table, performs a recursive lookup to find the outbound interface, and then forwards the packet out of that interface. This process continues for every hop (router) along the path to the packet’s destination.
MPLS forwarding reduces the lookup process by all the routers in the path of a packet. A router assigns a locally significant label (numerical value) for the directly connected prefixes that are connected to it and then advertises this label to prefix binding to neighboring routers. The neighboring router receives that label and creates a corresponding locally significant label. The process continues where a label exists for all the routes for all the routers in the routing domain.
When a packet is received, the longest match lookup is performed in the forwarding table, and then the associated label is identified. All further forwarding occurs based on the MPLS label. The local router contains a table that correlates its local label with the downstream router’s label, and outbound interface. Forwarding lookups are more explicit and do not require subsequent analysis by routers in the middle of the packet’s path.
Note
MPLS forwarding is more efficient than traditional route lookups and provides a method of tunneling packets that otherwise could not be forwarded in the underlay network. New technologies continue to be developed upon MPLS forwarding, which reinforces the brilliance of the technology.
MPLS networks forward traffic based on the outermost MPLS label of a packet. The MPLS labels are inserted after the Layer 2 information and before the IP headers (source IP and destination IP) in a packet, so none of the transit routers require the examination of the packet’s inner header or payload. As packets cross the core of the network, the source and destination IP addresses are never checked as long as a label exists in the packet. Only the PE routers need to know how to send the packets toward the customer edge (CE) router. MPLS VPNs are considered an overlay network because they are forwarded on the SP’s underlay network by using MPLS labels.
MPLS VPNs are categorized by two technologies on the PE router:
Layer 2 VPN (L2VPN): PE routers provide connectivity to CE routers by creating a virtual circuit between the nodes. An interface on the PE router is associated directly to the virtual circuit. Packets received on the interface are then associated with the virtual circuit, labeled with the circuit-id, then labeled for the remote PE’s IP address, and then forwarded toward the PE attaching to the other end of the circuit.
The PE routers do not participate in the routing of the devices in the private networks.
Layer 3 VPN (L3VPN): PE routers provide connectivity to CE routers by exchanging network prefixes with them. PE routers then exchange those network prefixes with remote PE routers that then advertise the prefixes to the remote CE routers.
PE routers forward network traffic based on IP addresses and not based off circuit-ids.
MPLS VPNs provide a method of segmenting network traffic, but do not encrypt network traffic. Encryption of network traffic is required to ensure data confidentiality or data integrity.
MPLS Layer 3 VPN (L3VPN) Overview
MPLS L3VPNs participate in the routing tables of the CE and PE routers. The technology is based on the following components:
Exchange of network prefixes between CE and PE routers.
Exchange of network prefixes between local PE and remote PE routers. This can be a direct BGP session or through a route-reflector (RR).
Exchange of MPLS labels that are used for forwarding packets between the local PE and remote PE routers.
Forwarding of packets based on outermost MPLS labels.
PE routers use a virtual context known as a Virtual Routing and Forwarding (VRF) context for each customer. Every VRF context provides a method for routers to maintain a separate routing and forwarding table for each customer on a router.
The PE routers should have an abundant amount of CPU and RAM to hold all the routes for every VRF configured on them. The PE routers must contain all the routes for a particular customer (VRF), whereas the CE routers maintain only the routes for the customer that it belongs to. The route table on the PE routers can be programmed via a static route at the local PE router or is learned dynamically from routes that are advertised from the CE router.
PE routers exchange the VRF’s routes with other PE routers using Multi-Protocol BGP (MP-BGP) using a special address family just for MPLS L3VPN networks. VPN labels are associated to each of the VRF’s routes to identify which VRF the routes belong to.
Virtual Routing and Forwarding
Virtual Routing and Forwarding (VRF) is a technology that creates separate virtual routers on a physical router. Router interfaces, routing tables, and forwarding tables are completely isolated between VRFs, preventing traffic from one VRF to forward into another VRF. VRFs are an essential component of the L3VPN architecture and provide increased router functionality through segmentation in lieu of using multiple devices. All router interfaces belong to the global VRF (also known as default VRF) until they are specifically assigned to a different VRF. The global VRF is identical to the regular routing table of non-VRF routers.
Figure 10-1 demonstrates three VRFs in one physical router—the default global VRF and two user-defined VRFs: VPN01 and VPN02.
Every VRF configured on a router maintains a separate routing table that allows for overlapping IP address ranges between different customers. Configuring VRFs on a router ensures that the paths are isolated, network security is increased, and encrypting traffic on the network is not needed to maintain privacy between VRFs.
Figure 10-2 displays a simple peer-to-peer topology where a service provider needs to deliver connectivity to two different customers while ensuring that routes are not mixed between customers.
By using VRFs, the service provider can use one router instead of two. Figure 10-3 displays the same topology except that PE1 is using VRFs to separate the customer’s network on PE1. CE1 and CE3 belong to VRF VPN01 for Customer A, and CE2 and CE4 belong to VRF VPN02 for Customer B. Configuration changes are needed only on PE1, and all the CE routers are configured normally (that is, no VRF configuration).
CE1 maintains connectivity with CE3, but neither router can see any VRF VPN02 networks (172.16.2.0/24, 172.16.4.0/24, 172.16.12.0/24, 172.16.14.0/24). CE2 maintains connectivity with CE4, but neither router can see any VRF VPN01 network (172.16.1.0/24, 172.16.3.0/24, 172.16.11.0/24, 172.16.13.0/24).
Route Distinguisher
A route distinguisher (RD) is a method of providing uniqueness to a network prefix regardless of which VRF context the route belongs to. Routes are exchanged between other service provider routers as a VPNv4 prefix that includes the following:
8-byte (64 bit) RD identifier.
4-bytes (32 bit) for the actual IPv4 network. IPv6 networks will use 16 bytes.
The RD identifier is commonly expressed by two 4-byte identifiers separated by a colon. The first 4 bytes of the RD generally refers to the autonomous system number (ASN) of the service provider, and the last 4 bytes of the RD is an incremental number. An example RD would be 100:1 within AS100’s network. Only one RD can be configured on a PE router per VRF. A VPN prefix is the RD combined with the actual IPv4 or IPv6 network.
Note
The RD is not used to link a network prefix to a specific VRF. It is used strictly to guarantee the uniqueness of a network prefix. The RD can be the same or a different value for the same VRF on two different PE routers based on design.
Route Target
BGP route targets (RTs) are extended BGP communities that identify the VPNv4 or VPNv6 routes that should be associated with a VRF. Route targets are an 8-byte value (64 bits) that are expressed by two 4-byte identifiers separated by a colon. The first set generally refers to the ASN of the service provider, and the second set is an incremental value. An example RT is 100:1 within AS100’s network.
In the router’s VRF definition, the RT is defined globally or for a specific address-family with one or more RTs for routes that are advertised. As the router advertises a VPN prefix to another PE router, the export RTs are added as a BGP path attribute to the VPN prefix. Upon receipt of a VPN prefix, the RTs are matched to the import RTs for the local router’s VRF. If the VPN prefix’s RTs match the local VRFs import RTs, the route is imported to that specific VRF.
Note
A router discards the VPN prefixes if the RT in the VPN prefix does not match any of the import RTs for the VRFs on the router. This is done to conserve router memory. Cisco refers to this as a route-target filter, and it is enabled by default.
A router can have multiple import RTs and multiple export RTs per VRF. The RTs do not have to match, but the export RTs of one router must match the import RTs of another router for the route to be associated to the VRF. Typically, a service provider keeps the RTs the same for import and export on all VRFs spread across all PE routers to simplify troubleshooting.
Note
The capability of having multiple RT path attributes increases scalability and functionality in the routing design. This concept is explained later in this chapter.
Multi-Protocol BGP (MP-BGP)
A BGP address-family identifier (AFI) correlates to a specific network protocol, such as IPv4, IPv6, and the like, and additional granularity through subsequent address-family identifier (SAFI), such as unicast and multicast. These attributes are carried inside BGP update messages and are used to carry network reachability information for different address families.
The additional length of a VPN prefix (RD + Network) requires a different address-family in BGP to exchange routes with other PE routers. MPLS L3VPN uses the AFI 1, SAFI 128 for IPv4 VPN prefixes and is referred to as the VPNv4 address-family. IPv6 network prefixes use the AFI 2, SAFI 128 and is referred to as VPNv6 address-family. For the remainder of this chapter, VPNv4 address families will be the implied address-family, but the same functionality exists with VPNv6 address-family too.
PE routers need to establish a VPNv4 BGP session with other PE routers. Like traditional IPv4 prefixes, a router does not advertise network prefixes learned from an IBGP peer to another IBGP peer. This means that a full mesh of VPNv4 BGP sessions must be formed between PE routers, or a route-reflector can be used.
Network Advertisement Between PE and CE Routers
All the LAN networks attached to the CE routers need to be advertised to the PE router so that it can advertise those routes to other PE routers. Network prefixes can be statically advertised with the network command, but dynamic redistribution from the connected interface database or from a routing protocol is considered more scalable.
The BGP VPNv4 address-family is how network prefixes are exchanged between PE routers. Traditionally the VPNv4 address-family is an IBGP session between other route reflectors (RR) and PE routers. The most intuitive solution is to use BGP as the PE-CE routing protocol because routes learned in the VRF context in BGP do not have to be redistributed into or out of BGP. The PE-CE BGP session is normally an EBGP session. Using other protocols like static routes, connected networks, Enhanced Interior Gateway Routing Protocol (EIGRP), Open Shortest Path First (OSPF), Intermediate-System to Intermediate System (IS-IS), and so on require redistribution (generally mutually with a tag-and-block technique to prevent routing loops) with the VRF context in BGP.
This book focuses only on BGP as the PE-CE routing protocol.
MPLS Layer 3 VPN Configuration
Now that an overview of the technologies and concepts has been provided, this chapter offers a brief overview of the components. Figure 10-4 displays a topology containing three PE routers: PE1 (IOS), PE2 (IOS-XR), and PE3 (NX-OS), and a VPNv4 route reflector (RR). This topology will be the primary topology to demonstrate how to troubleshoot BGP in a MPLS L3VPN network.
Table 10-1 displays a list of relevant networks, IP addresses, and BGP information for our topology. All three PE routers peer with the VPNv4 route-reflector at 192.168.100.100. The PE routers all use the same RD (1:1) and RT (1:1).
VRF Creation and Association
This section explains the VRF creation process and how a VRF is associated to an interface.
IOS VRF Creation
In IOS, the following steps are required to create a VRF and assign it to an interface:
Step 1. Define the VRF. The VRF is defined with the command vrf definition vrf-name. The VRF name is locally relevant and does not impact connectivity with other routers.
Step 2. Define the route distinguisher (RD). An RD ensures uniqueness of a route between multiple VRFs across multiple PE routers. The RD is configured with the command rd {as-number:nn | ip-address:nn}.
Step 3. Initialize the address-family. Initialize the appropriate address-family with the command address-family {ipv4 | ipv6}. The address-family can be IPv4, IPv6, or both. Failing to initialize the address-family prohibits the appropriate IPv4 or IPv6 addresses from being configured on a VRF-enabled interface.
Step 4. Identify BGP route targets. BGP route targets (RT) are extended BGP communities that identify the VPNv4 or VPNv6 routes that should be associated with a VRF. The command route-target [import | export | both] as-number:nn identifies the RTs used upon the import or export of the routes to other VPNv4 or VPNv6 BGP neighbors. If the import or export keyword is not included, then both is implied. Multiple route targets can be associated to a VRF.
Step 5. Associate the VRF to the interfaces. In interface configuration submode, specify the interface to be associated with the VRF with the interface parameter command vrf forwarding vrf-name. The VRF must be associated to the interface first before configuring an IP address. If an IP address is already configured on the interface, and the VRF is associated to the interface, IOS removes the IP address.
Step 6. Configure an IP address on the interface or subinterface. The IP address needs to be associated to the interface with the command ip address ip-address subnet-mask [secondary].
Note
Older IOS versions supported only IPv4 based VRFs and used the commands ip vrf vrf-name for the VRF definition and ip vrf forwarding vrf-name for associating the VRF to an interface. Eventually these commands will be deprecated.
The command vrf upgrade-cli multi-af-mode {common-policies | non-common-policies} [vrf vrf-name] only upgrades IPv4-based VRF to a multiprotocol VRF. Cisco recommends using the multiprotocol VRF vrf definition and vrf forwarding commands.
Example 10-1 demonstrates the VRF creation and assignment to interfaces on the IOS router PE1. Notice how the previous IP address was automatically removed from the interface when the VRF was assigned.
Example 10-1 VRF Creation and Interface Assignment for IOS
PE1-IOS# configuration t
Enter configuration commands, one per line. End with CNTL/Z.
PE1-IOS(config)# vrf definition VPN01
PE1-IOS(config-vrf)# rd 1:1
PE1-IOS(config-vrf)# address-family ipv4 unicast
PE1-IOS(config-vrf-af)# route-target 1:1
PE1-IOS(config-vrf-af)# interface gi0/2
PE1-IOS(config-if)# vrf forwarding VPN01
% Interface GigabitEthernet0/2 IPv4 disabled and address(es) removed due to enabling
VRF VPN01
PE1-IOS(config-if)# ip address 172.16.1.1 255.255.255.192
PE1-IOS(config-if)# interface loopback100
PE1-IOS(config-if)# vrf forwarding VPN01
PE1-IOS(config-if)# ip address 172.16.1.200 255.255.255.255
IOS XR VRF Creation
The following steps are required to create a multiprotocol VRF and assign it to an interface on an IOS XR node:
Step 1. Define the VRF. The VRF is defined with the command vrf vrf-name. The VRF name is locally relevant and does not impact connectivity with other routers.
Step 2. Initialize the address-family. Initialize the appropriate address-family with the command address-family {ipv4 | ipv6} unicast. The address-family can be IPv4, IPv6, or both. Failing to initialize the address-family prohibits the appropriate IPv4 or IPv6 address from being configured on a VRF-enabled interface.
Step 3. Identify BGP route targets. BGP route targets are extended BGP communities that identify the VPNv4 or VPNv6 routes that should be associated with a VRF. The import RT is configured with the command import route-target as-number:nn and identifies the RTs that should be imported to the VRF from VPNv4 prefixes. The command export route-target as-number:nn sets the RT on VPNv4 prefixes that are advertised to other PE routers. Multiple route targets can be associated to a VRF.
Step 4. Associate the VRF to the interfaces. In interface configuration submode, specify the interface to be associated with the interface parameter configuration command vrf vrf-name. The VRF needs to be associated to the interface first before configuring an IP address. Unlike IOS and NX-OS, the IP address is not automatically removed and prohibits the change from being committed. If an IP address is already configured, remove the IP address and commit the change first.
Step 5. Configure an IP address on the interface or subinterface. The IP address is configured by entering the commands ipv4 address ipv4-address subnet-mask.
Example 10-2 demonstrates the VRF creation and assignment to interfaces on the IOS XR router PE2. An IP address was not present on the Gi0/0/0/1 interface before the configuration because this can cause the commit to fail.
Example 10-2 VRF Creation and Interface Assignment for IOS-XR
RP/0/0/CPU0:PE2-XR# configuration t
RP/0/0/CPU0:PE2-XR(config)# vrf VPN01
RP/0/0/CPU0:PE2-XR(config-vrf)# address-family ipv4 unicast
RP/0/0/CPU0:PE2-XR(config-vrf-af)# import route-target 1:1
RP/0/0/CPU0:PE2-XR(config-vrf-af)# export route-target 1:1
RP/0/0/CPU0:PE2-XR(config-vrf-af)# interface gi0/0/0/1
RP/0/0/CPU0:PE2-XR(config-if)# vrf VPN01
RP/0/0/CPU0:PE2-XR(config-if)# ipv4 address 172.16.2.1 255.255.255.192
RP/0/0/CPU0:PE2-XR(config-if)# interface loopback 100
RP/0/0/CPU0:PE2-XR(config-if)# vrf VPN01
RP/0/0/CPU0:PE2-XR(config-if)# ipv4 address 172.16.2.200 255.255.255.255
RP/0/0/CPU0:PE2-XR(config-if)# commit
NX-OS VRF Creation
In NX-OS, the following steps are required to create a VRF and assign it to an interface:
Step 1. Define the VRF. The VRF is defined with the command vrf context vrf-name. The VRF name is locally relevant and does not impact connectivity with other routers.
Step 2. Define the route distinguisher. An RD ensures uniqueness of a route between multiple VRFs across multiple PE routers. The RD is configured with the command rd {as-number:nn | ip-address:nn}.
Step 3. Initialize the address-family. Initialize the appropriate address family with the command address-family {ipv4 | ipv6}. The address-family can be IPv4, IPv6, or both. Failing to initialize the address-family prohibits the appropriate IPv4 or IPv6 addresses from being configured on a VRF-enabled interface.
Step 4. Identify BGP route targets. BGP RTs are extended BGP communities that identify the VPNv4 or VPNv6 routes that should be associated with a VRF. The command route-target {import | export | both} as-number:nn identifies the route targets used on the import or export of the routes to other VPNv4 or VPNv6 BGP neighbors. Multiple route targets can be associated to a VRF.
Step 5. Associate the VRF to the interfaces. In interface configuration submode, specify the interface to be associated with the VRF with the interface parameter command vrf member vrf-name. The VRF needs to be associated to the interface first before configuring an IP address. If an IP address is already configured on the interface, and the VRF is associated to the interface, NX-OS removes the IP address.
Step 6. Configure an IP address on the interface or subinterface. The IP address needs to be associated to the interface with the command ip address ip-address /prefix-length [secondary].
Example 10-3 demonstrates the VRF creation and assignment to interfaces on the NX-OS device PE3. Notice how the previous IP address was automatically removed from the interface when the VRF was assigned.
Example 10-3 VRF Creation and Interface Assignment for NX-OS
PE3-NXOS# configuration t
PE3-NXOS(config)# vrf context VPN01
PE3-NXOS(config-vrf)# rd 1:1
PE3-NXOS(config-vrf)# address-family ipv4 unicast
PE3-NXOS(config-vrf-af-ipv4)# route-target both 1:1
PE3-NXOS(config-vrf-af-ipv4)# interface e2/2
PE3-NXOS(config-if)# vrf member VPN01
04:26:57 PE3-NXOS %CLIS-6-CLIS_SYSLOG_CMD_REMOVED: Command deleted from the NVDB: ip
address 172.16.3.1/26
Warning: Deleted all L3 config on interface Ethernet2/2
PE3-NXOS(config-if)# ip address 172.16.3.1/26
PE3-NXOS(config-if)# interface loopback100
PE3-NXOS(config-if)# vrf member VPN01
PE3-NXOS(config-if)# ip address 172.16.3.200/32
Verification of VRF Settings and Connectivity
After creating VRFs and assigning them to an interface, it is recommended to verify the changes. This section also explains some of the commands to view the routing table and provide connectivity tests with in the VRF context.
Viewing VRF Settings and Interface IP Addresses
IOS routers display a quick summary of the VRF(s) that include the VRF name, RD, IP protocols, and interfaces, with the command show vrf [vrf-name] [detail]. The vrf-name is optional, and the optional detail keyword provides additional information, such as route targets, and import/export route-maps.
IOS XR routers display the VRF name, RD, RT(s), and IP protocols with the command show vrf [vrf-name | all]. IOS XR includes the RD, import/export route targets and the address-families that have been initialized for that VRF.
NX-OS devices display the VRF information with the command show vrf [vrf-name | all] [interface | detail]. Only the internal VRF-ID and state is shown if the optional keywords are not used. The interface keyword displays the interfaces that are associated to the VRF.
Example 10-4 provides sample output of the commands. This is the first step of troubleshooting PE-CE connectivity in verifying that the CE is in the correct VRF on the PE.
Example 10-4 Display of VRF Interface Information
PE1-IOS# show vrf
Name Default RD Protocols Interfaces
VPN01 1:1 ipv4 Gi0/2
Lo100
RP/0/0/CPU0:PE2-XR# show vrf all
VRF RD RT AFI SAFI
VPN01 not set
import 1:1 IPV4 Unicast
export 1:1 IPV4 Unicast
PE3-NXOS# show vrf VPN01 interface
Interface VRF-Name VRF-ID Site-of-Origin
loopback100 VPN01 3 --
Ethernet2/2 VPN01 3 --
IOS routers display IP addresses in all VRFs (default, and custom defined) with the command show ip interface [brief]. The interface status can be narrowed to all VRFs or a specific VRF with the command show vrf {ipv4 | ipv6} interface [vrf-name] on IOS routers. Example 10-5 demonstrates the output.
Example 10-5 Viewing VRF IP Interfaces on IOS Routers
PE1-IOS# show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 10.255.1.89 YES NVRAM administratively down down
GigabitEthernet0/1 10.1.11.1 YES manual up up
GigabitEthernet0/2 172.16.1.1 YES manual up up
Loopback0 192.168.1.1 YES NVRAM up up
Loopback100 172.16.1.200 YES manual up up
PE1-IOS# show vrf ipv4 interface VPN01
Interface VRF Protocol Address
GigabitEthernet0/2 VPN01 up 172.16.1.1
Loopback100 VPN01 up 172.16.1.200
IOS XR and NX-OS devices only show the IP addresses in the default routing table with the command show ip interface brief. To see the IP addresses in the global routing table and all VRFs, use the command show ipv4 vrf all interface brief on IOS XR routers, and the command show ip interface brief vrf all on NX-OS devices. Example 10-6 provides sample output of these commands.
Example 10-6 Viewing VRF IP Interfaces on IOS XR and NX-OS Devices
RP/0/0/CPU0:PE2-XR# show ip vrf all interface brief
Interface IP-Address Status Protocol Vrf-Name
Loopback0 192.168.2.2 Up Up default
Loopback100 172.16.2.200 Up Up VPN01
GigabitEthernet0/0/0/0 10.1.22.2 Up Up default
GigabitEthernet0/0/0/1 172.16.2.1 Up Up VPN01
PE3-NXOS# show ip interface brief vrf all
IP Interface Status for VRF "default"(1)
Interface IP Address Interface Status
Lo0 192.168.3.3 protocol-up/link-up/admin-up
Eth2/1 10.1.33.3 protocol-up/link-up/admin-up
IP Interface Status for VRF "VPN01"(2)
Interface IP Address Interface Status
Lo100 172.16.3.200 protocol-up/link-up/admin-up
Eth2/2 172.16.3.1 protocol-up/link-up/admin-up
Viewing the VRF Routing Table
Examining the routing table from the VRF context is a vital component of troubleshooting. The commands for displaying the routing table on all three operating systems are as follows:
IOS routers allow only the viewing of the routing table of one VRF with the command show ip route vrf vrf-name.
IOS XR routers use the command show route vrf {vrf-name | all}.
NX-OS devices use the command show ip route vrf {vrf-name | all}.
IOS XR and NX-OS devices allow for the viewing of multiple VRF routing tables by using the all keyword in lieu of specifying a VRF name. With all three command sets, additional information can be seen by appending other keywords like network /prefix-length after the initial command. Example 10-7 displays sample content for all three types of routers.
Example 10-7 Viewing the VRF Routing Table
PE1-IOS# show ip route vrf VPN01
! Output omitted for brevity
Routing Table: VPN01
Gateway of last resort is not set
172.16.0.0/16 is variably subnetted, 10 subnets, 2 masks
C 172.16.1.0/26 is directly connected, GigabitEthernet0/2
L 172.16.1.1/32 is directly connected, GigabitEthernet0/2
B 172.16.1.100/32 [20/0] via 172.16.1.11, 1w3d
C 172.16.1.200/32 is directly connected, Loopback100
RP/0/0/CPU0:PE2-XR# show route vrf VPN01
! Output omitted for brevity
Gateway of last resort is not set
C 172.16.2.0/26 is directly connected, 1d11h, GigabitEthernet0/0/0/1
L 172.16.2.1/32 is directly connected, 1d11h, GigabitEthernet0/0/0/1
L 172.16.2.200/32 is directly connected, 1d11h, Loopback100
PE3-NXOS# show ip route vrf VPN01
! Output omitted for brevity
IP Route Table for VRF "VPN01"
172.16.3.0/26, ubest/mbest: 1/0, attached
*via 172.16.3.1, Eth2/2, [0/0], 1d10h, direct
172.16.3.1/32, ubest/mbest: 1/0, attached
*via 172.16.3.1, Eth2/2, [0/0], 1d10h, local
172.16.3.200/32, ubest/mbest: 2/0, attached
*via 172.16.3.200, Lo100, [0/0], 1d10h, local
*via 172.16.3.200, Lo100, [0/0], 1d10h, direct
VRF Connectivity Testing Tools
The ping and traceroute utilities are vital tools for troubleshooting path failure issues. When working with VRF contexts, the vrf must be specified.
For IOS and IOS XR routers, the command is ping vrf vrf-name destination-ip-address [source {interface-id | ip-address}] or the command traceoute vrf vrf-name destination-ip-address [source {interface-id | ip-address}]. NX-OS devices use the command ping destination-ip-address vrf vrf-name [source ip-address] or the command traceroute destination-ip-address vrf vrf-name [source ip-address].
Example 10-8 displays the use of the ping command on the PE routers with their local peer CE device.
Example 10-8 VRF Connectivity Test Tools
PE1-IOS# ping vrf VPN01 172.16.1.11
Sending 5, 100-byte ICMP Echos to 172.16.1.11, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/3/7 ms
RP/0/0/CPU0:PE2-XR# ping vrf VPN01 172.16.2.22
Sending 5, 100-byte ICMP Echos to 172.16.2.22, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
PE3-NXOS# ping 172.16.3.33 vrf VPN01
PING 172.16.3.33 (172.16.3.33): 56 data bytes
64 bytes from 172.16.3.33: icmp_seq=0 ttl=254 time=2.333 ms
64 bytes from 172.16.3.33: icmp_seq=1 ttl=254 time=2.237 ms
64 bytes from 172.16.3.33: icmp_seq=2 ttl=254 time=2.413 ms
64 bytes from 172.16.3.33: icmp_seq=3 ttl=254 time=2.856 ms
64 bytes from 172.16.3.33: icmp_seq=4 ttl=254 time=4.028 ms
--- 172.16.3.33 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 2.237/2.773/4.028 ms
MPLS Forwarding
MPLS forwarding needs to be enabled in the service provider network. There are multiple ways that MPLS labels can be exchanged between routers. MPLS Label Distribution Protocol (LDP) is common and is enabled fairly quickly.
On IOS routers the mpls ip command is placed under each of the interfaces in the service provider network that may forward labeled packets. On older IOS versions of code, MPLS may need to be enabled globally with the command mpls ip.
IOS XR routers require the usage of an MPLS package that must be installed before configuration. IOS XR routers initialize the LDP process with the command mpls ldp. All the interfaces listed under the LDP process will be LDP enabled. MPLS operations, administration, and maintenance (OAM) must be enabled with the command mpls oam so that the path can be verified with ping and traceroute commands.
NX-OS devices must enable MPLS-based services with the following commands in the order provided: install feature-set mpls, feature-set mpls, feature mpls l3vpn, and feature mpls ldp. After all the MPLS features have been enabled, LDP is enabled globally with the command mpls ip and then again under any MPLS interfaces.
Example 10-9 displays the basic configuration for enabling MPLS on the PE routers. All three PE routers have only one internal interface connecting to the service provider network. Only that interface needs MPLS enabled on it.
Example 10-9 Sample MPLS LDP Configuration
PE1-IOS
mpls ip
!
interface GigabitEthernet0/1
description to P2
ip address 10.1.11.1 255.255.255.0
mpls ip
PE2-XR
mpls oam
!
mpls ldp
router-id 192.168.2.2
interface GigabitEthernet0/0/0/0
PE3-NXOS
mpls ip
!
interface Ethernet2/1
description to P3
no switchport
mpls ip
ip address 10.1.33.3/24
ip router ospf 1 area 0.0.0.0
no shutdown
BGP Configuration for VPNv4 and PE-CE Prefixes
The BGP configuration consists of the following components:
Advertisement of VPNv4 prefixes to other PE routers or route reflector (RR).
Establishment of the BGP session between the PE and CE routers. The BGP configuration resides under the VRF context.
Route advertisement in the VRF context. Any routes that exist in the BGP VRF context are advertised as a VPNv4 prefix to other PE routers. This is where networks are specifically defined or redistributed.
The configuration for all three platforms varies slightly and is explained next:
IOS BGP Configuration for MPLS L3VPN
The following steps show how to configure BGP routing for MPLS L3VPN on an IOS router. The RD must be configured in the VRF definition before configuring BGP.
Step 1. Create the BGP routing process. Initialize the BGP process with the global command router bgp as-number.
Step 2. Disable the Default IPv4 address-family. IOS enables the IPv4 address-family by default. Disabling this behavior ensures that only the intended address-family is initialized. The IPv4 address-family is disabled with the command no bgp default ipv4-unicast.
Step 3. Define the VPNv4 neighbor session configuration. The other PE routers (or route reflectors) need to have the session parameters defined. The remote ASs must be defined with the command neighbor ip-address remote-as as-number and the BGP session needs to be sourced by a loopback address with a /32 netmask with the command neighbor ip-address update-source interface-id. Additional session configuration options such as authentication can be added as well.
Step 4. Initialize the VPNv4 address-family. The VPNv4 address-family needs to be initialized with the command address-family vpnv4 unicast.
Step 5. Activate the BGP VPNv4 address-family for the PE or RR session. Now that the BGP session has been defined and the VPNv4 address-family is initialized, the BGP neighbor needs to be activated under the VPNv4 address-family with the command neighbor ip-address activate.
Step 6. Enable BGP extended communities. MPLS L3VPN is dependent on the usage of extended BGP communities for the advertisement of route targets and other BGP path attributes. The command neighbor ip-address send-community {extended | both} is required.
Other BGP address-family commands, such as defining the route-reflector client or configuring route-maps, are defined here. This concludes the configuration of the VPNv4 configuration.
Step 7. Initialize the address-family for a specific VRF. Initialize the address-family with the BGP router configuration command address-family afi [safi] vrf vrf-name.
Step 8. Configure the neighbor’s CE configuration settings. Identify the BGP neighbor’s IP address and autonomous system number with the BGP router configuration command neighbor ip-address remote-as as-number. Additional neighbor configuration occurs under the BGP address-family VRF context for route-maps, BGP sessions, and the like. The BGP session options and address-family options are all listed under this section.
Step 9. Advertise or redistribute routes in the VRF. Route advertisement or redistribution occurs under the BGP address-family VRF context. Typically, most service providers redistribute the connected routes into the VRF because it includes the peering link at a minimum.
Example 10-10 displays the BGP configuration for the VPNv4 address-family to the route reflector (192.168.100.100) and the CE1 router (172.16.1.11) in remote AS100.
Example 10-10 IOS BGP VPNv4 and PE-CE Configuration
PE1-IOS# show run | s router bgp
router bgp 65000
bgp log-neighbor-changes
no bgp default ipv4-unicast
neighbor 192.168.100.100 remote-as 65000
neighbor 192.168.100.100 description VPNv4 Route-Reflector
neighbor 192.168.100.100 update-source Loopback0
!
address-family ipv4
exit-address-family
!
address-family vpnv4
neighbor 192.168.100.100 activate
neighbor 192.168.100.100 send-community extended
exit-address-family
!
address-family ipv4 vrf VPN01
redistribute connected
neighbor 172.16.1.11 remote-as 100
neighbor 172.16.1.11 activate
exit-address-family
IOS XR BGP Configuration for MPLS L3VPN
The following steps show how to configure BGP routing for MPLS L3VPN on an IOS XR router. The RD is defined as part of the BGP process:
Step 1. Create the BGP routing process. Initialize the BGP process with the global configuration command router bgp as-number.
Step 2. Initialize the VPNv4 address-family. The VPNv4 address-family needs to be initialized with the command address-family vpnv4 unicast.
Step 3. Define the VPNv4 neighbors. Configure the BGP VPNv4 neighbor’s IP address with the BGP router configuration command neighbor ip-address.
Step 4. Define the VPNv4 BGP neighbor’s session information. The remote AS must be defined with the command remote-as as-number. The BGP session needs to be sourced by a loopback address with a /32 netmask with the command neighbor ip-address update-source interface-id. Any other additional session information, such as passwords, can be defined here as well.
Step 5. Activate the BGP VPNv4 address-family for the PE or RR session. Activate the address-family for the BGP neighbor with the BGP neighbor configuration command address-family afi safi.
Note
Additional BGP address-family specific commands, such as defining a route-reflector client or route policies, are configured here. There is no need to enable extended communities because those are automatically advertised to IBGP peers by IOS XR routers. This concludes the VPNv4 configuration.
Step 6. Define the VRF in BGP. The VRF for the BGP instance is defined with the command vrf vrf-name.
Step 7. Define the VRF’s route distinguisher. The RD is configured with the command rd {as-number:nn | ip-address:nn}.
Step 8. Initialize the VRF’s address-family. Initialize the address-family for that VRF with the BGP router configuration command address-family afi safi so it can be associated to a BGP neighbor in that VRF. Any route advertisement or redistribution on the PE occurs under the VRF address-family.
Step 9. Define the VRF’s Neighbor’s CE IP Address. Configure the BGP neighbor’s IP address with the BGP router configuration command neighbor ip-address.
Step 10. Define the neighbor’s BGP autonomous system number. Configure the BGP neighbor’s autonomous system number with the BGP neighbor configuration command remote-as as-number. Any additional BGP session information is configured here.
Step 11. Activate the address-family for the BGP neighbor. Activate the address-family for the BGP neighbor with the BGP neighbor configuration command address-family afi safi. EBGP peers require the user of a route policy to allow the inbound/outbound exchange of BGP prefixes, as explained earlier in Chapter 4, “Troubleshooting Route Advertisement and BGP Policies.”
Example 10-11 displays the BGP configuration for the VPNv4 address-family to the route-reflector (192.168.100.100) and the CE2 router (172.16.2.22) in remote AS200.
Example 10-11 IOS XR BGP VPNv4 and PE-CE Configuration
RP/0/0/CPU0:PE2-XR# show run router bgp
router bgp 65000
address-family vpnv4 unicast
!
neighbor 192.168.100.100
remote-as 65000
update-source Loopback0
address-family vpnv4 unicast
!
!
vrf VPN01
rd 1:1
address-family ipv4 unicast
redistribute connected
!
neighbor 172.16.2.22
remote-as 200
address-family ipv4 unicast
route-policy PASS-ALL in
route-policy PASS-ALL out
Note
IOS XR nodes do not establish a BGP session if the router identifier (RID) is set to zero because dynamic RID allocation did not find any up loopback interfaces. The RID needs to be set manually with the BGP router configuration command bgp router-id router-id.
NX-OS BGP Configuration for MPLS L3VPN
The following steps show how to configure BGP routing for MPLS L3VPN on an NX-OS router.
Step 1. Create the BGP routing process. Initialize the BGP process with the global configuration command router bgp as-number.
Step 2. Initialize the VPNv4 address-family. The VPNv4 address-family needs to be initialized with the command address-family vpnv4 unicast.
Step 3. Define the VPNv4 neighbors. Configure the BGP VPNv4 neighbor’s IP address with the BGP router configuration command neighbor ip-address.
Step 4. Define the VPNv4 BGP neighbor’s session information. The VPNv4 neighbor’s session information needs to be defined. The remote-as must be defined with the command remote-as as-number. The BGP session needs to be sourced by a loopback address with a /32 netmask with the command update-source interface-id. Any other additional session information, such as passwords, can be defined here as well.
Step 5. Activate the BGP VPNv4 address-family for the BGP PE or RR session. Activate the address-family for the BGP neighbor with the BGP neighbor configuration command address-family afi safi.
Step 6. Enable BGP extended communities. MPLS L3VPN is dependent on the usage of extended BGP communities for the advertisement of route targets and other BGP path attributes. The command send-community {extended | both} is required. Additional BGP address-family specific commands, such as defining a route-reflector client or route policies, are configured here.
Step 7. Define the VRF in BGP. The VRF for the BGP instance is defined with the command vrf vrf-name.
Step 8. Initialize the VRF’s address-family. Initialize the address-family for that VRF with the BGP router configuration command address-family afi safi so it can be associated to a BGP neighbor in that VRF. Any route advertisement or redistribution on the PE occurs under the VRF address-family.
Step 9. Define the VRF’s neighbor’s CE IP address. Configure the BGP neighbor’s IP address with the BGP router configuration command neighbor ip-address.
Step 10. Define the neighbor’s BGP autonomous system number. Configure the BGP neighbor’s autonomous system number with the BGP neighbor configuration command remote-as as-number. Any additional BGP session information is configured here.
Step 11. Activate the address-family for the BGP neighbor. Activate the address-family for the BGP neighbor with the BGP neighbor configuration command address-family afi safi. Any additional address-family configuration settings such as route-maps can be defined here.
Example 10-12 displays the BGP configuration for the VPNv4 address-family to the route-reflector (192.168.100.100) and the CE3 router (172.16.3.33) in remote AS300.
Example 10-12 NX-OS BGP VPNv4 and PE-CE Configuration
PE3-NXOS# show run bgp
feature bgp
router bgp 65000
address-family vpnv4 unicast
neighbor 192.168.100.100
remote-as 65000
update-source loopback0
address-family vpnv4 unicast
send-community extended
vrf VPN01
address-family ipv4 unicast
redistribute direct route-map ALL
neighbor 172.16.3.33
remote-as 300
address-family ipv4 unicast
! The following section is part of the show run bgp output, but is actually part
! of the VRF creation.
vrf context VPN01
rd 1:1
address-family ipv4 unicast
route-target import 1:1
route-target export 1:1
Verification of BGP Sessions and Routes
Now that the BGP session has been configured for the VPNv4 address-family and between the PE and CE routers, it is recommended that the sessions are validated.
The relevant BGP information for a specific VRF is seen with the command show bgp {vpnv4 | vpnv6} unicast {all | rd route-distinguisher | vrf vrf-name} [labels | summary | neighbors [ip-address [advertised-routes | received]]] on IOS routers. The labels keyword displays the MPLS label that is first tagged with the packet. This is the label assigned from the router that learns the route from a CE router. If the vrf vrf-name option is used, the VPNv4 sessions are not included in the output.
Note
The VRF BGP table should be a smaller subset of routes that are contained in the VPNv4 address-family for a router. On smaller topologies, it may be hard to differentiate between the two commands.
Example 10-13 displays the output of iterations of the preceding commands to verify that the BGP session is established and that routes have been received.
Example 10-13 Display VPNv4 and PE-CE BGP Sessions on IOS
PE1-IOS# show bgp vpnv4 unicast vrf VPN01 summary
! Output omitted for brevity
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
172.16.1.11 4 100 21 26 21 0 0 00:15:49 2
PE1-IOS# show bgp vpnv4 unicast summary
! Output omitted for brevity
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
172.16.1.11 4 100 21 26 21 0 0 00:16:16 2
192.168.100.100 4 65000 23 20 21 0 0 00:12:26 6
PE1-IOS# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
* 172.16.1.0/26 172.16.1.11 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 172.16.1.11 0 0 100 ?
*> 172.16.1.200/32 0.0.0.0 0 32768 ?
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 200 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 300 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
PE1-IOS# show bgp vpnv4 unicast vrf VPN01
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
* 172.16.1.0/26 172.16.1.11 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 172.16.1.11 0 0 100 ?
*> 172.16.1.200/32 0.0.0.0 0 32768 ?
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 200 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 300 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
PE1-IOS# show bgp vpnv4 unicast vrf VPN01 labels
Network Next Hop In label/Out label
Route Distinguisher: 1:1 (VPN01)
172.16.1.0/26 172.16.1.11 25/nolabel
0.0.0.0 25/nolabel(VPN01)
172.16.1.100/32 172.16.1.11 26/nolabel
172.16.1.200/32 0.0.0.0 27/nolabel(VPN01)
172.16.2.0/26 192.168.2.2 nolabel/24012
172.16.2.100/32 192.168.2.2 nolabel/24013
172.16.2.200/32 192.168.2.2 nolabel/24012
172.16.3.0/26 192.168.3.3 nolabel/492287
172.16.3.100/32 192.168.3.3 nolabel/22
172.16.3.200/32 192.168.3.3 nolabel/492287
The command show bgp {vpnv4 | vpnv6} unicast [summary] displays the VPNv4 session with the PE or RRs on IOS XR and NX-OS devices. The command show bgp vrf {vrf-name | all}] [labels | summary | neighbors [ip-address [advertised-routes | received]]] provides additional information about the routes from the VRF context. The significant difference in the two commands is that IOS XR/NX-OS does not require the identification of a VRF, RD, or use the all keyword.
Example 10-14 displays the output of iterations of the commands that were just provided to verify that the BGP session is established and that routes have been received on the IOS XR router. Notice that the IP address for the BGP peer changes depending upon the command.
Example 10-14 Display VPNv4 and PE-CE BGP Sessions on IOS XR
RP/0/0/CPU0:PE2-XR# show bgp vpnv4 unicast summary
! Output omitted for brevity
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 22 22 22 22 22 0
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
192.168.100.100 0 65000 43 36 22 0 0 00:30:55 6
RP/0/0/CPU0:PE2-XR# show bgp vrf VPN01 ipv4 unicast summary
! Output omitted for brevity
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
172.16.2.22 0 200 40 40 22 0 0 00:33:33 2
RP/0/0/CPU0:PE2-XR# show bgp vrf VPN01 ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*> 172.16.2.0/26 0.0.0.0 0 32768 ?
* 172.16.2.22 0 0 200 ?
*> 172.16.2.100/32 172.16.2.22 0 0 200 ?
*> 172.16.2.200/32 0.0.0.0 0 32768 ?
*>i172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i172.16.3.100/32 192.168.3.3 0 100 0 300 ?
*>i172.16.3.200/32 192.168.3.3 0 100 0 ?
Processed 9 prefixes, 10 paths
RP/0/0/CPU0:PE2-XR# show bgp vrf VPN01 ipv4 unicast labels
! Output omitted for brevity
Route Distinguisher: 1:1 (default for vrf VPN01)
*>i172.16.1.0/26 192.168.1.1 25 nolabel
*>i172.16.1.100/32 192.168.1.1 26 nolabel
*>i172.16.1.200/32 192.168.1.1 27 nolabel
*> 172.16.2.0/26 0.0.0.0 nolabel 24012
* 172.16.2.22 nolabel 24012
*> 172.16.2.100/32 172.16.2.22 nolabel 24013
*> 172.16.2.200/32 0.0.0.0 nolabel 24012
*>i172.16.3.0/26 192.168.3.3 492287 nolabel
*>i172.16.3.100/32 192.168.3.3 22 nolabel
*>i172.16.3.200/32 192.168.3.3 492287 nolabel
Processed 9 prefixes, 10 paths
Example 10-15 displays the output of the previous commands to verify that the BGP session is established and that routes have been received on the NX-OS device.
Example 10-15 Display VPNv4 and PE-CE BGP Sessions on NX-OS Devices
PE3-NXOS# show bgp vpnv4 unicast summary
! Output omitted for brevity
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.100.100 4 65000 51 44 14 0 0 00:38:19 6
PE3-NXOS# show bgp vrf VPN01 ipv4 unicast summary
! Output omitted for brevity
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
172.16.3.33 4 300 39 41 13 0 0 00:32:21 2
PE3-NXOS# show bgp vrf VPN01 ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i172.16.2.100/32 192.168.2.2 0 100 0 200 ?
*>i172.16.2.200/32 192.168.2.2 0 100 0 ?
*>r172.16.3.0/26 0.0.0.0 0 100 32768 ?
* e 172.16.3.33 0 0 300 ?
*>e172.16.3.100/32 172.16.3.33 0 0 300 ?
*>r172.16.3.200/32 0.0.0.0 0 100 32768 ?
PE3-NXOS# show bgp vrf VPN01 ipv4 unicast labels
! Output omitted for brevity
Network Next Hop In label/Out label
*>i172.16.1.0/26 192.168.1.1 nolabel/25
*>i172.16.1.100/32 192.168.1.1 nolabel/26
*>i172.16.1.200/32 192.168.1.1 nolabel/27
*>i172.16.2.0/26 192.168.2.2 nolabel/24012
*>i172.16.2.100/32 192.168.2.2 nolabel/24013
*>i172.16.2.200/32 192.168.2.2 nolabel/24012
*>r172.16.3.0/26 0.0.0.0 492287/nolabel (VPN01)
* e 172.16.3.33 492287/nolabel (VPN01)
*>e172.16.3.100/32 172.16.3.33 22/nolabel (VPN01)
*>r172.16.3.200/32 0.0.0.0 492287/nolabel (VPN01)
Troubleshooting MPLS L3VPN
The previous sections explained the concepts and technologies used in a MPLS L3VPN. This section continues on the theory and explains the process for troubleshooting an MPLS L3VPN network.
The first step of troubleshooting is to verify that the control plane is working properly. This includes verification that the CE routers contain routes for the source and destination network in their routing table. Sometimes access to the CE routers is not possible, so viewing the routing table is restricted to the VRF tables on the PE routers. If the packet’s source or destination network is missing on either CE router, start tracing the missing route along the path from the source of the route’s origination toward the CE missing the network.
Figure 10-5 displays a smaller version of the topology that displays only two sites and the BGP tables for all the CE and PE routers. The CE routers are advertising their networks to the PE routers. The PE routers are advertising their local routes and routes learned from their attached CE to the remote PE. The remote PE then advertises the networks to the remote CE. Both CEs contain routes for the source and destination network in the topology.
Note
The Loopback 100 interface was added to all the PE routers as a way to check route advertisement between CE and PE or between PE routers. This technique accelerates the ability to verify route advertisement between PE and CE routers.
Production networks typically do not have a loopback interface in the VRF for this purpose, but they can be added as part of the troubleshooting step.
Default Route Advertisement Between PE-CE Routers
The CE routers need only a default route to forward packets toward the PE router. The PE router can have more routing entries for the VRF so that packets can be forwarded to the appropriate PE router. However, the PE routers need to know about the routes that are attached to the CE routers because BGP still needs to advertise the routes toward the PE routers.
Figure 10-6 demonstrates the concept where PE1 and PE2 advertise only a default route to the CE routers. The CE routers still advertise their local networks to the PEs. Notice that the routing table for the PE routers is the same as in Figure 10-5 when the CE routers receive the full routing table. In either scenario, the CE routers contain routes for the source and destination of the network traffic.
Problems with AS-PATH
Most customers use the same ASN on their CE routers, or keep the number of ASNs fairly low. Keeping the ASN count to a low number simplifies their operations and deployment of CE routers by network management systems because the configurations can be placed in standardized templates. This also allows the company to keep the routing policies standardized. Regardless of the design reasons, if two locations use the same ASN, there could be routing problems between these sites while using MPLS L3VPN to connect them.
The problem is demonstrated by revisiting the sample topology depicted in Figure 10-4, which has all the CE routers using a different ASN. CE1 uses AS100, CE2 uses AS200, and CE3 uses AS300.
After changing all the CE routers (and corresponding changes on the PE routers) to use the same ASN (AS100), connectivity between all CE devices is broken. But any PE router can ping any CE router or its associated networks. To simplify troubleshooting for this scenario, focus will be directed toward CE1 connecting to CE2.
Example 10-16 displays the BGP tables for CE1 and CE2 as the first step of troubleshooting. In the output, CE1 and CE2 receive all the routes from the PE routers, but they do not have any routes from the remote CE routers. Notice that the CE routers are missing the loopback networks from the remote CE routers.
Example 10-16 Viewing the BGP Table on CE1 and CE2
CE1# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
* 172.16.1.0/26 172.16.1.1 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 0.0.0.0 0 32768 ?
*> 172.16.1.200/32 172.16.1.1 0 0 65000 ?
*> 172.16.2.0/26 172.16.1.1 0 65000 ?
*> 172.16.2.200/32 172.16.1.1 0 65000 ?
*> 172.16.3.0/26 172.16.1.1 0 65000 ?
*> 172.16.3.200/32 172.16.1.1 0 65000 ?
CE2# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/26 172.16.2.1 0 65000 ?
*> 172.16.1.200/32 172.16.2.1 0 65000 ?
* 172.16.2.0/26 172.16.2.1 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.2.100/32 0.0.0.0 0 32768 ?
*> 172.16.2.200/32 172.16.2.1 0 0 65000 ?
*> 172.16.3.0/26 172.16.2.1 0 65000 ?
*> 172.16.3.200/32 172.16.2.1 0 65000 ?
Because the CE routers are missing some of the networks, it is time to check the PE routers to see if they have the CE’s loopback networks in the BGP table. The PE router’s VPNv4 BGP table is displayed in Example 10-17. All the CE loopback interfaces (172.16.1.100/32, 172.16.2.100/32, and 172.16.3.100/32) are present on PE1 and PE2.
Example 10-17 Looking for the Remote CE Loopbacks on the PE Routers
PE1-IOS# show bgp vpnv4 unicast vrf VPN01
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
* 172.16.1.0/26 172.16.1.11 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 172.16.1.11 0 0 100 ?
*> 172.16.1.200/32 0.0.0.0 0 32768 ?
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
RP/0/0/CPU0:PE2-XR# show bgp vrf VPN01 ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*> 172.16.2.0/26 0.0.0.0 0 32768 ?
* 172.16.2.22 0 0 100 ?
*> 172.16.2.100/32 172.16.2.22 0 0 100 ?
*> 172.16.2.200/32 0.0.0.0 0 32768 ?
*>i172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i172.16.3.200/32 192.168.3.3 0 100 0 ?
Processed 9 prefixes, 10 paths
Next, check for any outbound filtering of prefixes toward the CE routers. But there are no route maps or filtering techniques applied to the PE routers during configuration. Upon further examination, the CE prefixes all contain the same AS-Path. The CE devices are dropping the prefixes as part of BGP’s loop prevention mechanisms. Verify this by using the techniques shown in Chapter 4. Example 10-18 displays the BGP neighbor status on CE1 that shows that prefixes were dropped due to the AS-Path loop.
Example 10-18 Checking for AS-Path Loops on CE1
CE1# show bgp ipv4 unicast neighbors 172.16.1.1
! Output omitted for brevity
For address family: IPv4 Unicast
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
AS_PATH loop: n/a 2
Bestpath from this peer: 7 n/a
Total: 7 2
Figure 10-7 displays a smaller subsection of the topology where CE1 and CE2 both use AS100. In this topology, CE1 and CE2 advertise their prefixes to their local PE routers. These PE routers exchange routes with each other, and thereby have complete BGP tables. The PE routers then advertise the routes to the local CE router. Only the remote CE routes are discarded because of the AS-Path loop check. The local PE loopback networks are accepted by the CE routers.
This scenario occurs frequently with MPLS L3VPNs and can be overcome with the AS-overide feature. The AS-override feature operates under the address-family and checks the AS-Path for network prefixes. When an AS-Path matches the ASN of the peering router, the local PE’s ASN is used instead. AS-override is enabled on IOS PE devices with the command neighbor ip-address as-override within the VRF’s BGP configuration. IOS XR and NX-OS PE devices enable AS-override with the command as-override within the address-family configuration for the CE router.
Example 10-19 displays the relevant configuration on the PE routers to use the as-override feature.
Example 10-19 Configuration to Use as-override Feature
PE1-IOS
router bgp 65000
address-family ipv4 vrf VPN01
redistribute connected
neighbor 172.16.1.11 remote-as 100
neighbor 172.16.1.11 activate
neighbor 172.16.1.11 as-override
exit-address-family
PE2-XR
router bgp 65000
vrf VPN01
neighbor 172.16.2.22
address-family ipv4 unicast
as-override
PE3-NXOS
router bgp 65000
vrf VPN01
neighbor 172.16.3.33
address-family ipv4 unicast
as-override
Example 10-20 displays the BGP table on CE1 and CE2 after the as-override feature has been configured on all the PE routers. Notice that the loopback addresses from the remote CE devices have an AS-Path length of two and that the second ASN is the same as the PE router. This verifies that the feature is working as intended.
Example 10-20 Verification of Route Advertisements After as-override Configuration
CE1# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
* 172.16.1.0/26 172.16.1.1 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 0.0.0.0 0 32768 ?
*> 172.16.1.200/32 172.16.1.1 0 0 65000 ?
*> 172.16.2.0/26 172.16.1.1 0 65000 ?
*> 172.16.2.100/32 172.16.1.1 0 65000 65000 ?
*> 172.16.2.200/32 172.16.1.1 0 65000 ?
*> 172.16.3.0/26 172.16.1.1 0 65000 ?
*> 172.16.3.100/32 172.16.1.1 0 65000 65000 ?
*> 172.16.3.200/32 172.16.1.1 0 65000 ?
CE2# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/26 172.16.2.1 0 65000 ?
*> 172.16.1.100/32 172.16.2.1 0 65000 65000 ?
*> 172.16.1.200/32 172.16.2.1 0 65000 ?
*> 172.16.2.0/26 0.0.0.0 0 32768 ?
* 172.16.2.1 0 0 65000 ?
*> 172.16.2.100/32 0.0.0.0 0 32768 ?
*> 172.16.2.200/32 172.16.2.1 0 0 65000 ?
*> 172.16.3.0/26 172.16.2.1 0 65000 ?
*> 172.16.3.100/32 172.16.2.1 0 65000 65000 ?
*> 172.16.3.200/32 172.16.2.1 0 65000 ?
There are two alternatives to using the as-override feature:
The use of the allowas-in feature on the CE routers, which disables the AS-Path loop check only for that neighbor.
The PE routers advertise a default route to the CE routers. As explained earlier, the CEs forward the packets toward the PE router. The PE router then has the full routing table (no conflicts in AS-Path) so the PE can forward to the appropriate remote PE router.
This shrinks the size of the routing table on the CE but could cause traffic loss in certain scenarios. For example, imagine that every remote site is connected to two different SPs. The CE’s routing table would have only two default routes that point to the different SPs. If a remote link failed, the CE would not know about it because the PE is not advertising more specific routes for the path that does work properly.
Suboptimal Routing with VPNv4 Route Reflectors
Large SP networks use RRs to overcome the scalability issue with using an IBGP full mesh. However, because BGP advertises the best path only to other BGP peers, the RR placement can potentially cause suboptimal routing. RRs do not pass all the paths to a client by default. It only advertises the BGP best path, which is calculated based on the RR’s location in the network and not the route reflector client’s location.
Figure 10-8 demonstrates an environment where suboptimal routing occurs because of the loss of path information at the route reflectors. Notice the following elements in Figure 10-8:
CE1 and CE2 are advertising the 172.16.10.0/24 network to PE1 and PE2 accordingly.
PE1 and PE2 advertise the VPNv4 prefix of 172.16.10.0/24 with the RD 1:1 to RR.
The RR receives the prefix and stores them in the same topology database because they share the same RD.
The RR compares both paths and picks the path via PE2 as the best path because of the IGP metric.
The RR only advertises PE2’s path to PE3.
Looking at PE3’s routing table to PE1 (192.168.1.1) and PE2 (192.168.2.2), the optimal path from CE3 to the 172.16.10.0/24 network should be sent directly to PE1 across the 10.1.13.0/30 network. Unfortunately, the RR only advertised the path that uses PE2, so PE3 forwards traffic out of the suboptimal path to reach the 172.16.10.0/24 network. Notice that the BGP route on PE3 does a recursive lookup to the PE2’s next-hop address.
Example 10-21 verifies that the RR is advertising only one of the paths (from PE2) for the 172.16.10.0/24 prefix to PE3 router. This forces PE3 to forward traffic to PE2 even though PE1 is a better path.
Example 10-21 172.16.10.0/24 Network Prefixes on the Route Reflector
RR# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i 172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i 172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i 172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
*>i 172.16.10.0/24 192.168.2.2 0 100 0 100 ?
* i 192.168.1.1 0 100 0 100 ?
RR# show bgp vpnv4 unicast all neighbors 192.168.3.3 advertised-routes
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i 172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i 172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i 172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
*>i 172.16.10.0/24 192.168.2.2 0 100 0 100 ?
Total number of prefixes 10
Example 10-22 displays that PE3 has only one path from the RR to calculate the BGP best path. This is the suboptimal path to the 172.16.10.0/24 network.
Example 10-22 Only One Path to 172.16.10.0/24 on PE3
PE3-NXOS# show bgp vpnv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (VRF VPN01)
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i172.16.2.200/32 192.168.2.2 0 100 0 ?
*>r172.16.3.0/26 0.0.0.0 0 100 32768 ?
* e 172.16.3.33 0 0 100 ?
*>e172.16.3.100/32 172.16.3.33 0 0 100 ?
*>r172.16.3.200/32 0.0.0.0 0 100 32768 ?
*>i172.16.10.0/24 192.168.2.2 0 100 0 100 ?
If PE3 had received both paths (that is, a path from PE1 and PE2) from the RR, then PE3 could calculate the best path based on its IGP metric to reach each PE1 or PE2. By associating a unique RD to every VRF, the RR sees only one path for each RD and advertises all the unique VPN prefixes to PE3. Although there may be unique VPN prefixes, they have the same RT associated to them and are imported into the VRF table on the destination PE.
Note
Recall that the VPN prefix is the combination of the RD and a network prefix, and the function of the RD is to ensure that the VPN prefix is unique. The RD does not associate a VPNv4 prefix to a specific VRF; that is the role of the RT.
To explain the solution, imagine that the RD has been changed on PE2 to 1:2 and PE3 has changed its RD to 1:3 for the VPN01 VRF. Example 10-23 displays the VPNv4 addresses on the route reflector. Notice that all the routes advertised from a PE are contained in a different RD grouping. This is because the paths are considered unique.
Upon viewing the VPNv4 routes that are advertised to PE3, there are two paths to 172.16.10.0/24 (one from PE1 with RD 1:1 and another from PE2 with RD 1:2.
Example 10-23 Viewing of VPNv4 Prefixes on the Route Reflector
RR# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i 172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i 172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i 172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i 172.16.10.0/24 192.168.1.1 0 100 0 100 ?
Route Distinguisher: 1:2
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.10.0/24 192.168.2.2 0 100 0 100 ?
Route Distinguisher: 1:3
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
RR# show bgp vpnv4 unicast all neighbors 192.168.3.3 advertised-routes
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i 172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i 172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i 172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i 172.16.10.0/24 192.168.1.1 0 100 0 100 ?
Route Distinguisher: 1:2
*>i 172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i 172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i 172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i 172.16.10.0/24 192.168.2.2 0 100 0 100 ?
Route Distinguisher: 1:3
*>i 172.16.3.0/26 192.168.3.3 0 100 0 ?
*>i 172.16.3.100/32 192.168.3.3 0 100 0 100 ?
*>i 172.16.3.200/32 192.168.3.3 0 100 0 ?
Total number of prefixes 11
Note
Notice that the number of advertised prefixes is different between Example 10-21 and Example 10-23. This is to accommodate the extra path from the different RD for the 172.16.10.0/24 network.
Example 10-24 displays the BGP table on PE3. The first two groups of VPN prefixes are specific to RD 1:1 and 1:2 where they each have a path to the 172.16.10.0/24 network prefix. The last group (RD 1:3) contains all the routes from any RD (1:1 or 1:2) because the VPN01 VRF import RT matches the RTs in both paths. PE3 installs all the paths for the 172.16.10.0/24 regardless of the RDs and can now calculate the best path (from PE1 or PE2) for that specific VRF.
Note
In essence, the last grouping shows the output from the command show bgp ipv4 unicast vrf vrf-name.
Example 10-24 Verification of VPN Prefixes on PE3
PE3-NXOS# show bgp vpnv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i172.16.10.0/24 192.168.1.1 0 100 0 100 ?
Route Distinguisher: 1:2
*>i172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i172.16.2.200/32 192.168.2.2 0 100 0 ?
*>i172.16.10.0/24 192.168.2.2 0 100 0 100 ?
Route Distinguisher: 1:3 (VRF VPN01)
*>i172.16.1.0/26 192.168.1.1 0 100 0 ?
*>i172.16.1.100/32 192.168.1.1 0 100 0 100 ?
*>i172.16.1.200/32 192.168.1.1 0 100 0 ?
*>i172.16.2.0/26 192.168.2.2 0 100 0 ?
*>i172.16.2.100/32 192.168.2.2 0 100 0 100 ?
*>i172.16.2.200/32 192.168.2.2 0 100 0 ?
*>r172.16.3.0/26 0.0.0.0 0 100 32768 ?
* e 172.16.3.33 0 0 100 ?
*>e172.16.3.100/32 172.16.3.33 0 0 100 ?
*>r172.16.3.200/32 0.0.0.0 0 100 32768 ?
* i172.16.10.0/24 192.168.2.2 0 100 0 100 ?
*>i 192.168.1.1 0 100 0 100 ?
Although this technique provides additional paths for every PE to calculate the BGP best path, it consumes additional router memory. The size of the BGP table and amount of memory on the router should be evaluated with this design.
All three of the platforms support the capability to view routes based on a specific RD with the command show bgp vpnv4 unicast rd route-distinguisher. Example 10-25 shows the use of the command.
Example 10-25 Viewing All the Routes Based on the RD
PE1-IOS# show bgp vpnv4 unicast rd 1:1
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
* 172.16.1.0/26 172.16.1.11 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 172.16.1.11 0 0 100 ?
*> 172.16.1.200/32 0.0.0.0 0 32768 ?
RP/0/0/CPU0:PE2-XR# show bgp vpnv4 unicast rd 1:2
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 2:1 (default for vrf VPN01)
*> 172.16.2.0/26 0.0.0.0 0 32768 ?
* 172.16.2.22 0 0 100 ?
*> 172.16.2.100/32 172.16.2.22 0 0 100 ?
*> 172.16.2.200/32 0.0.0.0 0 32768 ?
PE3-NXOS# show bgp vpnv4 unicast rd 1:3
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 3:1 (VRF VPN01)
*>r172.16.3.0/26 0.0.0.0 0 100 32768 ?
* e 172.16.3.33 0 0 100 ?
*>e172.16.3.100/32 172.16.3.33 0 0 100 ?
*>r172.16.3.200/32 0.0.0.0 0 100 32768 ?
*>r172.16.3.200/32 0.0.0.0 0 100 32768 ?
Running the show bgp vpnv4 unicast rd route-distinguisher command on a PE router also includes routes with a matching RT and RD. This command is used best on a RR to include only the routes on a RR. In large SP environments, it can be difficult to trace a network prefix on a route reflector because the RDs are different for every location and VRF. A troubleshooting trick involves the temporary creation of a VRF on the RR that includes all the RTs so that all the VRF’s routes are visible. This allows for the use of show bgp commands from the VRF perspective versus the VPNv4 address-family.
Note
Newer versions of software support BGP Add-Path which allows for the advertisement of multiple paths to a peer. BGP Add-Path is covered in detail in Chapter 14, “BGP High Availability.”
Troubleshooting Problems with Route Targets
The last aspect for troubleshooting MPLS L3VPN is to understand the import and export of route targets (RT). As network prefixes are received from a CE router by a PE router, the export RTs are added to the prefix as part of the BGP path attributes in the VPN prefix. When a router receives a VPN prefix from another PE router (or RR), the network prefix is only installed into the VRF when the VPN prefix’s RT matches the VRF’s import RT. Any route that does not match the VRF’s import RT with the prefix RT is discarded by default to save router resources.
Note
BGP route reflectors (RR)s generally do not have a VRF configured, which would mean that it drops all the VPN prefixes. Making a BGP router a RR disables the filtering of routes that do not import into a VRF. This allows for all routes to remain present in the RR’s BGP database.
Figure 10-9 is a small subsection of the reference topology where PE1 is using an import/export RT of 1:1, and PE2 is using an import/export RT of 1:2 for the same VRF.
Connectivity between the CE devices is failing, and after reviewing the BGP table of the CE routers, network engineering realizes that the CE routers are not learning any of the other CE prefixes from the PE routers. Examining the remote PE’s routing table in Example 10-26 indicates that only the local routes and routes from the local CE are in the BGP table.
Example 10-26 PE Router’s BGP Table with Different Route Targets
PE1# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
* 172.16.1.0/26 172.16.11.1 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.1.100/32 172.16.11.1 0 0 100 ?
*> 172.16.1.200/32 0.0.0.0 0 32768 ?
PE2# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:2 (default for vrf VPN01)
* 172.16.2.0/26 172.16.22.1 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.2.100/32 172.16.22.1 0 0 100 ?
*> 172.16.2.200/32 0.0.0.0 0 32768 ?
The next step of troubleshooting is to check the VPN prefixes are on the RR. The local PE configuration should be checked after verifying that all the VPN prefixes are on the RR, and the RR displays that the routes are being advertised to the remote PE router. After examining the VRF configuration on both PE routers, the problem is found with the import RTs not matching the other PE router’s export RT.
Enabling BGP debugs provides another method of verification that the import RT does not match the RT of the VPN prefix. Example 10-27 displays how using BGP debugs provides verification of an invalid RT import statement.
Example 10-27 Verification of RT Import Failure with BGP Debug
PE1# debug bgp vpnv4 unicast updates in
BGP updates debugging is on (inbound) for address family: VPNv4 Unicast
! A soft reset is done with the RR to resent all of the VPNv4 prefixes again
PE1# clear bgp vpnv4 unicast * soft
! Output omitted for brevity
02:06:06: BGP(4): (base) 192.168.2.2 send UPDATE (format) 2:1:172.16.2.0/24, next
192.168.1.1, label 21, metric 0, path 200
02:06:07: BGP(4): 192.168.2.2 rcvd UPDATE w/ attr: nexthop 192.168.2.2, origin ?,
localpref 100, metric 0, merged path 200, AS_PATH , extended community RT:1:2
02:06:07: BGP(4): 192.168.2.2 rcvd 2:3:172.16.2.0/26, label 21 -- DENIED due to:
extended community not supported;
A quick solution is to add the other PE router’s export RTs to the local PE router’s import RTs. The new VRF configuration for PE1 and PE2 is displayed in Example 10-28.
Example 10-28 VRF Configuration with Multiple Import Route Targets
PE1-IOS
vrf definition VPN01
rd 1:1
!
address-family ipv4
route-target export 1:1
route-target import 1:1
route-target import 1:2
PE2-XR
vrf VPN01
address-family ipv4 unicast
import route-target
1:1
1:2
!
export route-target
1:2
Example 10-29 provides verification that the PE routers accept multiple RTs. It contains the output of PE1’s VRF definition and the BGP table for the prefixes learned from Site 2 (172.16.2.0/26) that have a RT of 1:2.
Example 10-29 Verification of Multiple Import RTs
PE1-IOS# show vrf detail VPN01
! Output omitted for brevity
VRF VPN01 (VRF Id = 1); default RD 1:1; default VPNID <not set>
Interfaces:
Lo100 Gi0/2
Address family ipv4 unicast (Table ID = 0x2):
Export VPN route-target communities
RT:1:1
Import VPN route-target communities
RT:1:1 RT:1:2
PE1-IOS# show bgp vpnv4 unicast all 172.16.2.0/26
BGP routing table entry for 1:1:172.16.2.0/26, version 41
Paths: (1 available, best #1, table VPN01)
Advertised to update-groups:
1
Refresh Epoch 2
Local
192.168.2.2 (metric 4) (via default) from 192.168.100.100 (192.168.100.100)
Origin incomplete, metric 0, localpref 100, valid, internal, best
Extended Community: RT:1:2
Originator: 192.168.2.2, Cluster list: 192.168.100.100
mpls labels in/out nolabel/24012
MPLS L3VPN Services
A benefit to MPLS L3VPN is that routes can be exchanged between two different VRFs on the same service provider. The service provider can advertise all the network prefixes from a VRF into another customer’s topology or be selective with the route advertisement. The process of advertisement networks between customers is referred to as VRF route leaking or just route leaking.
Note
It is important to note that routes must be leaked in both directions to establish connectivity. One direction is for the forwarding of the initial packet, and the other direction is for the return of the packets.
Route leaking is used during company mergers and acquisitions when they use the same SP. Assuming that there is no overlapping IP addresses, all the routes could be mutually leaked to every VRF so that both companies are fully interconnected.
Another use case is for cloud-based providers that provide application services to multiple customers, like email hosting. Instead of providing connectivity across a public network such as the Internet, they could provide connectivity through a common MPLS L3VPN provider. The cloud provider could use Network Address Translation (NAT) and firewalls in its infrastructure to maintain separation of customer network space.
Figure 10-10 provides a sample topology where two companies (A & B) are subscribing to company C’s email hosting service out of Site 2. The network design must accommodate the following:
All the sites in Company A must be able to reach the email servers attached to CE3 that is located in Site 2.
All the sites in Company B must be able to reach the hosted email servers attached to CE3 that is located in Site 2.
All of Company C’s email servers attached to CE3 must be able to reach all the networks in Company A’s and Company B’s sites.
Company A (CE1 and CE4) should not be able to reach any of the sites in Company B (CE2 and CE5).
Company B (CE2 and CE5) should not be able to reach any of the sites in Company A (CE1 and CE4).
This may seem like a challenging task that will require the use of many ACLs on routers for all three companies on a traditional network. If an additional network was added later—Company C’s network that hosts email services—how many devices would require a configuration change? Fortunately, this is a very common scenario that provides flexibility and scalability by using different RTs in a VRF.
A VRF can have multiple import and export RTs. The first step is to define the logic for the export RT, which follows:
An export RT (that is, 1:10) is selected for all the sites in Company A.
An export RT (that is, 1:20) is selected for all the sites in Company B.
An export RT (that is, 1:30) is selected for Company C’s site that is hosting email services. If there are normal sites for Company C’s day-to-day operations (such as corporate LAN or human resource systems), those are assigned a different RT (i.e. 31) to prevent external companies from receiving those routes.
The second step is to define all the import RTs for every site, which uses the following logic:
All sites in Company A import RT 1:10 (Company A sites) and 1:30 (Company C email hosted network on CE3). This allows users to send traffic to the email servers from Company A networks.
All sites in Company B import RT 1:20 (Company B sites) and 1:30 (Company C email hosted network on CE3). This allows user to send traffic to the email servers from Company B networks.
Company C’s email servers require routes to connect to all of Company A and B networks. PE2 imports RT 1:10 (all Company A sites) and 1:20 (all Company B sites) in to VRF VPN03.
Table 10-2 displays the RT import/export logic for the VRF assigned to each CE device.
Example 10-30 provides the BGP configuration for the PE routers in Figure 10-10. A unique RD has been assigned to every VRF where the first octet correlates to the VPN number and the second octet correlates to the site. Notice that the import/export RTs match those from Table 10-2.
Example 10-30 Configuration for BGP Services
PE1
vrf definition VPN01
rd 1:1
!
address-family ipv4
route-target export 1:10
route-target import 1:10
route-target import 1:30
exit-address-family
!
vrf definition VPN02
rd 2:1
!
address-family ipv4
route-target export 1:20
route-target import 1:20
route-target import 1:30
exit-address-family
!
router bgp 65000
no bgp default ipv4-unicast
neighbor 192.168.2.2 remote-as 65000
neighbor 192.168.2.2 update-source Loopback0
neighbor 192.168.3.3 remote-as 65000
neighbor 192.168.3.3 update-source Loopback0
!
address-family vpnv4
neighbor 192.168.2.2 activate
neighbor 192.168.2.2 send-community extended
neighbor 192.168.3.3 activate
neighbor 192.168.3.3 send-community extended
exit-address-family
!
address-family ipv4 vrf VPN01
redistribute connected
neighbor 172.16.11.1 remote-as 100
neighbor 172.16.11.1 activate
neighbor 172.16.11.1 as-override
exit-address-family
!
address-family ipv4 vrf VPN02
redistribute connected
neighbor 172.16.22.1 remote-as 200
neighbor 172.16.22.1 activate
neighbor 172.16.22.1 as-override
exit-address-family
PE2
vrf definition VPN03
rd 3:2
!
address-family ipv4
route-target export 1:30
route-target import 1:10
route-target import 1:20
exit-address-family
!
router bgp 65000
no bgp default ipv4-unicast
neighbor 192.168.1.1 remote-as 65000
neighbor 192.168.1.1 update-source Loopback0
neighbor 192.168.3.3 remote-as 65000
neighbor 192.168.3.3 update-source Loopback0
!
address-family vpnv4
neighbor 192.168.1.1 activate
neighbor 192.168.1.1 send-community extended
neighbor 192.168.3.3 activate
neighbor 192.168.3.3 send-community extended
exit-address-family
!
address-family ipv4 vrf VPN01
redistribute connected
neighbor 172.16.33.1 remote-as 300
neighbor 172.16.33.1 activate
neighbor 172.16.33.1 as-override
exit-address-family
PE3
vrf definition VPN01
rd 1:3
!
address-family ipv4
route-target export 1:10
route-target import 1:10
route-target import 1:30
exit-address-family
!
vrf definition VPN02
rd 2:3
!
address-family ipv4
route-target export 1:20
route-target import 1:20
route-target import 1:30
exit-address-family
!
router bgp 65000
no bgp default ipv4-unicast
neighbor 192.168.1.1 remote-as 65000
neighbor 192.168.1.1 update-source Loopback0
neighbor 192.168.2.2 remote-as 65000
neighbor 192.168.2.2 update-source Loopback0
!
address-family vpnv4
neighbor 192.168.1.1 activate
neighbor 192.168.1.1 send-community extended
neighbor 192.168.2.2 activate
neighbor 192.168.2.2 send-community extended
exit-address-family
!
address-family ipv4 vrf VPN01
redistribute connected
neighbor 172.16.44.1 remote-as 100
neighbor 172.16.44.1 activate
neighbor 172.16.44.1 as-override
exit-address-family
!
address-family ipv4 vrf VPN02
redistribute connected
neighbor 172.16.55.1 remote-as 200
neighbor 172.16.55.1 activate
neighbor 172.16.55.1 as-override
exit-address-family
Example 10-31 displays the import and export RTs for the VRFs on PE1, PE2, and PE3. The RDs are different for all the VRFs for every PE router, and the import/export RTs all match Table 10-2.
Example 10-31 Verification of VRF Route Target Import and Export
PE1# show vrf detail
! Output omitted for brevity
VRF VPN01 (VRF Id = 1); default RD 1:1; default VPNID <not set>
Interfaces:
Gi0/1
Address family ipv4 unicast (Table ID = 0x2):
Export VPN route-target communities
RT:1:10
Import VPN route-target communities
RT:1:10 RT:1:30
VRF VPN02 (VRF Id = 2); default RD 2:1; default VPNID <not set>
Interfaces:
Gi0/2
Address family ipv4 unicast (Table ID = 0x3):
Flags: 0x0
Export VPN route-target communities
RT:1:20
Import VPN route-target communities
RT:1:20 RT:1:30
PE2# show vrf detail
! Output omitted for brevity
VRF VPN01 (VRF Id = 1); default RD 3:2; default VPNID <not set>
Interfaces:
Gi0/1
Address family ipv4 unicast (Table ID = 0x2):
Export VPN route-target communities
RT:1:30
Import VPN route-target communities
RT:1:10 RT:1:20
PE3# show vrf detail
! Output omitted for brevity
VRF VPN01 (VRF Id = 1); default RD 1:3; default VPNID <not set>
Interfaces:
Gi0/1
Address family ipv4 unicast (Table ID = 0x2):
Export VPN route-target communities
RT:1:10
Import VPN route-target communities
RT:1:10 RT:1:30
VRF VPN02 (VRF Id = 2); default RD 2:3; default VPNID <not set>
Interfaces:
Gi0/2
Address family ipv4 unicast (Table ID = 0x3):
Export VPN route-target communities
RT:1:20
Import VPN route-target communities
RT:1:20 RT:1:30
Example 10-32 demonstrates the routing tables of CE1 (VRF: VPN01), CE2 (VRF: VPN02), and CE3 (VRF:VPN03). CE1 and CE2 have the route to the email servers installed in their routing table even though it resides in a different VRF. The significance is that CE3 provides connectivity to the email servers and requires connectivity to all Company A and Company B networks so it has imported routes from VPN01 and VPN02 VRFs.
Example 10-32 Verification of Routing Tables for MPLS Services
CE1# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 0.0.0.0 0 32768 ?
*> 172.16.3.0/24 172.16.11.2 0 65000 300 ?
*> 172.16.4.0/24 172.16.11.2 0 65000 65000 ?
* 172.16.11.0/24 172.16.11.2 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.33.0/24 172.16.11.2 0 65000 ?
*> 172.16.44.0/24 172.16.11.2 0 65000 ?
CE2# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*> 172.16.2.0/24 0.0.0.0 0 32768 ?
*> 172.16.3.0/24 172.16.22.2 0 65000 300 ?
*> 172.16.5.0/24 172.16.22.2 0 65000 65000 ?
* 172.16.22.0/24 172.16.22.2 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.33.0/24 172.16.22.2 0 65000 ?
*> 172.16.55.0/24 172.16.22.2 0 65000 ?
CE3# show bgp ipv4 unicast
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
*> 172.16.1.0/24 172.16.33.2 0 65000 100 ?
*> 172.16.2.0/24 172.16.33.2 0 65000 200 ?
*> 172.16.3.0/24 0.0.0.0 0 32768 ?
*> 172.16.4.0/24 172.16.33.2 0 65000 100 ?
*> 172.16.5.0/24 172.16.33.2 0 65000 200 ?
*> 172.16.11.0/24 172.16.33.2 0 65000 ?
*> 172.16.22.0/24 172.16.33.2 0 65000 ?
* 172.16.33.0/24 172.16.33.2 0 0 65000 ?
*> 0.0.0.0 0 32768 ?
*> 172.16.44.0/24 172.16.33.2 0 65000 ?
*> 172.16.55.0/24 172.16.33.2 0 65000 ?
Example 10-33 displays the VPNv4 prefixes on PE1. The prefixes in the RD 1:1 VPN01 VRF grouping include the routes with a RD of 1:1 (local), RD of 1:2 (Site 3), and RD of 3:2 (Site 3).
Example 10-33 PE1’s BGP Table with VPN Services
PE1-IOS# show bgp vpnv4 unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1 (default for vrf VPN01)
*> 172.16.1.0/24 172.16.11.1 0 0 100 ?
*>i 172.16.3.0/24 192.168.2.2 0 100 0 300 ?
*>i 172.16.4.0/24 192.168.3.3 0 100 0 100 ?
* 172.16.11.0/24 172.16.11.1 0 0 100 ?
*> 0.0.0.0 0 32768 ?
*>i 172.16.33.0/24 192.168.2.2 0 100 0 ?
*>i 172.16.44.0/24 192.168.3.3 0 100 0 ?
Route Distinguisher: 1:3
*>i 172.16.4.0/24 192.168.3.3 0 100 0 100 ?
*>i 172.16.44.0/24 192.168.3.3 0 100 0 ?
Route Distinguisher: 2:1 (default for vrf VPN02)
Network Next Hop Metric LocPrf Weight Path
*> 172.16.2.0/24 172.16.22.1 0 0 200 ?
*>i 172.16.3.0/24 192.168.2.2 0 100 0 300 ?
*>i 172.16.5.0/24 192.168.3.3 0 100 0 200 ?
* 172.16.22.0/24 172.16.22.1 0 0 200 ?
*> 0.0.0.0 0 32768 ?
*>i 172.16.33.0/24 192.168.2.2 0 100 0 ?
*>i 172.16.55.0/24 192.168.3.3 0 100 0 ?
Route Distinguisher: 2:3
*>i 172.16.5.0/24 192.168.3.3 0 100 0 200 ?
*>i 172.16.55.0/24 192.168.3.3 0 100 0 ?
Route Distinguisher: 3:2
Network Next Hop Metric LocPrf Weight Path
*>i 172.16.3.0/24 192.168.2.2 0 100 0 300 ?
*>i 172.16.33.0/24 192.168.2.2 0 100 0 ?
In Example 10-32, CE3’s BGP table contains the peering networks for the other company networks (172.16.11.0/24, 172.16.22.0/24, 172.16.44.0/24, and 172.16.55.0/24). Those network prefixes could be assigned a different RT on the PE routers (for example, RT 1:11 for Company A and RT 1:21 for Company B) as they are advertised to other PEs. The networks that should be imported into the other VRF should maintain the original RT for that VRF.
The VRF definitions would then include three RTs: the two for each VRF marking and one for the cloud provider (Company C’s). It is important to note that a route is exported only once. In other words, if VRF VPN01 imports a route from VRF VPN02, VRF VPN01 cannot export VRF VPN02’s routes as its own. VRF VPN03 must import from VRF VPN01 directly. This is a similar concept to how IBGP learned routes are not reflected to other IBGP peers.
A VRF can use multiple different RTs at a time. Trying to find routes that match a specific RT may seem like a daunting task on a route-reflector. However the solution involves the creation of an extended community list on IOS and NX-OS devices or a route policy on IOS XR routers. The extended community list or route policy is then applied against the VPNv4 address family, as shown in Example 10-34.
Example 10-34 Viewing Prefixes Based on RT
IOS-RR# config t
IOS-RR(config)# ip extcommunity-list 1 permit rt 1:10
IOS-RR(config)# exit
IOS-RR# show bgp vpnv4 unicast all extcommunity-list 1
! Output omitted for brevity
BGP table version is 21, local router ID is 192.168.100.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i 172.16.1.0/24 192.168.1.1 0 100 0 100 ?
*>i 172.16.11.0/24 192.168.1.1 0 100 0 ?
Route Distinguisher: 1:3
*>i 172.16.4.0/24 192.168.3.3 0 100 0 100 ?
*>i 172.16.44.0/24 192.168.3.3 0 100 0 ?
RP/0/0/CPU0:XR-RR# conf t
RP/0/0/CPU0:XR-RR(config)# route-policy RT
RP/0/0/CPU0:XR-RR(config-rpl)# if extcommunity rt matches-any ( 1:10) then
RP/0/0/CPU0:XR-RR(config-rpl)# pass endif
RP/0/0/CPU0:XR-RR(config-rpl)# end-policy
RP/0/0/CPU0:XR-RR(config)# commit
RP/0/0/CPU0:XR-RR(config)# end
RP/0/0/CPU0:XR-RR# show bgp vpnv4 unicast route-policy RT
! output ommitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i172.16.1.0/24 192.168.1.1 0 100 0 100 ?
*>i172.16.11.0/24 192.168.1.1 0 100 0 ?
Route Distinguisher: 1:3
*>i172.16.4.0/24 192.168.3.3 0 100 0 100 ?
*>i172.16.44.0/24 192.168.3.3 0 100 0 ?
*
NXOS-RR# config t
NXOS-RR(config)# ip extcommunity-list standard RT permit rt 1:10
NXOS-RR(config)# exit
NXOS-RR# show bgp vpnv4 unicast extcommunity-list RT
! output ommitted for brevity
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*>i172.16.1.0/24 192.168.1.1 0 100 0 100 ?
*>i172.16.11.0/24 192.168.1.1 0 100 0 ?
Route Distinguisher: 1:3
*>i172.16.4.0/24 192.168.3.3 0 100 0 100 ?
*>i172.16.44.0/24 192.168.3.3 0 100 0 ?
*>i172.16.44.0/24 192.168.3.3 0 100 0 ?
RT Constraints
As stated earlier, the PE router discards any VPN prefixes that contain an RT that does not match any of the import RTs for any of the VRFs. The filtering occurs on the PE router after it has received the prefixes. This is not much of an issue for environments that have a low number of VRFs or when the VRFs are consistently deployed across all VRFs.
However, most service providers have hundreds of PE routers that each hold thousands of VRFs on them. Only a small portion of the VRFs are located on a single router, which results in the PE router receiving a lot of routes that it does not need from the VPNv4 RR. This behavior causes a lot of wasted resources (CPU, memory, and network) between the PE and RR routers.
Figure 10-11 demonstrates the behavior where PE1 has two VRFs configured (VPN01 and VPN02) and PE2 has two VRFS configured (VPN02 and VPN03). Both PEs advertise their VPN prefixes to the RR. The RR advertises all VPN prefixes toward the PE routers. The VPN prefixes for the VPN03 VRF are not needed on PE1, and the VPN prefixes for VPN01 are not needed on PE2, but both unwanted prefixes are advertised.
The same concept applies to VPN prefixes that are exchanged between RRs. Not only are there unwanted updates from RR to PE, there are unwanted updates between RRs. Figure 10-12 demonstrates the following:
RR1 receives VPN prefixes from PE1 and PE2 (VPN01, VPN02, and VPN03).
RR1 advertises these VPN prefixes to RR2, which then advertises the VPN prefixes to PE3 and PE4.
RR2 receives VPN prefixes from PE3 and PE4 (VPN01, VPN03, and VPN04).
RR2 advertises these VPN prefixes to RR1, which then advertises the VPN prefixes to PE1 and PE2.
It should be apparent that the VPN03 VPN prefix is not needed on the RR2, PE3, or PE4 routers, and that the VPN04 VPN prefix is not needed on the RR1, PE1, or PE1 routers.
RT Constraint (RTC) is a mechanism to prevent the propagation of the VPN prefixes to a PE (or RR) that is not interested in the VPN prefix. RTC is defined in RFC 4684, and operates in a similar fashion to Outbound Route Filtering (ORF) of IPv4 prefixes.
RTC is based on the concept that every BGP peer advertises its list of RTs to a BGP peer. The BGP peer then applies a filter to the list of VPN prefixes it has and advertises only VPN prefixes that match the RTs that match the peer. In essence, the PE router tells the RR which VRFs it has and wants to receive.
On PE routers with low VPN membership, this functionality helps reduce the scaling requirements of the PE. The PE router does not have to spend resources on filtering out unwanted updates, and thus the memory and CPU requirements on the PE are reduced.
On the other hand, the resource requirements on the RR increase to accommodate holding a list of RTs because they are required to filter the VPN prefixes that it advertises. At the same time, the RR needs to process only the VPN routes used by its RR clients, and the resources that would be spend on the actual transmission of unwanted updates are saved.
RTC works through the usage of an additional address-family that contains all the RT membership information. This is known as the RT-Filter address-family. Routers that want to participate in RTC establish a VPNv4 (or VPNv6) session and a RT-Filter session.
A RR can advertise VPN prefixes to RTC and non-RTC clients at the same time. RT filtering occurs only for routers that establish a RT-Filter BGP session with it.
At the time of this writing, RTC is supported only on IOS and IOS XR routers. IOS initialize the RT-Filter address family with the command address-family rtfilter unicast. The BGP session is then activated, and route-reflector clients can be defined as well. An IOS XR router initializes the RT-Filter address family with the command address-family ipv4 rt-filter, and then the address-family is associated underneath each BGP peer as well.
Example 10-35 provides the sample configuration for PE1 (IOS) and PE2 (IOS XR).
Example 10-35 RTC Configuration for IOS and IOS XR
PE1-IOS
router bgp 65000
neighbor 192.168.100.100 remote-as 65000
neighbor 192.168.100.100 update-source Loopback0
!
address-family vpnv4
neighbor 192.168.100.100 activate
neighbor 192.168.100.100 send-community extended
exit-address-family
!
address-family rtfilter unicast
neighbor 192.168.100.100 activate
neighbor 192.168.100.100 send-community extended
exit-address-family
PE2-XR
router bgp 65000
address-family vpnv4 unicast
address-family ipv4 rt-filter
!
neighbor 192.168.100.100
remote-as 65000
update-source Loopback0
address-family vpnv4 unicast
address-family ipv4 rt-filter
Note
The RR uses the same configuration as Example 10-35, but also configures the PE routers as route-reflector clients.
The RT-Filter address-family always sets the next-hop IP address to the router that advertised the RT-Membership prefix. This allows for RT-Membership to cascade or follow the advertisement path across multiple RRs. The RT-Membership prefix is structured as follows:
The first 4 bytes are for the ASN of the advertising router.
The second octet reflects the BGP extended community type.
The third and fourth octets reflect the actual RT itself.
The contents of the RT-Filter table is displayed on IOS routers with the command show bgp rtfilter unicast {all | default | rt rt}. IOS XR routers use the command show bgp ipv4 rt-filter.
Example 10-36 displays RT-Membership prefixes on the RR in Figure 10-4. Notice that PE1 and PE2 have all advertised 65000:2:1:1, which signifies that both PE routers have RT 1:1 configured on them. PE1 has the additional RTs of 1:10, 1:11, and 1:12 configured as well.
Example 10-36 RT-Membership Prefixes on the Route Reflector
RR# show bgp rtfilter unicast all
! Output omitted for brevity
Network Next Hop Metric LocPrf Weight Path
0:0:0:0 0.0.0.0 0 i
* i 65000:2:1:1 192.168.2.2 100 32768 i
*>i 192.168.1.1 0 100 32768 i
*>i 65000:2:1:10 192.168.1.1 0 100 32768 i
*>i 65000:2:1:11 192.168.1.1 0 100 32768 i
*>i 65000:2:1:12 192.168.1.1 0 100 32768 i
Notice the 0:0:0:0 RT-Membership prefix that is advertised on the RR. This prefix is advertised to all the RT-Filter route-reflector clients to tell them that they should advertise all VPN prefixes to it. Failing to configure the route-reflector clients on the RR prohibits the advertisement of RT-Membership prefixes to the RR.
Note
PE devices that want to receive all VPN prefixes but do not want the BGP session to reset, which occurs when disabling the RT-Filter address-family, can advertise the default route to the RR with the neighbor default-originate command.
MPLS VPN Label Exchange
After the CE routers have routes for source and destination networks, the focus shifts toward the MPLS forwarding of the network traffic. As stated earlier, packets are forwarded by MPLS labels between PE routers to PE routers.
But how does a PE router differentiate one VRF’s network traffic from another VRF when there are thousands of VRFs on the same router? After talking about import and export RTs, some network engineers might think that the export RTs are included as part of the packet header. The RT is 8 bytes in length, and multiple RTs on a packet would make the packet headers too big.
Instead, VRF identification works by adding a second MPLS label. When a packet is encapsulated at the source PE, the MPLS VPN label (inner) first, and then a second forwarding MPLS label (outer) is added to the packet. Forwarding between the PEs uses the outer label, and the inner label is used by the destination PE to identify which VRF it belongs to. Figure 10-13 displays the packet capture of a MPLS L3VPN packet, which should help with the visualization of the MPLS labels.
The destination PE router assigns a unique local MPLS VPN label for every network prefix that is attached to a VRF. The MPLS VPN PE label is advertised as part of the BGP path attribute. Upon receipt of a packet, the destination PE router looks up the MPLS VPN label and can identify which VRF it is associated with and the outbound interface that the packet should be sent out on. Regardless of the source PE router, the MPLS VPN label will be the same for that VPN prefix.
Note
It is possible for the same MPLS VPN label to be used for two different VPN prefixes as long as the VPN prefixes are associated to different destination PEs. This is because the first differentiator is the packet’s next-hop, which correlates to a different PE router.
Example 10-37 displays the BGP tables for the sample topology in Figure 10-4. PE2 is advertising the 172.16.2.100/32 VPN prefix with the MPLS VPN label of 24013 to the other PE routers. Notice that the PE1 and PE3 routers have the same Out label for the 172.16.2.100/32 network.
If the MPLS VPN labels are not the same, the packet does not forward out the correct port. Additional troubleshooting is needed to identify why the MPLS VPN label changed between the PE routers.
PE1-IOS# show bgp vpnv4 unicast vrf VPN01 labels
Network Next Hop In label/Out label
Route Distinguisher: 1:1 (VPN01)
172.16.1.0/26 172.16.1.11 25/nolabel
0.0.0.0 25/nolabel(VPN01)
172.16.1.100/32 172.16.1.11 26/nolabel
172.16.1.200/32 0.0.0.0 27/nolabel(VPN01)
172.16.2.0/26 192.168.2.2 nolabel/24012
172.16.2.100/32 192.168.2.2 nolabel/24013
172.16.2.200/32 192.168.2.2 nolabel/24012
172.16.3.0/26 192.168.3.3 nolabel/492287
172.16.3.100/32 192.168.3.3 nolabel/22
172.16.3.200/32 192.168.3.3 nolabel/492287
RP/0/0/CPU0:PE2-XR# show bgp vrf VPN01 ipv4 unicast labels
! Output omitted for brevity
Network Next Hop Rcvd Label Local Label
Route Distinguisher: 1:1 (default for vrf VPN01)
*>i172.16.1.0/26 192.168.1.1 25 nolabel
*>i172.16.1.100/32 192.168.1.1 26 nolabel
*>i172.16.1.200/32 192.168.1.1 27 nolabel
*> 172.16.2.0/26 0.0.0.0 nolabel 24012
* 172.16.2.22 nolabel 24012
*> 172.16.2.100/32 172.16.2.22 nolabel 24013
*> 172.16.2.200/32 0.0.0.0 nolabel 24012
*>i172.16.3.0/26 192.168.3.3 492287 nolabel
*>i172.16.3.100/32 192.168.3.3 22 nolabel
*>i172.16.3.200/32 192.168.3.3 492287 nolabel
Processed 9 prefixes, 10 paths
PE3-NXOS# show bgp vrf VPN01 ipv4 unicast labels
! Output omitted for brevity
Network Next Hop In label/Out label
*>i172.16.1.0/26 192.168.1.1 nolabel/25
*>i172.16.1.100/32 192.168.1.1 nolabel/26
*>i172.16.1.200/32 192.168.1.1 nolabel/27
*>i172.16.2.0/26 192.168.2.2 nolabel/24012
*>i172.16.2.100/32 192.168.2.2 nolabel/24013
*>i172.16.2.200/32 192.168.2.2 nolabel/24012
*>r172.16.3.0/26 0.0.0.0 492287/nolabel (VPN01)
* e 172.16.3.33 492287/nolabel (VPN01)
*>e172.16.3.100/32 172.16.3.33 22/nolabel (VPN01)
*>r172.16.3.200/32 0.0.0.0 492287/nolabel (VPN01)
MPLS Forwarding
The first portion of this section focused on the control plane aspect of troubleshooting MPLS L3VPNs. The other component is to verify that MPLS forwarding is working properly. Enabling MPLS operations, administration, and maintenance (OAM) is required on IOS XR for this troubleshooting technique. After it is enabled, a router can ping or traceroute using MPLS labels to verify PE-to-PE MPLS connectivity. The destination address should match the IP address associated with the next-hop IP address shown in BGP (that is, a loopback interface with a /32 IP address).
The command structure is ping mpls ipv4 destination-network /prefix-length. If connectivity is successful in one direction, it needs to be checked in the other direction for return traffic. MPLS forwarding should be troubleshot as a unidirectional technology.
Example 10-38 displays the MPLS ping being performed on PE1, PE2, and PE3.
PE1-IOS# ping mpls ipv4 192.168.2.2/32
Sending 5, 100-byte MPLS Echos to 192.168.2.2/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/5/7 ms
RP/0/0/CPU0:PE2-XR# ping mpls ipv4 192.168.3.3/32
! Output omitted for brevity
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/8/10 ms
PE3-NXOS# ping mpls ipv4 192.168.1.1/32
! Output omitted for brevity
Type Ctrl-C to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/9/10 ms
Total Time Elapsed 57 ms
In the event that the MPLS ping fails, a MPLS traceroute can be executed with the same command structure of traceroute mpls ipv4 destination-network /prefix-length, which could help with isolating where the problem may exist.
Note
The troubleshooting of MPLS forwarding is outside of the scope of this book and can be found in other Cisco Press books, such as MPLS Fundamentals by Luc de Ghein.
Summary
This chapter provided a general overview of MPLS VPNs. MPLS L3VPNs provide privacy of customer networks through the usage of VRFs and MPLS forwarding between PE routers.
Troubleshooting MPLS L3VPN consists of troubleshooting the control plane (VPN prefix exchange via BGP) and MPLS forwarding. Most of the complexity comes from troubleshooting the control plane component of MPLS L3VPN. Common issues to look for when troubleshooting are the following:
AS-Path loops on the CE routers. This is overcome by the use of as-override on the PE router.
Verify that the import RTs match the export RTs for the specific VRF. Making these the same where possible simplifies troubleshooting.
Verify that MPLS VPN labels are the same on both PE routers.
If RTC is enabled, make sure that the RR is advertising a default route in the RT-Filter address-family toward the route-reflector clients. Verify that the RTs are being advertised toward the RR.
When a route is missing on the CE or PE router, start to check the path of route advertisements toward the PE advertising the route.
References
RFC 4360, BGP Extended Communities Attribute, Srihari Sangli, Dan Tappan, Yakov Rekhter, IETF, http://www.ietf.org/rfc/rfc4360.txt, February 2006.
RFC 4364, BGP/MPLS IP Virtual Private Networks (VPNs), Eric Rosen, Yakov Rekhter. IETF, https://tools.ietf.org/html/rfc4364, February 2006.
RFC 4684, Constrained Route Distribution for Border Gateway Protocol/Multiprotocol Label Switching (BGP/MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs), Marques, R. Bonica, L. Fange, et al., IETF, https://tools.ietf.org/html/rfc4684, November 2006.
Cisco. Cisco IOS Software Configuration Guides. http://www.cisco.com.
Cisco. Cisco IOS XR Software Configuration Guides. http://www.cisco.com.
Cisco. Cisco NX-OS Software Configuration Guides. http://www.cisco.com.