Chapter 2. Transport Independence

The primary focus of WANs is to provide the ability to exchange network traffic across geographic distances. In addition, scalability and cost are important factors for the design of a WAN. The provisioning and operation of a WAN can be a complex task for any network engineer because of the variety and behaviors of technologies available.

Telephone companies use the existing infrastructure and sell access between devices as dedicated leased lines. Leased lines provide high-bandwidth secure connections but lack the flexibility of dial-up. Regardless of link utilization, leased lines provide guaranteed bandwidth because the circuits are dedicated between sites in the SP environment.

Most SPs now install fiber-optic cables between locations; each cable contains multiple strands that support multiple connections. Only a portion of the cable strands is consumed, which means that additional capacity can be consumed internally or leased to other companies. Unused fiber-optic cables are called dark fiber circuits. Some companies specialize in locating and reselling dark fiber circuits. Customers are provided with a transport medium but are responsible for installing devices at each end for any routing or switching capabilities. Dark fiber is not really a service but more of an asset.

Organizations can take advantage of the full capability of fiber-optic cable (commonly 10 Gbps) by using dense wavelength-division multiplexing (DWDM) devices. These allow a 10 Gb interface to be assigned a specific optical wavelength, and because some fiber-optic cable supports multiple optical wavelengths, a single strand of cable supports multiple 10 Gb links. Bandwidth is guaranteed because only two devices can communicate on the same wavelength, and no wavelength conflicts with other wavelengths.

Technologies like X25, Frame Relay, and ATM introduced the concept of sharing a line through the use of virtual circuits. Virtual circuits provide a tunnel for traffic to follow a specific path in an SP’s network. They allow multiple conversations to exist on physical links, reducing the SP’s operational costs and allowing communication with multiple devices using the same physical interface.

Virtual circuits provide the following advantages:

Communication is allowed with more than two devices at the same time using the same interface.

Interfaces can operate with different bandwidth settings between devices.

The amount of bandwidth required at a remote site can vary depending upon the number of users and data services required at each site. Some remote sites may be able to support Ethernet connectivity, but other sites can connect only via serial links because of circuit length requirements. If two sites connect via two different transport media (such as Ethernet and serial) and must be able to communicate with each other, the SP typically provides the connectivity indirectly by routing (Layer 3) between the two sites.

Figure 2-1 illustrates an SP providing connectivity between Site 1 and Site 2 using different network media. There are CE routers located at the edge of the customer network and on the customer’s site. The CE routers connect to PE routers that are located at the SP’s site. The PE routers connect to provider (P) routers or other PE routers.

CE1 connects to PE1 via an Ethernet link (172.16.11.0/24), and CE3 connects to PE3 via a serial link (172.16.33.0/24). CE1 exchanges routes with PE1 via a dynamic routing protocol which is then advertised to P2 and then on to PE3. CE2’s routes are exchanged in an identical fashion toward CE1. In this model, the SP network provides network connectivity to both locations for both customers.

A peer-to-peer network distributes all aspects of the network among all its peers. A problem with peer-to-peer networks is that the SP participates in the network. Any existing networks in the SP’s network cannot be used in the customer’s network, and all IP address spaces are shared with everyone attached to it, including other customers.

In Figure 2-1, the peer-to-peer network is shared between two customers (Customer A and Customer B). Customer A cannot use the 172.16.2.0/24 network or the 172.16.4.0/24 network because Customer B is using them. This limitation can cause issues when two different companies use private IP addressing ranges (RFC 1918) and want to use the same network segment.

Broadband networks exist in two formats:

Cable modem: Connectivity is provided through the existing coaxial infrastructure used by cable television providers. Initial cable modem speeds were 1 to 2 Mbps but currently provide 100+ Mbps. Because the coaxial infrastructure is shared among several locations, the bandwidth is shared and can fluctuate based on a neighbor’s bandwidth usage. Although bandwidth may peak to what is stated in the SLA, setting the bandwidth to the average is recommended.

DSL: There are different variants of DSL, but in essence it uses the existing copper wires in the PSTN network that provide telephone service to a house. DSL uses higher frequencies that humans cannot hear to transmit data. It does not use a shared infrastructure but is generally limited to a small distance (about 2 to 5 km) from the telephone company’s head-end device. The farther away from the head-end device, the lower the available bandwidth. DSL can reach speeds in excess of 40 Mbps.

Broadband has since expanded in scale and is no longer considered a home consumer technology. Many telephone and cable companies also provide broadband services to businesses.

Reachability: Not all businesses operate in places where an SP has circuits. In some remote locations (such as mountains or islands) cables for connectivity may not exist. The coverage of cellular wireless networks enables businesses to take the network into geographic locations that were never available via wired connections.

Backup connectivity: The speed of cellular networks offers businesses the ability to run their operations over what was typically used as an out-of-band management link. This means that cellular wireless networks can provide a second connection where multiple SPs are not available. This includes voice and video support over cellular in the event of a primary network outage.

Timeliness: Terrestrial circuits can typically take up to 90 days to deploy at new locations. The quick activation of cellular allows businesses to bring locations and services online in days or even hours instead of weeks or months.

Ability to host networks that change in location: Some mobile vendors (such as food trucks) need network connectivity to verify credit card transactions. Another growing sector is the use of cellular wireless networks to track a patient’s vital signs while the patient is at home, reducing the patient’s need to visit a healthcare facility, or making care available to patients who are unable to travel.

Ability to host networks in motion: Some businesses require the ability to provide network connectivity to devices in motion. For example, passenger trains may provide WiFi service to their passengers, track critical on-board sensors, or combine GPS sensors to track the vehicle location.

A VPN provides connectivity to private networks over a public network, such as the Internet. A VPN operates by tunneling, encrypting the payload, or both. With VPN tunneling, packets destined to travel between private networks are encapsulated and assigned new packet headers that allow the packets to traverse the public network. A VPN tunnel is classified as an overlay network because the VPN network is built on top of an existing transport network, also known as an underlay network. After authenticating with the remote VPN endpoint, the VPN tunnel is established for packets to traverse to the private network. A VPN can leverage public network transport such as the Internet to provide global connectivity using only one transport.

Within VPN tunnels, the new packet headers provide a method of forwarding a packet across the public network without exposing the private network’s original packet headers. This allows the packet to be forwarded between the two endpoints without requiring any routers to extract information from the payload (original packet headers and data). After the packet reaches the remote endpoint, the VPN tunnel headers are decapsulated (removed). The endpoint checks the original headers and then forwards the packet through the appropriate interface to the private network.

There are multiple VPN protocols with various benefits and design considerations for encapsulating, encrypting, and transferring data across public networks. Internet Protocol Security (IPsec), Secure Sockets Layer (SSL), Datagram Transport Layer Security (DTLS), and Point-to-Point Tunneling Protocol (PPTP) are some of the most common protocols used today for VPNs. The following sections explain the most common VPN types.

Typically the VPN server defines any access or security policies on the client. The VPN server can require that all traffic from the client be sent across the VPN tunnel, known as full tunnel, or specify that traffic to only specific destination networks route across the VPN tunnel, known as split tunneling.

Each VPN endpoint must know the public IP address of the other. In addition, the endpoints must identify the interesting traffic (source and destination networks) that will use the VPN tunnel. If there is a mismatch of interesting traffic, either the packets are not encapsulated properly or the packets are not decapsulated and are dropped.

When the first VPN endpoint receives interesting traffic, the VPN tunnel is established. Private network packets are encapsulated, routed across the public network toward the remote endpoint, and decapsulated. Depending on the endpoint configuration, the VPN tunnel remains up indefinitely, or an idle timer can be configured. The idle timer is reset if packets come across the tunnel. If the timer reaches zero, the VPN tunnel is torn down.

Figure 2-2 illustrates a typical hub-and-spoke topology between R1, R2, and R3.

R1 is the hub, and R2 and R3 are spokes. Network latency from either spoke to the hub is 110 ms, which is generally an acceptable latency for most applications. The end-to-end latency between spokes (R2 to R3) is 220 ms, which may be acceptable for most applications but can cause problems for others. For example, VoIP recommends a maximum delay of 150 ms. Exceeding the recommended delay can affect call audio quality for users at R2 talking with users at R3.

Allocating proper bandwidth for the hub site is critical. The hub site is the typical destination for most network traffic, but traffic between other spokes must transit the hub circuit twice, one time inbound and another leaving the hub to the other branch site. Although oversubscription of spoke bandwidth is common, bandwidth oversubscription at the hub site should not be common.

Figure 2-3 shows all three sites with direct site-to-site connectivity. This illustrates a full-mesh topology because all sites are connected to all other sites regardless of size.

Full-mesh topologies using site-to-site VPN tunnels can introduce a variety of issues as the number of sites increases. Those issues are

Manageability of VPN endpoints: When a new site needs connectivity, the VPN tunnel interface needs to be configured on the new router and the remote router. The formula n(n−1) provides the number of VPN tunnel interfaces that must be configured, where n reflects the number of sites. In Figure 2-3 there are three sites, but the topology requires the creation of six VPN tunnel endpoints. The problem is further exacerbated when 10 sites are required because 90 VPN tunnel interfaces must be managed.

Consumption of router CPU and memory: Maintaining a high number of active VPN tunnels consumes more CPU and memory resources. All routers need to be sized accordingly to accommodate any load generated for maintaining the VPN tunnels.

Most VPN designs for large organizations typically place a hub in a major geographic region. The hub is connected via full-mesh, and then the interregion traffic is contained via a hub-and-spoke model.

An MPLS network forwards traffic based upon the outermost MPLS label of a packet. The MPLS labels are placed before the IP headers (source IP and destination IP), so none of the transit routers require examination of the packet’s IP header or payload. As packets cross the core of the network, the source and destination IP addresses are never checked as long as the forwarding label exists in the packet. Only the PE routers need to know how to send the packets toward the CE router. An MPLS VPN is considered an overlay network because network traffic is forwarded on the SP’s underlay network using MPLS labels.

All MPLS VPNs are categorized by two technologies on the PE router: Layer 2 and Layer 3 VPN.

Virtual Private LAN Service (VPLS) is an evolution of L2VPN that provides multisite access to the same bridge domain. A VPLS includes logic to maintain MAC address table mappings to a specific PE router and provides the capability of LAN switching to devices spread across large geographic areas.

An L2VPN can be set up with topologies of point-to-point, point-to-multipoint, or full-mesh. This technology allows for all packets to be exchanged between PE routers.

The PE routers must contain all the routes for a particular customer (VRF), whereas the CE routers require only a default route to the PE router. The route table on the PE routers can be programmed via a static route at the local PE router or is advertised from the CE router. A PE router exchanges the VRF’s routes with other PE routers using Multiprotocol Border Gateway Protocol (MBGP) using a special address family just for MPLS L3VPN networks. There are VPN labels associated to each of the VRF’s routes to identify which VRF the routes belong to.

A CE router can use only a static default route toward the closest PE router, and the PE routers are responsible for maintaining and exchanging the complete routing table for that VRF using MBGP. But the CE router can use a routing protocol to dynamically advertise networks to the PE routers. This allows the PE routers to have a complete routing table that is dynamically learned versus statically configured. BGP is commonly used to exchange routes between CE and PE routers because of its scalability and capability for route policy manipulation.

MPLS is enabled in the SP network to forward traffic between PE routers. A packet is forwarded based upon the outermost label of the packet that references the destination PE router. Only the edge PE routers need to examine other portions of the packet (internal MPLS labels, packet headers, and payload) as they send traffic to the CE router.

Using MPLS VPNs provides reliable bandwidth because the SP owns the entire infrastructure. Deploying a new site requires connecting a new circuit between the CE router and the PE router. Minor configuration is needed on the PE router with the new circuit, and then that remote site has full connectivity to all other sites in the MPLS VPN.

The primary difference between L2VPNs and L3VPNs is how the CE-PE relationship is handled. L2VPNs are concerned only from the Layer 2 perspective, whereas L3VPNs participate in the routing table of the customer networks. Both models provide security to the customer networks through circuit/network segmentation, and the SP network is invisible to the customer. Each model has its advantages and disadvantages.

Some organizations that are subject to stringent privacy regulations (such as healthcare, government, and financial) often require all traffic to be encrypted across all WAN links; other organizations encrypt traffic only where they feel vital data can be compromised. In instances like these, a second technology to encrypt and decrypt the traffic must be deployed.

Figure 2-4 illustrates this concept. The link connecting R1 and R2 allows 10 Mbps of traffic in each direction. R1 receives 20 Mbps of traffic locally, but R1 shapes the traffic down to a 10 Mbps traffic stream when sending on to R2. R2 receives a 15 Mbps traffic stream locally, but R2 shapes the traffic to a 10 Mbps traffic stream before sending it on to R1. If the QoS class buffers become full, packets should be dropped before traffic is sent across the link.

The same logic may not apply to multipoint topologies because sites may have different bandwidth connectivity. It is possible for one router to send more traffic than the remote network can accept. When this happens, bandwidth is wasted on the transmitting router’s link because those packets will ultimately be dropped. In addition, it is possible to prevent a recipient’s router from receiving traffic from a different site.

Figure 2-5 illustrates the concept. Company ABC is using an MPLS VPN for connectivity between its headquarters (HQ) and branch sites. A majority of the network traffic is from the Los Angeles headquarters site to the Miami branch site. R1’s circuit bandwidth has been sized to allow simultaneous communication with R2 and R3.

Unfortunately this causes problems instead. The problem is that R1 is not aware of R3’s bandwidth and transmits up to a rate of 10 Mbps, whereas R3 can receive only 5 Mbps of traffic. The SP drops traffic that exceeds 5 Mbps as it is being sent to R3. In addition to wasting some of R1’s bandwidth, R2 cannot communicate with R3 because R3’s link is oversubscribed.

The dynamic spoke-to-spoke behavior allows full-mesh connectivity without the additional management of providing full-mesh connectivity. The spoke-to-spoke tunnels are removed after a certain period of inactivity, freeing up router memory and CPU. Because the site-to-site tunnels are dynamic, the spoke routers do not require as much memory or CPU as the hub routers.

Figure 2-6 illustrates a DMVPN tunnel where R1 is the hub and R2, R3, and R4 are spoke routers. R2 and R4 establish a dynamic spoke-to-spoke tunnel for direct communication, and eventually that dynamic tunnel is torn down after it is no longer needed.

DMVPN uses the following technologies:

Multipoint generic routing encapsulated (mGRE) tunnels: mGRE is an overlay tunneling protocol that transports multiple protocols such as IPv4, IPv6, IPX (Internetwork Packet Exchange), and so on. Unlike IPsec VPN tunnels, GRE tunnels are assigned an actual interface and require addressing on the tunnel interface. Unlike traditional GRE tunnels which are point-to-point, mGRE supports more than two devices on the overlay network.

Next Hop Resolution Protocol (NHRP): Because of the dynamic nature of DMVPN, the DMVPN spoke routers need a method to associate the tunnel endpoint IP address with the transport IP address for other DMVPN routers. NHRP provides IP resolution to next-hop clients (NHCs) by registering and querying a next-hop server (NHS). Originally used for mapping IP addresses on .X25 networks, it is now used for resolving the tunnel endpoint addresses to the logical IP addresses of DMVPN network cloud. The protocol was defined in RFC 2332.

IPsec tunnel protection (optional): IPsec uses cryptography to authenticate and encrypt IP network traffic across networks. It supports Internet Key Exchange (IKE) v1 and v2, and the Suite-B next-generation encryption technologies.

DMVPN provides the following benefits over traditional site-to-site IPsec VPN:

Zero-touch hub: Adding more spoke routers does not require any additional configuration on the hub router. This simplifies the ongoing maintenance of the hub routers. Resolution of spoke endpoint IP addressing is accomplished with NHRP.

Spoke-to-spoke communication: Phases 2 and 3 of NHRP support dynamic spoke-to-spoke network connectivity. All the DMVPN routers obtain the benefits of a full-mesh topology but conserve router resources by creating tunnels between VPN endpoints only as needed. Spoke-to-spoke communication reduces latency and jitter, while avoiding consumption of bandwidth at hub locations.

Tunnel IP addressing: DMVPN uses GRE tunnels that use IP addressing in the overlay network. IPv4 or IPv6 addresses can be used in the overlay or to provide connectivity in the underlay. Routing protocols can run on top of the DMVPN tunnel, thereby allowing dynamic network updates versus manually updating the routing tables when using IPsec VPN tunnels.

Encryption is optional: The benefits of overlay routing can be obtained with MPLS L3VPN networks without unnecessary consumption of router resources for encryption. DMVPN tunnels operating on insecure networks (such as the Internet) typically encrypt the payload with IPsec technologies.

Multicast support: Native IPsec VPN tunnels do not support multicast traffic, which complicates the ability to transmit multicast traffic from branch sites to headquarters sites. DMVPN provides multicast support between hub-and-spoke devices for hub-to-spoke traffic flows.

Per-tunnel QoS: DMVPN supports the ability of the DMVPN hub to set different QoS and bandwidth policies for each tunnel to a spoke router based on the site’s connectivity model.

Ensuring that the network is always available for business purposes requires that there not be any SPOFs. Network architects plan for backup circuits that are active only when the primary circuit fails, or dual circuits that can be used at the same time. Network architects must also decide on the number of routers at each site, whether a router is dedicated to each circuit, or if both circuits should connect to the same router.

Using a single router with a single transport provides “three nines” (99.9%) of availability, which correlates to about four to nine hours of downtime a year. Using a single router with two transports provides “four nines” (99.995%) of availability, correlating to 26 minutes of downtime a year. Using two routers, each with its own transport, provides “five nines” (99.999%) of availability, which correlates to five minutes of downtime a year.

Figure 2-7 depicts a typical scenario for Company ABC, which uses two different providers and transports because it cannot locate two SPs that provide the same service at both the branch and headquarters. ABC uses MPLS L3VPN from one SP and creates an IPsec tunnel across the Internet for a backup circuit. R1 and R2 form an external BGP (EBGP) session to the MPLS L3VPN provider, and the routers use static routes and reverse-route injection to send traffic across the IPsec tunnel.

In scenarios like the one shown in Figure 2-7, one transport acts as a primary and the other as a backup, and there are multiple routing protocols on the WAN routers that require advanced configuration to ensure that the correct path is taken. The complexity increases when routes are exchanged between IGP (Enhanced IGRP [EIGRP], Open Shortest Path First [OSPF], and so forth) at each site and the WAN routing protocols, which could lead to suboptimal routing or routing loops.

A well-designed network architecture should take into account the operational support staff and reduce complexity where possible. It should account for junior-level engineers working in the network operations center (NOC) performing day-to-day management of the network. Redistribution between protocols should be kept to a minimum to prevent confusion and routing loops. The design should be modular to support future growth and allow for a simple migration between SPs to address business needs.

DMVPN provides transport independence through the use of full-mesh overlay routing. This technology allows organizations to use multiple WAN transports, because the transport type is associated to the underlay network and is irrelevant to the overlay network. The overlay network is consistent and normalized to the DMVPN tunnel.

Transport independence allows operational consistency for WAN architectures by providing the following:

Single routing domain: Traditional routing protocols were designed to solve the endpoint reachability problem in a hop-by-hop destination-only forwarding environment of unknown topology. The routing protocols choose only the best path based on statically assigned cost. Because DMVPN presents a flat topology, it provides a consistent topology that allows ECMP load balancing across DMVPN tunnels.

Consistent troubleshooting: The same process can be used for troubleshooting connectivity for the DMVPN tunnel and the underlay network. Even though the underlay transport changes, it relies on basic IP connectivity between tunnel endpoints.

Consistent topology: There is a consistent topology and methodology for deploying other services such as performance routing.

Figure 2-8 illustrates the same topology as before except that R1 and R2 use a DMVPN tunnel for each different transport (path). Company ABC can use the same routing protocol across both paths while maintaining a consistent topology. In addition, it can change one of the WAN transports at a later time without impacting the overall WAN design or operational support model.

Increasing bandwidth to existing primary paths, which can be expensive

Deploying additional network circuits, sometimes using low-cost residential circuits

Deploying application optimization technologies such as Cisco WAAS

Deploying intelligent load-balancing technologies such as PfR

Additional WAN bandwidth improves aggregate throughput but does not improve end-to-end delay or packet loss for critical applications. Furthermore, additional bandwidth can be expensive, especially in rural areas where faster connections may not be available. At the same time, the cost of Internet connectivity is generally decreasing while reliability is increasing, as shown in Figure 2-9. Enterprises are now qualifying the Internet transport as an alternative business-class WAN circuit.

Offloading network traffic to the Internet can help load-balance best-effort traffic (that is, lower-priority traffic) across both links to help deal with traffic from business VPN access. In addition, administrators can use local Internet access for a distributed Internet access model to offload employee traffic that goes directly to public cloud services (such as Google Apps, Salesforce.com, Office 365, and so on) from the private WAN altogether.

This approach uses the Internet connection not just as a backup but as a real component in dealing with WAN workloads. With the right network technologies to optimize the flows, administrators can reduce overall WAN transport bandwidth requirements and improve application performance.

Dual MPLS: The first transport is an MPLS VPN provided by one SP, and a second MPLS VPN provided by a second SP. Assuming that both providers use the same type of MPLS VPN (all MPLS L3VPN or all MPLS L2VPN), the same routing protocol can be used.

Dual hybrid: The first transport is an MPLS VPN provided by one SP, and Internet connectivity is a second transport. The Internet transport can be provided by the same or a different SP. Ideally the SPs would use different “last mile” circuits and routers to eliminate SPOFs. The Internet circuit is used to establish a VPN service to another site.

This model provides connectivity for a distributed Internet model for all or select Internet-based traffic. It is possible to allow Internet access only for IT-approved cloud applications.

Dual Internet: The first transport is Internet connectivity provided by one SP, and Internet connectivity is provided by a different SP for resiliency purposes. This model provides connectivity for a distributed Internet model too and does not have to be restricted to only VPN tunnels.

In all three models, deploying a DMVPN tunnel for each transport provides transport independence and simplifies the routing domain, operational model, and troubleshooting.

Implementing transport independence on existing circuits provides flexibility and leverage during circuit renewals. Typically, using the Internet as a transport maintains a lower cost than using an MPLS VPN for a transport and may produce savings to your company.

Figure 2-10 illustrates three of the Cisco Intelligent WAN (IWAN) models. Notice the connectivity to the Internet and cloud-based applications for all three models. Internet connectivity is available only through the headquarters in dual MPLS, whereas the hybrid and dual Internet models can provide Internet and cloud connectivity directly at the branch.

Properly designed network architecture takes into account the skill level of the network engineers who support and maintain the network. The architecture should be consistent and scalable to ensure consistency in the environment regardless of the technology. Using DMVPN on top of the WAN transports provides a consistent operational model and provides transport independence while

Simplifying the WAN with easy multihoming to SPs with a consistent design over all transports

Providing scalable full-mesh connectivity among all DMVPN routers

Providing a platform that supports proven robust security and cryptography

Providing zero-touch configuration on DMVPN hub routers as branch sites are added

DMVPN is explained in greater detail in the following chapters.