When an organization migrates resources and services to Azure, as a network architect or engineer, you must set up communication between the on-premises and Azure workloads securely and reliably. Azure ExpressRoute lets you connect your enterprise networks on-premises with Microsoft cloud services, such as Azure and Microsoft 365. Using ExpressRoute, you can connect your on-premises network to the Microsoft cloud without experiencing the delays, disruptions, and fluctuating bandwidth that can plague a VPN connection between a corporate network and the Microsoft cloud. At a very high level, this chapter focuses on how Azure ExpressRoute works, what it does, and when you should use it.

To get the most out of this chapter, you must have experience with networking concepts, including IP addressing, DNS, and routing. You must also know how to connect to networks such as VPNs or WANs, and you should be familiar with the Azure portal and Azure PowerShell.

By the end of this chapter, you will be able to design, deploy, and manage Azure ExpressRoute for your organization to connect between on-premises and Azure networks. You should also be able to design and implement a hybrid connectivity solution that will address the short- and long-term goals of your organization's global enterprise IT footprint.

Getting Started with Azure ExpressRoute

In addition to the Azure VPN gateway (covered in Chapter 2, “Design, Deploy, and Manage a Site-to-Site VPN Connection and Point-to-Site VPN Connection”), many organizations use the Azure ExpressRoute method to connect Azure virtual networks (VNets) to on-premises resources. This is because a VPN may not meet every business requirement. A VPN, for example, is limited to a maximum network speed of 1.25 Gbps. Business users who need higher speeds shouldn't opt for VPNs. In other words, all traffic is also sent over the public Internet, which can be prohibitively expensive for some end users.

Microsoft Azure ExpressRoute, which uses private connectivity, serves as an alternative method of connecting your on-premises networks to Microsoft Azure cloud services. It meets most enterprise organizations’ needs, including security and resiliency.

An ExpressRoute circuit represents a logical connection between your on-premises infrastructure and Microsoft cloud services through a connectivity provider, as shown in Figure 3.1.

Your organization can choose from a variety of ExpressRoute circuit types. You can connect each circuit to the on-premises infrastructure through different connectivity providers, either within the same region or in different regions.

Service providers help organizations connect their on-premises infrastructure to the Azure cloud using ExpressRoute. Network service providers offer a wide variety of services.

Organizations worldwide can gain performance advantages through an ExpressRoute connection since the connection is directly made through the ExpressRoute connectivity provider's infrastructure to the edge of Microsoft's network. However, even with ExpressRoute, you may experience suboptimal connectivity since the Internet is typically provided through partnerships and relationships among telecommunications providers.

The provider takes direct responsibility for setting up an optimized connection with the Microsoft network when the organization chooses a dedicated and private connection with a connectivity provider.

To provide fiber-optic connectivity at speeds of up to 10 Gbps, Azure offers a service via ExpressRoute. As shown in Figure 3.2, the ExpressRoute feature allows you to connect to Azure from your on-premises network through a Microsoft Enterprise Edge (MSEE) router. The MSEE router sits on the edge of its network, and in most cases, the connection will also be through a router in your on-premises network.

Since ExpressRoute does not traverse the public Internet, bandwidth is more reliable. You must trust the provider with the information that flows through the circuit to use the ExpressRoute configuration. With ExpressRoute Direct, you can connect directly to a physical port on the MSEE router without dealing with the service provider. You can also choose ExpressRoute Direct if the organization needs greater bandwidth.

Microsoft refers to ExpressRoute connections as circuits. The ExpressRoute service depends on ExpressRoute circuits and ExpressRoute gateways. There are different zones for ExpressRoute circuits that are deployed at peering locations or peering, or meet-me, locations. ExpressRoute gateways are deployed in Azure. You can choose from several plans that have different bandwidth limitations. Each circuit has a fixed bandwidth.

There is a mapping between a circuit and a connectivity provider and peering site. Each peering shares the bandwidth for the circuit. Circuits may peer with up to two peers. Different routings are used depending on the kind of service requested through these peerings, such as Microsoft peering and private peering:

• Through Microsoft peering, you can access Microsoft public services such as Microsoft 365, Dynamics 365, and Power Platform.
• Through private peering, you can access private Azure resources such as Azure VMs.

Border Gateway Protocol (BGP) is primarily used by autonomous systems to exchange network routing and reachability across autonomous systems. The internal BGP is known as iBGP (internal), and the external BGP is eBGP (external). Border Gateway Protocol 4 (BGP-4) is defined by RFC 4271.

Peerings consist of two independent BGP sessions, each configured redundantly for high availability. Resiliency requires that these sessions transit over different physical connections.

The public Internet is advertised with the IP addresses of Microsoft's cloud services. Microsoft also advertises the relevant IP prefixes for the services specified by the peerings for those circuits through the ExpressRoute BGP connection.

It is the responsibility of your network team to set up internal routing to send data to Microsoft. They must:

• Prioritize the route of Microsoft online services traffic through the subnet connected to ExpressRoute.
• Configure routing for Microsoft online services traffic via a BGP session for the subnet connected to ExpressRoute.

The traffic that flows to Microsoft datacenters will be routed to the appropriate service within the datacenter, and Microsoft manages that process.

Azure ExpressRoute has no up-front cost, no termination fees, and charges for only what the organization uses.

The following list details the critical features offered by Azure ExpressRoute:

• Layer 3 Connectivity   The Azure cloud network uses BGP (described earlier in this section), an industry-standard dynamic routing protocol to exchange routes between your organization's on-premises network, your instances in Azure, and Microsoft's public addresses. As part of Microsoft's cloud service, ExpressRoute establishes multiple BGP sessions with the on-premises networks for different traffic profiles.
• Redundancy   A Microsoft Enterprise Edge (MSEE) router is located at each ExpressRoute location and connects two ExpressRoute circuits. Microsoft wants a double BGP connection from the connectivity provider/network edge to each MSEE. Network connectivity service providers use redundant devices to ensure that your connections are properly handed over to Microsoft. Redundant devices/Ethernet services may not be deployed depending on your decision at the organization's end.
• Connectivity to Microsoft Cloud Services   Connecting to Azure and Microsoft 365 via ExpressRoute enables access to your services securely and reliably.
• Connectivity to All Regions within a Geopolitical Region   Connecting to Microsoft via a peering location will allow organizations to access regions within the geopolitical region. For example, if you connect to Microsoft in Amsterdam through ExpressRoute, you have access to all Microsoft cloud services hosted in Northern and Western Europe.
• Global Connectivity with ExpressRoute Premium   ExpressRoute Premium allows your organization to extend connectivity across geopolitical boundaries. If connecting to Microsoft via ExpressRoute from Amsterdam, for example, your organization can use all Microsoft cloud services regardless of where the cloud is hosted. The cloud services can also be accessed in South America or Australia and North and West Europe. However, national clouds are excluded from consideration.

  On a standard ExpressRoute circuit, an organization can have 10 virtual network connections, whereas Premium consumers can have 100 virtual network connections.

  When ExpressRoute is deployed within the same geographic region as Dynamics 365, ExpressRoute Premium is not required.

• Local Connectivity with ExpressRoute Local   When you enable Local SKU, you can transfer data to an ExpressRoute location close to the Azure region. Local includes data transfer in its port charge.
• On-Premises Connectivity between Locations with ExpressRoute Global Reach   Connecting ExpressRoute circuits from on-premises networks to ExpressRoute Global Reach allows data exchange between on-premises sites.

  For example, imagine an oil and gas company with a private datacenter in Abu Dhabi connected to an ExpressRoute circuit in the UAE North region. Another private datacenter in Dubai is connected to an ExpressRoute circuit in UAE Central. These two ExpressRoute circuits allow the organization to connect its private datacenters with ExpressRoute Global Reach.

  Microsoft's network will carry the cross-datacenter traffic from the on-premises networks in this configuration.

• Rich Connectivity Partner Ecosystem   Across ExpressRoute's ecosphere, connectivity providers, and system integrators are continuously growing.
• ExpressRoute Direct   ExpressRoute Direct offers customers a direct connection to the Microsoft global network through peering locations worldwide. Dual 100 Gbps connectivity is provided by ExpressRoute Direct, which supports active/active connections. Organizations can take advantage of the ExpressRoute Direct service through a service provider.
• Connectivity to National Clouds   For specific segments and geopolitical regions, Microsoft operates fully isolated cloud environments.
• Bandwidth Choices and Dynamic Scaling   Azure customers can choose between a variety of ExpressRoute bandwidth options. There are various bandwidths to choose from, such as 50 Mbps, 100 Mbps, 200 Mbps, 500 Mbps, 1 GB, 2 GB, 5 GB, and 10 GB. Without hindering an organization's connections, ExpressRoute circuit bandwidth can be increased (under the best conditions).

ExpressRoute Deployment Model

In four different deployment models, you can build and configure connectivity between consumers’ on-premises networks and the Microsoft cloud. Network connectivity service providers may offer one or more connectivity deployment models, such as cloud exchange co-location, point-to-point Ethernet connection, any-to-any (IPVPN) connection, and ExpressRoute Direct.

Let's start with the co-located at a cloud exchange deployment model, shown in Figure 3.3. In this case, you can order virtual cross-connections to the Microsoft cloud through the co-location provider's Ethernet exchange. Through cross-connections on Layer 2 or 3, Microsoft cloud infrastructure and your on-premises infrastructure in co-location facilities can be integrated.

The second model is point-to-point Ethernet connection, through which you can connect Microsoft cloud customers’ on-premises datacenters or remote offices. Microsoft cloud customers’ on-premises datacenters can connect to Microsoft cloud via managed Layer 3 or Layer 2 connections through point-to-point Ethernet. Figure 3.4 shows the deployment model for a point-to-point Ethernet connection.

In the third network model, any-to-any (IPVPN), you can integrate Microsoft's cloud WAN with your organization's WANs. A VPN provider (likely an MPLS VPN) provides connectivity between cloud client datacenters and branch offices. Connecting a Microsoft cloud to your WAN makes it appear like any other branch office. Your WAN provider typically manages Layer 3 connectivity.

Figure 3.5 shows the any-to-any (IPVPN) deployment model.

The fourth deployment model is direct from ExpressRoute sites; you can connect directly into Microsoft's global network from a peering location strategically placed around the globe. The ExpressRoute Direct router provides dual 100 Mbps or 10-Gbps connectivity. Active/active connectivity is supported at scale. Figure 3.6 shows the deployment model direct from ExpressRoute sites.

Choosing Between the Network Service Provider and ExpressRoute Direct

Azure ExpressRoute enables you to build a secure connection among their on-premises datacenter/branch offices and the Azure cloud. It is necessary to have an existing Azure account to access ExpressRoute. If the desired connectivity provider is not supported, ExpressRoute customers must establish a relationship with a supported connectivity provider or connect to Microsoft cloud services through an exchange.

ExpressRoute Direct enables you to connect directly into Microsoft's global network at peering locations strategically distributed worldwide. Connecting ExpressRoute routers requires the cooperation of the local carrier and co-location provider, which will be the only means for cloud customers to use ExpressRoute Direct. Direct connections to the Microsoft global backbone are available through ExpressRoute Direct.

Port pairings will be billed at a fixed amount in ExpressRoute Direct. There is no additional charge for standard circuits and premium circuits. Billing for egress is based on the zone where the peering location is located.

When you create the ExpressRoute Direct resource or after either or both links are enabled, ExpressRoute Direct's port pairs are billed 45 days later. Customers are given a 45-day grace period to complete the cross-connection process with their co-location provider.

Azure storage and other big data services can be accessed via ExpressRoute Direct for massive data ingestion. Circuits on ExpressRoute Direct 100 Gbps are available with a 40 Gbps or 100 Gbps circuit SKU. Port pairs are only 100 Gbps or 10 Gbps, and there is no limit on the number of virtual circuits created. Circuit sizes are 100 Gbps ExpressRoute Direct and 10 Gbps ExpressRoute Direct.

The following are benefits of ExpressRoute:

• ExpressRoute extends your on-premises networks over a private connection to Azure and Microsoft 365 clouds using a connectivity provider.
• Using Ethernet networks, the ExpressRoute platform allows you to establish IPVPNs between any location and any IP address. A co-location facility can provide your organization with virtual cross-connections.
• Connections over ExpressRoute are typically faster and more secure since they never cross the public Internet.

The following are benefits of ExpressRoute Direct:

• Ingestion of massive amounts of data into services like Azure Storage and Cosmos DB is possible.
• Banking, government, and retail are some industries that require physical isolation because of regulations.
• Distribution of circuits is based on business units.

For the Microsoft AZ-700 exam, keep in mind the significant differences between ExpressRoute via a network service provider and via ExpressRoute Direct, shown in Table 3.1.

Designing and Deploying Azure Cross-Region Connectivity between Multiple ExpressRoute Locations

Through ExpressRoute, on-premises infrastructure can be seamlessly connected to Azure services. Let's review some design decisions you need to make before deploying an ExpressRoute circuit.

ExpressRoute must be configured according to several prerequisites. The inevitable result of these can be unexpected costs and activities that hurt the project and other services.

ExpressRoute does not provide a physical connection; instead, it offers private connectivity over an established physical connection. A connectivity provider must first set up the physical connectivity. Connecting ExpressRoute to existing ExpressRoute partners can be done via various Microsoft network service providers.

The following are the seven important design areas to focus on for ExpressRoute. These are each discussed in the sections that follow.

• Select an ExpressRoute circuit SKU.
• Select a peering location.
• Estimate price based on the chosen SKU.
• Select the appropriate ExpressRoute circuit.
• Select a billing model.
• Design for high availability.
• Select a business continuity and disaster recovery design pattern.

Select a High Availability Design

To ensure high availability, you must maintain the redundancy of ExpressRoute circuits throughout your entire end-to-end network.

In a nutshell:

• Redundancy is required for the on-premises network.
• Cloud service providers should maintain redundancy.

Having redundancy implies at a minimum preventing single-point failures. Network devices can also be powered and cooled redundantly, which will further improve availability.

The rest of this section discusses the factors you should consider when designing for high availability. Let's start with a physical layer:

• If both the primary and secondary connections of ExpressRoute circuits are terminated on the same customer premises equipment (CPE), availability of the consumer's on-premises networks is compromised.
• If you configure the primary and secondary connections through the same CPE port, your partner is forced to compromise high availability on their network segment.

You can terminate the two connections over different subinterfaces, or you can merge them within your partner's network. (A “partner” offers a wide variety of network connectivity and datacenters. Partners may be able to provide fast, reliable, and private connections between your organization's on-premises infrastructure and Azure.) If ExpressRoute circuits are terminated in different locations, however, you can compromise the performance of the connection. A substantial difference in network latency between primary and secondary connections that terminate at different locations will result in suboptimal network performance if traffic is actively load-balanced. Figure 3.7 illustrates this situation.

Active/active connections are configured for ExpressRoute circuits on the Microsoft network. However, the redundant connections on an ExpressRoute circuit can be forced to operate in passive/active mode by a route advertisement. The standard techniques used to make one path preferred over another include more specific route advertising and prepending BGP autonomous system (AS) paths.

To improve high availability, Microsoft recommends operating both the connections of an ExpressRoute circuit in active/active mode. If the connections work in an active/active design pattern, the Microsoft network will load-balance the traffic across the links on a per-flow basis. If the active path fails for one of the connections of an ExpressRoute circuit, the active/passive mode may fail for the other connection as well. Most failures when switching over are due to a passive connection that is not managed correctly and is advertising stale routes. In addition, whenever an ExpressRoute circuit fails, running both primary and secondary connections in active/active mode results in about half of the flows being rerouted. The mean time to recovery will improve significantly by implementing a proactive approach.

Microsoft prefers using AS path prepending during a maintenance period (or in an unplanned event affecting a connection) to move traffic to the healthy connection. You will need to ensure that the traffic can route over the healthy path when path prepend is configured from Microsoft. The required route advertisements must be configured appropriately to avoid any service disruption.

A Microsoft peering service is designed to facilitate communication between public servers. The common practice, shown in Figure 3.8, is for on-premises Private Endpoints to be network address translated (NATed) with a public IP on the partner or customer network before being handed off for peering. How you use NAT influences how quickly you recover following the failure of one ExpressRoute connection, assuming that they are both used in active/active mode.

A common NAT pool should be your first choice for traffic distribution to split traffic between the primary and secondary connections. There is no requirement for common NAT pools to introduce a single point of failure to compromise high availability before breaking the traffic. After primary or secondary connections fail, the NAT pool is still accessible. Consequently, networks can reroute packets to recover faster from failures.

A NAT pool with independent NATs should be your second choice for traffic distribution between primary and secondary connections. In an ExpressRoute circuit, NAT is applied after traffic is divided between the primary and secondary connections. Traffic will return to the same edge device from which it exited. To meet the stateful requirements of NAT, each primary and secondary device is assigned an independent NAT pool.

ExpressRoute is unable to reach the appropriate NAT pool if the connection fails. All broken network flows need to be reestablished by either TCP or the application layer after the corresponding timeout. When Azure fails, it cannot reach on-premises servers until the ExpressRoute circuit's primary or secondary connections have been restored.

A key design consideration is that an ExpressRoute circuit cannot reach the on-premises server if a port for the IP address from the NAT pool is mapped to it via NAT choice 2. This will occur if a connection is lost during NAT choice 2 (a NAT pool with independent NATs).

Both the private and Microsoft peering connections under ExpressRoute support Bidirectional Forwarding Detection (BFD). You can activate BFD on ExpressRoute to speed the detection of link failures between MSEE devices and ExpressRoute routers—that is, between customer edge routers (CEs) and provider edge routers/switches (PEs). In this book, we refer to this as CE/PE. Your edge routing devices or partner edge routing devices can be configured using ExpressRoute.

BFD is supported over private peering in ExpressRoute. Microsoft Enterprise Edge and their BGP neighbors on the on-premises side experience reduced failure detection times due to BFD. BFD accelerates failure recovery by allowing for more time to recover from failures.

Pick a Business Continuity and Disaster Recovery Design Pattern

ExpressRoute is designed for high availability so that Microsoft resources have carrier-grade connectivity. There are no single points of failure within the ExpressRoute network. In the previous section, you read about design considerations to maximize ExpressRoute circuit availability. In this section, you will read about geo-redundant ExpressRoute circuits that can provide disaster recovery.

The use case for the business continuity and disaster recovery (BC/DR) design pattern is that cloud customers, Microsoft, network service providers, or any other cloud service provider can face opportunities or situations where a complete regional service degrades. A significant reason for this is natural disasters. So, if business relies on mission-critical applications or business continuity, a disaster recovery deployment pattern is a must.

Any Azure region can serve as a failover site, whether your business runs its mission-critical apps on-premises or in Azure. To ensure mission-critical operations are not interrupted during ExpressRoute connectivity, Microsoft recommends that disaster recovery plans include geo-redundant network connectivity.

Difficulties in managing various ExpressRoute circuits start with asymmetrical routing. You create parallel paths when they interconnect two or more networks simultaneously, using more than one connection. Without best architectural design practices, similar methods could lead to asymmetrical routing. In the case of stateful entities (NAT, firewalls), asymmetric routing could obstruct traffic flow. ExpressRoute peering does not involve stateful entities like NAT or firewalls, so you will not commonly encounter them. Therefore, asymmetrical routing with private peering over ExpressRoute won't obstruct traffic flow.

Although you may or may not have stateful entities, the network's performance may not be harmonious if they load balance traffic across geo-redundant parallel paths. These geo-redundant parallel paths can pass through the same metros or different metros.

For example, the Azure region Tokyo has Tokyo and Tokyo2. Architects could design redundant connections by connecting two Azure locations with two parallel paths in the identical metro area. In this design, when the on-premises applications fail, Microsoft's end-to-end latency remains roughly constant. It may, however, no longer be possible to connect both paths during a natural disaster such as a tsunami or earthquake.

With parallel paths, you need to use the Premium SKU for both circuits to choose a location outside the geopolitical region. Choose a second location in the same geopolitical region when you need redundancy from multiple metro areas. This configuration has an advantage because there is a lower chance of a natural disaster causing an outage of both links, resulting in increased end-to-end latency.

Use Case 1

The small business unit network connectivity with the BC/DR pattern is shown in Figure 3.9. This figure shows established geo-redundant ExpressRoute connectivity between an on-premises network and an Azure VNet within an Azure region. An active preferred path is indicated by the solid gray line, and the dotted line shows a standby path.

When designing ExpressRoute connectivity for disaster recovery, follow these best practices:

• Apply geo-redundant ExpressRoute circuits.
• Apply diverse service provider network(s) for different ExpressRoute circuits.
• Develop each of the ExpressRoute circuits with a high availability pattern.
• Terminate the different ExpressRoute circuits in different locations on the on-premises network.

Azure uses equal-cost multipath (ECMP) routing to load-balance all ExpressRoute routes if you advertise identical routes over all ExpressRoute paths. In contrast, for geo-redundant ExpressRoute circuit paths (especially latency), you must consider different network performance aspects. You may prefer the ExpressRoute circuit with minimal latency because it will perform more consistently during regular operation.

Azure can support multiple ExpressRoute circuits by using any of the following techniques:

• Perform more specific route advertising over a preferred ExpressRoute circuit(s) than over other ExpressRoute circuits.
• Set up a higher connection weight for the connection between the virtual network and the ExpressRoute circuit preferred by the virtual network.
• Advertise routes via the AS path prepended to ExpressRoute circuits that have a lower preference.

Figure 3.10 illustrates the influence of route advertisements on ExpressRoute path selection. This example of SMB network connectivity with the BC/DR pattern illustrates IP addresses advertised by the active preferred path through ExpressRoute 1 and as /24 through ExpressRoute 2 over a /24 address range via the standby path.

In Azure's normal state, /25 is more specific than /24, so traffic destined to 10.1.10.0/24 is sent over ExpressRoute 1. It is only through ExpressRoute 2 that the VNet might receive the route advertisement 10.1.11.0/24 in this failed state; therefore, the standby is utilized.

The next technique is setting up a higher connection weight for the connection between the virtual network and the ExpressRoute circuit preferred by the virtual network. Figure 3.11 shows ExpressRoute path selection using connection weight. Connectivity weight is set to 0 by default. This example shows ExpressRoute 1 with a weight of 100. VNets will prefer the connection with the highest weight among route prefixes advertised over more than one ExpressRoute circuit.

The standby circuit is used in this scenario when both connections of ExpressRoute 1 go down because the VNet would only see the 10.1.10.0/24 route advertisements via ExpressRoute 2 when ExpressRoute 1 fails.

In the final technique, routes are advertised with the AS path prepended to ExpressRoute circuits that have a lower preference. Figure 3.12 demonstrates how AS path prefixes can be used to influence ExpressRoute path selection. This figure illustrates the default behavior of eBGP in terms of route advertisements over ExpressRoute 1. Also, autonomous system numbers (ASNs) are prefixed to the AS path of routes advertised over ExpressRoute 2 by the on-premises network. As part of the eBGP route selection process, if VNet receives the same route over multiple ExpressRoute circuits, it prefers the route with the shortest AS path.

ExpressRoute 1 would continue to advertise 10.1.10.0/24 only if both connections failed. If both connections failed, the longer AS path would no longer be of any importance. When both connections fail, the standby circuit is used.

To avoid asymmetric flows, if you influence Azure to prioritize one ExpressRoute over another, the on-premises network must also prioritize the same ExpressRoute path for Azure-bound traffic. On-premises networks use the local preference value to decide which ExpressRoute circuit to prioritize over another. A local preference value is part of the iBGP protocol, and the BGP route with the highest local preference value is preferred.

Microsoft recommends that you actively manage your ExpressRoute circuits and periodically conduct disaster recovery drills and failover operations when using ExpressRoute circuits as standby.

Use Case 2

This use case is an example of enterprise distributed network connectivity with the BC/DR pattern. Let's consider the example shown in Figure 3.13. Using ExpressRoute circuits in two other peering locations, an organization has two on-premises locations connected to two different infrastructures as service deployments in two separate Azure regions.

In disaster recovery, all traffic flows through a local ExpressRoute circuit between Azure and on-premises networks in a constant state. All traffic flows between Azure and the remote ExpressRoute circuit handle the on-premises network if the local ExpressRoute circuit fails.

• Choice 1   The bolded lines in Figure 3.14 show traffic between on-premises site 1 networks and Azure VNet1. The light lines at the bottom of the figure indicate the paths for traffic flow between Azure VNet2 and on-premises site 2 networks. Solid straight lines represent the desired direction in a steady state. They show the related ExpressRoute circuit that carries steady-state traffic flow when the corresponding dashed lines are broken.

  

  Connectivity weight influences VNets to choose on-premises network-bound traffic to be routed through local peering locations instead of ExpressRoute. You must ensure symmetrical reverse traffic flow to complete the design. The ExpressRoute circuit is preferred when iBGP is used between on-premises BGP routers (when ExpressRoute circuits go out on-premises).

• Choice 2   This choice is depicted in Figure 3.15. In this diagram, bolded lines represent routes between Azure region 1 VNet1 and the on-premises networks. On-premises cloud networks and Azure region 2 VNet2 are shown with light lines to show the traffic flow paths. All traffic between the VNets and the on-premises locations passes primarily over Microsoft's backbone when doing business, represented by solid lines. The data will go through the interconnection between on-premises locations only in the failover state.

Changing the interconnection configuration between the on-premises and Azure locations is necessary to influence the choice of network route for traffic bound for Azure. However, it would help if you determined which interconnection link is preferable within their on-premises network based on their routing protocol. You can choose the wireless networks’ local or metric preferences for iBGP (Open Shortest Path First [OSPF] or Intermediate System to Intermediate System [IS-IS]), and a more specific route because the AS path prefix will influence the VNet path selection more precisely.

For the Microsoft AZ700 exam, keep in mind that:

• Microsoft peering allows organizations to consume a subset of supported services with route filters.
• Online services provided by Microsoft (Microsoft 365 and Azure PaaS) are connected through Microsoft peering.

Choosing an Appropriate ExpressRoute SKU and Tier

Creating a virtual network gateway is the first step toward connecting Azure virtual networking and on-premises networks via ExpressRoute. Virtual network gateways serve two primary purposes:

• Used when a network exchanges IP routes with another network
• For network traffic routing

This section will explore methods for choosing gateway types, gateway SKUs, and predicted performance based on SKU.

• Gateway Type   You need to specify several settings when creating a virtual network gateway. One of the required configurations, GatewayType, defines whether the gateway is a VPN or ExpressRoute. The two gateway types are as follows:
  • You can use VPN gateways to send encrypted traffic across the public Internet, also known as the Microsoft VPN gateway. VNet-to-VNet, site-to-site, and point-to-site connections all use a VPN gateway.
  • You can use the ExpressRoute gateway to send network traffic on a private connection. In the ExpressRoute configuration, this is referred to as the ExpressRoute gateway.

  The number of virtual network gateways per gateway type is limited to one per virtual network. For Example You can use one virtual network gateway that uses -GatewayType VPN and one that uses -GatewayType ExpressRoute.

• Gateway SKUs   You can create virtual network gateways by specifying the gateway SKUs they want to use. You can allocate more CPUs and network bandwidth to the gateway when choosing a higher gateway SKU. Therefore, throughput to a virtual network can be higher thanks to the gateway.

  Gateways for virtual networks powered by ExpressRoute are available in the following SKUs:

  • Standard
  • HighPerformance
  • UltraPerformance

  You can generally resize AzVirtualNetworkGateway PowerShell cmdlets to upgrade gateways to more powerful models using the Resize-AzVirtualNetworkGateway PowerShell cmdlet. You can upgrade from Standard and HighPerformance SKUs. It would be best to re-create a non-availability-zone gateway to upgrade it to the UltraPerformance SKU. The gateway does not have to be deleted and rebuilt to upgrade an AZ-enabled SKU. Keep in mind that replacing a gateway may result in downtime.

  Table 3.3 lists the characteristics and tiers of ExpressRoute SKUs.

  Gateway SKU	VPN gateway and ExpressRoute coexistence	Max. number of circuit connections
  ERGw1Az/Standard SKU	Yes	4
  ERGw2Az/HighPerformance SKU	Yes	8
  ErGw3Az/UltraPerformance SKU/	Yes	16

• Gateway Subnet   It is helpful to create gateway subnets before creating ExpressRoute gateways. Subnets for virtual network gateway hosts and services are in the gateway subnet. You can deploy virtual network gateways to gateway subnets that have been configured to use ExpressRoute gateway settings. Never add anything (such as more virtual machines) to the gateway subnet. GatewaySubnet must be the name of the gateway subnet where you want the virtual network gateway VMs and services to be deployed.

  For gateway VMs and gateway services, a gateway subnet is populated with IP addresses. You must specify how many IP addresses the gateway subnet contains when you create the subnet. Depending on the configuration, you may need more IP addresses.

  A configuration combining ExpressRoute with a VPN gateway, for example, requires a larger gateway subnet than most others. You should expect subnets to have enough IP addresses to accommodate any additional configurations. You can create a gateway subnet as small as /29; however, Microsoft recommends making at least a /27 gateway subnet (/27, /26, etc.) if enough address space is available. If 16 ExpressRoute circuits are interconnected, the gateway subnet should be /26. Microsoft recommends an a /64 IPv6 range. This range can accommodate almost any configuration, including, for example, a dual-stack gateway subnet.

  Azure availability zones (AZs) can also be used for ExpressRoute gateway deployments. Separating them in this manner protects on-premises network connectivity to Azure against zone-level failures on both a physical and a logical level.

  As of this writing, the following SKU types exist for ExpressRoute with zone-redundant gateways:

• ErGw1AZ
• ErGw2AZ
• ErGw3AZ

Other deployment options are available to fit your needs. You can deploy a virtual network gateway to a specific zone with the SKUs listed here. For ExpressRoute IPv6-based private peering, choose an AZ SKU that you can deploy in a dual-stack gateway subnet.

Designing and Deploying ExpressRoute Global Reach

You can access many Microsoft cloud services such as Azure and Microsoft 365 from private datacenters or corporate networks. ExpressRoute offers private and reliable connections to the Microsoft cloud for on-premises network users.

For example, consider the scenario as shown in Figure 3.16; you have a branch office at an on-premises site 1 and site 2. Both sites have high-speed connectivity to Azure services; however, the branches cannot communicate directly.

The ExpressRoute Global Reach service allows you to link ExpressRoute circuits to form private networks between on-premises cloud networks. As shown in Figure 3.17, adopting ExpressRoute Global Reach allows the on-premises network at site 2 (10.2.10.0/24) to directly exchange data with site 1 through ExpressRoute circuits in Microsoft's global network.

One of the principal use cases for ExpressRoute Global Reach is to connect an organization's branch offices worldwide. By leveraging ExpressRoute Global Reach, you can use existing service providers to connect their branches in their primary country. This is useful for remote offices located outside the region, where the current service provider does not provide services. You can use the ExpressRoute Global Reach link and connect their branch location office to the primary office.

To take advantage of this benefit, you must have the Premium SKU with the ExpressRoute Global Reach add-on.

At of this writing, ExpressRoute Global Reach is supported in the following countries: Australia, Canada, Denmark, France, Germany, Hong Kong SAR, India, Ireland, Japan, South Korea, Netherlands, New Zealand, Norway, Singapore, South Africa (Johannesburg only), Sweden, Switzerland, Taiwan, the United Kingdom, and the United States.

Let's now look at the steps for deploying ExpressRoute Global Reach.

Deploying ExpressRoute Global Reach

First, you should identify the ExpressRoute circuits that you wish to use. When two ExpressRoute circuits are located in supported countries/regions and created at different peering locations, you can enable ExpressRoute Global Reach between them.

$ckt_1 = Get-AzExpressRouteCircuit -Name "CloudConsumer_ER1_name" 
-ResourceGroupName "CloudConsumer_resource:group"
$ckt_2 = Get-AzExpressRouteCircuit -Name "CloudConsumer_ER2_name" 
-ResourceGroupName ""CloudConsumer_resource:group""

Add-AzExpressRouteCircuitConnectionConfig 
-Name 'CloudConsumer_connection_name' -ExpressRouteCircuit $ckt_1 -
PeerExpressRouteCircuitPeering $ckt_2.Peerings[0].Id 
-AddressPrefix '10.0.0.0/29'

"/subscriptions/{CloudConsumer_subscription_id}/resourceGroups
/{CloudConsumer_resource:group}/providers/Microsoft.Network
/expressRouteCircuits/{CloudConsumer_circuit_name}/peerings
/AzurePrivatePeering".

Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt_1

$ckt_2 = Get-AzExpressRouteCircuit -Name "CloudConsumer_circuit_2_name" -
ResourceGroupName "CloudConsumer_resource:group"
Add-AzExpressRouteCircuitAuthorization -ExpressRouteCircuit $ckt_2 -Name 
"Name_for_auth_key"

Add-AzExpressRouteCircuitConnectionConfig -Name 'CloudConsumer_connection_name' -
ExpressRouteCircuit $ckt_1 -PeerExpressRouteCircuitPeering 
"circuit_2_private_peering_id" -AddressPrefix '__.__.__.__/29' -AuthorizationKey 
'########-####-####-####-############'

Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt_1

$ckt_1 = Get-AzExpressRouteCircuit -Name "CloudConsumer_circuit_1_name" -
ResourceGroupName "CloudConsumer_resource:group"
Remove-AzExpressRouteCircuitConnectionConfig -Name "Cloud_connection_name" -
ExpressRouteCircuit $ckt_1
Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt_1

Designing and Deploying ExpressRoute FastPath

A virtual network's FastPath protocol improves communications between your on-premises and Azure virtual networks by accelerating the data path. Network routes and traffic are routed through ExpressRoute's virtual network gateway. Through FastPath, network traffic is sent directly to VMs within the virtual network, bypassing the gateway.

FastPath is available on all ExpressRoute circuits. It still requires a virtual network gateway to be built to exchange routes between virtual and on-premises networks. When you build the virtual network gateway, you need to specify the gateway SKU. As a result of selecting a higher gateway SKU, the gateway is allocated more CPUs and network bandwidth, increasing the virtual network's bandwidth.

Microsoft offers ExpressRoute virtual network gateway SKUs such as Standard, HighPerformance, UltraPerformance, ErGw1Az, ErGw2Az, and ErGw3Az. However, to use ExpressRoute FastPath, the virtual network gateway must be either UltraPerformance or ErGw3AZ.

When designing FastPath, keep in mind that it comes with these limitations and doesn't support the following subsystem integration:

• UDRs (user-defined routes) on the gateway subnet
• Basic load balancer
• Private Link

Typically, ExpressRoute FastPath is controlled and enabled by engineers if their virtual network gateways are UltraPerformance or ErGw3AZ. FastPath enhances on-premises cloud connectivity with Azure virtual networks and improves packet-per-second rates.

You can enable ExpressRoute networking directly to VMs connected to a local or peering virtual network through FastPath and virtual network peering, bypassing the ExpressRoute virtual network gateway in the data path.

The following is a deployment example that configures FastPath on a new connection:

$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG" 
$gw = Get-AzVirtualNetworkGateway -Name "SybexGateway" 
-ResourceGroupName "SybexRG" 
$connection = New-AzVirtualNetworkGatewayConnection 
-Name "SybexConnection" -ResourceGroupName "SybexRG" 
-ExpressRouteGatewayBypass -VirtualNetworkGateway1 $gw -PeerId $circuit.Id -
ConnectionType ExpressRoute 
-Location "SybexLocation"

The following is a deployment example that updates an existing connection to enable FastPath:

$connection = Get-AzVirtualNetworkGatewayConnection -Name "SybexConnection" 
-ResourceGroupName "SybexRG" 
$connection.ExpressRouteGatewayBypass = $True
Set-AzVirtualNetworkGatewayConnection 
-VirtualNetworkGatewayConnection $connection

The following is a deployment example that removes an ExpressRoute link from the subscription to the gateway. You use the Remove-AzVirtualNetworkGatewayConnection command to eliminate the link between the gateway and the circuit:

Remove-AzVirtualNetworkGatewayConnection "MyConnection" -ResourceGroupName "MyRG"

Evaluate Private Peering Only, Microsoft Peering Only, or Both

ExpressRoute circuits can be set up by peering in three Azure domains: Azure public, Azure private, and Microsoft. Each peering is configured identically on a pair of routers in an active/active or load-sharing configuration for high availability. Two Azure services are based on the IP address schemes: Azure public and Azure private. Figure 3.18 depicts two types of peering.

Cloud services, such as virtual machines and cloud services deployed within a virtual network, can be connected to Azure through the private peering domain. Azure recognizes the private peering domain as an extension of the core network. Azure VNets can be configured to be bidirectionally connected to the on-premises core network. Connecting to VMs and cloud services directly via their private IP addresses is enabled by Azure private peering. You can connect more than one virtual network to the private peering domain.

Microsoft 365 was created to be accessed securely and reliably via the Internet. Because of this, Microsoft recommends ExpressRoute for specific scenarios.

Connectivity to Microsoft online services (Microsoft 365 and Azure PaaS services) occurs through Microsoft peering. Microsoft enables bidirectional connectivity between the consumer's WAN and Microsoft's cloud services through the peering routing domain. You must connect to Microsoft cloud services only over public IP addresses owned by your organization or your network connectivity providers.

Table 3.4 shows the baseline for evaluating Azure public, Azure private, and Microsoft peerings.

ExpressRoute circuits can support one or more enabled routing domains. Support for more than one can be achieved by dividing them into multiple routing domains. You can combine all routing domains onto a single VPN to form a single domain.

Microsoft recommends that public and Microsoft peering links be configured to connect to the DMZ, whereas private peering is linked directly to the core network.

There are different peering types, and each requires its own BGP session. Pairs of BGP sessions ensure high availability. You must configure and manage the routes if they connect via layer 2 connectivity providers.

Network Performance Monitor (NPM) can be used to monitor ExpressRoute circuits’ availability, connectivity, and bandwidth utilization. Monitoring the health of Azure private peering and Microsoft peering is performed by NPM.

Setting Up Private Peering

By using Azure's private peering domain, Azure compute services, such as IaaS and PaaS, can be connected within a virtual network. Azure will peer with the organization's network over the private peering domain. Azure VNets and core networks can be connected bidirectionally. Connecting directly to VMs and cloud services is possible with peering.

ExpressRoute circuits can be configured with private peering. You can set up peerings in any order you wish. The peering setup for each network must be performed individually, however.

To connect to Microsoft cloud services through ExpressRoute, you must meet a few prerequisites:

• Azure must be valid and active.
• You must connect to the Microsoft cloud via an ExpressRoute connection or through a cloud exchange service provider.
• There must be redundancy of peering locations, disaster recovery, routing, NAT, quality of service (QoS), and network security requirements of each peering location.
• An active ExpressRoute circuit is needed.
• Ensure that both sides of the tunnel use the key if the security requirements call for a shared key/MD5 hash. The maximum number of characters is 25.

ExpressRoute workflows are configured to make provisioning and routing of circuits easy. This rest of this section outlines the steps to provision an ExpressRoute circuit from end to end, accomplishing the following tasks:

• Validate all prerequisites.
• Order connectivity or configure ExpressRoute Direct.
  • ExpressRoute partner model
  • ExpressRoute Direct model
• Create an ExpressRoute circuit.
  • ExpressRoute partner model
  • ExpressRoute Direct model
• Start consuming the ExpressRoute circuit.
• Set up routing domains.
  • Azure private peering
  • Microsoft peering
• Network service provider provisions connectivity.

Configure the ExpressRoute circuit. Check the provider's status to ensure that the connectivity provider fully provisions the circuit before continuing.

If your network connectivity provider offers managed Layer 3 services, you can request that their network connectivity provider enable Microsoft peering. However, if the network connectivity provider doesn't manage routing for you, you can take the following steps.

To configure private peering for the circuit, ensure that you have the following information before getting started:

• You need an IP address space not reserved for virtual networks, which comprises two subnets. The primary link will use one subnet, and the secondary link will use the other. You will assign the first usable IP address to your organization's router from each subnet since Microsoft uses the second functional IP address for its router. Here are the options:
  • IPv4: Two /30 subnets
  • IPv6: Two /126 subnets
  • Both: Two /30 subnets and two /126 subnets
• The VLAN ID on which the peering is to be established. You should use the same VLAN ID on both primary and secondary links.
• The AS number for peering can be a 2-byte or 4-byte AS number. You can do peering using your own AS numbers, except for AS numbers 65515 to 65520.
• The route from the on-premises edge router must be advertised through BGP to Azure.
• The MD5 hash may be used depending on security needs, but it is optional.

The following is a deployment example of configuring Azure private peering for the circuit:

Add-AzExpressRouteCircuitPeeringConfig -Name "AzurePvtPeering" 
-ExpressRouteCircuit $ckt -PeeringType AzurePrivatePeering 
-PeerASN 100 -PrimaryPeerAddressPrefix "10.0.0.0/30" 
-SecondaryPeerAddressPrefix "10.0.0.10/30" -VlanId 200
Set-AzExpressRouteCircuit -ExpressRouteCircuit $ckt

Next is a deployment example of configuring Azure private peering for the circuit with an MD5 hash:

Add-AzExpressRouteCircuitPeeringConfig -Name "AzurePvtPeering" 
-ExpressRouteCircuit $ckt -PeeringType AzurePrivatePeering 
-PeerASN 100 -PrimaryPeerAddressPrefix "10.0.0.0/30" 
-SecondaryPeerAddressPrefix "10.0.0.10/30" -VlanId 200 
-SharedKey "AX1YBZ2EC3D4"

And here is a deployment example for getting Azure private peering details:

$ckt = Get-AzExpressRouteCircuit -Name "ExpressRouteARMCircuit" 
-ResourceGroupName "SybexExpressRouteResourceGroup"
Get-AzExpressRouteCircuitPeeringConfig -Name "AzurePvtPeering" 
-ExpressRouteCircuit $ckt

Next, we'll explore a method for creating and getting the Azure private peering configuration for an ExpressRoute circuit through PowerShell.

Setting Up Microsoft Peering

The Internet is a secure and reliable way to access Microsoft 365. Consequently, ExpressRoute is recommended for this.

Azure PaaS services and Microsoft 365 services are connected by Microsoft peering. The Microsoft peering routing domain allows bidirectional connectivity between WANs and Microsoft cloud services. To access Microsoft cloud services, you must use public IP addresses owned or controlled by your organization or your connectivity providers. They should also adhere to all cloud service regulations.

ExpressRoute circuits can be configured with Microsoft peering. You can set up peerings in any order you wish. The peering setup for each network must be performed individually, however.

For your organization to connect to Microsoft cloud services through ExpressRoute, the following prerequisites must be met:

• Microsoft Azure must be valid and active.
• You must connect to the Microsoft cloud via an ExpressRoute connection or through a cloud exchange service provider.
• There must be redundancy of peering locations, disaster recovery, routing, NAT, QoS, and network security requirements of each peering location.
• An active ExpressRoute circuit is needed.
• Ensure that both sides of the tunnel use the key if the security requirements call for a shared key/MD5 hash. The maximum number of characters is 25.

Configure the ExpressRoute circuit. Check the provider's status to ensure that the connectivity provider fully provisions the circuit before continuing.

If the network connectivity provider offers managed Layer 3 services, you can request that your network connectivity provider enable Microsoft peering. However, follow the process described next if the network connectivity provider doesn't manage the route. Then you to have configure Microsoft peering for the circuit.

Ensure that you have the following prerequisites in place before getting started:

• You need a pair of subnets owned by your organization and registered within a cloud registry. There will be a primary link and a secondary link, and one subnet will be used for each connection. A router will be assigned the first usable IP address from each subnet and Microsoft's router will be assigned the second usable IP address. Here are three options for this pair of subnets:
  • IPv4: Two /30 subnets. All are public IPv4 addresses.
  • IPv6: Two /126 subnets. These are required publicly available IPv6 addresses.
  • IPv4 and IPv6: Two /30 subnets and two /126 subnets.
• You need the VLAN ID on which the peering is to be established. You should use the same VLAN ID on both primary and secondary links.
• The AS number for peering can be a 2-byte or 4-byte AS number.
• List all prefixes you plan to advertise over BGP. You can send a comma-separated list if you plan to send a set of prefixes. Those prefixes must be registered in an RIR (Regional Internet Registry) or IRR (Internet Routing Registry).
• Your advertising prefixes registered to peer AS numbers can be specified if they are not registered to peering AS numbers.
• For the Routing Registry Name, you may specify an RIR/IRR rather than an AS number or prefix.
• The MD5 hash may be used depending on your security needs, and it is optional.

Let's use Azure PowerShell to create and obtain the Microsoft peering configuration for an ExpressRoute circuit.

The following is a deployment example of configuring Microsoft peering for the circuit:

Add-AzExpressRouteCircuitPeeringConfig -Name "MSPeering" -ExpressRouteCircuit 
$ckt -PeeringType MicrosoftPeering -PeerASN 100 
-PeerAddressType IPv4 -PrimaryPeerAddressPrefix "124.0.0.0/30" 
-SecondaryPeerAddressPrefix "124.0.0.4/30" -VlanId 300 
-MicrosoftConfigAdvertisedPublicPrefixes "124.1.0.0/24" 
-MicrosoftConfigCustomerAsn 23 
-MicrosoftConfigRoutingRegistryName "ARIN"
Set-AzExpressRouteCircuit 
-ExpressRouteCircuit $ckt

Next is a deployment example for getting Microsoft peering for the circuit:

$ckt = Get-AzExpressRouteCircuit -Name "ExpressRouteARMCircuit" 
-ResourceGroupName ""ExpressRouteResourceGroup""
Get-AzExpressRouteCircuitPeeringConfig -Name ""MSPeering"" 
-ExpressRouteCircuit $ckt

Building and Configuring an ExpressRoute Gateway

In this section, you build and configure a gateway for a preexisting VNet. The steps in Exercise 3.1 walk you through the process.

Next, in Exercise 3.2, you create a virtual network gateway in the Azure portal.

Connect a Virtual Network to an ExpressRoute Circuit

This section covers connecting VNets to ExpressRoute circuits using PowerShell. A virtual network may belong to the same subscription or a different subscription as the ExpressRoute circuit it's connecting to.

Here are important deployment considerations:

• An ExpressRoute circuit can link up to 10 virtual networks. A standard ExpressRoute circuit allows only virtual networks in the same geographic region.
• VNets can be linked to an unlimited number of ExpressRoute circuits. Subscriptions for ExpressRoute circuits can be the same, different, or a mix of both.
• You can link virtual networks outside the ExpressRoute circuit's geopolitical region by enabling the ExpressRoute Premium add-on. Using the Premium add-on, you can also connect up to 10 virtual networks to the ExpressRoute circuit.
• To establish a connection from the ExpressRoute circuit to the ExpressRoute virtual network gateway, the number of IP address spaces advertised by the local or peer virtual networks must be equal to or less than 200 (as of this writing). Following the successful creation of a connection, you can add up to 1,000 IP address spaces to local or peer virtual networks.
• The previous sections discussed the prerequisites, routing requirements, and workflows that should be considered when constructing ExpressRoutes.
• Make sure you have your Azure private peering configured for circuits.
• Set up Azure private peering and establish BGP peering between the on-premises network and Microsoft for end-to-end connectivity.
• Create and provision virtual networks and virtual network gateways for every organization.

The following cmdlet connect a virtual network gateway to an ExpressRoute circuit running in the same subscription. You must be run this code after the virtual network gateway has been created and can be linked:

$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
$gw = Get-AzVirtualNetworkGateway 
-Name "SybexExpressRouteGw" 
-ResourceGroupName "SybexRG"
$connection = New-AzVirtualNetworkGatewayConnection 
-Name "ERConnection" -ResourceGroupName "SybexRG" 
-Location "UAE North" 
-VirtualNetworkGateway1 $gw -PeerId $circuit.Id 
-ConnectionType ExpressRoute

Consumers of Azure services can share an ExpressRoute circuit among multiple subscriptions. Figure 3.19 is a simple diagram showing how ExpressRoute circuits are shared across multiple subscriptions.

Each of the multitenant subscriptions within the large cloud represents different departments within the enterprise. Each department can own an ExpressRoute circuit. Tenants use their subscriptions to deploy their services but can connect back through an ExpressRoute circuit to the on-premises network. Another subscription within the company may use the ExpressRoute circuit.

The subscription owner will be charged for bandwidth and connectivity for ExpressRoute circuits. All virtual networks share the bandwidth.

At a high level, there are two roles within the Azure environment. The circuit owner is a power user who can access the ExpressRoute circuit resource and has full authority to create, review, add, and delete authorization. They can create authorization for circuit users to redeem.

The circuit user has the lowest authority to redeem and release connections. Circuit users own virtual network gateways that are not included in the same subscription as an ExpressRoute circuit. The circuit users may redeem authorizations (per virtual network).

Modifying and revoking authorizations are at the discretion of the circuit owner. Revoking an authorization results in all link connections being deleted from the subscription whose access was revoked.

For circuit users to connect their virtual network gateways to the ExpressRoute circuit, the circuit owner creates an authorization, which establishes an authorization key. Authorizations can be used only once. The following code shows how this is done:

$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
Add-AzExpressRouteCircuitAuthorization 
-ExpressRouteCircuit $circuit -Name "SybexAuthorization1"
Set-AzExpressRouteCircuit 
-ExpressRouteCircuit $circuit
$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
$auth1 = Get-AzExpressRouteCircuitAuthorization 
-ExpressRouteCircuit $circuit -Name "SybexAuthorization1"

The next example shows how a circuit owner reviews an authorization key:

$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
$authorizations = Get-AzExpressRouteCircuitAuthorization 
-ExpressRouteCircuit $circuit

Here is how you add an authorization key (the required role is circuit owner):

$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
Add-AzExpressRouteCircuitAuthorization 
-ExpressRouteCircuit $circuit -Name "SybexAuthorization2"
Set-AzExpressRouteCircuit -ExpressRouteCircuit $circuit
$circuit = Get-AzExpressRouteCircuit -Name "SybexCircuit" 
-ResourceGroupName "SybexRG"
$authorizations = Get-AzExpressRouteCircuitAuthorization 
-ExpressRouteCircuit $circuit

The following code shows deleting an authorization key (required role is circuit owner):

Remove-AzExpressRouteCircuitAuthorization 
-Name "SybexAuthorization2" 
-ExpressRouteCircuit $circuit
Set-AzExpressRouteCircuit -ExpressRouteCircuit $circuit

The circuit user must obtain the peer ID and the circuit owner's authorization key. Globally unique identifiers (GUIDs) are used for authorization keys.

You can use this command to check the peer ID:

Get-AzExpressRouteCircuit -Name ""SybexCircuit"" -ResourceGroupName ""SybexRG""

Circuit users can redeem link authorizations by running the following cmdlets; the minimum required role is circuit user.

$id = ""/subscriptions/********************************/resourceGroups/
ERCrossSubTestRG/providers/
Microsoft.Network/expressRouteCircuits/SybexCircuit"  
$gw = Get-AzVirtualNetworkGateway -Name "ExpressRouteGw" 
-ResourceGroupName "SybexRG"
$connection = New-AzVirtualNetworkGatewayConnection 
-Name "SybexER_01" -ResourceGroupName "RemoteResourceGroup" 
-Location "UAE North" -VirtualNetworkGateway1 $gw 
-PeerId $id -ConnectionType ExpressRoute -AuthorizationKey 
"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"

Consumer virtual networks can be connected to multiple ExpressRoute circuits. Several ExpressRoute circuits can provide the same prefix. As shown in the following code, you can modify a connection's RoutingWeight to specify which connection to use for traffic destined for this prefix. That connection receives traffic.

$connection = Get-AzVirtualNetworkGatewayConnection -Name 
"SybexVirtualNetworkConnection" 
-ResourceGroupName "SybexRG"
$connection.RoutingWeight = 100
Set-AzVirtualNetworkGatewayConnection 
-VirtualNetworkGatewayConnection $connection

Recommend a Route Advertisement Configuration

Organizations must set up and manage the ExpressRoute to connect to Microsoft cloud services. Some connectivity providers offer managed services for setting up and managing the routing. Find out if your organization's network connectivity providers supply this service. Otherwise, follow these guidelines:

• You must reserve a few blocks of IP addresses to configure routing between your networks and MSEE routers.
• High availability configurations using Microsoft do not support router redundancy protocols—for example, Hot Standby Router Protocol (HSRP) and Virtual Router Redundancy Protocol (VRRP). Microsoft's solution uses a pair of redundant BGP sessions for high availability.
• Public or private IPv4 addresses can be used for private peering by cloud service consumers. These addresses aren't advertised online. Private peering allows Microsoft to isolate traffic end to end, so overlapping addresses with other customers is impossible.
• Customers of Microsoft's cloud services can connect to the Microsoft peering path. Services on the list include Microsoft 365 services, such as Exchange Online, SharePoint Online, Skype for Business, and Microsoft Teams. Connectivity between Microsoft peering sites is bidirectional. For traffic to reach Microsoft cloud services, valid public IPv4 addresses must be used.
• A session between the MSEEs and your routers is established via eBGP. Routing exchanges are done using BGP. There is no requirement for BGP authentication. MD5 hashes can be configured if necessary.
• Microsoft uses AS 12076 for Azure public, Azure private, and Azure peering. From 65515 to 65520, Microsoft reserves ASNs for internal use. AS numbers in both 16- and 32-bit formats are supported.
• The Azure private peering allows for advertising up to 4,000 IPv4 addresses and 100 IPv6 addresses. The ExpressRoute Premium add-on allows you to increase this number to 10,000 IPv4 prefixes. During a BGP session for Azure public and Microsoft peering, Microsoft will accept up to 200 prefixes.
• Transit routers cannot be configured with ExpressRoute. For transit routing services, you must rely on the providers of their network connectivity.
• Azure private peering sessions can only use default routes. All traffic from the associated virtual networks will be routed to your network in such a case. Default routes into private peering will result in the Internet path from Azure being blocked. Services hosted in Azure must rely on corporate edges to route traffic to and from the Internet.
• The following items must be in place before you can connect to Azure services and infrastructure services:
  • Peering between Azure public endpoints is enabled.
  • Each subnet requiring Internet connectivity can be connected to the cloud by using user-defined routing.
  • Peering paths with Microsoft with routes tagged with appropriate community values will be advertised publicly by Microsoft. Microsoft, however, will not honor any community values tagged to routes advertised to Microsoft.

Configure Encryption over ExpressRoute

ExpressRoute ensures the privacy and integrity of data traversing between Microsoft's network and the cloud as part of its encryption capabilities.

Data is encrypted at the media access control (MAC) level or Network Layer 2 by MACsec, an IEEE standard. The MACsec protocol can encrypt the physical links between your networks and Microsoft's network devices when connecting to Microsoft via ExpressRoute Direct. By default, MACsec is not enabled on ExpressRoute Direct ports. Azure Key Vault lets cloud users store their MACsec keys for encryption in the cloud.

The MACsec protocol encrypts all traffic on a physical link with a key that belongs to one entity (i.e., the customer). This protocol is only available with ExpressRoute Direct. MACsec encrypts all network control traffic, BGP data, and customer data traffic once enabled.

As an alternative to preshared keys, Microsoft supports MACsec configuration. This means you need to update your keys on your devices and Microsoft's via an API. You will lose connectivity when a critical mismatch between the two sides occurs. Microsoft strongly recommends scheduling a maintenance window for the configuration change. After you switch your network traffic to another link, you can update the ExpressRoute Direct configuration on one link to reduce downtime. When MACsec is configured and there is a key mismatch, your end user loses connectivity to Microsoft. Microsoft won't fall back to unencrypted connections, because that would expose your data.

MACsec encryption and decryption happen in hardware on the routers Microsoft uses, which has no impact on performance. Nonetheless, you should check with your network device vendor to see whether MACsec has any performance implications.

As of this writing, Microsoft supports the following encryption ciphers:

• GCM-AES-128
• GCM-AES-256
• GCM-AES-XPN-128
• GCM-AES-XPN-256

The IPsec standard is an Internet Engineering Task Force (IETF) standard. At the IP level, IPsec encrypts data at Network Layer 3. You can use the IPsec protocol to encrypt connections between your on-premises and VNets on Azure.

MACsec secures your physical connections to Microsoft. Your physical network and your virtual network on Azure are connected and secured using IPsec. You can enable IPsec independently.

Deploy Bidirectional Forwarding Detection

Bidirectional Forwarding Detection (BFD) is supported both by ExpressRoute over private peering and by Microsoft peering. By configuring BFD over ExpressRoute, you can speed up the link failure detection between MSEE devices and routers that connect to your ExpressRoute circuits (CE/PE). If you have chosen a managed Layer 3 connection service, you can configure ExpressRoute over your edge routing devices or your partner's edge routing devices.

You can enable ExpressRoute circuits through Layer 2 connections or managed Layer 3 connections. If there is more than one Layer 2 device in the ExpressRoute path, the overlying BGP session is responsible for detecting link failures.

MSEE devices are configured to keep alive for 60 seconds and to hold for 180 seconds in BGP. When a link fails, it may take up to 3 minutes for traffic to be switched to an alternative link.

You can configure a lower BGP hold-and-keep-alive time on your edge peering device. BGP sessions will be established based on the lower value when the BGP timers don't match the peer devices. As little as 3 seconds can be specified as the BGP keep-alive period and as little as 10 seconds as the hold time. BGP is a process-intensive protocol, so Microsoft does not recommend setting a very aggressive timer.

BFD is configured by default on all newly created ExpressRoute private peering interfaces on MSMEs. So, you only need to configure BFD on your primary and secondary devices to enable BFD. Two steps are involved in configuring BFD: configuring the BFD on the interface and linking it to the BGP session.

BFD peers determine their transmission rate based on which of them is slower. A 300-millisecond transmission/receive interval is set for MSEE's BFDs. Depending on the scenario, the interval may be set as high as 750 milliseconds. These intervals can be made longer by configuring a higher value, but they cannot be made shorter.

Diagnose and Resolve ExpressRoute Connection Issues

In this section you learn how to check and troubleshoot Microsoft Azure ExpressRoute connectivity. A connection provider typically facilitates a private connection that extends an on-premises network into the Microsoft cloud with ExpressRoute. ExpressRoute connections usually consist of three different network zones:

• An organization's network
• Network service provider
• Microsoft Azure datacenter

The following list shows various checkpoints you need to verify and validate:

• Verify CircuitProvisioningState is enabled and not disabled.
• Verify ServiceProviderProvisioningState is provisioned.
• Validate Peering Configuration.
• Validate ARP.
• Validate BGP and routes on the MSEE.
• Confirm the traffic flow.

Follow the steps in Exercise 3.3 to troubleshoot Azure ExpressRoute connectivity.

Summary

In this chapter, you learned about ExpressRoute and how to design an Azure consumer network with ExpressRoute and perform required ExpressRoute configuration. You also learned how to decide on the appropriate SKU based on business requirements. We discussed ExpressRoute Global Reach and ExpressRoute FastPath, and you gained a detailed understanding about ExpressRoute peering, private peering, and Microsoft peering.

You should now be able to design, deploy, and manage Azure ExpressRoute for your organization to connect between on-premises and Azure networks. You should also be able to design and implement a hybrid connectivity solution that will address the short- and long-term goals of your organization's global enterprise IT footprint.

Exam Essentials

• Understand the difference between a provider and a direct model (ExpressRoute Direct).   Using ExpressRoute Direct, you can connect directly to Microsoft's global network at peering locations strategically distributed worldwide. With ExpressRoute Direct, you can have dual 10 Gbps or 100 Gbps connections and active/active connectivity at scale. ExpressRoute uses a service provider to provide fast connectivity to existing infrastructure and enable onboarding. Ethernet and MPLS providers are supported. ExpressRoute uses service provider–supported SKUs of 50 Mbps up to 10 Gbps.
• Be able to design and implement Azure cross-region connectivity between multiple ExpressRoute locations.   Based on specific organizational requirements, ExpressRoute can be designed and implemented in various ways.

  The following services can be accessed via ExpressRoute connections:

  • Microsoft Azure services
  • Microsoft 365 services

  These provide support for all regions within a geographic region, with access to all regions within that region from any of the peering locations.

• Know the appropriate ExpressRoute SKU and tier.   Three different circuit solutions are offered by ExpressRoute: Local, Standard, and Premium. These SKU types have different pricing structures from which you can choose: metered (pay by the gigabyte used) or an unlimited version.
• Understand how to design and deploy ExpressRoute.   ExpressRoute enables you to seamlessly connect on-premises networks to Azure services. Before deploying an ExpressRoute circuit, you must choose ExpressRoute circuit SKUs, select a peering location, choose the right ExpressRoute circuit, and decide on a billing model.
• Be able to design and deploy ExpressRoute FastPath.   ExpressRoute virtual network gateways are used to route network traffic and exchange network routes. FastPath is designed to improve the performance of the data path between your on-premises network and your virtual network. With FastPath enabled, network traffic is sent directly to VMs within the virtual network, bypassing the gateway.
• Understand the peering associated with an ExpressRoute circuit.   Two peering options are associated with an ExpressRoute circuit: Azure private and Microsoft. For high availability, every peering is configured identically on two routers (active/active or load-sharing configurations). Azure services are divided into Azure public and Azure private, based on the IP address scheme used.
• Know how to create and configure an ExpressRoute gateway.   You must first create a virtual network gateway in order to connect your Azure virtual network with your on-premises network through ExpressRoute. A virtual network gateway exchanges IP routes between networks and routes traffic. FastPath requires an UltraPerformance or ErGw3AZ virtual network gateway. If you plan to use FastPath with ExpressRoute-based IPv6-based private peering, ensure that the SKU is ErGw3AZ. Remember that FastPath is only available on circuits using ExpressRoute Direct.
• Understand how to connect a virtual network to an ExpressRoute circuit.   Using the Azure portal, you can create a connection to link a virtual network to an Azure ExpressRoute circuit. The virtual networks you connect to your Azure ExpressRoute circuit can either be in the same subscription or part of another subscription. A standard ExpressRoute circuit can connect up to 10 virtual networks. Standard ExpressRoute circuits require all virtual networks to be in the same geopolitical region. The maximum number of ExpressRoute circuits in a single VNet is 16. ExpressRoute circuits can be in the same subscription, different subscriptions, or both.
• Know how to configure a route advertisement.   To configure routing between your network and the MSEE's routers, you need to reserve a few blocks of IP addresses. To configure peerings, you can use private IP addresses or public IP addresses. Azure must not use overlapping address ranges for configuring routes and creating virtual networks. You must use public IP addresses to set up the BGP sessions. Through Routing Internet Registries and Internet Routing Registries, Microsoft must be able to verify ownership of IP addresses.
• Know how to configure encryption over ExpressRoute.   Through an Azure ExpressRoute circuit, you can establish an IPsec/IKE VPN connection from your on-premises network to Azure using Azure Virtual WAN. Using this technique, on-premises and Azure virtual networks can be connected securely over ExpressRoute without going over the public Internet or using public IP addresses. Setting up the connection is straightforward:
  • Connect ExpressRoute to a private peering and ExpressRoute circuit.
  • Set up a VPN connection.

  This configuration includes ExpressRoute and VPN paths for routing between on-premises and Azure networks.

• Know how to implement Bidirectional Forwarding Detection.   Bidirectional Forwarding Detection (BFD) is supported over private and Microsoft peering in ExpressRoute. You can use BFD over ExpressRoute to increase the speed of link failure detection between MSEE and routers that are configured for your ExpressRoute circuit (CE/PE). ExpressRoute can be configured on your edge routing devices or your Partner Edge routing devices (if you choose managed Layer 3 connection service).
• Understand how to diagnose and resolve ExpressRoute connection issues.   ExpressRoute extends an on-premises network into the Microsoft cloud over a private connection that a connectivity provider commonly facilitates. ExpressRoute connectivity traditionally involves three distinct network zones: customer network, provider network, and Microsoft datacenter. Logical steps in troubleshooting an ExpressRoute circuit start with verifying circuit provisioning and state and moving on to validating peering configuration, validating ARP, validating BGP and routes on the MSEE, confirming the traffic flow, performing a private peering connectivity test, and verifying availability of the virtual network gateway.

Review Questions