Chapter 11. Troubleshooting IPv4 Routing Protocols
This chapter covers the following exam topics:
2.0 Routing Technologies
2.4 Configure, verify, and troubleshoot single area and multiarea OSPFv2 for IPv4 (excluding authentication, filtering, manual summarization, redistribution, stub, virtual-link, and LSAs)
2.6 Configure, verify, and troubleshoot EIGRP for IPv4 (excluding authentication, filtering, manual summarization, redistribution, stub)
To troubleshoot a possible IPv4 routing protocol problem, first focus on interfaces, and then on neighbors. The routing protocol configuration identifies the interfaces on which the router should use the routing protocol. After identifying those interfaces, a network engineer can look at the neighbors each router finds on each interface, searching for neighbors that should exist but do not.
This chapter focuses on issues related to these two main branches of logic: on which interfaces should a router enable the routing protocol, and which neighbor relationships should each router create. This chapter relies on the configuration discussed in Chapter 8 for OSPFv2 and in Chapter 10 for EIGRP. This chapter’s troubleshooting discussions emphasize how to find incorrect configuration problems by using only show and debug commands.
This chapter first briefly introduces a few broad concepts related to troubleshooting problems with routing protocols. The next major section examines problems related to which interfaces on which a router enables the routing protocol, with the final major section focusing of routing protocol neighbor relationships. Note that the entire chapter moves back and forth between discussing both Enhanced Interior Gateway Routing Protocol (EIGRP) and Open Shortest Path First Version 2 (OSPFv2).
“Do I Know This Already?” Quiz
The troubleshooting chapters of this book pull in concepts from many other chapters, including some chapters in CCENT/CCNA ICND1 100-105 Official Cert Guide. They also show you how to approach some of the more challenging questions on the ICND2 and CCNA R&S exams. Therefore, it is useful to read these chapters regardless of your current knowledge level. For these reasons, the troubleshooting chapters do not include a “Do I Know This Already?” quiz. However, if you feel particularly confident about troubleshooting OSPFv2 and EIGRP, feel free to move to the “Chapter Review” section near the end of this chapter to bypass the majority of the chapter.
Foundation Topics
Perspectives on Troubleshooting Routing Protocol Problems
Because a routing protocol’s job is to fill a router’s routing table with the currently best routes, it makes sense that troubleshooting potential problems with routing protocols could begin with the IP routing table. Given basic information about an internetwork, including the routers, their IP addresses and masks, and the routing protocol, you could calculate the subnet numbers that should be in the router’s routing table and list the likely next-hop routers for each route. For example, Figure 11-1 shows an internetwork with six subnets. Router R1’s routing table should list all six subnets, with three connected routes, two routes learned from R2 (172.16.4.0/24 and 172.16.5.0/24), and one route learned from R3 (172.16.6.0/24).
So, one possible troubleshooting process is to analyze the internetwork, look at the routing table, and look for missing routes. If one or more expected routes are missing, the next step would be to determine whether that router has learned any routes from the expected next-hop (neighbor) router. The next steps to isolate the problem differ greatly if a router is having problems forming a neighbor relationship with another router, versus having a working neighbor relationship but not being able to learn all routes.
For example, suppose that R1 in Figure 11-1 has learned a route for subnet 172.16.4.0/24 in Figure 11-1 but not for subnet 172.16.5.0/24. In this case, it is clear that R1 has a working neighbor relationship with R2. In these cases, the root cause of this problem might still be related to the routing protocol, or it might not. For example, the problem may be that R2’s lower LAN interface is down. However, if R1 did not have a route for both 172.16.4.0/24 and 172.16.5.0/24, R1’s neighbor relationship with R2 could be the problem.
Troubleshooting routing protocol problems in real internetworks can be very complex—much more complex than even the most difficult CCNA R&S exam questions. Defining a generic troubleshooting process with which to attack both simple and complex routing protocol problems would require a lot of space and be counterproductive for preparing for the CCNA R&S exams. This chapter instead offers a straightforward process for attacking routing protocol problems—specifically, problems similar to the depth and complexity of the CCNA R&S exams.
If an exam question appears to be related to a problem with a routing protocol, you can quickly identify some common configuration errors with the following process—even if the question does not list the configuration. The process has three main tasks:
Step 1. Examine the internetwork design to determine on which interfaces the routing protocol should be enabled and which routers are expected to become neighbors.
Step 2. Verify whether the routing protocol is enabled on each interface (as per Step 1). If it isn’t, determine the root cause and fix the problem.
Step 3. Verify that each router has formed all expected neighbor relationships. If it hasn’t, find the root cause and fix the problem.
For instance, as noted with asterisks in Figure 11-2, each router should enable the routing protocol on each of the interfaces shown in the figure. Also, routing protocol neighbor relationships should form between R1 and R2, and R1 and R3, but not between R2 and R3.
While the concepts outlined in Figure 11-2 should be somewhat obvious by now, this chapter discusses how some of the most common configuration mistakes can impact the interfaces used by a routing protocol and whether a routing protocol creates neighbor relationships.
Interfaces Enabled with a Routing Protocol
This section examines the second major troubleshooting step outlined in the previous section of the chapter: how to verify the interfaces on which the routing protocol has been enabled. Both EIGRP and OSPF configuration enable the routing protocol on an interface by using the network router subcommand. For any interfaces matched by the network commands, the routing protocol tries the following two actions:
Attempt to find potential neighbors on the subnet connected to the interface
Advertise the subnet connected to that interface
At the same time, the passive-interface router subcommand can be configured so that the router does not attempt to find neighbors on the interface (the first action just listed), but still advertises the connected subnet (the second action).
Three show commands are all that is needed to know exactly which interfaces have been enabled with EIGRP and which interfaces are passive. In particular, the show ip eigrp interfaces command lists all EIGRP-enabled interfaces that are not passive interfaces. The show ip protocols command essentially lists the contents of the configured network commands for each routing protocol and a separate list of the passive interfaces. Comparing these two commands identifies all EIGRP-enabled interfaces and those that are passive.
For OSPF, the command works slightly differently, with the show ip ospf interface brief command listing all OSPF-enabled interfaces (including passive interfaces). Using this command, along with the list of passive interfaces listed by the show ip protocols command, again identifies all fully enabled OSPF interfaces as well as all passive interfaces.
Table 11-1 summarizes the commands that identify the interfaces on which OSPFv2 and EIGRP are enabled for easier reference.
Note
All the commands in Table 11-1 list the interfaces regardless of interface status, in effect telling you the results of the network and passive-interface configuration commands.
So, for the major troubleshooting step covered in this section, the task is to use the commands in Table 11-1 and analyze the output. First, an EIGRP example will be shown, followed by an OSPF example.
EIGRP Interface Troubleshooting
This section shows a few examples of the commands in the context of Figure 11-3, which is used in all the examples in this chapter.
This example includes four routers, with the following scenario in this case:
R1 and R2 are configured correctly on both LAN interfaces.
R3 is mistakenly not enabled with EIGRP on its G0/1 interface.
R4 meant to use a passive-interface G0/1 command because no other routers are off R4’s G0/1 LAN. However, R4 has instead configured a passive-interface G0/0 command.
This example begins by showing the working details between Routers R1 and R2, and then moves on to discuss the issues related to R3 and R4.
Examining Working EIGRP Interfaces
Examples 11-1 and 11-2 list configuration and show commands for R1 and R2, respectively. Each lists the related configuration, the show ip eigrp interfaces and show ip protocols command, and the EIGRP-learned routes on each router.
Example 11-1 EIGRP Interfaces Problem: R1 Commands
R1# show running-config
! only pertinent lines shown
router eigrp 99
network 10.0.0.0
!
R1# show ip eigrp interfaces
EIGRP-IPv4 Interfaces for AS(99)
Xmit Queue PeerQ Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable Un/Reliable SRTT Un/Reliable Flow Timer Routes
Gi0/0 3 0/0 0/0 2 0/0 50 0
Gi0/1 0 0/0 0/0 0 0/0 0 0
R1# show ip protocols
*** IP Routing is NSF aware ***
Routing Protocol is "eigrp 99"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP-IPv4 Protocol for AS(99)
Metric weight K1=1, K2=0, K3=1, K4=0, K5=0
NSF-aware route hold timer is 240
Router-ID: 1.1.1.1
Topology : 0 (base)
Active Timer: 3 min
Distance: internal 90 external 170
Maximum path: 4
Maximum hopcount 100
Maximum metric variance 1
Automatic Summarization: disabled
Maximum path: 4
Routing for Networks:
10.0.0.0
Routing Information Sources:
Gateway Distance Last Update
10.1.1.2 90 09:55:51
10.1.1.3 90 00:02:00
Distance: internal 90 external 170
R1# show ip route eigrp
! Legend omitted for brevity
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
D 10.1.22.0/24 [90/30720] via 10.1.1.2, 00:00:40, GigabitEthernet0/0
Example 11-2 EIGRP Interfaces Problem: R2 Commands
R2# show running-config
! only pertinent lines shown
router eigrp 99
network 10.1.0.0 0.0.255.255
R2# show ip eigrp interfaces
EIGRP-IPv4 Interfaces for AS(99)
Xmit Queue PeerQ Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable Un/Reliable SRTT Un/Reliable Flow Timer Routes
Gi0/0 2 0/0 0/0 1 0/1 50 0
Gi0/1 0 0/0 0/0 0 0/0 0 0
R2# show ip protocols
*** IP Routing is NSF aware ***
Routing Protocol is "eigrp 99"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP-IPv4 Protocol for AS(99)
Metric weight K1=1, K2=0, K3=1, K4=0, K5=0
NSF-aware route hold timer is 240
Router-ID: 2.2.2.2
Topology : 0 (base)
Active Timer: 3 min
Distance: internal 90 external 170
Maximum path: 4
Maximum hopcount 100
Maximum metric variance 1
Automatic Summarization: disabled
Maximum path: 4
Routing for Networks:
10.1.0.0/16
Routing Information Sources:
Gateway Distance Last Update
10.1.1.3 90 00:02:30
10.1.1.1 90 09:56:20
Distance: internal 90 external 170
R2# show ip route eigrp
! Legend omitted for brevity
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
D 10.1.11.0/24 [90/30720] via 10.1.1.1, 00:03:25, GigabitEthernet0/0
The show ip eigrp interfaces command output on both R1 and R2 shows how both R1 and R2 have configured EIGRP using process ID 99, and that EIGRP has been enabled on both G0/0 and G0/1 on both these routers. This command lists only interfaces on which EIGRP has been enabled, excluding passive interfaces.
The highlighted parts of the show ip protocols command output on each router are particularly interesting. These sections show the parameters of the configured network commands. The show ip protocols command lists a separate line under the header “Routing for Networks,” one for each configured network command. Example 11-1’s output suggests R1 has a network 10.0.0.0 configuration command (as shown at the beginning of the example), and Example 11-2’s “10.1.0.0/16” suggests R2 has a network 10.1.0.0 0.0.255.255 command.
Examining the Problems with EIGRP Interfaces
The next few pages now look at the problems caused by the configuration on Routers R3 and R4.
First, Example 11-2 gives brief insight into the current problem caused by R3. The end of R2’s show ip protocols command (Example 11-2) lists two routing information sources: 10.1.1.1 (R1) and 10.1.1.3 (R3). However, R2 has learned only one EIGRP route (10.1.11.0/24), as shown in the show ip route eigrp command output. When working properly, R2 should learn three EIGRP routes—one for each of the other LAN subnets shown in Figure 11-3.
Example 11-3 shows the root cause on R3. First, R3’s show ip eigrp interfaces command lists G0/0, but not G0/1, so a problem might exist with how EIGRP has been configured on G0/1. The configuration at the top of the example lists the root cause: an incorrect network command, which does not enable EIGRP on R3’s G0/1 interface.
Example 11-3 EIGRP Problems on R3
R3# show running-config
! lines omitted for brevity
router eigrp 99
network 10.1.1.3 0.0.0.0
network 10.1.13.3 0.0.0.0
auto-summary
R3# show ip eigrp interfaces
EIGRP-IPv4 Interfaces for AS(99)
Xmit Queue PeerQ Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable Un/Reliable SRTT Un/Reliable Flow Timer Routes
Gi0/0 2 0/0 0/0 1 0/1 50 0
R3# show ip protocols
*** IP Routing is NSF aware ***
Routing Protocol is "eigrp 99"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP-IPv4 Protocol for AS(99)
Metric weight K1=1, K2=0, K3=1, K4=0, K5=0
NSF-aware route hold timer is 240
Router-ID: 3.3.3.3
Topology : 0 (base)
Active Timer: 3 min
Distance: internal 90 external 170
Maximum path: 4
Maximum hopcount 100
Maximum metric variance 1
Automatic Summarization: disabled
Maximum path: 4
Routing for Networks:
10.1.1.3/32
10.1.13.3/32
Routing Information Sources:
Gateway Distance Last Update
10.1.1.2 90 00:05:14
10.1.1.1 90 00:05:14
Distance: internal 90 external 170
The root cause of R3’s problem is that R3 has a network 10.1.13.3 0.0.0.0 configuration command, which does not match R3’s 10.1.33.3 G0/1 IP address. If the configuration was not available in the exam question, the show ip protocols command could be used to essentially see the same configuration details. In this case, the show ip protocols command on R3 lists the text “10.1.13.3/32” as a reference to the contents of the incorrect network command’s parameters, with “/32” translating to a wildcard mask of 32 binary 0s, or decimal 0.0.0.0.
R3’s incorrect configuration means that two actions do not happen on R3’s G0/1 interface. First, R3 does not try to find neighbors on its G0/1 interface, which is not a big deal in this case. However, R3 also does not advertise subnet 10.1.33.0/24, the connected subnet off R3’s G0/1 interface.
Moving on to R4’s problem, Example 11-4 shows why R1 and R2 do not learn R4’s 10.1.44.0/24 subnet. In this case, on R4, the engineer could have correctly used a passive-interface gigabitethernet0/1 router subcommand because no other routers should exist off R4’s G0/1 interface. However, the engineer mistakenly made R4’s G0/0 interface passive.
Example 11-4 EIGRP Problems on R4
R4# show running-config
! lines omitted for brevity
router eigrp 99
passive-interface GigabitEthernet0/0
network 10.0.0.0
auto-summary
R4# show ip eigrp interfaces
EIGRP-IPv4 Interfaces for AS(99)
Xmit Queue PeerQ Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable Un/Reliable SRTT Un/Reliable Flow Timer Routes
Gi0/1 0 0/0 0/0 0 0/1 0 0
R4# show ip protocols | begin Routing for Networks
Routing for Networks:
10.0.0.0
Passive Interface(s):
GigabitEthernet0/0
Routing Information Sources:
Gateway Distance Last Update
Distance: internal 90 external 170
Note
The last command on the example, show ip protocols | begin Routing for Networks, lists the command output, but starting with the line with the literal case-sensitive string Routing for Networks. You can use this feature with any output from a command when you prefer to view only later lines of the command’s output.
To find this mistake without the configuration, Example 11-4 lists two useful commands. R4’s show ip eigrp interfaces command omits the (G0/0) passive interface, which means that R4 will not attempt to find EIGRP neighbors off that interface. Also, the highlighted part of R4’s show ip protocols command output lists G0/0 as a passive interface, which again means that R4 does not even attempt to become neighbors with others off its G0/0 interface.
OSPF Interface Troubleshooting
OSPF has the same basic requirements as EIGRP for interfaces, with a few exceptions. First, EIGRP routers need to use the same autonomous system number (ASN) as their neighboring routers, as configured in the router eigrp asn global configuration command. OSPF routers can use any process ID on the router ospf process-id command, with no need to match their neighbors. Second, OSPF requires that the interfaces connected to the same subnet be assigned to the same OSPF area, whereas EIGRP has no concept of areas.
Example 11-5 shows a mostly working OSPF internetwork, again based on Figure 11-3. The problem in this case relates to the area design, as shown in Figure 11-4, the revised version of Figure 11-3. All subnets should be placed into area 0. However, the engineer made a configuration mistake on R2, putting both its interfaces into area 1. As a result, R2’s G0/0 interface breaks the OSPF design rule of being in the same subnet as R1, R3, and R4, but not being in the same OSPF area.
Example 11-5 begins to break down the problem by looking at the status of OSPF on the router interfaces of R1 and R2, using the show ip ospf interface brief command.
Example 11-5 show ip interface brief on R1 and R2
R1> show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi0/1 1 0 10.1.11.1/24 1 DR 0/0
Gi0/0 1 0 10.1.1.1/24 1 DROTH 2/2
! The following command is from R2
R2> show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi0/1 2 1 10.1.22.2/24 1 WAIT 0/0
Gi0/0 2 1 10.1.1.2/24 1 WAIT 0/0
From a general perspective, the show ip ospf interface brief command lists output similar to the show ip eigrp interface command, with one line for each enabled interface. The show ip ospf interface command, not shown in the example, lists detailed OSPF information for each interface.
Specific to this problem, the output in Example 11-5 shows that R1 and R2 both have OSPF enabled on both LAN interfaces. However, this command also lists the area number for each interface, with R2 having both LAN interfaces in area 1. Also, these commands repeat the IP address and mask of the interfaces, so together, you can see that R1’s 10.1.1.1/24 address is in the same subnet as R2’s 10.1.1.2/24 address, putting these two routers in the same subnet but in different OSPF areas.
Example 11-6 shows another way to look at the problem, with the show ip protocols commands on both R1 and R2. Because this command lists the OSPF network commands in shorthand form, it can point toward a possible configuration error, even if the configuration is not available.
Example 11-6 Finding OSPF Configuration Errors with show ip protocols R1 and R2
R1> show ip protocols
*** IP Routing is NSF aware ***
Routing Protocol is "ospf 1"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 1.1.1.1
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
10.0.0.0 0.255.255.255 area 0
Routing Information Sources:
Gateway Distance Last Update
2.2.2.2 110 00:14:32
3.3.3.3 110 00:14:32
10.1.44.4 110 00:14:42
Distance: (default is 110)
R1> show ip route ospf
! Legend omitted for brevity
10.0.0.0/8 is variably subnetted, 6 subnets, 2 masks
O 10.1.33.0/24 [110/2] via 10.1.1.3, 00:15:32, GigabitEthernet0/0
O 10.1.44.0/24 [110/2] via 10.1.1.4, 00:15:42, GigabitEthernet0/0
! Now moving to Router R2
R2> show ip protocols
*** IP Routing is NSF aware ***
Routing Protocol is "ospf 2"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 2.2.2.2
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
10.0.0.0 0.255.255.255 area 1
Routing Protocol is "ospf 2"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 2.2.2.2
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
10.0.0.0 0.255.255.255 area 1
Routing Information Sources:
Gateway Distance Last Update
Distance: (default is 110)
R2>
Nov 15 12:16:39.377: %OSPF-4-ERRRCV: Received invalid packet: mismatched area
ID, from backbone area must be virtual-link but not found from 10.1.1.1,
GigabitEthernet0/0
Interestingly, a closer look at R2’s show ip protocols command output, particularly the highlighted portion, points out the configuration error. As usual, the section with the heading “Routing for Networks:” points to a shorthand version of the configuration. In this case, the highlighted phrase “10.0.0.0 0.255.255.255 area 1” is actually the exact syntax of the one network command on Router R2, minus the word network, or network 10.0.0.0 0.255.255.255 area 1. Because Figure 11-4 shows the design should put all interfaces in area 0, reconfiguring this command to instead be network 10.0.0.0 0.255.255.255 area 0 would solve this particular problem.
The end of the example also shows an unsolicited log message generated by Router R2, notifying the console user that this router has received a Hello from a router in a different area.
As you check the interfaces, you could also check several other details. It makes sense to go ahead and check the interface IP addresses, masks, and interface status values by using the show interfaces and show ip interface brief commands. In particular, it is helpful to note which interfaces are up/up, because a router will send no packets (including routing protocol packets) out interfaces that are not in an up/up state. These interface verification checks are part of the IPv4 troubleshooting topics in both the ICND1 and ICND2 exam topics, and are discussed in Chapter 21, “Troubleshooting IPv4 Routing,” so they are not repeated here.
Neighbor Relationships
This final major section of the chapter examines the large number of facts that each router must check with each potential neighbor before the two routers become neighbors.
At a very basic level, routing protocols can easily create neighbor relationships using a Hello protocol. First, the routing protocol must be enabled on an interface. In addition, the interface may not be configured as a passive interface, because that stops the routing protocol from sending the Hello messages.
Beyond this basic process, the routing protocols actually check several other parameters to find out whether the routers should become neighbors. Both OSPF and EIGRP use Hello messages, and these messages each list information used to perform some basic verification checks. For example, as just shown in earlier Example 11-5, an OSPF router should not become neighbors with another router in another area because all routers on a common subnet should be in the same OSPF area by design.
After an EIGRP or OSPF router hears a Hello from a new neighbor, the routing protocol examines the information in the Hello, and compares that information with the local router’s own settings. If the settings match, great. If not, the routers do not become neighbors. Because there is no formal term for all these items that a routing protocol considers, this book just calls them neighbor requirements.
Table 11-2 lists the neighbor requirements for both EIGRP and OSPF. Following the table, the next few pages examine some of these settings for both EIGRP and OSPF, again using examples based on Figure 11-3.
Note
Even though it is important to study and remember the items in this table, when reading this chapter the first time, just keep reading. When later reviewing the chapter or part, make sure you remember the details in the table.
Unlike most of the neighbor requirements listed in Table 11-2, the first three requirements have very little to do with the routing protocols themselves. The two routers must be able to send packets to each other over the physical network to which they are both connected. To do that, the router interfaces must be up/up, and they must be in the same subnet. In addition, the routers must not be using an ACL that filters the routing protocol traffic.
For instance, OSPF sends many messages to the well-known multicast IP addresses 224.0.0.5 and 224.0.0.6, whereas EIGRP uses 224.0.0.10. An ACL command like access-list 101 deny ip any host 224.0.0.10, in an inbound ACL on a router interface, would filter incoming EIGRP packets. Or, an ACL command like access-list 102 deny ospf any any could filter all OSPF traffic. Even more difficult to notice is an ACL that has lots of permit commands that match different TCP and UDP port numbers, but does not match the routing protocol explicitly, so the routing protocol packets match the implicit deny any at the end of the ACL. So, take extra care to watch for ACLs, especially when it seems like all the routing protocol configuration looks good.
In practice, before examining the rest of the details of why two routers do not become neighbors, confirm that the two routers can ping each other on the local subnet. If the ping fails, investigate all the Layer 1, 2, and 3 issues that could prevent the ping from working (such as an interface not being up/up). The details of troubleshooting IPv4 routing (that is, packet forwarding) can be found in several places, including Chapter 21, “Troubleshooting IPv4 Routing.” Additionally, the ICND1 Cert Guide includes other related details, including a chapter about IPv4 troubleshooting tools such as ping and traceroute; that ICND1 chapter is made available to you in this ICND2 book as a DVD Appendix J, “IPv4 Troubleshooting Tools.”
Now, on to the specific discussions about EIGRP and OSPF. Because the details differ slightly between the two routing protocols, this section first examines EIGRP, followed by OSPF.
Note
This section assumes that the routing protocol has actually been enabled on each required interface, as covered earlier in this chapter in the “Interfaces Enabled with a Routing Protocol” section.
EIGRP Neighbor Verification Checks
Any two EIGRP routers that connect to the same data link, and whose interfaces have been enabled for EIGRP and are not passive, will at least consider becoming neighbors. To quickly and definitively know which potential neighbors have passed all the neighbor requirements for EIGRP, just look at the output of the show ip eigrp neighbors command. This command lists only neighbors that have passed all the neighbor verification checks.
Example 11-7 shows an example of the show ip eigrp neighbors command, with the four routers from Figure 11-3 again. In this case, all the routers have been configured correctly, so each has a neighbor relationship with the other three routers on the same LAN subnet.
Example 11-7 R1 show ip eigrp neighbors Command with All Problems Fixed
R1# show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(99)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 10.1.1.3 Gi0/0 13 00:00:20 1 100 0 31
2 10.1.1.4 Gi0/0 13 00:00:43 80 480 0 10
0 10.1.1.2 Gi0/0 13 00:13:52 1 100 0 20
If the show ip eigrp neighbors command does not list one or more expected neighbors, the first problem isolation step should be to find out if the two routers can ping each other’s IP addresses on the same subnet. If that works, start looking at the list of neighbor verification checks, as relisted for EIGRP here in Table 11-3. Table 11-3 summarizes the EIGRP neighbor requirements, while noting the best commands with which to determine which requirement is the root cause of the problem.
Of the four rows of requirements listed in Table 11-3, the first two have already been discussed in this chapter, and do not need further discussion.
For EIGRP authentication (the third item in the table), EIGRP supports the capability for routers to trust routers as EIGRP neighbors only if the routers share the same security key (password); if that check fails, the neighbor relationship fails. By default, routers do not attempt EIGRP authentication, which allows the routers to form EIGRP neighbor relationships. If one router uses authentication, and the other does not, they will not become neighbors. If both use authentication, they must use the same authentication key to become neighbors.
The last item in the table, EIGRP K-values, refers to the EIGRP metric components and the metric calculation. These K-values are variables that basically enable or disable the use of the different components in the EIGRP composite metric. Cisco recommends leaving these values at their default settings, using only bandwidth and delay in the metric calculation. The K-value settings must match before two routers will become neighbors; you can check the K-values on both routers with the show ip protocols command.
EIGRP Neighbor Troubleshooting Example
Example 11-8 shows three problems that can cause EIGRP routers to fail to become neighbors. This example uses the usual design for this chapter, as repeated in Figure 11-5. The figure shows the same routers, and same interfaces, but with the following problems:
R2 has been configured with IP address 10.1.2.2/24 in a different subnet than R1, R3, and R4.
R3 has been configured to use ASN 199 with the router eigrp 199 command instead of ASN 99, as used on the other three routers.
R4 has been configured to use message digest 5 (MD5) authentication, whereas the other routers use no authentication.
R1 can actually detect two of the problems using local commands and messages, as shown in Example 11-8. R1 generates an unsolicited log message for the mismatched subnet problem, and a debug command on R1 can reveal the authentication failure. The example shows some running commentary inside the example.
Example 11-8 Common Problems Preventing the Formation of EIGRP Neighbors (R1)
! First, R1 has no neighbor relationships yet. R1 uses ASN (process) 99.
R1# show ip eigrp neighbors
EIGRP-IPv4 Neighbors for AS(99)
R1#
! Next, R1 generates a log message, which shows up at the console, stating
! that the router with IP address 10.1.2.2 is not on the same subnet as R1.
!
*Nov 15 16:19:14.740: %DUAL-6-NBRINFO: EIGRP-IPv4 99: Neighbor 10.1.2.2
(GigabitEthernet0/0) is blocked: not on common subnet (10.1.1.1/24)
! Next, R1 enables a debug that shows messages for each packet received from R4,
! which uses the wrong password (authentication key string)
!
R1# debug eigrp packets
EIGRP Packets debugging is on
(UPDATE, REQUEST, QUERY, REPLY, HELLO, IPXSAP, PROBE, ACK, STUB, SIAQUERY,
SIAREPLY)
R1#
*Nov 15 16:20:30.865: EIGRP: Gi0/0: ignored packet from 10.1.1.4, opcode = 5
(authentication off or key-chain missing)
Example 11-8 shows some evidence of the mismatched subnet with R2, and the invalid authentication problem with R4. Note that the ICND2 200-105 and CCNA 200-125 exam topics specifically state that both OSPF and EIGRP authentication are excluded from the exam topics. However, even without knowing the details, it is easy to imagine that if one router’s EIGRP process uses authentication with a defined password, and the other does not, that authentication will fail. The result? Neighbor relationships do not form.
Example 11-8 shows details about two of the problems, but not any details about the incorrect ASN configured on R3. Example 11-9 shows those details by listing excerpts from two show commands on R3, both of which identify the ASN configured on that router. By using these same commands on all the routers, you could note that R1, R2, and R4 use ASN 99, whereas R3 uses 199, as shown in Example 11-9.
Example 11-9 Displaying the Incorrect ASN (199) on R3
R3# show ip protocols
Routing Protocol is "eigrp 199"
!
! The first line of output from show ip eigrp interfaces lists ASN 199
!
R3# show ip eigrp interfaces
EIGRP-IPv4 Interfaces for AS(199)
Xmit Queue Mean Pacing Time Multicast Pending
Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes
Gi0/0 0 0/0 0 0/1 0 0
Gi0/1 0 0/0 0 0/1 0 0
OSPF Neighbor Troubleshooting
Similar to EIGRP, a router’s show ip ospf neighbor command lists all the neighboring routers that have met all the requirements to become an OSPF neighbor as listed in Table 11-2. So, the first step in troubleshooting OSPF neighbors is to look at the list of neighbors.
Example 11-10 lists the output of a show ip ospf neighbor command on Router R2, from Figure 11-4. All four routers sit on the same LAN subnet, in area 0, with correct configurations, so all four routers form a valid OSPF neighbor relationship.
Example 11-10 Normal Working show ip ospf neighbors Command on Router R2
R2# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
1.1.1.1 1 FULL/BDR 00:00:37 10.1.1.1 GigabitEthernet0/0
3.3.3.3 1 2WAY/DROTHER 00:00:37 10.1.1.3 GigabitEthernet0/0
4.4.4.4 1 FULL/DR 00:00:31 10.1.1.4 GigabitEthernet0/0
First, note that the neighbor IDs, listed in the first column, identify neighbors by their router ID (RID). For this example network, all four routers use an easily guessed RID. Further to the right, the Address column lists the interface IP address used by that neighbor on the common subnet.
A brief review of OSPF neighbor states (as explained in Chapter 7) can help you understand a few of the subtleties of the output in the example. A router’s listed status for each of its OSPF neighbors—the neighbor’s state—should settle into either a 2-way or full state under normal operation. For neighbors that do not need to directly exchange their databases, typically two non-designated router (DR) routers on a LAN, the routers should settle into a 2-way neighbor state. In most cases, two neighboring routers need to directly exchange their full link-state databases (LSDB) with each other. As soon as that process has been completed, the two routers settle into a full neighbor state.
In Example 11-10, Router R4 is the DR, and R1 is the backup DR (BDR), so R2 and R3 (as non-DRs) do not need to directly exchange routes. Therefore, R2’s neighbor state for R3 (RID 3.3.3.3) in Example 11-10 is listed as 2-way.
Note
Notably, OSPF neighbors do not have to use the same process ID on the router ospf process-id command to become neighbors. In Example 11-10, all four routers use different PIDs.
If the show ip ospf neighbor command does not list one or more expected neighbors, you should confirm, even before moving on to look at OSPF neighbor requirements, that the two routers can ping each other on the local subnet. But if the two neighboring routers can ping each other, and the two routers still do not become OSPF neighbors, the next step is to examine each of the OSPF neighbor requirements. Table 11-4 summarizes the requirements, listing the most useful commands with which to find the answers.
This topic looks at a couple of OSPF neighbor problems using the usual four-router network from Figure 11-4, with all interfaces in area 0. However, the following problems have been introduced into the design:
R2 has been configured with both LAN interfaces in area 1, whereas the other three routers’ G0/0 interfaces are assigned to area 0.
R3 is using the same RID (1.1.1.1) as R1.
R4 has been configured with a Hello/dead timer of 5/20 on its G0/0 interface, instead of the 10/40 used (by default) on R1, R2, and R3.
Figure 11-6 shows these same problems for reference.
Finding Area Mismatches
Earlier in this chapter, the “OSPF Interface Troubleshooting” section showed how to use the show ip ospf interface command to list the area numbers and find OSPF area mismatches. This next topic shows how to see that same issue using the debug ip ospf adj command, as shown in Example 11-11. This command lists messages related to OSPF neighbor adjacency events, and shows messages that identify the area mismatch (with R2).
Example 11-11 Finding Mismatched Area Problem with R1 debug
R1# debug ip ospf adj
OSPF adjacency events debugging is on
R1#
*Nov 15 13:42:02.288: OSPF-1 ADJ Gi0/0: Rcv pkt from 10.1.1.2, area 0.0.0.0,
mismatched area 0.0.0.1 in the header
R1#
R1# undebug all
All possible debugging has been turned off
As noted in Table 11-4, the debug ip ospf adj command helps troubleshoot mismatched OSPF area problems. The first part of the highlighted message in the example lists shorthand about a received packet (“Rcv pkt”) from 10.1.1.2, which is R2’s IP address. The rest of the message mentions R1’s area (0.0.0.0), and the area claimed by the other router (0.0.0.1). (Note that the message lists the 32-bit area number as a dotted-decimal number.)
This particular example focuses on the symptom (that a neighbor relationship does not start), and the debug messages that identify the problem (mismatched areas). However, finding the configuration error may take some work, because the problem could be more complex than just having the wrong area number configured on a command.
One harder-to-notice configuration error happens when the configuration has multiple network commands, with different area numbers, that all happen to match one interface’s IP address. IOS stores the OSPF network commands to the configuration in the same order they are configured (which is the same order listed in the output of show running-config). IOS processes the commands in sequence, so that the first network command that matches a particular interface is used to set the OSPF area number.
For instance, imagine a router with interface G0/1 configured with IP address 1.1.1.1. The OSPF configuration lists the following two network commands, in that order. Both would match the interface IP address of 1.1.1.1, so IOS uses the first command, which lists area 1. IOS would not use the second command, even though it uses a wildcard mask that is more specific.
network 1.0.0.0 0.255.255.255 area 1
network 1.1.1.1 0.0.0.0 area 0
Another tricky configuration error that can result in an area mismatch occurs when configuring both the network OSPF subcommand and the ip ospf interface subcommand on the same router. IOS supports using both on the same router at the same time. However, IOS does not prevent a case in which a network command attempts to enable OSPF in one area, and the ip ospf interface subcommand attempts to enable OSPF in a different area. When that happens, IOS uses the area number defined in the ip ospf interface subcommand.
For instance, with the two network commands just listed, if the ip ospf 1 area 5 command was configured on that router’s interface, that interface would be in area 5; IOS would prefer that setting over any OSPF network command.
Note
Using both network router subcommands and ip ospf interface subcommands allows an easier migration from the older to newer style OSPF configuration. However, most enterprises today would use either network commands or ip ospf commands in one router.
Finding Duplicate OSPF Router IDs
Next, Example 11-12 shows R1 and R3 both trying to use RID 1.1.1.1. Interestingly, both routers automatically generate a log message for the duplicate OSPF RID problem between R1 and R3; the end of Example 11-12 shows one such message. For the exams, just use the show ip ospf commands on both R3 and R1 to easily list the RID on each router, noting that they both use the same value.
Example 11-12 Comparing OSPF Router IDs on R1 and R3
! Next, on R3: R3 lists the RID of 1.1.1.1
!
R3# show ip ospf
Routing Process "ospf 3" with ID 1.1.1.1
Start time: 00:00:37.136, Time elapsed: 02:20:37.200
! lines omitted for brevity
! Back to R1: R1 also uses RID 1.1.1.1
R1# show ip ospf
Routing Process "ospf 1" with ID 1.1.1.1
Start time: 00:01:51.864, Time elapsed: 12:13:50.904
Supports only single TOS(TOS0) routes
Supports opaque LSA
Supports Link-local Signaling (LLS)
Supports area transit capability
Supports NSSA (compatible with RFC 3101)
Event-log enabled, Maximum number of events: 1000, Mode: cyclic
Router is not originating router-LSAs with maximum metric
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
Maximum wait time between two consecutive SPFs 10000 msecs
Incremental-SPF disabled
Minimum LSA interval 5 secs
Minimum LSA arrival 1000 msecs
LSA group pacing timer 240 secs
Interface flood pacing timer 33 msecs
Retransmission pacing timer 66 msecs
Number of external LSA 0. Checksum Sum 0x000000
Number of opaque AS LSA 0. Checksum Sum 0x000000
Number of DCbitless external and opaque AS LSA 0
Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Number of areas transit capable is 0
External flood list length 0
IETF NSF helper support enabled
Cisco NSF helper support enabled
Reference bandwidth unit is 100 mbps
Area BACKBONE(0) (Inactive)
Number of interfaces in this area is 3
Area has no authentication
SPF algorithm last executed 00:52:42.956 ago
SPF algorithm executed 9 times
Area ranges are
Number of LSA 1. Checksum Sum 0x00C728
Number of opaque link LSA 0. Checksum Sum 0x000000
Number of DCbitless LSA 0
Number of indication LSA 0
Number of DoNotAge LSA 0
Flood list length 0
*May 29 00:01:25.679: %OSPF-4-DUP_RTRID_NBR: OSPF detected duplicate router-id
1.1.1.1 from 10.1.1.3 on interface GigabitEthernet0/0
First, focus on the problem: the duplicate RIDs. The first line of the show ip ospf command on the two routers quickly shows the duplicate use of 1.1.1.1. To solve the problem, assuming R1 should use 1.1.1.1 and R3 should use another RID (maybe 3.3.3.3), change the RID on R3, and restart the OSPF process. To do so, use the router-id 3.3.3.3 OSPF subcommand and use the EXEC mode command clear ip ospf process.
Also, take a moment to read over the log message generated on each router when a duplicate RID exists.
Finally, note that the show ip ospf commands in Example 11-12 also show a common false positive for a root cause of OSPF neighbor problems. OSPF PIDs—the number of the router ospf command—do not have to match. Note that in Example 11-12 that same first line of output shows that R3 uses the router ospf 3 command, per the phrase “Process ospf 3,” whereas R1 uses the router ospf 1 command, as noted with the phrase “Process ospf 1.” These mismatched numbers are not a problem.
Finding OSPF Hello and Dead Timer Mismatches
Finally, consider the problem created on R4, with the configuration of a different Hello timer and dead timer as compared with the default settings on R1, R2, and R3. Whereas EIGRP allows neighbors to use a different Hello timer, OSPF does not, so this mismatch prevents R4 from becoming neighbors with any of the other three OSPF routers.
Example 11-13 shows the easiest way to find the mismatch, using the show ip ospf interface command on both R1 and R4. This command lists the Hello and dead timers for each interface, as highlighted in the example. Note that R1 uses 10 and 40 (Hello and dead), whereas R4 uses 5 and 20.
Example 11-13 Finding Mismatched Hello/Dead Timers
R1# show ip ospf interface G0/0
GigabitEthernet0/0 is up, line protocol is up
Internet Address 10.1.1.1/24, Area 0, Attached via Network Statement
Process ID 1, Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 1.1.1.1, Interface address 10.1.1.1
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
! lines omitted for brevity
! Moving on to R4 next
!
R4# show ip ospf interface Gi0/0
GigabitEthernet0/0 is up, line protocol is up
Internet Address 10.1.1.4/24, Area 0, Attached via Network Statement
Process ID 4, Router ID 10.1.44.4, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.1.44.4, Interface address 10.1.1.4
No backup designated router on this network
Timer intervals configured, Hello 5, Dead 20, Wait 20, Retransmit 5
! lines omitted for brevity
The debug ip ospf hello command can also uncover this problem because it lists a message for each Hello that reveals the Hello/dead timer mismatch, as shown in Example 11-14.
Example 11-14 Finding Mismatched Hello/Dead Timers with debug
R1# debug ip ospf hello
OSPF hello events debugging is on
R1#
*Nov 15 14:05:10.616: OSPF-1 HELLO Gi0/0: Rcv hello from 10.1.44.4 area 0 10.1.1.4
*Nov 15 14:05:10.616: OSPF-1 HELLO Gi0/0: Mismatched hello parameters from 10.1.1.4
*Nov 15 14:05:10.616: OSPF-1 HELLO Gi0/0: Dead R 20 C 40, Hello R 5 C 10 Mask R
255.255.255.0 C 255.255.255.0
Although debug messages can be a little difficult to understand, a few comments make the meaning of these messages much clearer. The highlighted message uses a C to mean “configured value”—in other words, the value on the local router, or R1 in this case. The R in the message means “received value,” or the value listed in the received Hello. In this case
“Dead R 20 C 40” means that R1 received a Hello with a dead timer set to 20, while R1’s configured value is set to 40.
“Hello R 5 C 10” means that R1 received a Hello with the Hello timer set to 5, while R1’s configured value is set to 10.
Note that any IP subnet mismatch problems could also be found with this same debug, based on the received and configured subnet masks.
Other OSPF Issues
This last short discussion in this chapter looks at these two additional topics: shutting down the routing protocol process and the interface maximum transmission unit (MTU) size.
Shutting Down the OSPF Process
Cisco uses the IOS shutdown command in several contexts. You can use the shutdown command in interface configuration mode to disable the interface so that it no longer sends and receives packets. Cisco IOS switches allow the shutdown command in VLAN configuration mode, causing the switch to stop forwarding frames in that VLAN. In both cases, the shutdown command does not remove any configuration; it simply causes IOS to stop a particular function. Then, the no shutdown command in the same command mode re-enables that function.
IOS allows both the OSPFv2 and EIGRP routing protocol processes to be disabled and enabled with the shutdown and no shutdown commands, respectively, in routing protocol configuration mode. When a routing protocol process is shut down, IOS
Brings down any existing neighbor relationships
Does not form new neighbor relationships
Quits sending Hello messages
Does not remove routing protocol configuration
Basically, shutting down the routing protocol process gives the network engineer a way to stop using the routing protocol on that router, without having to remove all the configuration.
From a troubleshooting perspective, on the exam, what would you expect to see if a small design was configured perfectly, except that one router’s OSPF process was shut down? First, the router with the shutdown routing protocol process would not have any OSPF neighbors, and other routers would not list that router as a neighbor. But because the OSPF shutdown subcommand does not remove any configuration, the show ip ospf interfaces command still shows evidence that OSPF is configured on the interfaces.
Example 11-15 shows an example on Router R5, as shown in Figure 11-7. R5 is a different router than the one used in earlier examples, but it begins the example with two OSPF neighbors, R2 and R3, with router IDs 2.2.2.2 and 3.3.3.3. The example shows the OSPF process being shut down, the neighbors failing, and those two key OSPF show commands: show ip ospf neighbor and show ip ospf interface brief.
Example 11-15 Shutting Down an OSPF Process, and the Resulting Neighbor States
R5# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
2.2.2.2 1 FULL/DR 00:00:35 10.1.12.2 GigabitEthernet0/1
3.3.3.3 1 FULL/DR 00:00:33 10.1.13.3 GigabitEthernet0/2
R5# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
R5(config)# router ospf 1
R5(config-router)# shutdown
R5(config-router)# ^Z
R5#
*Mar 23 12:43:30.634: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on GigabitEthernet0/1
from FULL to DOWN, Neighbor Down: Interface down or detached
*Mar 23 12:43:30.635: %OSPF-5-ADJCHG: Process 1, Nbr 3.3.3.3 on GigabitEthernet0/2
from FULL to DOWN, Neighbor Down: Interface down or detached
R5#
R5# show ip ospf neighbor
R5#
R5# show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi0/1 1 0 10.1.12.1/24 1 DOWN 0/0
Gi0/2 1 0 10.1.13.1/24 1 DOWN 0/0
The two show commands point out a couple of particularly important facts. First, before the shutdown, the show ip ospf neighbor command lists two neighbors. After the shutdown, the same command lists no neighbors at all. Second, the show ip ospf interface brief command does list the interfaces on which OSPF is enabled, on the local router’s own IP addresses. However, it lists a state of DOWN, which is a reference to the neighbor’s state.
Mismatched MTU Settings
The MTU size defines a per-interface setting used by the router for its Layer 3 forwarding logic, defining the largest network layer packet that the router will forward out each interface. For instance, the IPv4 MTU size of an interface defines the maximum size IPv4 packet that the router can forward out an interface.
Routers often use a default MTU size of 1500 bytes, with the ability to set the value as well. The ip mtu size interface subcommand defines the IPv4 MTU setting, and the ipv6 mtu size command sets the equivalent for IPv6 packets.
In an odd twist, two OSPFv2 routers can actually become OSPF neighbors, and reach 2-way state, even if they happen to use different IPv4 MTU settings on their interfaces. However, they fail to exchange their LSDBs. Eventually, after trying and failing to exchange their LSDBs, the neighbor relationship also fails.
The concepts behind what happens with an MTU mismatch work the same with both OSPFv2 and OSPFv3. In Chapter 23, “Implementing OSPF for IPv6,” the “The Issue of IPv6 MTU” section shows an example of this particular problem with OSPFv3. Read that section for a little more detail about this issue.
Chapter Review
One key to doing well on the exams is to perform repetitive spaced review sessions. Review this chapter’s material using either the tools in the book, DVD, or interactive tools for the same material found on the book’s companion website. Refer to the “Your Study Plan” element for more details. Table 11-5 outlines the key review elements and where you can find them. To better track your study progress, record when you completed these activities in the second column.
Command References
Tables 11-7, 11-8, and 11-9 list configuration, verification, and debug commands used in this chapter. As an easy review exercise, cover the left column in a table, read the right column, and try to recall the command without looking. Then repeat the exercise, covering the right column, and try to recall what the command does.
Table 11-7 Chapter 11 Configuration Command Reference
Table 11-8 Chapter 11 show Command Reference
Table 11-9 Chapter 11 debug Command Reference