Chapter 14. Troubleshooting IPsec VPNs
Introduction
When crypto is configured on Cisco IOS, at a minimum several components are used, such as IP connectivity, IKEv2, IPsec and GRE encapsulation. All these technologies are complex and often are made up of a number of components themselves. Having a base knowledge of how the protocol or feature in question operates will always help in any troubleshooting situation.
Many designers and administrators of VPN technologies soon become lost when issues occur; however, by following some simple steps they can usually find the root cause of the issue in a timely manner. If the root cause cannot be found, following a proven methodology will allow the area of issue to be determined, so it can then be examined by people with greater expertise, such as members of the Cisco Technical Assistance Center (TAC).
When the components of IPsec VPNs are segmented into the following functions and the related protocols, it supports the creation of a methodology that can be followed to correctly identify broken behavior when it is necessary to troubleshoot IPsec VPNs.
Table 14-1 illustrates the functions and protocols used when troubleshooting IPsec VPNs.
The functions are listed top-down, so before an IKE session can be successfully established, for example, there must be IP connectivity between both peers. This might require that static routes or a dynamic routing protocol be used. It’s also worth noting that IKEv2 uses UDP port 500, with Encapsulation Security Payload and Authentication Header using IP protocol 50 and 51 respectively (or potentially UDP port 4500 if NAT Traversal is active). It could be the case that IKE traffic will be permitted; however Encapsulation Security Payload or Authentication Header could be blocked on the transport network. For this reason, IP connectivity should be tested at every stage of the protocol negotiation when the ports/protocols change.
IKEv2 (if used) will use one or more methods for authentication and authorization; these steps are listed as steps 2.1 and 2.2, which are a subset of IKEv2, because an IKE SA would not be established unless authentication and authorization were successful.
Once the IKEv2 SA is created, the subsequent IPsec Security Associations must use the same encapsulation method. If one peer used GRE and the other a VTI, then the negotiation of the IPsec Security Association would fail. The negotiated traffic selectors and cryptographic protection must also match between peers for an IPsec Security Association to be established successfully.
Once the IPsec Security Association has been created, the relevant IP routing must be correct for traffic to be encrypted/decrypted by both parties.
Tools of Troubleshooting
There are various methods to enable troubleshooting, such as show commands, debug messages, packet capture, IKEv2 error diagnostics, and event-trace monitoring. Each has varying degrees of impact on the system when enabled. Generally, the more verbose the debugging messages are, the heavier the impact. If the output is being sent to a local buffer, it will have less impact than if it is sent to the console, which has less impact than if it is sent to a syslog server.
When troubleshooting, take care to ensure that the tools and features enabled by troubleshooting do not overload a system.
The following sections list some of the most common issues seen by Cisco TAC and the methods used to troubleshoot them. These are listed in the order of impact on the system.
Show Commands
The ability to obtain insightful information from the command-line interface (CLI) can be crucial when troubleshooting. Generally speaking running show commands to gain insight put little stress on a system.
In situations where remote access (SSH or telnet) to the CLI is obtained over a low-bandwidth link, if the output of a show command is extremely verbose, it could have an impact on the traffic over the link.
The commands listed in chapter 13, “Monitoring IPsec VPNs,” can be used when troubleshooting specific aspects of the generation of an IPsec Security Association.
Syslog Messages
Messages from syslog can give an insight into normal and abnormal behavior. With a well-engineered system, normal events will generally generate a small number of syslog messages; even in failure scenarios, the telemetry gained via syslog messages can be a very lightweight method to obtain insightful information. Although the output of debugging messages is obtained via syslog, we define syslog messages as those messages obtained under normal conditions without enabling any specific debugs.
Event-Trace Monitoring
Event tracing works by reading informational messages from specific Cisco IOS software subsystems that are logged into system memory. Trace messages that are stored in memory can be displayed on the screen or saved to a file for later analysis. Messages are saved in binary format, without applying additional processing or formatting. Because of this binary format, event tracing can collect informational messages faster and for a longer time than nonbinary formats.
When a device starts behaving abnormally, it is useful to know the events that caused the abnormal state. Turning debug on when abnormal behavior is happening generally won’t help in troubleshooting, because it records data after the cause of the issue. Event-trace monitoring provides a black-box type of function that constantly records information about the state of subsystems. This feature is enabled by default and allows a historical view of events that led up to the problem state.
When debugging could potentially increase the load on an already overloaded system, event-trace monitoring allows logging information to be stored and retrieved without putting any more stress on the system.
The following example illustrates gathering information pertaining to IKEv2 and IPsec from the event-trace monitoring feature.
Under the monitor event-trace crypto ikev2 submenu you can view errors, events, or exceptions.
Router#show monitor event-trace crypto ikev2 ?
error Show IKEv2 Errors
event Show IKEv2 Events
exception Show IKEv2 Exceptions
Under the crypto ikev2 error submenu a number of options exist. Errors can be viewed for certain time periods.
Router#show monitor event-trace crypto ikev2 error ?
all Show all the traces in current buffer
back Show trace from this far back in the past
clock Show trace from a specific clock time/date
from-boot Show trace from this many seconds after booting
latest Show latest trace events since last display
parameters Parameters of the trace
Under the monitor event-trace crypto ipsec submenu you can view errors, events, or exceptions
Router#show monitor event-trace crypto ipsec ?
error Show IPSEC Error
event Show IPSEC Events
exception Show IPSEC Exception
Similar to IKEv2, under the crypto ipsec error submenu a number of options exist. Errors can be viewed for certain time periods.
Router#show monitor event-trace crypto ipsec error ?
all Show all the traces in current buffer
back Show trace from this far back in the past
clock Show trace from a specific clock time/date
from-boot Show trace from this many seconds after booting
latest Show latest trace events since last display
parameters Parameters of the trace
The following example illustrates the output retrieved from the event-trace IKEv2 error conditions. This data was recorded without enabling any debugging commands.
Router#show monitor event-trace crypto ikev2 error all
SA ID:1 SESSION ID:1 Remote: 10.10.10.1/500 Local:
10.10.60.6/500
SA ID:1 SESSION ID:1 Remote: 10.10.10.1/500 Local:
10.10.60.6/500 : Auth exchange failed
SA ID:1 SESSION ID:1 Remote: 10.10.10.1/500 Local:
10.10.60.6/500 Negotiation aborted due to ERROR: Auth
exchange failed
ID:0 SESSION ID:0 Optional profile description not
updated in PSH
The size of the event trace or its location can be changed with the global monitor event-trace command. This is illustrated in the following example.
Router(config)#monitor event-trace crypto ikev2 error ?
dump-file Set name of trace dump file
size Set size of trace
stacktrace Trace call stack at tracepoints; clear the trace buffer first
<cr>
Debugging
In many cases, knowing how a protocol operates gives an engineer the ability to diagnose an issue. When troubleshooting a VPN configured via IKEv2, unless the engineer is highly experienced in the configuration, in many cases debugging must be enabled to diagnose an error.
When an engineer is dealing with issues relating to an IPsec VPN using IKEv2, the most common protocols that will require debugging are IKEv2 and IPsec.
IKEv2 Debugging
Cisco IOS supports debugging of IKEv2 by allowing detailed messages to be logged for certain events pertaining to IKEv2.
IKEv2 debugging has a number of verbose options, with standard IKEv2 debugging giving information about important events when the IKEv2 Security Association is generated, such as policy, key and profile lookup, IKEv2 message exchanges, timer events, and the invocation of external components such as AAA, PKI, and callbacks.
The following example illustrates enabling standard IKEv2 debugging.
Router#debug crypto ikev2
IKEv2 default debugging is on
There are a number of more verbose options for debugging IKEv2 that are specific to certain features. These should be enabled only when issues are seen with certain features or when more verbose troubleshooting is required.
The following example illustrates the options available when enabling IKEv2 debugging.
Router#debug crypto ikev2 ?
client Client
cluster IKEv2 Cluster load-balancer debugging
error IKEv2 Error debugging
internal IKEv2 Internal debugging
packet IKEv2 Packet debugging
IKEv2 client debugging supports debugging of FlexVPN client features. This feature is enabled with the debug crypto ikev2 client flexvpn command.
IKEv2 cluster debugging supports debugging the IKEv2 cluster feature. There are two additional options, with the detail and error keywords enabling detailed verbose debugging and error conditions respectively.
IKEv2 error debugging allows error conditions to be reported. Although this debugging level is not exhaustive and will not give an exact indication of the cause of the issue, it can be used.
IKEv2 internal debugging enables the logging of debug messages for IKEv2 state machine transitions, user/group authorization attributes, merging, and config mode exchanges. This debugging level offers limited insight for users and should only be enabled by Cisco TAC.
IKEv2 packet debugging lets the IKEv2 packet be dumped to the logging facility. This lets the raw IKE data to be examined in decrypted format. The hexdump option will print the raw hex data.
The following example illustrates the output of a IKE_SA_INIT message that was obtained after enabling the debug crypto ikev2 packet command. After the initial message indicating the exchange type, the full IKE message is shown, with the contents of each payload described in detail.
IKEv2-PAK:(SESSION ID = 1,SA ID = 1):Next payload: SA, version: 2.0 Exchange type:
IKE_SA_INIT, flags: INITIATOR Message id: 0, length: 464
Payload contents:
SA Next payload: KE, reserved: 0x0, length: 48
last proposal: 0x0, reserved: 0x0, length: 44
Proposal: 1, Protocol id: IKE, SPI size: 0, #trans: 4 last transform: 0x3,
reserved: 0x0: length: 12
type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_2048_MODP/Group 14
KE Next payload: N, reserved: 0x0, length: 264
DH group: 14, Reserved: 0x0
N Next payload: VID, reserved: 0x0, length: 24
VID Next payload: VID, reserved: 0x0, length: 23
VID Next payload: NOTIFY, reserved: 0x0, length: 21
NOTIFY(NAT_DETECTION_SOURCE_IP) Next payload: NOTIFY, reserved: 0x0, length: 28
Security protocol id: IKE, spi size: 0, type: NAT_DETECTION_SOURCE_IP
NOTIFY(NAT_DETECTION_DESTINATION_IP) Next payload: NONE, reserved: 0x0, length: 28
Security protocol id: IKE, spi size: 0, type: NAT_DETECTION_DESTINATION_IP
The following illustrates the output of a IKE_SA_INIT message which was obtained after enabling the debug crypto ikev2 packet hexdump command. After the initial message indicating the exchange type, the payload contents is described in detail. For certain payloads the data is output in raw hex. Finally we see the raw hex of the packet as sent on the wire.
IKEv2-PAK:(SESSION ID = 1,SA ID = 1):Next payload: SA, version: 2.0 Exchange type:
IKE_SA_INIT, flags: INITIATOR Message id: 0, length: 464
Payload contents:
SA Next payload: KE, reserved: 0x0, length: 48
last proposal: 0x0, reserved: 0x0, length: 44
Proposal: 1, Protocol id: IKE, SPI size: 0, #trans: 4 last transform: 0x3,
reserved: 0x0: length: 12
type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 2, reserved: 0x0, id: SHA256
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 4, reserved: 0x0, id: DH_GROUP_2048_MODP/Group 14
KE Next payload: N, reserved: 0x0, length: 264
DH group: 14, Reserved: 0x0
45 B7 59 62 46 9A 5A 61 7A 93 FC 63 DC 5E 87 AC
50 AA DB 2F 11 39 19 F5 7F E4 6C C3 57 39 60 E6
C9 ED E4 A1 4E A1 87 87 12 79 D7 DD 68 83 06 2B
32 25 C2 D6 01 AD F3 4F 40 23 2C 02 AA 16 6A 25
45 B5 D2 61 B9 DD 4C 72 95 27 08 13 E4 FE E1 32
19 F3 5A A3 0E EF 59 2A CC C4 A2 97 E4 04 73 21
82 BB 71 4B 85 CB BE 33 55 89 61 D8 28 4A B7 CB
32 18 AB A0 CB 5D AB C1 4D 82 3A 4B 51 41 B4 A4
5B 56 C2 73 B3 B0 26 67 57 05 55 9D C2 BE 46 BC
9F 1A EA 6C CA D3 07 65 02 D4 EC 32 A9 B3 94 ED
F6 D6 EE 54 FB 74 30 F4 E3 DE E2 57 0B 50 4E 88
E9 3C 8A 35 44 6F 34 42 DA B6 8D 49 B7 0A 8A 68
90 A9 C7 39 C5 35 98 42 0F B4 91 F3 31 E7 42 D9
7D 20 43 AE B5 C4 B0 34 2C 16 15 1E 9A 68 F4 F6
C3 11 BE 46 AC C7 25 44 23 35 AF 6E EB 6B 74 CC
14 BF F2 40 7D E6 48 B7 9A 3C C1 C4 0B A3 26 BD
N Next payload: VID, reserved: 0x0, length: 24
E4 77 64 DC 98 E7 3E 85 7C 41 CA A1 21 50 2F 0F
6C AC AE A4
VID Next payload: VID, reserved: 0x0, length: 23
43 49 53 43 4F 2D 44 45 4C 45 54 45 2D 52 45 41
53 4F 4E
VID Next payload: NOTIFY, reserved: 0x0, length: 21
40 55 3D 1F 3C D1 36 43 82 0D 0F C5 15 ED 49 9E
NOTIFY(NAT_DETECTION_SOURCE_IP) Next payload: NOTIFY, reserved: 0x0, length: 28
Security protocol id: IKE, spi size: 0, type: NAT_DETECTION_SOURCE_IP
0D BE 1C 55 0D 3F A6 61 EE C8 F3 1D 8C 6B 46 9D
E7 09 23 75
NOTIFY(NAT_DETECTION_DESTINATION_IP) Next payload: NONE, reserved: 0x0, length: 28
Security protocol id: IKE, spi size: 0, type: NAT_DETECTION_DESTINATION_IP
53 CD 98 45 F3 B9 1F 39 8E EB 2C CB C2 AA CF DE
64 C2 85 19
IKEv2 Tx packet(on wire):
F4 9D AD A2 60 CB 37 3F 00 00 00 00 00 00 00 00
21 20 22 08 00 00 00 00 00 00 01 D0 22 00 00 30
00 00 00 2C 01 01 00 04 03 00 00 0C 01 00 00 0C
80 0E 00 80 03 00 00 08 02 00 00 05 03 00 00 08
03 00 00 0C 00 00 00 08 04 00 00 0E 28 00 01 08
00 0E 00 00 45 B7 59 62 46 9A 5A 61 7A 93 FC 63
DC 5E 87 AC 50 AA DB 2F 11 39 19 F5 7F E4 6C C3
57 39 60 E6 C9 ED E4 A1 4E A1 87 87 12 79 D7 DD
68 83 06 2B 32 25 C2 D6 01 AD F3 4F 40 23 2C 02
AA 16 6A 25 45 B5 D2 61 B9 DD 4C 72 95 27 08 13
E4 FE E1 32 19 F3 5A A3 0E EF 59 2A CC C4 A2 97
E4 04 73 21 82 BB 71 4B 85 CB BE 33 55 89 61 D8
28 4A B7 CB 32 18 AB A0 CB 5D AB C1 4D 82 3A 4B
51 41 B4 A4 5B 56 C2 73 B3 B0 26 67 57 05 55 9D
C2 BE 46 BC 9F 1A EA 6C CA D3 07 65 02 D4 EC 32
A9 B3 94 ED F6 D6 EE 54 FB 74 30 F4 E3 DE E2 57
0B 50 4E 88 E9 3C 8A 35 44 6F 34 42 DA B6 8D 49
B7 0A 8A 68 90 A9 C7 39 C5 35 98 42 0F B4 91 F3
31 E7 42 D9 7D 20 43 AE B5 C4 B0 34 2C 16 15 1E
9A 68 F4 F6 C3 11 BE 46 AC C7 25 44 23 35 AF 6E
EB 6B 74 CC 14 BF F2 40 7D E6 48 B7 9A 3C C1 C4
0B A3 26 BD 2B 00 00 18 E4 77 64 DC 98 E7 3E 85
7C 41 CA A1 21 50 2F 0F 6C AC AE A4 2B 00 00 17
43 49 53 43 4F 2D 44 45 4C 45 54 45 2D 52 45 41
53 4F 4E 29 00 00 15 46 4C 45 58 56 50 4E 2D 53
55 50 50 4F 52 54 45 44 29 00 00 1C 01 00 40 04
0D BE 1C 55 0D 3F A6 61 EE C8 F3 1D 8C 6B 46 9D
E7 09 23 75 00 00 00 1C 01 00 40 05 53 CD 98 45
F3 B9 1F 39 8E EB 2C CB C2 AA CF DE 64 C2 85 19
IPsec Debugging
Although IKEv2 provides insight into the traffic selectors that are exchanged to create an IPsec Security Association, detailed debugging of IPsec can be enabled separately.
IPsec debugging will give details of the protocol (Encapsulation Security Payload or Authentication Header), transform, and SPI used when constructing the IPsec Security Association.
IPsec debugging can be enabled with the debug crypto ipsec command.
A number of more verbose IPsec debugging commands can be enabled. The following example illustrates the additional options available when enabling IPsec debugging.
Router#debug crypto ipsec ?
client Client Debug
error IPSEC errors
ha IPSEC High Availability
hw-request IPSEC hw-request
message IPSEC message
metadata CTS metadata
states IPSEC states
Key Management Interface Debugging
Key Management Interface (KMI) is a message-based programmatic interface between the IOS IPsec and key management processes such as IKEv1 and IKEv2. KMI defines a set of well-defined messages between IKE and IPsec.
Table 14-2 describes the KMI messages sent from IPsec to IKEv2. Note that the KMI messages use the KEY_ENG prefix.
Table 14-3 describes the KMI messages sent from IKEv2 to IPsec. Note that the KMI messages use the KEY_MGR prefix.
KMI debugging is a very useful debugging tool in isolating a problem between IKE and IPsec. The KMI debugging is enabled using the debug crypto kmi command.
The following example illustrates the debug crypto kmi output for an IKE/IPsec session establishment on an IPsec sVTI-based initiator.
KMI: (Session ID: 1) IPSEC key engine sending message KEY_ENG_REQUEST_SAS to
Crypto IKEv2.
KMI: (Session ID: 1) Crypto IKEv2 received message KEY_ENG_REQUEST_SAS from IPSEC
key engine.
KMI: (Session ID: 1) Crypto IKEv2 sending message KEY_MGR_VALIDATE_IPSEC_PROPOSALS
to IPSEC key engine.
KMI: (Session ID: 1) IPSEC key engine received message KEY_MGR_VALIDATE_IPSEC_
PROPOSALS from Crypto IKEv2.
KMI: (Session ID: 1) Crypto IKEv2 sending message KEY_MGR_CREATE_IPSEC_SAS to
IPSEC key engine.
KMI: (Session ID: 1) IPSEC key engine received message KEY_MGR_CREATE_IPSEC_SAS
from Crypto IKEv2.
KMI: IPSEC key engine sending message KEY_ENG_NOTIFY_INCR_COUNT to Crypto IKEv2.
The following example illustrates the debug crypto kmi output for an IKE/IPsec session establishment on an IPsec dVTI-based responder.
KMI: (Session ID: 3) Crypto IKEv2 sending message KEY_MGR_IKMP_READY to IPSEC key
engine.
KMI: (Session ID: 3) IPSEC key engine received message KEY_MGR_IKMP_READY from
Crypto IKEv2.
KMI: IPSEC key engine sending message KEY_ENG_IPSEC_READY to Crypto IKEv2.
KMI: Crypto IKEv2 received message KEY_ENG_IPSEC_READY from IPSEC key engine.
KMI: (Session ID: 3) Crypto IKEv2 sending message KEY_MGR_VALIDATE_IPSEC_PROPOSALS
to IPSEC key engine.
KMI: (Session ID: 3) IPSEC key engine received message
KEY_MGR_VALIDATE_IPSEC_PROPOSALS from Crypto IKEv2.
KMI: (Session ID: 3) Crypto IKEv2 sending message
KEY_MGR_CREATE_IPSEC_SAS to IPSEC key engine.
KMI: (Session ID: 3) IPSEC key engine received message KEY_MGR_CREATE_IPSEC_SAS
from Crypto IKEv2.
KMI: IPSEC key engine sending message KEY_ENG_NOTIFY_INCR_COUNT to Crypto IKEv2.
KMI: Crypto IKEv2 received message KEY_ENG_NOTIFY_INCR_COUNT from IPSEC
key engine.
PKI Debugging
Public key infrastructure (PKI) debugging events can be generated to give detailed information about certificate enrollment or renewal. The applications that use PKI, such as IKEv2, can generate specific debug messages when using the PKI trustpoint to perform signing and/or validation. Debugging can be enabled to give detailed information regarding the PKI’s interaction with the crypto engine when performing operations with the cryptographic key material.
Conditional Debugging
By default Cisco IOS provides a debugging mechanism that works in an all-or-none model, whereby specific debugging is either enabled or disabled. This is not efficient in debugging problems in a large-scale VPN deployment, where many remote devices are connecting to a hub, causing a large number of debugging messages to be generated.
Conditional debugging allows a user to define a set of condition filters, so that the system outputs only debug messages related to those predefined conditions based on filter checking at run time. The debug filters include a number of parameters, such as peer IP address, destination (subnet with mask), connection-id, and SPI.
When multiple debug conditions are set, the debug message is printed only if all the conditions are met.
When a debug message would be output, if the debug condition parameter is not available for comparison, then the debug message is suppressed. For example, when conditional debugging is based on an IKEv2 profile, the IKEv2 profile is available on a responder only after it receives the IKE_AUTH request (in the third message), so until then the messages are suppressed. Be aware that in this instance you will not see any debug message that occurred in the IKE_SA_INIT exchange.
This behavior can be changed with the debug crypto condition unmatched command. In this instance all available messages are generated, but this can soon fill most log buffers for busy systems.
Router#debug crypto condition ?
unmatched Output debugs even if no context available
Conditional debugging can be used to minimize the risk of consuming CPU cycles and to create debug messages that pertain to certain peers only.
The debug crypto condition ikev2 profile profile_name command can be used to enable IKEv2 debugging for peers that associate with a certain IKEv2 profile.
The debug crypto condition peer [group group] [hostname hostname] [ipv4 ipaddress] [subnet subnet mask] [username username]] command can be used to enable debug messages to be generated for one or more remote peers. This is a very handy command when troubleshooting connectivity issues on VPN gateways that are serving multiple clients. When troubleshooting issues on VPN gateways with many active clients, if conditional debugging is not used the amount of logging information generated can overwhelm many administrators.
Conditional debugging can be seen as a filter and should be used in conjunction with IKEv2 or IPsec debugging.
IP Connectivity
IKEv2 uses UDP as a transport protocol over port 500 or port 4500 (when NAT is detected). When NAT or PAT is used on the transport network, the source port used by the UDP traffic can be translated from UDP/500 to any UDP port. When IPsec VPNs are deployed over the Internet or a network that uses NAT or PAT, access control lists that pertain to IKEv2 should be configured to allow UDP traffic from any source port.
To validate IP connectivity between hosts, the ping command, which uses ICMP echo and echo-reply, can be used to check that an IP-level path exists. It should be noted that if a firewall or access-control device permits ICMP but not the protocols and/or ports used by IKE, Encapsulation Security Payload, or Authentication Header, then the results of the ping command might be misleading.
We must assume that the physical interface that is used as the source of the tunnel is physically connected and in the up state; this might seem an obvious check, but if the interface is down for any reason, then traffic cannot be sent or received on the physical network interface card (NIC).
There are cases where traffic is fragmented at the IP layer with fragments being lost in transit. In this case the fragmented traffic is never reassembled, so it will not be passed up the stack and hence appear in IKE debugs. Embedded Packet Capture (EPC) is an onboard packet capture facility that enables the system to capture packets flowing into, through, and from the device. The captured traffic can be analyzed locally or saved and exported for offline analysis by using a tool such as Wireshark.
The following example illustrates configuring an EPC to capture traffic on an interface.
A capture buffer is defined, which is a temporary buffer for storing the captured packets. Various options can be selected when the buffer is defined, such as size, maximum packet size, and circular/linear. In this example, the maximum size of packets is set to 1600 bytes to ensure the full packet is captured. Depending on the size of MTU and the requirement as to what is to be captured, this value should be adjusted accordingly.
Router#monitor capture buffer buf1 size 2048 max-size 1600 linear
Note that if certain traffic is being captured, a filter can also be applied to limit the capture to desired traffic. Define an access control list (ACL) within config mode and apply the filter to the buffer:
Router(config)#ip access-list buf-filter
Router(config-ext-nacl)#permit udp any any eq 500
Router#monitor capture buffer buf1 filter access-list buf-filter
A capture point is created, which defines the location where the capture occurs. In this case only incoming traffic is captured. The capture point also defines whether the capture occurs for IPv4 or IPv6 and in which switching path.
Router#monitor capture point ip cef point1 GigabitEthernet0/0 in
The buffer is attached to the capture point:
Router#monitor capture point associate point1 buf1
The capture is started:
Router#monitor capture point start point1
%BUFCAP-6-ENABLE: Capture Point point1 enabled
Once the capture is active, ensure that the traffic to be captured is generated. This could be as simple as disabling and enabling a tunnel interface (using shutdown and no shutdown commands) to ensure that IKE traffic is generated. Once the intended traffic has been generated, the capture can be stopped.
Router#monitor capture point stop point1
%BUFCAP-6-DISABLE: Capture Point point1 disabled
The contents of the capture can be viewed (this data is in hexadecimal). This hex information can be converted into a human-readable packet format (for example, pcap) using any tool that converts raw hexadecimal into pcap format.
Router#show monitor capture buffer buf1 dump
UTC May 21 2016 : IPv4 LES CEF : Gi0/0 None
2D2EBFA0: A44C110E $L..
2D2EBFB0: B4A0C89C 1DFEBF20 080045C0 00420058 4 H..~? [email protected]
2D2EBFC0: 0000FE11 610DAC10 0101AC10 016401F4 ..~.a.,...,..d.t
<removed>
The capture can be exported in pcap format so it can then be opened in a tool such as Wireshark.
monitor cap buffer buf1 export flash:/buf1.pcap
The following example shows the IKEv2 debugging output from a client that does not have IP connectivity with its peer. We see that the IKEv2 SA_INIT packet is sent; however, this packet is retransmitted a number of times, which is a sure indication that the packet did not reach the peer, or if it did, the returning packet was not received.
IKEv2:(SESSION ID = 1,SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial
negotiation),
Num. transforms: 15
AES-CBC AES-CBC AES-CBC SHA512
SHA384 SHA256 SHA1 MD5 SHA512 SHA384 SHA256 SHA96 MD596 DH_GROUP_1536_MODP/Group 5
DH_GROUP_1024_MODP/Group 2
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.20.3:500/VRF i0:f0]
Initiator SPI : 502A6C10FFD6F8BC - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP)
NOTIFY(NAT_DETECTION_DESTINATION_IP)
IKEv2:(SESSION ID = 1,SA ID = 1):Insert SA
FlexClient#
IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.20.3:500/VRF i0:f0]
Initiator SPI : 502A6C10FFD6F8BC - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP)
NOTIFY(NAT_DETECTION_DESTINATION_IP)
IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet
In this case, the reason why IP connectivity failed between both peers must be determined.
There are cases where IP connectivity can be lost within the IKE_AUTH exchange. When certificates are used, if the IKE datagram becomes fragmented and IKEv2 fragmentation is not used, then connectivity can fail. In the following example we see that the initiator sends the IKE_AUTH packet; however, the reply to this is not received, and the IKE_AUTH is retransmitted a number of times until it fails. This can be an indication that fragmentation occurred, for the IKE_SA_INIT exchange was successful.
IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDi CERT CERTREQ NOTIFY(HTTP_CERT_LOOKUP_SUPPORTED) AUTH CFG SA TSi TSr
NOTIFY(INITIAL_CONTACT) NOTIFY(USE_TRANSPORT_MODE) NOTIFY(SET_WINDOW_SIZE)
NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.50.5:500/VRF i0:f0]
Initiator SPI : DB0340BE40E55EE9 - Responder SPI : 756425ADB92B135A Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR
IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.50.5:500/VRF i0:f0]
Initiator SPI : DB0340BE40E55EE9 - Responder SPI : 756425ADB92B135A Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR
%LINK-3-UPDOWN: Interface Tunnel1, changed state to up
IKEv2:(SESSION ID = 1,SA ID = 1):Retransmitting packet
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.50.5:500/VRF i0:f0]
Initiator SPI : DB0340BE40E55EE9 - Responder SPI : 756425ADB92B135A Message id: 1
IKEv2 IKE_AUTH Exchange REQUEST
Payload contents:
ENCR
Note
If a VRF-aware IPsec is used in which the IVRF and FVRF are different, then ensure that the peer’s tunnel source address is reachable via the FVRF. This is a common mistake when VRF-aware IPsec is used. The show ip route vrf vrf_name command can be used to examine the FVRF routing table.
VPN Tunnel Establishment
IKEv2 is the protocol used to negotiate cryptographic algorithms in the IKE_SA_INIT exchange, authenticate both parties and establish IPsec Security Associations in the IKE_AUTH exchange, maintain the IKEv2 SA using INFORMATIONAL messages, and rekey using the CREATE_CHILD_SA exchange. Based on this knowledge, the steps involved in the creation of an IKEv2 SA can be followed, and when the observed behavior is not as intended debugging can be enabled to diagnose the cause of the failure.
IKEv2 Diagnose Error
If there is an error in the IKEv2 exchange, this will be logged internally. The errors can be viewed using the show crypto ikev2 diagnose error command. The following example illustrates a number of error conditions which were generated on an IKEv2 node.
Router#show crypto ikev2 diagnose error
Exit Path Table - status: enable, current entry 4, deleted 0, max allow 50
Error(1): Failed to locate an item in the database
-Traceback= 9D64A28z 8CBE65Ez 8C66C40z 8CA6FF5z 8C746E0z 8C5359Az 8C5A25Ez
8C59F5Fz 8C97FE2z 8C979DBz 8C6A95Dz 8CC2BC3z
Error(1): A supplied parameter is incorrect
-Traceback= 9D64A28z 8CBE65Ez 8C66C40z 8CC273Ez 8CFA2F1z 8CF4038z 8C8B7F2z
8C7B455z 8C7AFE0z 8C5549Dz 8C5A25Ez 8C59F5Fz 8C6FDCAz 8C6EC98z 8C6A95Dz 8CC2BC3z
Error(1): AAA author not configured in IKEv2 profile
-Traceback= 9D64A28z 8CBE65Ez 8C66C40z 8CC6D2Dz 8CC6AC2z 8CC9661z 8CC9537z
8CC2B8Fz
Error(1): We requested a certificate, but the peer supplied none
-Traceback= 9D64A28z 8CBE65Ez 8C66C40z 8C7BABBz 8C7B6B8z 8C553F3z 8C5A25Ez
8C59F5Fz 8C97FE2z 8C979DBz 8C6A95Dz 8CC2BC3z
The diagnose error message log can be cleared with the clear crypto ikev2 diagnose error command. This allows for errors to be cleared, connectivity tested, and then errors checked again.
As the same error message can be generated from multiple places in the code, the traceback decode helps Cisco TAC determine the exact location where the message was generated.
Troubleshooting the IKE_SA_INIT Exchange
The following example looks to highlight how misconfigured proposals can cause the IKE_SA_INIT exchange to fail. We will focus on two devices that are misconfigured.
As both parties are negotiating the cryptographic algorithms that will be used to secure the IKEv2 SA, a common issue that causes the IKE_SA_INIT exchange not to be successful is mismatching policies between peers.
Table 14-4 details the proposal that is configured on the initiator.
Table 14-5 details the proposal that is configured on the responder.
Obviously these attributes do not match, so an IKEv2 SA will not be established.
On the initiator, we see that the IKE_SA_INIT exchange is started, and the proposal sent to the peer,
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Start PKI Session
IKEv2:(SA ID = 1):[PKI -> IKEv2] Starting of PKI Session PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key,
DH Group 1
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):Request queued for computation of DH key
IKEv2:IKEv2 initiator - no config data to send in IKE_SA_INIT exch
IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_SA_INIT message
IKEv2:(SESSION ID = 1,SA ID = 1):IKE Proposal: 1, SPI size: 0
(initial negotiation),
Num. transforms: 4
AES-CBC SHA512 SHA512 DH_GROUP_768_MODP/Group 1
IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.10.10.1:500/From
10.10.50.5:500/VRF i0:f0]
Initiator SPI : D007314789FFB97A - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP)
NOTIFY(NAT_DETECTION_DESTINATION_IP)
IKEv2:(SESSION ID = 1,SA ID = 1):Insert SA
On the responder when the proposal is not compatible, we see that the IKE_SA_INIT message was received. However the policies within this did not match the locally configured proposal, so the NO_PROPOSAL_CHOSEN message is sent.
IKEv2:(SESSION ID = 9,SA ID = 1):Processing IKE_SA_INIT message
IKEv2:(SESSION ID = 9,SA ID = 1):Received Policies: : Failed to find a matching
policyProposal 1: AES-CBC-128 SHA512 SHA512 DH_GROUP_768_MODP/Group 1
IKEv2:(SESSION ID = 9,SA ID = 1):Expected Policies: : Failed to find a matching
policyProposal 1: AES-CBC-128 SHA256 SHA256 DH_GROUP_2048_MODP/Group 14
Proposal 2: AES-CBC-128 SHA256 SHA256 DH_GROUP_2048_MODP/Group 14
IKEv2:(SESSION ID = 9,SA ID = 1):: Failed to find a matching policy
IKEv2:(SESSION ID = 9,SA ID = 1):Sending no proposal chosen notify
The responder generates a reply containing a Notify payload of type NO_PROPOSAL_CHOSEN, which indicates that the proposals were not compatible.
The following example illustrates the debug messages seen on the initiator, where the notify payload sent from the responder can be seen, indicating that no proposal was chosen.
IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 10.10.10.1:500/To
10.10.50.5:500/VRF i0:f0]
Initiator SPI : D007314789FFB97A - Responder SPI : 63922088658CEA7B Message id: 0
IKEv2 IKE_SA_INIT Exchange RESPONSE
Payload contents:
NOTIFY(NO_PROPOSAL_CHOSEN)
To rectify this situation, ensure that the IKEv2 proposals are compatible between both peers. The show crypto ikev2 proposal command can be used to verify the configured IKEv2 proposals. The IKEv2 proposal can be changed by configuring the relevant cryptographic algorithms within the IKEv2 proposal. The following example illustrates that the incorrect attributes are configured:
Router#show crypto ikev2 proposal
IKEv2 proposal: default
Encryption : AES-CBC-128
Integrity : SHA512
PRF : SHA512
DH Group : DH_GROUP_768_MODP/Group 1
The following example illustrates that the relevant IKEv2 proposal is then changed to include the correct integrity and Diffie-Hellman group.
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router (config)#crypto ikev2 proposal default
Router (config-ikev2-proposal)#integrity sha256
Router (config-ikev2-proposal)#group 14
Router #show crypto ikev2 proposal
IKEv2 proposal: default
Encryption : AES-CBC-128
Integrity : SHA256
PRF : SHA256
DH Group : DH_GROUP_2048_MODP/Group 14
Within the SA_INIT exchange, cookie notification can be used. If a responder replied with the cookie notification payload and the initiator did not understand this for some reason, then connectivity would fail.
Note
The IKEv2 redirect feature can be used in the SA_INIT exchange; having a client redirected to another VPN gateway should occur only if supported by both the initiator and the responder.
Troubleshooting the IKE_AUTH Exchange
Within the IKE_AUTH exchange, the peers authenticate each other. Each peer also exchanges traffic selectors that are negotiated to form the IPsec Security Association.
Within IKEv2, the authentication is bidirectional and not negotiated. Both the initiator and responder must be configured as to what authentication method is used. This requires that the method the initiator uses to authenticate itself and the method that is set on the responder must match.
The following example illustrates the syslog messages that are generated on a responder when debugging an IKEv2 negotiation where the authentication method proposed by the initiator does not match what is configured on the responder.
IKEv2:(SESSION ID = 32,SA ID = 2):Get peer's authentication method
IKEv2:(SESSION ID = 32,SA ID = 2):: Peer's configured auth method mis-matches with
proposed auth method
IKEv2:(SESSION ID = 32,SA ID = 2):Verification of peer's authentication data
FAILED
IKEv2:(SESSION ID = 32,SA ID = 2):Sending authentication failure notify
IKEv2:(SESSION ID = 32,SA ID = 2):Building packet for encryption.
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
To rectify this situation, the correct authentication method needs to be configured on both the initiator and the responder. This can be achieved by the authentication command within the IKEv2 profile.
Authentication
There are three methods of authentication when IKEv2 is used: digital signature using RSA or ECDSA, PSK, and EAP.
Troubleshooting RSA or ECDSA Authentication
As authentication with certificates involves a number of components, we shall recap how this form of authentication occurs. Having an understanding of the traffic flows provides a step-by-step method to follow when troubleshooting.
Figure 14-1 illustrates the decisions that occur when digital signatures are used as the authentication method.
1. Assuming that both peers are using RSA or EC digital signatures to prove their authenticity, within the IKE_SA_INIT exchange the responder gives a hint to the initiator about the trustpoints it supports. This is achieved by sending a SHA-1 hash of the public key of the certificate authority contained within the CERTREQ payload.
2. When the initiator begins the IKE_AUTH exchange, if the responder indicated a PKI that is configured within the IKEv2 profile, this PKI is selected; otherwise the first trustpoint configured within the IKEv2 profile is used.
3. The initiator uses the private key that relates to the certificate of the selected trustpoint that is used to sign the AUTH payload. The AUTH payload is sent to the peer, along with the associated certificate, in the CERT payload. The CERTREQ payload is also included by the initiator in the IKE_AUTH exchange; this allows a peer with multiple trustpoints configured to select the correct one.
4. The responder will check that it has a valid trustpoint configured in the IKEv2 profile and for the certificate presented. The responder will check the digital signature of the presented certificate to ensure that the certificate authority signed this.
5. The certificate’s not-before and not-after dates are checked to ensure that the certificate is being used within the permitted time. If revocation checking is enabled, then a Certificate Revocation List (CRL) will be queried, or an Online Certificate Status Protocol (OCSP) request will be sent. The AUTH payload is verified using the public key contained within the certificate to validate that it was signed using the corresponding private key.
6. Assuming that the authentication was successful, the responder will itself prove its authenticity by signing the AUTH payload; the trustpoint used will be selected from what the initiator indicated it supports within the CERTREQ payload. The AUTH payload is then sent in the IKE_AUTH response message.
7. On receipt of this payload, the initiator will check that it has a valid trustpoint configured in the IKEv2 profile and for the certificate presented. The initiator will check the digital signature of the presented certificate to ensure that the certificate authority signed this.
8. The certificate’s not-before and not-after dates are checked to ensure that the certificate is being used within the permitted time. If a revocation checking is enabled then a Certificate Revocation List (CRL) will be queried, or an Online Certificate Status Protocol (OCSP) request is sent. The AUTH payload is verified using the public key contained within the certificate to validate that it was signed using the corresponding private key.
With Cisco IOS, the IKE subsystem will make calls to the PKI subsystem to perform the previously described cryptographic processes. Once this process has been completed, the PKI subsystem will signal to IKE if it was successful or not. We will now examine potential issues that can occur within the previously described steps.
When the responder receives an IKE_SA_INIT, it will reply with the CERTREQ to inform the peer of the supported trustpoints. If the initiator is not enrolled in any of these, then the message in the following example will be seen on the initiator if IKEv2 debugging (debug crypto ikev2) is enabled. The responder will then use the certificate and related key material of first configured trustpoint within the IKEv2 profile to sign the authentication payload.
IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_SA_INIT message
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
IKEv2:% Received cert hash is invalid, using configured trustpoints from profile
for signing
The configured trustpoints can be shown with the show crypto pki trustpoints [ status | label [status] ] command.
The following example illustrates a correctly configured trustpoint, with corresponding RSA keypair.
FlexClient#show crypto pki trustpoints PKI status
Trustpoint PKI:
Issuing CA certificate configured:
Subject Name:
cn=ca.cisco.com
Fingerprint MD5: 8E628B19 7251C76A 543A76B0 8961EAEB
Fingerprint SHA1: 46D32784 A5F7CF3F 2CC86337 6B0221E5 A5C5B4CD
Router General Purpose certificate configured:
Subject Name:
hostname=flexclient.cisco.com
Fingerprint MD5: DA640991 D2BA5328 F05594AE 09E2E235
Fingerprint SHA1: 67C92EAB 24C7CDA1 B28E2696 1C5593D2 A23F0AB2
State:
Keys generated ............. Yes (General Purpose, non-exportable)
Issuing CA authenticated ....... Yes
Certificate request(s) ..... Yes
To view the local certificates, use the show crypto pki certificates [ trustpoint-name [verbose] ] command.
The following example illustrates viewing two certificates that are associated with a trustpoint. The local certificate that is used to authenticate the device along with the CA certificate is shown. This is a very handy command for verifying that the certificate has not expired.
FlexClient#show crypto pki certificates PKI
Certificate
Status: Available
Certificate Serial Number (hex): 02
Certificate Usage: General Purpose
Issuer:
cn=ca.cisco.com
Subject:
Name: flexclient.cisco.com
hostname=flexclient.cisco.com
Validity Date:
start date: 19:03:45 JST Dec 21 2015
end date: 19:03:45 JST Dec 21 2016
Associated Trustpoints: PKI
Storage: nvram:s1#PKI.cer
CA Certificate
Status: Available
Certificate Serial Number (hex): 01
Certificate Usage: Signature
Issuer:
cn=ca.cisco.com
Subject:
cn=ca.cisco.com
Validity Date:
start date: 19:02:16 JST Dec 21 2015
end date: 19:02:16 JST Dec 21 2018
Associated Trustpoints: 20 PKI
Storage: nvram:s1#1CA.cer
If the verbose keyword is used, more detailed output can be viewed. This command will show the key usage, which indicates for what purpose the key within the certificate can be used. The following example illustrates viewing the key usage fields of a certificate.
FlexClient#show crypto pki certificates verbose
Certificate
Status: Available
Version: 3
Certificate Serial Number (hex): 02
...
X509v3 extensions:
X509v3 Key Usage: A0000000
Digital Signature
Key Encipherment
X509v3 Subject Key ID: 193A5E31 B1B5DECB C1172311 A61FEF7E E049CED0
When the Cisco AnyConnect client is used, name verification is performed on the server certificate. The following rules are applied for the purposes of IPsec name verification:
If a Subject Alternative Name extension is present with relevant attributes, name verification is performed solely against the Subject Alternative Name. Relevant attributes include DNS Name attributes for all certificates, and additionally include IP address attributes if the connection is being performed to an IP address.
If a Subject Alternative Name extension is not present or is present but contains no relevant attributes, name verification is performed against any common name attributes found in the subject of the certificate.
If a certificate uses a wildcard for the purposes of name verification, the wildcard must be in the first (left-most) subdomain only, and additionally it must be the last (right-most) character in the subdomain. Any wildcard entry not in compliance is ignored for the purposes of name verification.
Certificate Attributes
Key Usage is an X509v3 certificate extension that is described in RFC 2459, “Internet X.509 Public Key Infrastructure Certificate and CRL Profile.” The key usage extension defines the purpose (for example, encipherment, signature, certificate signing) of the key contained in the certificate.
RFC 4945, “The Internet IP Security PKI Profile of IKEv1/ISAKMP, IKEv2, and PKIX,” describes the Key Usage attribute when using IKEv2.
According to the following extract from RFC 4945, it is standard practice to set the key usage to digital signature; however, Cisco IOS will also set key encipherment. It is recommended to set both attributes.
Since we are talking about using the public key to validate a signature, if the
KeyUsage extension is present, then at least one of the digitalSignature or the
nonRepudiation bits in the KeyUsage extension MUST be set (both can be set as
well). It is also fine if other KeyUsage bits are set.
The following flow describes the logic for the certificate validation when key usage is employed.
If no KU extension is present, continue.
If a KU is present but doesn’t include digitalSignature or nonrepudiation, reject the certificate.
If none of the above is true, continue.
Note
When IKEv2 is used with Cisco AnyConnect connections require that if a server certificate contains Key Usage, the attributes must contain digitalSignature AND (KeyAgreement OR KeyEncipherment).
Extended Key Usage (EKU) is another X509v3 extension defined in RFC 3280, “Internet X.509 Public Key Infrastructure Certificate and CRL Profile.” This field indicates one or more purposes for which the certified public key may be used, in addition to or in place of the basic purposes indicated in the key usage extension field.
When Cisco AnyConnect is used, if the server certificate contains an EKU, the attributes must contain serverAuth (for SSL and IPsec) or ikeIntermediate (for IPsec only). Note that server certificates are not required to have a KU or an EKU to be accepted.
For certificate-based authentication, the FlexVPN server and Microsoft Windows 7 client certificates must have an Extended Key Usage (EKU) field as follows:
For the client certificate, EKU field = client authentication certificate.
For the server certificate, EKU field = server authentication certificate.
Debugging Authentication Using PKI
The following example illustrates the IKEv2 debug messages that are seen on the responder when the initiator attempts to validate itself by using a certificate that isn’t configured on any trustpoints. We see that the presented certificate cannot be validated against any locally configured trustpoints.
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
IKEv2:% Received cert hash is invalid, using configured trustpoints from profile
for signing
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Getting cert chain for the trustpoint 1
IKEv2:(SA ID = 1):[PKI -> IKEv2] Getting of cert chain for the trustpoint PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 1,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain FAILED
IKEv2:(SESSION ID = 1,SA ID = 1):: Platform errors
IKEv2:(SESSION ID = 1,SA ID = 1):Verify cert failed
IKEv2:(SESSION ID = 1,SA ID = 1):Verification of peer's authentication data FAILED
IKEv2:(SESSION ID = 1,SA ID = 1):Sending authentication failure notify
IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
When an initiator attempts to authenticate itself, the trustpoint that is used must be configured within the IKEv2 profile using the pki trustpoint command. If no trustpoints are configured, then authentication will fail.
If the client has authenticated the trustpoint but not enrolled to it (and obtained a certificate), then authentication will fail. Even if the trustpoint is configured within the IKEv2 profile, it can only be used once enrolled.
Assuming that the initiator has a trustpoint configured, it will look to sign the AUTH payload with the private key related to the certificate contained in the trustpoint; this is then sent to the peer.
Certificate Expiry
Even if the local certificate has expired, it is still used for authentication. It is the responsibility of the receiver of the authentication payload to validate that the certificate is not expired.
If the responder should receive a certificate from the initiator that is not within the notBefore and notAfter lifetimes (expired), then IKE will fail. If IKEv2 debugging is enabled, the log messages show in the following example will be seen.
IKEv2:Current time is more than cert validity time
IKEv2:(SESSION ID = 32,SA ID = 1):Failed to set IKE policy: Failed to build
certificate payload
IKEv2:(SESSION ID = 32,SA ID = 1):Verification of peer's authentication data
FAILED
IKEv2:(SESSION ID = 32,SA ID = 1):Sending authentication failure notify
IKEv2:(SESSION ID = 32,SA ID = 1):Building packet for encryption.
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
To ensure that the time is accurate, use Network Time Protocol (NTP). The command show ntp associations will list all configured NTP servers.
The PKI functions for rollover and re-enrollment can be affected if there is no authoritative time source. For this reason, you should always configure authenticated NTP on all of the routers that perform PKI functions.
The following example illustrates a client with an NTP server configured and synchronized. The asterisk (*) denotes that the peer is synced.
FlexClient#show ntp associations
address ref clock st when poll reach delay offset disp
*~10.10.10.1 127.127.1.1 8 17 64 1 0.000 0.000 189.44
* sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured
NTP has a number of debugging options, including clock adjustment behavior. The following example illustrates NTP packet debugging, in which NTP packets can be seen being sent and received to/from the NTP server.
FlexClient#debug ntp packet
NTP packets debugging is on
NTP message sent to 10.10.10.1, from interface 'Ethernet0/0' (10.10.20.3).
NTP message received from 10.10.10.1 on interface 'Ethernet0/0' (10.10.20.3).
NTP uses the User Datagram Protocol (UDP) and uses port 123 for communication. In the following example, we see that the NTP server is not synchronized. After NTP packet debugging is enabled, we see that two-way communication fails between the client and the NTP server.
FlexClient#show ntp associations
address ref clock st when poll reach delay offset disp
~10.10.10.1 .INIT. 16 - 64 0 0.000 0.000 15937.
* sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured
FlexClient#debug ntp packet
NTP packets debugging is on
NTP message sent to 10.10.10.1, from interface 'NULL' (0.0.0.0).
Note
If a certificate expires and connectivity is still required, a trustpoint can be configured to ignore expired certificates by entering the match certificate command with the allow expired-certificates option within the trustpoint configuration. The following example illustrates the application of a certificate map to allow expired certificates. The configuration for the certificate map is not shown. Note that this is contrary to security best practice, but it may be required in certain situations.
crypto pki trustpoint pki
match certificate cert_map allow expired-certificate
Matching Peer Using Certificate Maps
If certificate maps are used to anchor a presented certificate to an IKEv2 profile, the attributes within the certificate must be matched within the certificate map.
The following example illustrates the output from IKEv2 debugging of a successful connection when a certificate map is used to anchor a client connection to an IKEv2 profile.
IKEv2:(SESSION ID = 39,SA ID = 1):Searching policy based on peer's identity
'hostname=flexclient.cisco.com' of type 'DER ASN1 DN'
IKEv2:Optional profile description not updated in PSH
IKEv2:Searching Policy with fvrf 0, local address 10.10.10.1
IKEv2:Using the Default Policy for Proposal
IKEv2:Found Policy 'default'
IKEv2:Found matching IKEv2 profile 'default'
The following example illustrates output from IKEv2 debugging when an unsuccessful connection occurs when a certificate map is used to anchor a client connection to an IKEv2 profile. Note the “failed to locate an item in the database” message.
IKEv2:(SESSION ID = 40,SA ID = 1):Searching policy based on peer's identity
'hostname=flexclient.cisco.com' of type 'DER ASN1 DN'
IKEv2:% IKEv2 profile not found
IKEv2:(SESSION ID = 40,SA ID = 1):: Failed to locate an item in the database
IKEv2:(SESSION ID = 40,SA ID = 1):Verification of peer's authentication data
FAILED
IKEv2:(SESSION ID = 40,SA ID = 1):Sending authentication failure notify
IKEv2:(SESSION ID = 40,SA ID = 1):Building packet for encryption.
The certificate map can be viewed within the running configuration by using the command shown in the following example.
Router#show running-config | section certificate map
crypto pki certificate map certmap 10
subject-name eq hostname=flexclient.cisco.com
The IKEv2 profile must have a match statement within it; this is indicated by the message seen in the following example when there is an attempt to generate a new IKEv2 profile.
Router(config)#crypto ikev2 profile test
IKEv2 profile MUST have:
1. A local and a remote authentication method.
2. A match identity or a match certificate or match any statement.
Certificate Revocation
If revocation checking is enabled for the trustpoint, then the IKEv2 subsystem will make calls to the PKI subsystem. The PKI component will perform the revocation checking, which involves downloading a CRL or performing an OCSP request, depending on the configuration.
The following example illustrates a certificate being received when IKEv2 and PKI validation debugging has been enabled with the commands debug crypto ikev2 and debug crypto pki validation respectively.
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): 'pki'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Getting cert chain for the trustpoint pki
IKEv2:(SA ID = 1):[PKI -> IKEv2] Getting of cert chain for the trustpoint PASSED
IKEv2:(SESSION ID = 5,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 5,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain FAILED
IKEv2:(SESSION ID = 5,SA ID = 1):
%PKI-6-PKI_CRL_DOWNLOADED: CRL download notification sent for Issuer = cn=sub.
IKEv2:(SESSION ID = 5,SA ID = 1):Save pubkey
IKEv2:(SESSION ID = 5,SA ID = 1):Verify peer's authentication data
IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Verify signed authentication data
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] Verification of signed authentication
data PASSED
It can be seen that although the validation of the certificate chain fails because the CRL was not downloaded, the CRL is then downloaded and authentication is successful.
The following example shows the debug messages when there is an attempt to download a CRL; however the validation fails, and its failure generates an authentication failure notify payload.
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Getting cert chain for the trustpoint pki
IKEv2:(SA ID = 1):[PKI -> IKEv2] Getting of cert chain for the trustpoint PASSED
IKEv2:(SESSION ID = 8,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 8,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain FAILED
IKEv2:(SESSION ID = 8,SA ID = 1):
Router#
IKEv2:: Negotiation context locked currently in use
Router#
%PKI-3-SOCKETSELECT: Failed to select the socket.
IKEv2:(SESSION ID = 8,SA ID = 1):Verify cert failed
IKEv2:(SESSION ID = 8,SA ID = 1):Verification of peer's authentication data FAILED
IKEv2:(SESSION ID = 8,SA ID = 1):Sending authentication failure notify
IKEv2:(SESSION ID = 8,SA ID = 1):Building packet for encryption.
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
Further investigation is required to determine why the verification failed. PKI debugging can be enabled to determine why the PKI component failed. The following example illustrates the messages generated after PKI debugging has been enabled.
CRYPTO_PKI: (A0011) Requesting CRL from SCEP
CRYPTO_PKI: locked trustpoint pki, refcount is 2
CRYPTO_PKI: http connection opened via VRF OUTSIDE
CRYPTO_PKI: Sending HTTP message
CRYPTO_PKI: Reply HTTP header:
HTTP/1.0
Host: 10.10.30.3
Router#
IKEv2:: Negotiation context locked currently in use
Router#
%PKI-3-SOCKETSELECT: Failed to select the socket.
It can be seen that an HTTP request is sent out, but the reply is not received. The message “failed to select the socket” indicates this.
To investigate further, TCP debug messages are enabled with the debug ip tcp command. The following example shows the debug messages generated. The messages indicate that the TCP socket is being opened, however the TCP SYN does not receive a reply.
tcp0: O CLOSED 10.10.30.3:80 10.10.10.1:48085 seq 3276603876
OPTS 4 SYN WIN 4128
tcp0: R SYNSENT 10.10.30.3:80 10.10.10.1:48085 seq 3276603876
OPTS 4 SYN WIN 4128
TCP0: timeout #1 - timeout is 4000 ms, seq 3276603876
TCP: (48085) -> 10.10.30.3(80)
In this case a firewall was blocking access to the CRL Distribution Point (CDP). Other explanations that would cause the same symptoms are; the incorrect CDP could be configured in a certificate, or the HTTP daemon on the server hosting the CDP could be malfunctioning. In these scenarios, debugging HTTP would be more prudent.
The show crypto pki counters command can be used to see counters relating to PKI behavior on the device. In the following example, it can be seen that a number of CRL fetches failed.
R1#show crypto pki counters
PKI Sessions Started: 50
PKI Sessions Ended: 50
PKI Sessions Active: 0
Successful Validations: 4
Failed Validations: 46
Bypassed Validations: 0
Pending Validations: 0
CRLs checked: 56
CRL - fetch attempts: 49
CRL - failed attempts: 45
CRL - rejected busy fetching: 1
AAA authorizations: 0
The output from debugging a successful authentication using PKI is shown in the following example. Note that the “PASSED” message indicates that the device successfully performed the cryptographic functions.
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): 'pki'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Getting cert chain for the trustpoint pki
IKEv2:(SA ID = 1):[PKI -> IKEv2] Getting of cert chain for the trustpoint PASSED
IKEv2:(SESSION ID = 200,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 200,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain PASSED
IKEv2:(SESSION ID = 200,SA ID = 1):Save pubkey
IKEv2:(SESSION ID = 200,SA ID = 1):Verify peer's authentication data
IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Verify signed authentication data
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] Verification of signed authentication
data PASSED
Trustpoint Configuration
When digital signatures are used for authentication, one or more trustpoints can be defined within the IKEv2 profile. This is used to define which trustpoint should be used for authentication.
A trustpoint can be configured just for signing the local authentication payload by including the sign keyword, as illustrated in the following example.
Router(config-ikev2-profile)#pki trustpoint 1 sign
Or it can be configured just for verification of the remote peer by including the verify keyword, as illustrated here:
Router(config-ikev2-profile)#pki trustpoint 1 verify
If neither keyword is included, both signing and verification are performed with the same trustpoint.
Trustpoint Selection
When only a single trustpoint is configured on a device, there can be no discrepancy between peers as to which trustpoint is used. If more than one trustpoint is configured, then the CERTREQ payload will be used to give an indication of what PKI to use.
The following example illustrates the debug messages seen on an initiator when the received CERTREQ payload within the IKE_SA_INIT message did not match any locally configured trustpoints in the IKEv2 profile. In this instance, the first trustpoint from the IKEv2 profile will be used.
On the initiator (received SA_INIT)
IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_SA_INIT message
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
IKEv2:% Received cert hash is invalid, using configured trustpoints from profile
for signing
The following example illustrates the debug messages seen on a responder when the received CERTREQ payload within the IKE_AUTH message did not match any locally configured CA certificates.
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Retrieving trustpoint(s) from received
certificate hash(es)
IKEv2:(SA ID = 1):[PKI -> IKEv2] Retrieved trustpoint(s): NONE
IKEv2:% Received cert hash is invalid, using configured trustpoints from profile
for signing
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Getting cert chain for the trustpoint 1
IKEv2:(SA ID = 1):[PKI -> IKEv2] Getting of cert chain for the trustpoint PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 1,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain FAILED
IKEv2:(SESSION ID = 1,SA ID = 1):: Platform errors
IKEv2:(SESSION ID = 1,SA ID = 1):Verify cert failed
IKEv2:(SESSION ID = 1,SA ID = 1):Verification of peer's authentication data FAILED
IKEv2:(SESSION ID = 1,SA ID = 1):Sending authentication failure notify
IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
NOTIFY(AUTHENTICATION_FAILED)
On receipt of the IKE_AUTH packet, the responder will match the exchange to an IKEv2 profile and then perform cryptographic checks on the presented certificate.
If there are multiple IKEv2 profiles, ensure that there are no overlapping match statements within profiles. An IKEv2 profile that is associated with a tunnel interface should be used only by that tunnel interface; however, when a device is acting as a responder, the selection of the IKEv2 profile is based on the match condition within the IKEv2 profile. The profile that is configured on the tunnel interface must match the profile that is selected using the peer’s IKE identity. If this does not match, the negotiation will be aborted.
Should both IKEv2 profiles have overlapping match statements and the incorrect profile is selected, the debug messages within the following example will be seen. In this case, two tunnel interfaces with different tunnel destination addresses and IPsec and IKEv2 profiles are configured. Although IKE traffic is received from the correct tunnel destination IP address, the incorrect IKEv2 profile is matched (which is not applied to the IPsec profile protecting the tunnel interface), as the IKEv2 profile is used to match the peer and determine the protection the other IKEv2 profile will be used.
map_db_check_isakmp_profile profile did not match
map_db_find_best did not find matching map
IPSEC(ipsec_process_proposal): proxy identities not supported
IKEv2:(SA ID = 1):There was no IPSEC policy found for received TS
IKEv2:(SA ID = 1):Sending TS unacceptable notify
The IKE system will make calls to the PKI system; from a simplistic perspective the PKI system will validate that the certificate used has not expired and check that the AUTH payload was cryptographically signed with the private key that relates to the public key contained within the certificate.
Pre-Shared Key
When pre-shared keys are used for authentication, each peer can use a shared secret to authenticate itself to the peer, or the peer to itself. The keys are held within the keyring, which is a database of identities with associated keys. The identity can be populated with the same value as an IKE identity, but as the following section describes, this is not mandatory. The keys must match between peers. For the initiator, the identity within the keyring must be configured with either an IP address or hostname. The responder must have the identity within the keyring set to IP address or IKE identity.
The keyring can be configured locally or can be obtained from RADIUS using AAA.
The local or remote keywords can be included when the key is defined. This allows asymmetrical keys to be defined with different keys on each peer. If the configuration uses asymmetrical keys, ensure that the local key on one peer matches the remote key on the other peer.
The only method to verify that the local keyring has been configured correctly is the show running-configuration command. In the following example we see a single IKEv2 keyring configured.
Routert#show running-config | section ikev2 keyring
crypto ikev2 keyring key1
peer 10.10.10.1
address 10.10.10.1
pre-shared-key cisco
If RADIUS is used, a symmetrical key can be defined on the RADIUS server, or asymmetrical keys can be configured. Table 14-6 illustrates the different methods of configuration for pre-shared keys when RADIUS is used.
If a keyring that is configured within an IKEv2 profile is used and does not have the identity of the peer defined, the debug message shown in the following example will be seen.
IKEv2:% Getting preshared key from profile keyring key1
IKEv2:% key not found.
IKEv2:Failed to initiate sa
If an IP address is used as the IKEv2 identity, as the identity within the keyring can be configured as a subnet, ensure that the presented identities will always reside within one of the configured subnets.
When AAA is used with RADIUS to obtain the pre-shared key, the RADIUS server must be reachable, and the username used as the identity for the lookup must be configured on the RADIUS server. The following section describes troubleshooting AAA with RADIUS.
The name mangler function can be used with AAA to obtain the identity. If the name mangler is used, be sure to verify that it is correctly configured to extract the relevant part of the identity that is used as the lookup. The following example illustrates the IKEv2 debug messages where a presented identity type of flexclient.cisco.com of type FQDN is seen. The hostname (flexclient) is extracted and then used as the lookup. In this scenario, ensure that there is an account with the username flexclient on the RADIUS server.
IKEv2:(SESSION ID = 12,SA ID = 1):Peer's authentication method is 'PSK'
IKEv2:(SESSION ID = 12,SA ID = 1):Get peer's preshared key for flexclient.cisco.
com
IKEv2:(SA ID = 1):[IKEv2 -> AAA] Password request sent
IKEv2:IKEv2 local AAA - user author request for 'flexclient'
IKEv2:(SA ID = 1):[AAA -> IKEv2] Received password response
Cisco IOS provides the ability to configure the pre-shared key as a hexadecimal value. If this method is used, ensure that the corresponding peer has the same shared secret configured. This can be verified by using an ASCII-to-HEX conversion tool.
The following example illustrates a configuration in which one peer has a pre-shared key configured using the ASCII value of “cisco” and its peer has the equivalent hexadecimal value.
Router1#show run | section ikev2 keyring
crypto ikev2 keyring key1
peer 10.10.10.1
address 10.10.10.1
pre-shared-key cisco
Router2#show run | section ikev2 keyring
crypto ikev2 keyring key1
peer 10.10.10.100
address 10.10.10.100
pre-shared-key hex 636973636f
Extensible Authentication Protocol (EAP)
It is very common for architectures of large-scale deployments of VPN clients to use a backend database to manage the user accounts. In this case, EAP is a common authentication method that allows a RADIUS server to store the client credentials and perform the authentication. When EAP is used, the FlexVPN server will pass through the client credentials and allow the EAP server on the RADIUS device to make the accept or reject decision based on the attributes supplied by the client.
When EAP is used, the client must authenticate the VPN headend (and responder) using digital signatures, so a certificate must be configured on the FlexVPN server. The FlexVPN server certificate must have an Extended Key Usage (EKU) of server-auth.
The EKU can be verified by examining the certificate. The following example illustrates how to verify the EKU with the show crypto pki certificate verbose command.
Router#show crypto pki certificates verbose
...
X509v3 extensions:
X509v3 Key Usage: 86000000
Digital Signature
Key Cert Sign
CRL Signature
X509v3 extensions:
X509v3 Extended Key Usage:
TLS Web Server Authentication
TLS Web Client Authentication
When IKEv2 is used with EAP, the components are the client, the FlexVPN server, and the RADIUS server. The client and FlexVPN server will communicate over IKEv2; however, EAP messages are exchanged within the IKE_AUTH exchange. The FlexVPN server and RADIUS server communicate using RADIUS, where the EAP messages are transported within the EAP Message attribute.
Figure 14-2 illustrates the components of the EAP exchange, with the protocol flows.
The RADIUS server configuration can be verified by examining the running configuration, as shown in the following example.
Router#show run | section radius
aaa group server radius radius_group
server-private 10.10.20.1 auth-port 1812 acct-port 1813 key cisco123
aaa authentication login rad1 group radius_group
RADIUS connectivity can be verified with the test aaa command, as illustrated in the following example. RADIUS debugging was enabled to highlight the connectivity to the RADIUS server and verify that the username (test) and password (lab) presented to the RADIUS server was successfully authenticated.
router#test aaa group radius_group test lab new-code
RADIUS/ENCODE(00000000):Orig. component type = INVALID RADIUS/ENCODE(00000000):
dropping service type, "radius-server attribute 6 on-for-login-auth" is off
RADIUS(00000000): Config NAS IP: 10.10.10.1
RADIUS(00000000): sending
RADIUS/ENCODE: Best Local IP-Address 10.10.10.1 for Radius-Server 10.10.20.1
RADIUS(00000000): Send Access-Request to 10.10.20.1:1812 id 1812/1, len 50
RADIUS: authenticator CA DB F4 9B 7B 66 C8 A9 - D1 99 4E 8E A4 46 99 B4
RADIUS: User-Password [2] 18 *
RADIUS: User-Name [1] 6 "test"
RADIUS: NAS-IP-Address [4] 6 10.10.10.1
RADIUS: Received from id 1812/1 10.10.20.1:1812, Access-Accept, len 44
Assuming that the client initiates the IKEv2 session and the IKE_SA_INIT exchange was successful, the EAP messages will be exchanged in the IKE_AUTH exchange. To denote that EAP is being used, the client excludes the in the first IKE_AUTH request, at this point the FlexVPN server will prove its own identity by sending its own AUTH payload.
The following example illustrates the output that was generated from debugging IKEv2 on a FlexVPN client when EAP is used; it shows the IKE_AUTH request without the AUTH payload.
IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDi CERTREQ NOTIFY(HTTP_CERT_LOOKUP_SUPPORTED) CFG SA TSi TSr
NOTIFY(INITIAL_CONTACT) NOTIFY(USE_TRANSPORT_MODE) NOTIFY(SET_WINDOW_SIZE)
NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
If the client is Cisco IOS, then the IKE identity must not be configured as an IP address or Distinguished name (DN); in these cases the connection will fail, and the following debug message will be seen on the FlexVPN server.
IKEv2:Cannot use IP address as EAP identity
IKEv2:Received response from authenticator
IKEv2:(1): Authenticator sent NULL EAP message
IKEv2:(1): Extensible Authentication Protocol failed
IKEv2:(1): Auth exchange failed
IKEv2:(1): Abort exchange
Some third-party peers, such as Windows 7 and above, will send the identity as an IP address that will be rejected by the FlexVPN server. For peers that send the identity as an IP address, it is necessary to configure query-identity so that the headend sends an EAP Identity Request to the client during IKE_AUTH. The EAP identity is used later for authorization requests; note that even if we receive an EAP identity from the client, it can still be overridden by the EAP server once the EAP authentication has completed.
Figure 14-3 illustrates the behavior of IKEv2 on Cisco IOS without query-identity enabled. The IKE identity is used in the EAP response.
Figure 14-4 illustrates the behavior of IKEv2 on Cisco IOS with query-identity enabled. The IKE identity is not used in the EAP response. The FlexVPN server will query the client for the identity within an EAP request which is then used.
On receipt of the IKE_AUTH, the FlexVPN server will process the identity. The following example illustrates a debug message taken from a client using MS-CHAPv2 that shows the identity, which in hexadecimal is 636973636F, but in ASCII is “cisco.” The same identity is then used as the EAP User-Name.
IKEv2:Exchange type: IKE_AUTH, flags: INITIATOR
IKEv2:Message id: 0x2, length: 68
REAL Decrypted packet:
00 00 00 0E 02 3B 00 0A 01 63 69 73 63 6F
EAP Next payload: NONE, reserved: 0x0, length: 14
Code: response: id: 59, length: 10
Type: identity
The FlexVPN server will relay the identity to the RADIUS server within the Access-Request message. The following IKEv2 and RADIUS debug messages show a RADIUS request with the username of “cisco,” which was used as the IKEv2 identity.
IKEv2:(1): Processing EAP response
RADIUS/ENCODE(00000013):Orig. component type = VPN IPSEC
RADIUS: AAA Unsupported Attr: service-type [345] 4 1 RADIUS/
ENCODE(00000013): dropping service type, "radius-server attribute 6
on-for-login-auth" is off
RADIUS(00000013): Config NAS IP: 0.0.0.0
RADIUS(00000013): Config NAS IPv6: ::
RADIUS/ENCODE(00000013): acct_session_id: 9
RADIUS(00000013): sending
RADIUS/ENCODE: Best Local IP-Address 10.10.10.10.1 for Radius-Server 10.10.10.100
RADIUS(00000013): Send Access-Request to 10.10.10.10.100:1812 id 1645/10, len 82
RADIUS: authenticator 10 16 B6 71 69 BE 3C 8F - D4 9B 77 8B AF 5A 9E ED
RADIUS: Service-Type [6] 6 Login [1] RADIUS:
Calling-Station-Id [31] 13 "10.10.20.100"
RADIUS: User-Name [1] 7 "cisco"
RADIUS: EAP-Message [79] 12
RADIUS: 02 3B 00 0A 01 63 69 73 63 6F [ ;cisco]
RADIUS: Message-Authenticato[80] 18 RADIUS: 4F DE 44 97 7B 04 0E A2 C9 FC 55
FA 72 2F 53 34 [ OD{Ur/S4]
RADIUS: NAS-IP-Address [4] 6 10.10.10.1
RADIUS(00000013): Sending a IPv4 Radius Packet
Once the Access-Challenge is received on the FlexVPN server from the RADIUS server, it will be relayed via IKEv2 back to the client. This is illustrated in the following example.
RADIUS: Received from id 1812/30 10.10.10.100:1812, Access-Challenge, len 98
RADIUS: authenticator 02 E3 BC 5C 7B EF DC 38 - 0C 14 12 4A 4F A0 6C 08
RADIUS: Framed-IP-Address [8] 6 172.16.1.1
RADIUS: Service-Type [6] 6 Framed [2]
RADIUS: Framed-Protocol [7] 6 PPP [1]
RADIUS: EAP-Message [79] 24
RADIUS: 01 3C 00 16 04 10 67 30 CF 35 9B 2E A0 56 85 F4 51 7B 0F 01 7F 6B
[ <g05.VQ{k]
RADIUS: Message-Authenticato[80] 18
RADIUS: E1 11 71 E1 DC 3A DB 30 89 3D 62 A8 F0 FC 63 FB [ q:0=bc]
RADIUS: State [24] 18
RADIUS: 16 47 B3 FF 16 7B B7 6A 0B 77 42 E6 F7 FC 0A 7B [ G{jwB{]
RADIUS(00000013): Received from id 1811/30
RADIUS/DECODE: EAP-Message fragments, 22, total 22 bytes
RADIUS: Constructed " ppp negotiate"
IKEv2:(SA ID = 1):[AAA -> IKEv2] Successful response received
IKEv2:Received response from authenticator
IKEv2:(SESSION ID = 5,SA ID = 1):Generating EAP request
IKEv2:(SESSION ID = 5,SA ID = 1):Building packet for encryption.
Payload contents:
EAP
IKEv2:(SESSION ID = 5,SA ID = 1):Sending Packet [To 10.10.10.10:500/From
10.10.10.1:500/VRF i0:f0]
Initiator SPI : CEC6599A1DDEDAD4 - Responder SPI : 7044B33369C2FC4A Message id: 2
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
ENCR
...
IKEv2:(SA ID = 1):[AAA -> IKEv2] Successful response received
IKEv2:Received response from authenticator
IKEv2:(SESSION ID = 5,SA ID = 1):Generating EAP request
IKEv2:(SESSION ID = 5,SA ID = 1):Building packet for encryption.
Payload contents:
EAP
If the timeout keyword is used on the FlexVPN server when EAP is configured as the remote authentication method, as the following example illustrates, then the FlexVPN server must receive a reply to the EAP messages sent in the IKE_AUTH exchange within the timeout period.
authentication remote eap timeout <seconds>
When a remote access client is used with EAP, user authorization is enabled by default if AAA user authentication is configured. When the command aaa authorization user is enabled with the cached attribute, the RADIUS attributes that are received for the user account during the authentication phase, such as the assigned IP address for the client, are automatically applied to the client’s connection.
With EAP, user authorization is only possible with an external RADIUS server and not a locally configured user.
Debugging should not be focused only on the FlexVPN server. If the client has the incorrect EAP type configured, then this can be viewed by the logs on the RADIUS server.
The connectivity between the FlexVPN server and the RADIUS server must be stable, and the shared secret that is used for RADIUS authentication must be configured on both the FlexVPN server and the RADIUS server.
Authorization
When a client requests attributes via a configuration payload, authorization must be enabled on the FlexVPN server. The attributes that are sent to the peer by a configuration payload can be obtained either locally via the IKEv2 authorization policy or via RADIUS.
The username used to make the lookup as to which policy should be applied can be obtained by using the peer’s IKE identity, or by using the name mangler function to extract a part of the presented identity.
When local authorization is used, all attributes are stored locally, and no external connectivity is required. The IKEv2 system will make calls to the AAA system, which will then return the attributes contained within the IKEv2 authorization policy.
When attributes that are used for authorization are obtained via RADIUS, the IKE system will query the AAA system, which in turn will make the RADIUS requests.
When authorization is enabled, the identity lookup is performed after the authentication phase has completed.
The following example shows IKEv2 and AAA authorization debug messages from a FlexVPN server using local authorization. This illustrates the authentication completing and the authorization occurring. The IKEv2 system makes a call to the AAA system, which returns the authorization response containing the attributes.
IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Verify signed authentication data
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] Verification of signed authentication
data PASSED
IKEv2:(SESSION ID = 17,SA ID = 1):Processing INITIAL_CONTACT
IKEv2:Using mlist default and username service1 for group author request
AAA/BIND(0000002B): Bind i/f
AAA/AUTHOR (0x2B): Pick method list 'default'
IKEv2:(SA ID = 1):[IKEv2 -> AAA] Authorisation request sent
IKEv2:(SA ID = 1):[AAA -> IKEv2] Received AAA authorisation response
IKEv2:(SESSION ID = 17,SA ID = 1):Received valid config mode data
If local authorization is used and RADIUS services are not required to be enabled (no aaa new-model and local aaa authorization list), the local keyword can be used within the IKEv2 profile configuration when configuring local authorization, as illustrated in the following example.
aaa authorization group cert list default default local
When authorization is used, if the client requests an IP address via configuration payload and the FlexVPN server cannot allocate this, then authentication will fail. The request for an IP address must be fulfilled, and so this request will cause the authentication to fail if an address cannot be allocated. The following example shows the debug message that will be seen if an IP address cannot be allocated.
IKEv2:Error constructing config reply
IKEv2:(SESSION ID = 31,SA ID = 1):Error in retrieving config mode data to send
IKEv2:(SESSION ID = 31,SA ID = 1):Fail to alloc IP addr
If authorization is used to apply raw configuration commands to a virtual-access interface, the syntax of the command is not checked before it is applied. If the syntax of a command is incorrect and would not be accepted by the CLI, the IKE session will fail. The following example illustrates this behavior. The configuration that is applied to a virtual-access interface via authorization is incorrect. The letter “a” is missing from the word “access-group,” which will result in the IKE session failing.
aaa attribute list service1
attribute type interface-config "ip ccess-group service1 in" protocol ip
This incorrect configuration being applied generates the following IKEv2 debug messages.
IKEv2:AAA clone request failed, error 7
IKEv2:(SESSION ID = 0,SA ID = 1):Error while applying per user attributes
Be mindful when applying configuration via AAA authorization to the virtual-access interface. The order of the commands must be the same as when applying manually. For example if a virtual-access interface is assigned to a VRF with the command vrf forwarding, this command must be applied before an IP address is applied to the interface, because the application of the vrf forwarding command will remove any configured IP addresses.
The following example shows which debug commands can be used to give a verbose output when AAA authorization is used.
debug aaa authorization
debug aaa attr
To determine the username that is used for the authorization lookup, AAA protocol debugging can be enabled with the debug aaa proto {local | radius} command. The following example illustrates how this debugging is enabled along with IKEv2 debugging; the authorization policy named “service1” is used.
IKEv2:(SA ID = 1):[IKEv2 -> AAA] Authorisation request sent
AAA/LOCAL/AUTHEN: starting
AAA/LOCAL/AUTHEN(17): authorizing crypto service1
IKEv2:(SA ID = 1):[AAA -> IKEv2] Received AAA authorisation response
IKEv2:(SESSION ID = 7,SA ID = 1):Received valid config mode data
IKEv2:Config data received:
RADIUS combines both authentication and authorization within the Access-Accept message. When using authentication and authorization, separate message exchanges will occur between the FlexVPN server and the RADIUS server. If user authorization is enabled for PSK or EAP, the cached keyword can be included within the aaa configuration to cache the RADIUS attributes received in the PSK retrieval or EAP success respectively. If it is required to apply attributes received from RADIUS to user authorization for PSK or EAP, then ensure that the cached keyword is included within the AAA authorization configuration, as illustrated in the following example.
crypto ikev2 profile default
aaa authorization user {psk|eap} cached
The creation of the virtual-access interface can be debugged with the command debug vtemplate. This command has a number of options, with cloning, event, and error being more relevant when it is necessary to debug issues when authorization is used to apply configuration.
The following example shows the messages that were generated when debug vtemplate cloning was enabled and a dynamically generated virtual-access interface was created; aaa authorization was enabled where the commands applied to the virtual-access interface can be seen.
VT[Vi1]:Applying config commands on process "VTEMPLATE Background Mgr" (373)
VT[Vi1]:default ip mtu 1420
VT[Vi1]:default ip tcp adjust-mss 1400
VT[Vi1]:default ip access-group subscriber1 in
VT[Vi1]:default ip verify unicast source reachable-via rx
VT[Vi1]:default ip unreachables
VT[Vi1]:default ip unnumbered loopback1
VT[Vi1]:default ip access-group service1 in
VT[Vi1]:end
VT[Vi1]:Remove cloneblk AAA from vaccess with vtemplate/AAA
VT[Vi1]:Unclone vaccess, 3 command(s) to be removed
VT[Vi1]:default tunnel protection ipsec profile default
VT[Vi1]:default tunnel vrf OUTSIDE
VT[Vi1]:default tunnel source Ethernet0/0
VT[Vi1]:end
Data Encryption
Once the IPsec Security Association is created and data is transmitted and protected by IPsec, any issues with data encryption and integrity protection will obviously involve IPsec.
Debugging IPsec
When an IPsec Security Association is created, peers will negotiate what to protect by using traffic selectors and how this will be protected by using transforms that are contained within a proposal.
Although IKEv2 can perform a negotiation in which the most specific selectors are used between peers, this is relevant only when using crypto maps and dVTI (tunnel mode ipsec) and not for GRE or VTI interfaces.
To verify the IPsec transform set that is used, use the show crypto ipsec transform-set command. The following example illustrates this command being executed on a device with two transform sets configured.
Router#show crypto ipsec transform-set
Transform set default: { esp-aes esp-sha-hmac }
will negotiate = { Transport, },
Transform set aes-sha256: { esp-aes esp-sha256-hmac }
will negotiate = { Transport, },
The transform set must be referenced within the IPsec profile that is used to protect the intended interface. The following example illustrates viewing a default profile with the relevant transform set applied.
Router#show crypto ipsec profile
IPSEC profile default
IKEv2 Profile: default
Security association lifetime: 4608000 kilobytes/3600 seconds
Responder-Only (Y/N): N
PFS (Y/N): Y
DH group: group14
Mixed-mode : Disabled
Transform sets={
aes-sha256: { esp-aes esp-sha256-hmac } ,
If the initiator sends an IPsec transform that does not match on the responder, the responder will fail to find a matching policy. The following example shows IKEv2 debugging output from a responder that received a transform that it did not support. It can be seen that in addition to the authentication data, a NO_PROPOSAL_CHOSEN notify message is sent in return.
IKEv2:(SESSION ID = 473,SA ID = 1):Set received config mode data
IKEv2:(SESSION ID = 473,SA ID = 1):Processing IKE_AUTH message
IKEv2:% DVTI create request sent for profile default with PSH index 1.
IKEv2:(SESSION ID = 473,SA ID = 1):
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state
to down
IKEv2:% DVTI Vi1 created for profile default with PSH index 1.
IKEv2:IPSec policy validate request sent for profile default with psh index 1.
IKEv2:(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal -
FAILED.
IKEv2:(SESSION ID = 473,SA ID = 1):Received Policies: : Failed to find a matching
policyESP: Proposal 1: 3DES SHA256 Don't use ESN
IKEv2:(SESSION ID = 473,SA ID = 1):Expected Policies: : Failed to find a matching
policy
IKEv2:(SESSION ID = 473,SA ID = 1):: Failed to find a matching policy
IKEv2:(SESSION ID = 473,SA ID = 1):Sending no proposal chosen notify
***
IKEv2:(SESSION ID = 473,SA ID = 1):Building packet for encryption.
Payload contents:
VID IDr CERT AUTH NOTIFY(NO_PROPOSAL_CHOSEN)
On the initiator, the IKEv2 debugs can be slightly confusing. The AUTH payload is received from the responder, with the notify payload indicating that no proposal was chosen. Because this is an IKE_AUTH payload, the IKEv2 profile is firstly matched, this can be seen where the authentication messages indicate successful authentication. After the successful authentication the IKE session is created; however, because no proposal was chosen, the IKE SA is deleted and a DELETE message is sent to the peer to indicate that the IKE session has been torn down.
IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 10.10.10.1:500/To
10.10.50.5:500/VRF i0:f0]
Initiator SPI : 58C298669EA6A95F - Responder SPI : 38F270737BD3EE39 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
VID IDr CERT AUTH NOTIFY(NO_PROPOSAL_CHOSEN)
IKEv2:(SESSION ID = 1,SA ID = 1):Process auth response notify
...
IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's policy
IKEv2:(SESSION ID = 1,SA ID = 1):Peer's policy verified
IKEv2:(SESSION ID = 1,SA ID = 1):Get peer's authentication method
IKEv2:(SESSION ID = 1,SA ID = 1):Peer's authentication method is 'RSA'
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Validating certificate chain
IKEv2:(SA ID = 1):[PKI -> IKEv2] Validation of certificate chain PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):Save pubkey
IKEv2:(SESSION ID = 1,SA ID = 1):Verify peer's authentication data
IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Verify signed authentication data
IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] Verification of signed authentication
data PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):Check for EAP exchange
IKEv2:Using mlist default and username default for group author request
IKEv2:(SA ID = 1):[IKEv2 -> AAA] Authorisation request sent
IKEv2:(SA ID = 1):[AAA -> IKEv2] Received AAA authorisation response
IKEv2:(SA ID = 1):[IKEv2 -> PKI] Close PKI Session
IKEv2:(SA ID = 1):[PKI -> IKEv2] Closing of PKI Session PASSED
IKEv2:(SESSION ID = 1,SA ID = 1):IKEV2 SA created; inserting SA into database.
SA lifetime timer (86400 sec) started
IKEv2:(SESSION ID = 1,SA ID = 1):Session with IKE ID PAIR (hostname=R1.cisco.com,
hostname=R5.cisco.com,cn=r5,o=cisco,ou=service1) is UP
IKEv2:IKEv2 MIB tunnel started, tunnel index 1
IKEv2:(SESSION ID = 1,SA ID = 1):Checking for duplicate IKEv2 SA
IKEv2:(SESSION ID = 1,SA ID = 1):No duplicate IKEv2 SA found
IKEv2:(SESSION ID = 1,SA ID = 1):Queuing IKE SA delete request reason: unknown
IKEv2:(SESSION ID = 1,SA ID = 1):Sending DELETE INFO message for IPsec SA [SPI:
0xAEF5E4BE]
IKEv2:(SESSION ID = 1,SA ID = 1):Building packet for encryption.
Payload contents:
DELETE
An IPsec Security Association will fail to be established if the tunnel interface types do not match. VTI does not use any form of encapsulation, and GRE interfaces use GRE; these are incompatible. The following example illustrates the IKEv2 debug output that was generated on a responder that was configured with a GRE interface when the initiator was configured with a VTI interface. The TS unacceptable message indicates that the interface types are incompatible. The authentication and authorization (if configured) will be successful; however, because the IPsec policy does not match the locally configured policy, the IKE session will fail.
IKEv2:(SESSION ID = 1067,SA ID = 1):Processing IKE_AUTH message
IKEv2:% DVTI create request sent for profile default with PSH index 1.
IKEv2:(SESSION ID = 1067,SA ID = 1):
%LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access1, changed state
to down
IKEv2:% DVTI Vi1 created for profile default with PSH index 1.
IKEv2:IPSec policy validate request sent for profile default with psh index 1.
IKEv2:(SA ID = 1):[IPsec -> IKEv2] Callback received for the validate proposal -
FAILED.
IKEv2:(SESSION ID = 1067,SA ID = 1):: There was no IPSEC policy found for
received TS
IKEv2:(SESSION ID = 1067,SA ID = 1):Sending TS unacceptable notify
The following example shows the output that was generated by using the debug crypto ikev2 packet debugging command. This output illustrates the IPsec proposal; the traffic selectors can be seen, which indicate all traffic being protected, and GRE is not used, as indicated by the proto id being 0.
SA Next payload: TSi, reserved: 0x0, length: 44
last proposal: 0x0, reserved: 0x0, length: 40
Proposal: 1, Protocol id: ESP, SPI size: 4, #trans: 3 last transform: 0x3,
reserved: 0x0: length: 12
type: 1, reserved: 0x0, id: AES-CBC
last transform: 0x3, reserved: 0x0: length: 8
type: 3, reserved: 0x0, id: SHA256
last transform: 0x0, reserved: 0x0: length: 8
type: 5, reserved: 0x0, id: Don't use ESN
TSi Next payload: TSr, reserved: 0x0, length: 24
Num of TSs: 1, reserved 0x0, reserved 0x0
TS type: TS_IPV4_ADDR_RANGE, proto id: 0, length: 16
start port: 0, end port: 65535
start addr: 0.0.0.0, end addr: 255.255.255.255
TSr Next payload: NOTIFY, reserved: 0x0, length: 24
Num of TSs: 1, reserved 0x0, reserved 0x0
TS type: TS_IPV4_ADDR_RANGE, proto id: 0, length: 16
start port: 0, end port: 65535
start addr: 0.0.0.0, end addr: 255.255.255.255
IPsec Anti-Replay
IPsec provides anti-replay protection against the duplication of encrypted packets or generation of traffic outside of the sequence number window by assigning a monotonically increasing sequence number to each encrypted packet. The receiving IPsec endpoint maintains a list of which packets it has already processed on the basis of these numbers with the use of a sliding window of all acceptable sequence numbers.
Figure 14-5 illustrates the behavior of the anti-replay window. Encapsulation Security Payload packets with a sequence number outside of the window will be discarded.
If a packet is received outside of the sliding window, the message illustrated in the following example is generated.
%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed connection id=#, sequence
number=#
This message can be generated because of malicious or nonmalicious behavior. If an attacker captured encrypted Encapsulation Security Payload packets and replayed these, the message would be generated. Likewise, a message would be generated if an attacker sent random Encapsulation Security Payload packets that could not be verified. The message can also occur if traffic comes out of order, if parallel paths exist on the transport network, or if QoS is applied and results in some packets being queued. Unequal packet processing paths can occur if large IPsec packets that require IP reassembly before decryption are delayed while smaller packets are passed in a system under load.
The following example illustrates a method for determining the IPsec Security Association that was the owner of a packet failing the anti-replay check. The connection ID is 771, with the packet having a sequence number of 1193.
%CRYPTO-4-PKT_REPLAY_ERR: decrypt: replay check failed
connection id=771, sequence number=1193
The IPsec Security Association Database can be viewed to determine which IPsec Security Association relates to connection ID 771.
Router#show crypto ipsec sa | in peer|conn id
current_peer 10.1.1.1 port 500
conn id: 771, flow_id: SW:771, sibling_flags 80000046, crypto-map:
Tunnel0-head-0
conn id: 772, flow_id: SW:772, sibling_flags 80000046, crypto-map:
Tunnel0-head-0
On devices running IOS-XE since 3.7 / 15.2(4)S, details of the IPsec Security Association are included in the syslog message, as shown below.
%IPSEC-3-REPLAY_ERROR: IPSec SA receives anti-replay error, DP Handle 6,
src_addr 10.10,10.1, dest_addr 10.10.10.2, SPI 0x1a2b3c4d
This message does not show the Encapsulation Security Payload sequence number, unlike IOS, which will show the sequence number. On the ASR if the sequence number is required, the ASR datapath packet tracing feature can be used to capture traffic that is transmitted over the dataplane.
The following illustrates the method to enable the ASR datapath packet tracing feature to determine the Encapsulation Security Payload sequence number of dropped traffic due to the anti-replay check.
Step 1. Set up the platform conditional debugging filter in order to match traffic from the peer device.
Router#debug platform condition ipv4 10.2.0.200/32 ingress
Router#debug platform condition start
Step 2. Enable packet tracing with the copy option in order to copy the packet header information. The size is set to 100 because only the Encapsulation Security Payload header is required to determine the sequence number.
Router#debug platform packet enable
Router#debug platform packet-trace packet 64
Router#debug platform packet-trace copy packet input l3 size 100
Step 3. When anti-replay errors are detected, use the packet trace buffer to identify the packet dropped due to replay, and the Encapsulation Security Payload sequence number can be found in the packet copied.
Router#show platform packet-trace summary
Pkt InputOutput State Reason
0 Gi1/0/0 Tu1 CONS Packet Consumed
1 Gi1/0/0 Tu1 CONS Packet Consumed
2 Gi1/0/0 Tu1 DROP 053 (IpsecInput)
3 Gi1/0/0 Tu1 DROP 053 (IpsecInput)
4 Gi1/0/0 Tu1 CONS Packet Consumed
Step 4. The previous output shows that packet numbers 2 and 3 were dropped, so they should be examined further.
Router#show platform packet-trace packet 2
Packet: 2 CBUG ID: 2
Summary
Input : GigabitEthernet1/0/0
Output: Tunnel1
State : DROP 053 (IpsecInput)
Timestamp : 3233497953773
Path Trace
Feature: IPV4
Source : 10.1.1.1
Destination : 10.1.1.2
Protocol: 50 (ESP)
Feature: IPSec
Action: DECRYPT
SA Handle : 3
SPI : 0x4c1d1e90
Peer Addr : 10.1.1.1
Local Addr: 10.1.1.2
Feature: IPSec
Action: DROP
Sub-code : 019 - CD_IN_ANTI_REPLAY_FAIL
Packet Copy In
45000428 00110000 fc329575 0a0200c8 0a010064 4c1d1e90 000000df 790aa252
e9951cd9 57024433 d97c7cb8 58e0c869 2101f1ef 148c2a12 f309171d 1b7a4771
d8868af7 7bae9967 7d880197 46c6a079 d0143e43 c9024c61 0045280a d57b2f5e
23f06bc3 ab6b6b81 c1b17936 98939509 7aec966e 4dd848d2 60517162 9308ba5d
The Encapsulation Security Payload sequence number has an offset of 24 that starts from the IP header, as emphasized in bold and italics in the previous output. In this particular example, the Encapsulation Security Payload sequence number for the dropped packet is 0xdf, which is 223 in decimal.
The default anti-replay window size in Cisco IOS is 64 packets; however this is configurable to a maximum of 1024 packets by using the command set security-association replay window-size {64 | 128 | 256 | 512 | 1024 } within the IPsec profile or by using the global command crypto ipsec security-association replay window-size {64 | 128 | 256 | 512 | 1024 }.
To disable the anti-replay check, use the command set security-association replay window-size disable within the IPsec profile or use the global command crypto ipsec security-association replay window-size disable.
If IPsec traffic is received that is dropped by the anti-replay function and the anti-replay window is set as the default 64 packets, it is worthwhile to set it to the maximum of 1024 packets. If dropped traffic continues, then take packet captures of all messages at both hosts to fully diagnose the cause. By viewing the packet captures, the packet flow can be examined to see what traffic is being sent out of order or injected into the flow.
Data Encapsulation
The GRE protocol is stateless and only encapsulates traffic within the GRE header. This results in very little room for error or the chance of untoward behavior occurring.
Mismatching GRE Tunnel Keys
If GRE tunnel keys are used and these are not aligned between peers, then the IPsec Security Association will be created. The GRE tunnel key is not negotiated, but it is a locally configured value that is added to the GRE header of the outgoing packets.
Once the IPsec Security Association forms, traffic will be sent into the tunnel interface, which is protected by IPsec. At the peer device when traffic is decrypted and de-encapsulated, if the tunnel keys do not match the packet will be discarded.
This can be a tricky issue to diagnose, for the IPsec Security Association will indicate that traffic has been encapsulated by the sender and de-encapsulated by the receiver; however, IP connectivity will fail over the tunnel interface.
A simple method to troubleshoot this issue is to view the configuration on the tunnel interface to ensure that the key is the same on both peers.
The output in the following example illustrates how to confirm the configured GRE tunnel key.
Router#show interface tunnel 1
Tunnel1 is up, line protocol is down
Hardware is Tunnel
...
interface <OK>
Tunnel protocol/transport GRE/IP
Key 0x2, sequencing disabled
Checksumming of packets disabled
Overlay Routing
Once an IPsec Security Association is successfully created, IP packets can flow over it. The Routing Information Base (RIB) must have entries to the destination prefixes for all traffic that is to be sent into the tunnel. The RIB can be populated manually with static routes, IKEv2 routing, or a dynamic routing protocol. Manually configuring routing is not dynamic unless it is integrated with enhanced object tracking; however it offers minimal overhead. IKEv2 routing allows for prefixes to be distributed between peers without the overhead that is required in a routing protocol. Routing protocols allow dynamic routing decisions to occur over the tunnel, but it come with the cost of running the routing protocol and keepalives.
Static Routing
When static routing is used with the next hop as the tunnel interface, the route will be installed in the RIB only when the tunnel is in the up/up state. The following example illustrates a configuration that will only install the 192.1680.0/16 prefix into the RIB only if the tunnel interface is in the up state.
ip route 192.168.0.0 255.255.0.0 Tunnel1
With this configuration, you are at the mercy of the tunnel; if IKEv2 DPD is not enabled, then if the tunnel peers lose connectivity the tunnel can be in the up state, even though there is no IP connectivity between peers.
IKEv2 Routing
If IKEv2 routing is used, then prefixes are exchanged in the initial IKE authentication exchange via the configuration payload. These prefixes stay statically installed in the RIB for the duration of the IKEv2 SA and can be changed only when a new IKEv2 session is created.
If a FlexVPN client has concurrent IKEv2 SAs to multiple FlexVPN servers and IKEv2 routing is used, if prefixes are added or removed on the FlexVPN servers a new IKEv2 SA is required to make the client aware of these changes.
When IKEv2 routing is used, the prefixes are obtained locally from the IKEv2 authorization policy or via AAA and RADIUS. Ensure that the correct prefixes are used and that these are obtainable when AAA occurs.
Third-party remote access clients, which do not use split tunneling commonly, do not require a default route sent via the configuration payload, although this is implementation specific.
If FlexVPN clients are used with IKEv2 routing, ensure that both the client and server advertise the tunnel interface IP address via configuration payload. This is achieved by using aaa authorization and the route set interface command being enabled within the IKEv2 authorization policy.
The following example illustrates a FlexVPN server that has a number of connected clients. The clients are assigned IP addresses via a configuration payload. One of the clients is unreachable; although it was allocated an IP address, this was not installed into the RIB, which results in loss of connectivity.
Router#show crypto session brief
Status: A- Active, U - Up, D - Down, I - Idle, S - Standby, N - Negotiating
K - No IKE
ivrf = (none)
Peer I/F Username Group/Phase1_id Uptime Status
10.10.60.6 Vi2 hostname=Router2.cisco.com,cn 00:01:01
UA
10.10.50.5 Vi1 hostname=Router1.cisco.com,cn 00:31:38
UA
For the peer router1.cisco.com, we see the IP address assigned via the configuration payload is 192.168.1.14.
Router#show crypto ikev2 sa detailed
IPv4 Crypto IKEv2 SA
Tunnel-id Local Remote fvrf/ivrf Status
2 10.10.10.1/500 10.10.50.5/500 OUTSIDE/none READY
Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth
sign: RSA, Auth verify: RSA
Life/Active Time: 86400/220 sec
CE id: 1026, Session-id: 19
Status Description: Negotiation done
Local spi: B6D1DDD763BE26D7 Remote spi: 8639ADD29DBB80B2
...
Assigned host addr: 192.168.1.14
Initiator of SA : No
Note
We would expect to see the assigned IP address within the Remote subnets: field; however this field is not shown as the remote prefix was not installed.
Within the RIB there is no route for the 192.168.1.14 prefix.
Router#show ip route 192.168.1.14
% Subnet not in table
By debugging IKEv2, we see that the FlexVPN client did not update the IP address assigned to it via the configuration payload with the config-set command.
IKEv2:(SESSION ID = 1,SA ID = 1):Config-type: Config-set
IKEv2:(SESSION ID = 1,SA ID = 1):Attrib type: app-version, length: 256, data:
Cisco IOS Software,
..
On the remote client the ability to update the peer of the assigned IP address had been removed, which can be seen from the following output, which is missing the route set interface statement.
Router1#show crypto ikev2 authorization policy
IKEv2 Authorization Policy : default
route accept any tag : 1 distance : 1
By default the prefixes installed when IKEv2 routing is used will use an administrative distance of 1. If the administrative distance has been changed with the route accept command within the IKEv2 authorization policy, ensure that no other prefixes exist with a lower administrative distance unless required.
If using the route set local command within the IKEv2 authorization policy to apply certain prefixes to the RIB for certain peers, ensure that the correct IP address is applied and reachable via the peer.
Dynamic Routing Protocols
When running routing protocols over an IPsec Security Association, the IP address of the tunnel interface will be used to source and receive traffic pertaining to establishing and updating the routing protocol. The IP address of the peer must be reachable via the tunnel interface. The routing adjacency will form once the tunnel has been established and prefixes distributed between routing neighbors.
The following example illustrates a simple connectivity test between tunnel interfaces; the ping command is used to validate that the peer’s IP address is reachable from the source of the local tunnel interface.
Router#ping 192.168.1.1 source tunnel 1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.1.19
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
When routing protocols are used over a tunnel interface, ensure that the tunnel destination address is not advertised within the tunnel. Figure 14-6 illustrates that the tunnel destination is advertised from within the tunnel interface, which will result in traffic entering the tunnel and being encapsulated and protected by IPsec. The lookup for the tunnel destination address will be within the tunnel, which would result in traffic being encapsulated once more; however, at this point the packet will be dropped and the syslog message seen in the following example will be generated.
%LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to up
%TUN-5-RECURDOWN: Tunnel0 temporarily disabled due to recursive routing
%LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel0, changed state to down
%LINEPROTO-5-UPDOWN:
To rectify the flapping tunnel interface, remove the offending prefix (203.0.0.0/16) that is advertised within the tunnel so that the tunnel destination is reachable only via the tunnel source interface and not within the tunnel itself.
Many routing protocols will have an initial handshake, during which both peers establish a neighborship and then exchange prefixes. The initial handshake usually requires packets of a relatively small size. When large amounts of prefixes are exchanged, the packet size will grow, and the packet size may then reach the maximum transmission unit size resulting in traffic being fragmented. If IPsec traffic between hosts is dropped because of fragmentation issues, it is common for a routing protocol to establish a neighborship; then if the routing updates fail to be passed between peers, the neighborship will drop, resulting in flapping neighbors.
In this case, the IP MTU of the tunnel can be manually lowered with the ip mtu command, or the MTU of the tunnel can be automatically adjusted with PMTUD by using the tunnel path-mtu-discovery command, which is applied to the tunnel interface.
Summary
This chapter gives an understanding of how to undertake troubleshooting of IKE and IPsec. Knowing the operation of protocols will assist an engineer when troubleshooting issues with IPsec VPNs.
A key take away from this chapter is to understand the protocol flows? of working architectures, so that when untoward behavior occurs, it can easily be spotted.
Following a defined methodology will allow the cause of the issue to be located. The steps listed in Table 14-1 allow a layered approach to troubleshooting to be conducted.
The following tables can be used as a reference for commands that can be used when diagnosing behavior and enabling debugging.
Table 14-7 lists useful show commands that can be used to verify IKEv2 and IPsec behavior.
Table 14-8 lists useful debug commands that can be used to verify IKEv2 and IPsec behavior.
Table 14-9 lists useful commands that can be used to verify interactions when AAA, RADIUS, and EAP are used.
Table 14-10 lists useful show commands that can be used to verify PKI operations.
Table 14-11 lists useful debug commands that can be used to verify PKI operations.
References
http://www.cisco.com/c/en/us/td/docs/security/vpn_client/anyconnect/anyconnect31/release/notes/anyconnect31rn.html Release Notes for Cisco AnyConnect Secure Mobility Client, Release 3.1
http://www.cisco.com/c/en/us/support/docs/content-networking/adaptive-session-redundancy-asr/117858-technote-asr-00.html ASR Datapath Packet Tracing Feature