Cisco SDWAN troubleshooting Scenario : Port-hop
Cisco SDWAN troubleshooting Scenario: Port-hop
Today we are going to talk about one of the troubleshooting scenario where we were facing the continuously rebooting of Cisco SDWAN control connections after every 5 mins. The platform is Cisco SDWAN C8000v virtual router
Fig 1.1- Cisco SDWAN troubleshooting Scenario : Port-hop
- Router : Cisco SDWAN Router C8000v in Azure Platform
- Version : 17.9.04a
⭐Related : HSECK9 License for Cisco C8000v SDWAN Router
⭐Related : Resizing Cisco Virtual Router C8000v VM in Azure
⭐Related : Cisco SDWAN Workflow : Managing HSEC Licenses C8000V/CSR1000V
Step 1: Check the control connections on the router, you see it is up from last 4 mins and 12 seconds. It will again reboot after completing 5 mins
NDNA_c8000v#sh sdwan control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT ORGANIZATION LOCAL COLOR PROXY STATE UPTIME ID
------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart dtls 10.10.10.11 1 1 10.10.3.5 12646 17.23.12.11 12646 NDNA-111 gold No up 0:00:04:12 0
vsmart dtls 10.10.10.12 2 1 10.10.3.15 12646 17.23.12.25 12646 NDNA-111 gold No up 0:00:04:12 0
vmanage dtls 10.10.10.10 1 0 10.10.3.12 13046 17.23.12.88 13046 NDNA-111 gold No up 0:00:04:12 0
Step 2: Now checked again after like a minute now and you will notice, it is showing 8 seconds now which means it is rebooted again.
NDNA_c8000v#sh sdwan control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT ORGANIZATION LOCAL COLOR PROXY STATE UPTIME ID
------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart dtls 10.10.10.11 1 1 10.10.3.5 12646 17.23.12.11 12646 NDNA-111 gold No up 0:00:00:08 0
vsmart dtls 10.10.10.12 2 1 10.10.3.15 12646 17.23.12.25 12646 NDNA-111 gold No up 0:00:00:08 0
vmanage dtls 10.10.10.10 1 0 10.10.3.12 13046 17.23.12.88 13046 NDNA-111 gold No up 0:00:00:08 0
Step 3: Check the router IOS-XE version
NDNA_c8000v#sh ver
Cisco IOS XE Software, Version 17.09.04a
Cisco IOS Software [Cupertino], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.9.4a, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Fri 20-Oct-23 10:32 by mcpre
Step 4: For troubleshooting, move the router to CLI mode
First check the mode in which router is working, if we see below in red, the template is attached to the router which means the router is in controller mode.
First check the mode in which router is working, if we see below in red, the template is attached to the router which means the router is in controller mode.
Personality: vEdge
Model name: C8000V
Device role: cEdge-SDWAN
Services: None
vManaged: true
Commit pending: false
Configuration template: AZURE-NDNA-V01
Chassis serial number: XXXXXXXXXXXXXX
Step 5: As we said earlier, move the router from controller mode to CLI mode in order to do packet captures on the router.
Once you moved, run the below script in order to capture the packets on the interface with the source and the destination IPs as shown below :
10.10.1.23 is the interface IP which is used for control connection
10.10.1.23 is the interface IP which is used for control connection
17.23.12.88 is the vManage IP
Interface : Gi1
!
ip access-list extended CAP-Filter
10 permit ip host 10.10.1.23 host 17.23.12.88
20 permit ip host 17.23.12.88 host 10.10.1.23
exit
monitor capture CAP access-list CAP-Filter interface GigabitEthernet1 both buffer circular size 25
monitor capture CAP limit pps 1000000
monitor capture CAP access-list CAP-Filter both buffer circular size 25
monitor capture CAP start
monitor capture CAP stop
!
Step 6: Now run below commands to get debugs
NDNA_c8000v# debug platform software sdwan vdaemon all high
NDNA_c8000v# monitor logging process vdaemon internal
NDNA_c8000v# monitor logging process vdaemon internal
Step 7: Once you run the above commands, you will see logs related to the interfaces
You will see that in debug logs , TLOC Disable ... Why ?
2024/04/19 17:47:59.779970993 {vdaemon_R0-0}{255}: [event] [18342]: (debug): Disabling tloc GigabitEthernet1.
2024/04/19 17:47:59.780001093 {vdaemon_R0-0}{255}: [misc] [18342]: (ERR): Delta preference value added to TLOC pref.
2024/04/19 17:47:59.780003193 {vdaemon_R0-0}{255}: [misc] [18342]: (ERR): Sending TLOC: ifname:GigabitEthernet3 color:gold spi:18915 smarts:2 manages:1 state:DOWN LR encap:0 LR hold time:7000 bw:0, down-bw 0 range: 0-0,adapt period 0 up-bw range 0-0 up_fia 0 capability:0x3f
Step 9: Check the interface for port-hop and you will see port-hop is enabled. Now disable the port hop and you will see the control connections will be stable
interface GigabitEthernet1
tunnel-interface
encapsulation ipsec weight 1
no border
color gold restrict
no last-resort-circuit
no low-bandwidth-link
no vbond-as-stun-server
vmanage-connection-preference 5
port-hop
carrier default
nat-refresh-interval 5
hello-interval 1000
hello-tolerance 12
no allow-service all
no allow-service bgp
allow-service dhcp
allow-service dns
allow-service icmp
allow-service sshd
no allow-service netconf
no allow-service ntp
no allow-service ospf
no allow-service stun
allow-service https
no allow-service snmp
no allow-service bfd
exit
exit
appqoe
no tcpopt enable
no dreopt enable
no httpopt enable
!
Step 9: Check the control connection after disabling port-hop on the interface , you will see it is up from last 19 min. and stable.
NDNA_c8000v#sh sdwan control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT ORGANIZATION LOCAL COLOR PROXY STATE UPTIME ID
------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart dtls 10.10.10.11 1 1 10.10.3.5 12646 17.23.12.11 12646 NDNA-111 gold No up 0:00:19:02 0
vsmart dtls 10.10.10.12 2 1 10.10.3.15 12646 17.23.12.25 12646 NDNA-111 gold No up 0:00:19:02 0
vmanage dtls 10.10.10.10 1 0 10.10.3.12 13046 17.23.12.88 13046 NDNA-111 gold No up 0:00:19:02 0
Conclusion
For this specific version of 17.9.04a, port-hop needs to disabled if you triggered with the reboot of the control connections after some time which leads to have this issue.
Hope it will help in case you have issue with the control connection "DISTLOC"
Continue Reading...
- Finding the Right SD-WAN Vendor for Your Business
- The Evolution: Exploring the Origins of SD-WAN discussions
- Discover the Power of Multitenancy with Versa SDWAN!
- Introduction to Silver-Peak SDWAN Solution
- Introduction to VeloCloud SD-WAN Solution
- Introduction to Palo Alto's Prisma (CloudGenix) SDWAN
- Introduction to Fortinet SDWAN
Cisco SDWAN
