Latest

Cisco Viptela SDWAN : 10 Error Codes while checking Control Connections on vEdge/cEdge

Today we are going to talk about some of the troubleshooting steps when we got errors on the Cisco Viptela SDWAN control connections. These are the basics of the types of issues we see normally on the control connections.

Whenever we are checking why control connections are down for the TLOCs we used in our WAN, we saw some of the error codes while running the command "show control connections-history" in the vEdge devices or "show sdwan control local-properties" on the cEdges. The error is showing local or remote on the TLOCs with the following error codes 

Fig 1.1- Cisco SDWAN Issues

Error Code 1: BIDNTVRFD, CRTREJSER, SERNTPRES
Whenever you see this kind of error on the TLOC these are basically indicates that we are missing serial numbers. As you see there are three different error codes and these are defined as:

  • BIDNTVRFD indicates a missing serial number for vBond orchestrators.
  • CRTREJSER indicates a missing serial number for vEdge routers and vSmart controllers.
  • SERNTPRES on a vBond orchestrator indicates a serial number mismatch between vSmart controllers.

Resolution
When ever you get this kinds of errors make sure you check the serial number on the vManage screen and update, Once update please send it to all controllers

Process

  • In vManage NMS, select the Configuration --> Certificates screen
  • In the vEdge List tab, select the device whose serial number is missing.
  • Click Send to Controllers.

Error Code 2: CTORGNMMIS
CTORGNMMIS defines the Organization name mismatch in the certificate

Resolution
Use organization-name command on the vEdges/cEdges as shown below for the vEdges.
NDNA_vEdge(config)# system organization-name "NDNA"

Error Code 3: DCONFAIL: DTLS Connection Failure
This is one of the most seen issue on the control connections. This error code shows that vEdge router does not establish DTLS connections to controllers in the overlay network.

Resolution

  • Verify that the next hop is reachable
  • Verify that a default gateway is installed in the route table
  • Verify that the DTLS ports are open in any firewall you use
  • Verify that the default gateway is correctly mapping the IP address to the MAC address correctly.
  • Verify reachability

Error Code 4: DISCVBD or SYSIPCHNG
This is one of the rare issues but due to vEdge/cEdge router experiences transient control connection errors. 

Resolution:
As discuss with Cisco on this and reverted back that these issues are part of normal operation of the overlay network. They have no impact on production traffic, and they resolve by themselves, with no action required.

Error Code 5: DISTLOC
This is also a common issue we saw on TLOCs which is represented as Disabled TLOC. A TLOC identifies the physical interface where a vEdge router connects to the WAN transport network or to a NAT gateway. 

Resolution:

  • Check the Control connections have been cleared
  • Check if the TLOC color has been changed
  • Check if the system IP has changed

Error Code 6: LISFD
This error is also rare error when we deployed sdwan solution. This error indicates Socket error messages.

Resolution

  • Check if overlay network contains duplicate IP addresses, especially duplicate transport addresses. Once you removed duplicate IP, the issue may resolved.
  • Check if packets have been corrupted, resolve the corrupted packets, the issue may resolved.
  • Check if vEdge router receives a reset request from the remote device. Once the connection setup again, it will resolved.
  • Check if vEdge router and the vSmart controller are not both using DTLS or TLS ports.
  • Check forwarding ports are open, if not we will face this issue.

Error Code 7: NOVMCFG
This may be the simple error which shows that vEdge Router Template Not Attached in vManage NMS and is when we are using ZTP.

Resolution:
If the device is not attached with a template on the vManage at the time of ZTP and you will see no config. Check and assign the template on the vManage for the device which has issue.

Error Code 8: RDSIGFBD, TXCHTOBD
This is also a rare case where you found this error message which describes as Board ID Not Initialized.

Resolution
Try to reset the board-ID. It will take some time but will resolve your issue.

Error Code 9: VB_TMO, VM_TMO, VP_TMO, VS_TMO
These are also very common errors and the issues is basically when the peer timeout occurs if the vEdge router loses reachability to a controller in the overlay network.

Resolution:

  • Troubleshoot reachability to the controller using ping, traceroute, and rapid ping
  • Increase the hello-interval and hello-tolerance values on the interface to prevent packet loss.
  • Check if the underlay rate-limiting the TLS/DTLS packets may be the issue.

Error Code 10: VECRTREV, VSCRTREV
This is another error which may not be common but due to the certificate revoked

Resolution:

  • Most importantly you need to check the "clock" and is within vBond' s certificate validity range
  • Check the root cert corruption on vEdge