ACI Common Operations and Troubleshooting Tools

System health and faults

ACI gathers and correlates information into Health Scores and Statistics, both help guide our troubleshooting process.

We can see the health score of ACI components in various granularity levels. For example, there is a health score for an Application Profile, and a health score for each EPG within it.

The sum total of the faults in the ACI fabric can be seen under System –> Faults:

aci-system-faults

Viewing the ACI topology

Go to Fabric -> Inventory -> Topology

The topology feature displays only the active components, i.e. failed devices or devices put in maintenance are not displayed here, but rather under Unreachable Nodes or Unmanaged Fabric Nodes. Therefore this menu is not enough to have a complete view during any troubleshooting gig.

ACI presents a couple of options when we right-click on a node in the topology. We can for example click on Navigate To button to go to the page details of a desired node. For example, right-clicking on Leaf1-101 then clicking on Navigate To takes us to the Leaf1-101 page.

after_clicking_navigate_to

When we click once on a device, ACI highlights it and we can clearly see which links and devices are connected to it. For example, clicking on Leaf2-102 highlights its link to Spine1-201.

Another interesting feature within the Topology menu is that we can read the configured interface policy group configured on any interface of the fabric node. To do that, go to Interface and Policies tab and, in Select Node drop-down list, select the desired fabric node:

For example, on leaf 101 we have interfaces eth1/3 to eth1/8 are associated to interface policy groups. Among them we have Eth1/4 as a member of a Virtual Port Channel interface policy group and the rest are member of port channel interface policy groups. Eth1/1, eth1/2 and eth1/9 are not associated to any interface policy group:

The Interface menu allows us to present graphically all interfaces of a particular ACI node at once, and have a quick glance at their administrative status. Click on the menu Interface first:

Click on the “+” sign to add switches to the current view then select your preferred switches:

add_switches
available_fabric_switches

Clicking once on a port displays action buttons such as Enable, Disable and Reset. Besides, we can read the operational configuration of the port:

Removing the switch from the view is simple: click on the “x” symbol. This does not delete the switch from the fabric LOL, it simply removes it from the Topology Interface view:

Decommissioning and recommissioning ACI fabric switches

Operations and Troubleshooting with CLI

  • We can view configuration values with either the GUI or the CLI.
    • in CLI, we have the possibility of doing it with the NX-OS style commands or with the Linux commands; a Nexus switch is based on Linux.
  • When creating a normal port channel or a VPC, APIC associates to it in the background a numerical value. But we can not see it through a normal “show” command, but rather through a “show port-channel extended”
  • The same is true for “show vlan”; we need to issue the “show vlan extended” to see the internally associated VLANs.
  • The CLI in APIC does not support the pipe “|” symbol.

Extending the ACI fabric

  • new leafs should have a leaf ID numbered higher than 100
  • new spines should have a Spine ID numbered higher than 200.
  • Fabric Extenders can be attached to ACI fabric. Not all FEXs are supported by ACI. A FEX can attach to only one leaf.

Firmware Upgrade

Performing an upgrade of the firmware, whether it is for the APIC or for the fabric nodes, has slight differences before and after ACI version 4.

Let us learn these differences.

APIC firmware upgrade

In ACI version 3 we define a Controller Upgrade policy:

You define an APIC firmware policy first, that defines which firmware version (Target Firmware Version) and when to perform the upgrade (Apply Policy). We can launch the firmware upgrade immediately or we can define a Start Time.

In ACI version 4 however, both the firmware policy (i.e which firmware version) and the schedule policy are configurable from the same menu:

Fabric node firmware upgrade

Prior to ACI version 4, we define what is called firmware groups and maintenance groups. Firmware groups designate simply groups of fabric nodes that will receive the same firmware version.

Clients with critical ACI environments follow generally the four-group method; in other words, they define four maintenance groups:

  • red Leaf group
  • blue Leaf group
  • red Spine group
  • blue Spine group

and start by upgrading Leaf groups first.::

A maintenance group defines when and how to perform the upgrade associated with a particular firmware group:

  • the “when”: whether to execute the upgrade immediately or to use a Scheduler
  • the “how”: when the upgrade process is launched it must obey some rules. For example:
    • not exceeding the concurrCap value, which is the maximum simultaneous switches being upgraded at a time
    • only one switch per VPC.

For example. let’s create two fabric node firmware maintenance groups:

  • Red group: two leafs and the spine
  • Blue group: the two other leafs.

A Scheduler is automatically created for each fabric node firmware maintenance group we configured. That is why we see Schedulers named after our previously created Red-group and Blue-group:

Configuration management

Configuration management in ACI is performed with either snapshots or backups.

  • Snapshots
    • are fast (couple of clicks) to store a config or to restore it
    • store settings of the fabric or of the tenants
    • do not store the complete configuration
  • Backups
    • used to store and restore the ACI configuration
    • require an external server
    • in a backup operation, a backup file is generated. In a restore, an import file is needed
    • require an export policy in the save operation, and an import policy in the restore operation.
    • two types of backup/restore operation:
      • best effort:
        • when a difference in Shads is encountered, the portion of data is ignored.
        • when the imported config belongs to an ACI version is different from the current running, the difference is ignored.
      • atomic:
        • when a difference in Shads is encountered, the portion of data is ignored.
        • when the imported config belongs to an ACI version different from the current running, the backup/restore operation is aborted.

ACI Optimizer

  • simulates a topology with an X number of ressources ( BD, EPG, leafs, spines, …) in order to analyze the performance impact associated with scalability. This tool helps to make the right decision when thinking about expanding the ACI fabric.

On-Demand Troubleshooting Diagnostic Tests

We configure on-demand diagnostic tests under Fabric Policies. In other words, each test we configure is a fabric policy. The results are to be seen under the respective devices, under Inventory, whether it is a chassis test or a line card test, etc.

We go to Fabric Policies –> Troubleshooting Policies –> On-Demand Diagnostics. Then we right-click on any type of diagnostic policies and create one:

on-demand hardware troubleshooting diagnostics

rightclick -> Create Fabric Port On-Demand Diag Policy. A new window pops up. Give a name to our on-demand diag policy.

Notice the Admin State too. I leave it to Start which means “launch the test”.

I am going to create a Fabric Port On-demand Diag test on leaf 104. Let us first display which interfaces are used in the fabric on Leaf 104:

apic1# fabric 104 show lldp neighbors

Node 104 (Leaf4-104)
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic1 Eth1/1 120 eth2-1
Spine1-201 Eth1/54 120 BR Eth1/36
Total entries displayed: 2
%
apic1#

So I’ll run the test on fabric port Eth1/54. Go to Apply to Fabric Ports

Select port Eth1/54 from Leaf 104:

and click Update

oh, and under Diagnostic Tests, double-click on the field “No Tests” and select which test we want to execute:

Submit the whole on-demand diag test:

A summary of all the configured Fabric Port on-demand Diagnostic tests

Visibility and Troubleshooting Tool

  • allows to see:
    • drop statistics
    • traffic statistics
  • inspects contract deny logs, and permit logs when the leaf in question is a Nexus EX.
  • SPAN sessions. A SPAN session can be set up from the fabric menu, the tenant menu, or the Visibility and Troubleshooting tool menu.
  • Traceroute:
    • which is not to be confused with itraceroute Nexus-OS command.
    • involve endpoints and devices external to the fabric.
  • Atomic counters:
    • count packets of a specific protocol between any two points in the fabric.
    • are reset every 30 seconds
    • are synchronized between the emitting leaf and the receiving leaf, unlike in legacy networks where the sum total of packets on endpoint A does not equal that on endpoint B.
    • begin to be displayed after 90 seconds of running the atomic counter test.

End Point Tracker

End Point Tracker ( aka EP Tracker) : allows to answer questions such as:

  • on which leaf and port is the endpoint?
  • which encapsulation does the endpoint use?
  • where was the endpoint historically connected to?

Traffic Maps

  • Traffic Maps graphically displays dropped transmitted and received packets, thus helping identify bottlenecks on the fabric.
  • use atomic counters

SPAN / Port Mirroring

ACI supports these types of SPAN:

  • Local SPAN: source and destination are on the same leaf
  • Fabric SPAN: source port can be any fabric port, even a spine port.
  • Tenant SPAN: source and destination on the same tenant
  • Virtual SPAN: source is a virtual NIC, on a virtual machine.
  • RSPAN: mirrored traffic is sent over a remote VLAN
  • ERSPAN:
    • mirrored traffic is sent over IP using GRE tunnel technology.
    • has two versions: version 1 (aka type 1) and version 2 (type 2).
      • both versions use GRE encapsulation
      • The difference between them is that version 2 includes an ERSPAN header in the GRE-encapsulated packet.
    • not every packet analysis station supports both ERSPAN versions.
  • Generally a SPAN session supports different destination types. Some leaf hardware models do not support all destination types.
  • Copy service

Nexus Broker

In Nexus 3000 and 9300 the leaf switch can consolidate multiple ingress mirrored traffics and forward them, using OpenFlow, accordingly to different destinations. This may be useful when we want to interpret each mirrored traffic on a separate packet analysis station.

Network Audit Trail

Network Audit Trail in ACI is similar to the AAA framework. In fact, ACI already integrates the functionalities that AAA brings. We can perform network audit fabric-wide or per object.

acidiag

The ACI DIAGnostics command is self explanatory, as opposed to its usage options:

apic1# acidiag
usage: acidiag [-h] [-v]
{avread,fnvread,fnvreadex,rvread,rvreadle,crashsuspecttracker,bootother,bootcurr,journal,logs,telemetry,hwcheck,dbgtoken,validateimage,validatenginxconf,version,preservelogs,platform,verifyapic,bond0test,linkflap,touch,run,installer,start,stop,restart,dmestack,dmecore,reboot}

acidiag: error: too few arguments
%
apic1#

For example to read all VTEP addresses of the fabric switches:

apic1# acidiag fnvread 
ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
101 1 Leaf1-101 SAL18464AC7 10.0.224.67/32 leaf active 0
102 1 Leaf2-102 SAL18464ADA 10.0.224.68/32 leaf active 0
103 1 Leaf3-103 SAL18380V1W 10.0.224.66/32 leaf active 0
104 1 Leaf4-104 SAL18432WL7 10.0.224.64/32 leaf active 0
201 1 Spine1-201 SAL184642GL 10.0.224.65/32 spine active 0

Total 5 nodes

Connecting to fabric nodes from the APIC

Either with:

fabric {node ID} {command_to_be_executed}

For example I display the interfaces under fabric node ID 103 (which is a leaf here). By the way, it is show interface not show interfaces:

apic1# fabric 103 show interfaces

----------------------------------------------------------------
Node 103 (Leaf3-103)
----------------------------------------------------------------
Incorrect command "show interfaces"

% apic1# fabric 103 show interface
----------------------------------------------------------------
Node 103 (Leaf3-103)
----------------------------------------------------------------
mgmt0 is up
admin state is up,
Hardware: GigabitEthernet, address: f8c2.8823.7e04 (bia f8c2.8823.7e04)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is routed
full-duplex, 1000 Mb/s
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
EtherType is 0x0000
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Rx
5896 input packets 0 unicast packets 2421 multicast packets
3475 broadcast packets 1438123 bytes
Tx
8 output packets 0 unicast packets 8 multicast packets
--More-- 0 broadcast packets 752 bytes

Ethernet1/1 is down (sfp-missing)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: f8c2.8823.7e05 (bia f8c2.8823.7e05)
MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
--More-- 30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
0 input packets 0 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Ethernet1/2 is down (sfp-missing)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: f8c2.8823.7e06 (bia f8c2.8823.7e06)
--More-- MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped never
Last clearing of "show interface" counters never
0 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 0 broadcast packets
--More-- 0 input packets 0 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 0 multicast packets 0 broadcast packets
0 output packets 0 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Ethernet1/3 is up (out-of-service)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: f8c2.8823.7e07 (bia f8c2.8823.7e07)
MTU 9000 bytes, BW 1000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
--More-- FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 1d12h
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 0 bits/sec, 0 packets/sec
30 seconds output rate 0 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 0 bps, 0 pps; output rate 0 bps, 0 pps
RX
0 unicast packets 0 multicast packets 39 broadcast packets
39 input packets 2652 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
--More-- 0 input with dribble 39 input discard
0 Rx pause
TX
0 unicast packets 4356 multicast packets 0 broadcast packets
4356 output packets 1481040 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause

Ethernet1/4 is down (sfp-missing)
admin state is up, Dedicated Interface
Hardware: 1000/10000 Ethernet, address: f8c2.8823.7e08 (bia f8c2.8823.7e08)
MTU 9000 bytes, BW 0 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 10 Gb/s
FEC (forward-error-correction) : disable-fec
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
--More-- % apic1#

or by establishing an SSH connection to the node with the following command:

ssh {node name}
apic1# ssh 103 
ssh: connect to host 103 port 22: Invalid argument
% apic1#
% apic1# sssh leaf-3-103

Password: ! -- the same password as for connecting to APIC --!
Last login: Sat May 9 23:02:54 2020 from 10.0.0.1
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Leaf3-103#

to go back to the APIC:

Leaf3-103# exit
logout
Connection to leaf3-103 closed.
% apic1#

Let us connect to the spine

apic1# ssh spine1-201

Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
Spine1-201#

And try to connect from there to a leaf. It won’t be possible:

Spine1-201# ssh leaf3-103
ssh: connect to host leaf3-103 port 22: No route to host
Spine1-201#
Spine1-201#

iping

Issue iping with the -h option to print all possible options:

Spine1-201# 
Spine1-201# iping -h

usage: iping [ -d set the SO_DEBUG option]
[ -D enable debug information ]
[ -F enable do not fragment bit in IP header ]
[ -L receive packets on supplied interface ]
[ -n enable printing host IP address than resolved name ]
[ -q quiet output ]
[ -r disable routing of the packets, send only to directly connected hosts ]
[ -v output in verbose format ]
[ -V <vrf-name> name of the VRF through which destination is reachable ]
[ -c <count> no of packets to send ]
[ -i <wait> no of seconds to wait before sending next packet ]
[ -p <pattern> packet payload pattern ]
[ -s <packetsize> size of packets to send ]
[ -t <timeout> wait for seconds to receive reply ]
[ -S <source ip/interface> send packet with given source-ip or IP of given interface and
send packet out of that interface ]
<host> destination host-name or ip address
Spine1-201#

example: we try to ping another VTEP address. Remember we can quickly from CLI determine all VTEP addresses with acidiag fnvread command:

apic1# acidiag fnvread 
ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
101 1 Leaf1-101 SAL18464AC7 10.0.224.67/32 leaf active 0
102 1 Leaf2-102 SAL18464ADA 10.0.224.68/32 leaf active 0
103 1 Leaf3-103 SAL18380V1W 10.0.224.66/32 leaf active 0
104 1 Leaf4-104 SAL18432WL7 10.0.224.64/32 leaf active 0
201 1 Spine1-201 SAL184642GL 10.0.224.65/32 spine active 0

Total 5 nodes

% apic1# ssh spine1-201 Password:
Last login: Sun May 10 04:51:30 2020 from 10.0.0.1
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php

Spine1-201# iping- -V overlay-1 10.0.224.67
PING 10.0.224.67 (10.0.224.67) from 10.0.224.65: 56 data bytes
64 bytes from 10.0.224.67: icmp_seq=0 ttl=63 time=0.54 ms
64 bytes from 10.0.224.67: icmp_seq=1 ttl=63 time=0.482 ms
64 bytes from 10.0.224.67: icmp_seq=2 ttl=63 time=0.368 ms
64 bytes from 10.0.224.67: icmp_seq=3 ttl=63 time=0.327 ms
64 bytes from 10.0.224.67: icmp_seq=4 ttl=63 time=0.433 ms

--- 10.0.224.67 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.327/0.429/0.54 ms
Spine1-201#

There is also its sister command in the IPv6 world: iping6

Spine1-201# 
Spine1-201#
Spine1-201# iping6 -h
iping6: option requires an argument -- 'h'

usage: iping6 [-dDnqrv] [-V vrf] [-c count] [-i wait] [-p pattern] [-s packetsize] [-t timeout] [-S source interface/IPv6 address] host
Spine1-201#

References and further reading

Leave a Comment