When configuring a Cisco network from scratch, you can get things up and running flawlessly provided that you’ve configured everything properly. But chances are that in your day-to-day work you won’t get to build a network from scratch. You’ll inherit an existing network, and it will likely have some minor misconfigurations. In the worst case, it may even be broken. But even if your network isn’t broken by a misconfiguration, routers and switches do eventually wear out. Ports stop working, IOS images become corrupted, and power supplies fail.
The bottom line is that you’ll eventually encounter various problems that appear to be network-related. Here are some examples:
Some will obviously be related to a configuration issue on a router or switch, whereas others will be more ambiguous. Before you can begin troubleshooting the network, you have to determine whether the cause of the problem resides somewhere in your Cisco network or elsewhere. To do that, you’ll use two tools: debugging commands and the logging buffer.
Debugging commands are IOS commands that generate detailed messages about what a specific technology is doing at that moment. Most network folks refer to these messages as debugs. IOS can display these messages to the terminal console or write them to the logging buffer. The logging buffer is a spot in RAM that holds the output of various events, such as an interface going down, a new OSPF adjacency, and, of course, debug messages.
This chapter will show you how to use logging and debugging to gather detailed information about some of the technologies you’ve configured throughout the book. The idea is to help you collect enough information to either begin troubleshooting on your own or enlist the help of a colleague or vendor.
To be clear, troubleshooting every technology you’ve learned is beyond the scope of this book. I won’t cover every possible debug, and I won’t interpret all of the debugging messages. There also won’t be much in the way of “Try it now” exercises. If you do want to learn in-depth troubleshooting, you’ll find some training resources in the last chapter of the book.
Before you begin, I offer this word of caution: do not enable debugs in a production environment without getting appropriate permission. Turning on debugs creates additional load on the CPU, and although most debugs are safe, some can overwhelm a router or switch to the point that it stops forwarding traffic. I’ll warn you about the riskier debug commands, but don’t perform any of the commands in this chapter in a production environment unless the network is already down. Let’s get started!
By default, IOS stores all debug messages in the logging buffer, which stores up to 4,096 bytes by default. Once the log becomes full, IOS overwrites the oldest events first. The defaults notwithstanding, it’s a good idea to verify that the logging buffer is configured properly.
Choose a router or switch and execute the following command on it:
show logging
You should see something similar to the following:
Syslog logging: enabled (0 messages dropped, 0 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
No Active Message Discriminator.
No Inactive Message Discriminator.
Console logging: level debugging, 58 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 8 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
File logging: disabled
Persistent logging: disabled
No active filter modules.
Trap logging: level informational, 62 message lines logged
Logging Source-Interface: VRF Name:
Log Buffer (4096 bytes):
Notice that Console logging and Buffer logging are set to the debugging level. This means that once you enable debugs, IOS will display any debug messages to the terminal console and write them to the logging buffer. Also, notice that the logging buffer is 4,096 bytes. This roughly translates to over 50 lines, which is plenty in most cases. But just in case, let’s double it to 8,192 bytes.
Execute the following global configuration commands to configure IOS to send debug outputs to the logging buffer:
logging buffered debugging
Increase the size of the buffer to 8192 bytes:
logging buffered 8192
Optionally, you can clear the log to make it easier to read. Execute the following command at the enable prompt:
clear logging
By the way, once the logging buffer fills up, IOS will just overwrite the oldest events. It won’t stop logging, so don’t fret too much about the size. Once you’ve done all that, you’re ready to start debugging!
In most cases you can use the IOS inline help feature to figure out which debugs you need to turn on. But as you’ll see, some debug commands aren’t intuitive or easy to find. Your best bet is to search the internet for the command you’re looking for. In the following pages, I’ll give you some examples of how to enable and use various debugs, so you can see how it’s done.
In this first example, Executive-PC1 can’t access any network resources. After verifying that the PC has physical connectivity, you check to make sure Port Security isn’t blocking the PC’s MAC address:
Switch2#show port-security
Secure Port MaxSecureAddr CurrentAddr SecurityViolation Security Action
(Count) (Count) (Count)
---------------------------------------------------------------------------
Fa0/21 1 1 172 Restrict
---------------------------------------------------------------------------
Total Addresses in System (excluding one mac per port) : 0
Max Addresses limit in System (excluding one mac per port) : 6144
It appears that Port Security is blocking something, but it’s not clear whether it’s Executive-PC1 or another device. To get more information, you need to turn on debugging for Port Security.
Enable Port Security debugs by issuing the following command at an enable prompt:
debug port-security
Leave the debug running for about a minute; then turn it off:
undebug all
Finally, view the log buffer:
show logging
You’ll see the following:
*Mar 2 19:49:26.156: PSECURE: psecure_packet_enqueue: psecure receives
a packet: addr = 0800.2759.d9fd, swidb = Fa0/21, vlan = 700,
linktype = NullPak
*Mar 2 19:49:26.156: PSECURE: Read:197, Write:198
*Mar 2 19:49:26.156: PSECURE: swidb = FastEthernet0/21 mac_addr =
0800.2759.d9fd vlanid = 700
*Mar 2 19:49:26.156: PSECURE: Violation/duplicate detected upon receiving
0800.2759.d9fd on vlan 700: port_num_addrs 1 port_max_addrs
1 vlan_addr_ct 1: vlan_addr_max 1 total_addrs 0: max_total_addrs 6144
*Mar 2 19:49:26.156: PSECURE: Security violation, TrapCount:197
Although not without some cryptic elements, this output is fairly straightforward. Violation/duplicate detected upon receiving 0800.2759.d9fd on vlan 700 gives you the MAC address that Port Security is blocking and even what VLAN it’s on. Armed with this information, you can determine whether that MAC address belongs to Executive-PC1 or another machine. You can then adjust Port Security to allow it (refer to chapter 5 for more details).
In this example, Executive-PC1 isn’t receiving a DHCP-assigned IP address from Switch2. To get to the bottom of it, you enable the DHCP server debugs. A word of warning: this debug can generate a lot of output depending on how many DHCP clients are using the server. Do not run this in a production environment unless you’re doing real troubleshooting!
Clear the logging buffer:
clear logging
Debug DHCP server events:
debug ip dhcp server events
Wait about 10 seconds; then turn the debug off:
undebug all
View the logging buffer:
show logging
Here’s an excerpt from the output:
*Mar 2 20:03:36.744: DHCPD: subnet [172.31.70.1,172.31.70.254] in address pool Executives is empty. *Mar 2 20:03:36.744: DHCPD: Sending notification of ASSIGNMENT FAILURE: *Mar 2 20:03:36.744: DHCPD: htype 1 chaddr 0800.2759.d9fd *Mar 2 20:03:36.744: DHCPD: remote id 020a0000ac1f46fe0c000000 *Mar 2 20:03:36.744: DHCPD: interface = Vlan700 *Mar 2 20:03:36.744: DHCPD: class id 4d53465420352e30 *Mar 2 20:03:36.744: DHCPD: out_vlan_id 0 *Mar 2 20:03:36.744: DHCPD: Sending notification of ASSIGNMENT_FAILURE: *Mar 2 20:03:36.744: DHCPD: due to: POOL EXHAUSTED *Mar 2 20:03:36.744: DHCPD: htype 1 chaddr 0800.2759.d9fd *Mar 2 20:03:36.744: DHCPD: remote id 020a0000ac1f46fe0c000000 *Mar 2 20:03:36.744: DHCPD: interface = Vlan700 *Mar 2 20:03:36.744: DHCPD: class id 4d53465420352e30 *Mar 2 20:03:36.744: DHCPD: out_vlan_id 0
You’ll sometimes have to sift through murky output like this to find the cause. In this case, the two lines indicating pool Executives is empty and POOL EXHAUSTED are the ones with the valuable information. Executive-PC1 isn’t able to get an IP address because there are no addresses available for assignment in the pool!
It’s worth noting that show commands can give you much of the same information a debug gives you. For example, the show ip dhcp pool command would tell you that the DHCP pool has no available addresses. But it wouldn’t tell you the specific MAC address of the PC requesting a DHCP-assigned IP address, nor would it give you a timestamp of when that request occurred. Only debugging can give you a nearly real-time stream of what IOS is doing under the hood.
Most debugging commands are fairly obvious, but some require you to dig a bit to find them. The debug commands for the VLAN Trunking Protocol (VTP) fall into this category.
In this example, Switch1 is a VTP server and Switch2 is a VTP client, and Switch2 isn’t receiving a certain VLAN from Switch1. The command to debug VTP events is nested under a different set of debug commands that isn’t so intuitive.
Issue the following command to enable debugging for VTP events:
debug sw-vlan vtp events
Getting the output isn’t as speedy as for the previous debugs. You may have to wait up to five minutes for the VTP server to send an advertisement. Once you’ve given it some time, turn off the debug and check the log:
undebug all show logging
You’ll see the following:
*Mar 1 00:38:07.833: VTP LOG RUNTIME: Summary packet received, domain = cisco, rev = 11, followers = 1, length 77, trunk Fa0/24
*Mar 1 00:38:07.833: VTP LOG RUNTIME: Summary packet rev 11 greater than domain cisco rev 10
*Mar 1 00:38:07.833: VTP LOG RUNTIME: Domain cisco currently not in updating state
*Mar 1 00:38:07.833: VTP LOG RUNTIME: pdu len 77, #tlvs 1
*Mar 1 00:38:07.833: VTP LOG RUNTIME: Subset packet received, domain = cisco, rev = 11, seq = 1, length = 348
*Mar 1 00:38:07.833: VTP LOG RUNTIME: MD5 digest failing
calculated = B0 21 87 A1 4F 00 0C F4 14 7C C5 68 C6 84 A2 60
transmitted = B8 96 82 DE 39 52 A7 F7 6E 9C 60 EF 16 E4 77 23
This one is a bit more difficult to decipher. As a rule, whenever you encounter an unfamiliar debug and aren’t sure what to make of it, look for words that have a negative connotation. The last item in the log says MD5 digest failing, which means the VTP passwords on the client and server don’t match. That’s not obvious unless you’re familiar with the inner workings of VTP, but it doesn’t need to be. Most of the time you can copy and paste the error into your favorite search engine and get a quick answer on what it means.
For the final example, I’m going to show you how to track changes to the IP routing table. As you recall from chapter 16, dynamic routing protocols such as OSPF and EIGRP automatically install the best route to a network in the IP routing table. If a WAN connection between routers goes down, it can cause network performance problems, especially if the connection repeatedly goes down and comes back up. IP routing table debugs can tell you exactly when such events take place.
Take a look at the OSPF routes in the IP routing table on Switch1:
Switch1#show ip route ospf
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override
Gateway of last resort is not set
2.0.0.0/32 is subnetted, 1 subnets
O 2.2.2.2 [110/2] via 10.0.99.2, 03:51:43, FastEthernet0/24
10.0.0.0/8 is variably subnetted, 5 subnets, 2 masks
O 10.0.21.0/30 [110/2] via 10.0.99.2, 03:51:43, FastEthernet0/24
[110/2] via 10.0.12.2, 03:51:53, Vlan999
172.31.0.0/16 is variably subnetted, 3 subnets, 2 masks
O 172.31.70.0/24 [110/2] via 10.0.99.2, 03:51:43, FastEthernet0/24
Notice that the 2.2.2.2 network is reachable out of the FastEthernet0/24 interface with 10.0.99.2 (Switch2) as the next hop. To see what happens behind the scenes when this link goes down, you can enable debugs for the IP routing table.
On a router or layer-3 switch, enable IP routing debugs:
debug ip routing
Watch what happens when the link to Switch2—FastEthernet0/24—goes down:
Switch1#debug ip routing IP routing debugging is on Switch1# 00:20:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down 00:20:20: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down *Mar 1 00:20:20.232: is_up: FastEthernet0/24 0 state: 0 sub state: 1 line: 0 00:20:20: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/24 from FULL to DOWN, Neighbor Down: Interface down or detached *Mar 1 00:20:20.232: RT: interface FastEthernet0/24 removed from routing table *Mar 1 00:20:20.232: RT: del 10.0.99.0 via 0.0.0.0, connected metric [0/0] *Mar 1 00:20:20.232: RT: delete subnet route to 10.0.99.0/30 *Mar 1 00:20:20.232: RT: del 10.0.99.1 via 0.0.0.0, connected metric [0/0] *Mar 1 00:20:20.232: RT: delete subnet route to 10.0.99.1/32 *Mar 1 00:20:23.688: RT: updating ospf 2.2.2.2/32 (0x0): via 10.0.12.2 Vl999 *Mar 1 00:20:23.688: RT: closer admin distance for 2.2.2.2, flushing 1 routes *Mar 1 00:20:23.688: RT: add 2.2.2.2/32 via 10.0.12.2, ospf metric [110/3] *Mar 1 00:20:23.688: RT: updating ospf 172.31.70.0/24 (0x0): via 10.0.12.2 Vl999 *Mar 1 00:20:23.688: RT: closer admin distance for 172.31.70.0, flushing 1 routes *Mar 1 00:20:23.688: RT: add 172.31.70.0/24 via 10.0.12.2, ospf metric [110/3] *Mar 1 00:20:23.688: RT: del 10.0.21.0 via 10.0.99.2, ospf metric [110/2]
Switch1 removes the 10.0.99.0/30 subnet from its IP routing table, as indicated by the delete subnet route to 10.0.99.0/30 entry. Following that, Switch2 updates the route for 2.2.2.2/32 to use a different path via 10.0.12.2.
Keep in mind that the output of an IP routing debug probably won’t make much sense unless you’re looking at a network diagram. What’s important is that you know how to enable the debug and view its output should you ever need it.
At the beginning of the chapter I told you that by default IOS stores all debugging messages in the logging buffer. But you may have noticed that IOS also writes several non-debugging messages to the log as well.
Pick a switch or router and reboot it. Don’t save the running configuration!
reload Proceed with reload? [confirm] 00:03:06: %SYS-5-RELOAD: Reload requested by admin on console. Reload Reason: Reload command.
When it comes back up, view the logging buffer:
show logging
You may see a dozen or more messages like these:
*Mar 1 00:00:33.546: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down *Mar 1 00:00:35.508: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan *Mar 1 00:00:37.295: %DC-4-FILE_OPEN_WARNING: Not able to open flash:/dc_profile_dir/dc_default_profiles.txt *Mar 1 00:00:37.295: %DC-6-DEFAULT_INIT_INFO: Default Profiles DB not loaded. 00:00:38: %SYS-6-CLOCKUPDATE: System clock has been updated from 00:00:38 UTC Mon Mar 1 1993 to 20:00:38 EST Sun Feb 28 1993, configured from console by console. 00:00:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down 00:00:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan600, changed state to down 00:00:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan999, changed state to down 00:00:39: %SYS-5-CONFIG_I: Configured from memory by console
Keep in mind that debugs are off, and the logging buffer gets cleared when you reboot, so these aren’t debug messages. Most of these messages are notifications. The term notifications refers to one of eight logging severity levels, or logging levels for short. To see what they are, follow the next “Try it now.”
View the inline help by tying the global configuration command logging buffered ?.
IOS displays the relevant inline help as follows:
Switch1(config)#logging buffered ? <0-7> Logging severity level <4096-2147483647> Logging buffer size alerts Immediate action needed (severity=1) critical Critical conditions (severity=2) debugging Debugging messages (severity=7) discriminator Establish MD-Buffer association emergencies System is unusable (severity=0) errors Error conditions (severity=3) filtered Enable filtered logging informational Informational messages (severity=6) notifications Normal but significant conditions (severity=5) warnings Warning conditions (severity=4) xml Enable logging in XML to XML logging buffer <cr>
The eight logging levels are numbered in ascending order from most important to least important. For example, emergencies has a severity level of 0, whereas notifications has a severity level of 5. The highest level of 7 is for debugging. What’s important to note is that each logging level implicitly includes all levels below it. For example, when you set the logging level to level 7, debugging, IOS will still log messages with a lower severity level.
Pick a router or switch and configure the logging severity level to level 4, warnings:
logging buffered warnings
Save your running configuration and reload:
write memory reload
Finally, view the logging buffer:
show logging
You should see much less output:
*Mar 1 00:00:37.337: %DC-4-FILE_OPEN_WARNING: Not able to open flash:/ dc_profile_dir/dc_default_profiles.txt 00:00:42: %LINK-3-UPDOWN: Interface FastEthernet0/12, changed state to up
Notice that each entry has a modulus (%) followed by capital letters, a dash, and a number. The number indicates the severity level. For example, the %LINK-3-UPDOWN event has a severity level of 3, errors.
The logging buffer is a nice tool, but it has a major drawback: it’s stored in RAM, which means it’s limited in size and it gets cleared when the device reboots or powers off. When you need to check the logs across multiple devices, logging into each one and viewing the logging buffer can become a tedious, time-consuming process. To make your life easier, you can configure your Cisco devices to send logs to a syslog server.
Configuring a syslog server is well beyond the scope of this book, but most paid network monitoring tools include one. There are also free offerings, Kiwi being one of the more popular ones. The syslog server sits on the network and continually receives logs from devices that are configured to send them. Just like with the logging buffer, you specify logging severity levels to control what types of logging messages IOS sends to the syslog server.
To configure a router or switch to send its logs to a syslog server, you need two things:
Choose a router or switch and configure it to send logs to the syslog server at the IP address 1.2.3.4:
logging host 1.2.3.4 00:27:11: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 1.2.3.4 port 514 started - CLI
Set the logging trap level to 7, debugging:
logging trap debugging
Verify your configuration with a show logging.
Starting about 20 lines into the output, you should see the following:
Trap logging: level debugging, 29 message lines logged
Logging to 1.2.3.4 (udp port 514, audit disabled,
link up),
2 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
Enabling syslogging doesn’t stop IOS from logging to the logging buffer. For consistency, it’s a good idea to maintain the same logging level for both the logging buffer and the syslog.
For the most part, logging and debugging commands are intuitive. Table 19.1 lists the ones you saw in this chapter. As always, use the inline help to locate specific debug commands as the need arises.
You can use the command undebug all to turn off all debugs. But for some inconceivable reason, Cisco has made it possible to turn on all debugs simultaneously using the debug all command. In this hands-on lab, you’ll explore the consequences of having too much debugging going at once.
Perform the following on your lab router or switch. Do not perform any of these commands on production equipment!