Content

| 4.3 |  | Troubleshooting Switched Ethernet Networks |
| 4.3.6 |  | Troubleshooting Ethernet switch flooding |

LAN switches use forwarding tables, called Content Addressable Memory (CAM), to direct traffic to specific ports based on the VLAN number and the destination MAC address of the frame. When there is no entry corresponding to the frame's destination MAC address in the incoming VLAN, the unicast frame will be sent to all forwarding ports within the respective VLAN. This is called flooding. Limited flooding is part of the normal switching process. However, there are situations when continuous flooding can cause adverse performance effects on the network.

Note that most modern switches including the Catalyst 2900 XL, 3500 XL, 2950, 3550, 4000, 5000, and 6000 maintain Layer 2 forwarding tables per VLAN.

Low-bandwidth links can become saturated with large amounts of flooded traffic. When this happens, network performance will degrade, occasionally causing a loss of connectivity.

Server S1 in VLAN 1 is running a bulk data transfer backup to server S2 in VLAN 2. Server S1 has its default gateway pointing to router A's VLAN 1 interface. Server S2 has its default gateway pointing to router B's VLAN 2 interface. Packets from S1 to S2 will follow this path:

> S1–VLAN 1–-switch A–-router A–-VLAN 2–-switch B–-VLAN 2–-S2 (orange line)

Packets from S2 to S1 will go along the following path:

> S2–VLAN 2–switch B–router B–VLAN 1–switch A–flooded to VLAN 1–S1 (red line)

Note that with such an arrangement, switch A will not "see" traffic from the S2 MAC address in VLAN 2. This is because the source MAC address will be rewritten by router B and the packet will only arrive in VLAN 1. This means that every time switch A needs to send the packet to the S2 MAC address, the packet will be flooded to VLAN 2. The same situation will occur with the S1 MAC address on switch B.

This behavior is called asymmetric routing. Packets follow different paths depending on the direction. Asymmetric routing is one of the two most common causes of flooding.

Returning to the above example, all packets contained in the data transfer between the two servers will be flooded to all ports on Switch A that belong to VLAN 2 and all ports on Switch B that are members of VLAN1. Any workstation attached to one of these ports, in this example Workstation W, will receive all packets of the conversation between S1 and S2. Suppose the server backup takes 50 Mbps of bandwidth, which will saturate 10 Mbps links. The result to the PCs will be either a complete connectivity outage, or significant performance degradation.

This flooding is due to asymmetric routing, and may stop when server S1 sends a broadcast packet such as Address Resolution Protocol (ARP). Switch A will flood this packet to VLAN 1 and switch B will receive and learn the MAC address of S1. Since the switch is not receiving traffic constantly, this forwarding entry will eventually age out and flooding will resume. The same process applies to S2.

There are different approaches to limit the flooding caused by asymmetric routing. The approach is normally to bring the router's ARP timeout and the switches' forwarding table-aging time close to each other. This will cause the ARP packets to be broadcast. Relearning must occur before the L2 forwarding table entry ages out.

A typical scenario where this kind of issue might be observed is when there are redundant Layer 3 switches (such as a Catalyst 6000 with Mutilayer Switch Feature Card (MSFC)) configured to load-balance with Hot Standby Router Protocol (HSRP). In this case, one switch will be active for even numbered VLANs and the other one will be active for odd numbered VLANs.

Another common issue caused by flooding is STP Topology Change Notification (TCN). TCN is designed to correct forwarding tables after the forwarding topology has changed. This is necessary to avoid a connectivity outage, since after a topology change some destinations previously accessible via particular ports might become accessible via different ports. TCN operates by shortening the forwarding table aging time, such that if the address is not relearned it will age out and flooding will occur.

TCNs are triggered by a port that is transitioning to or from the forwarding state. After the TCN and even if the particular destination MAC address has aged out, flooding should not happen. If it does occur, it will be brief in most cases since the address will be relearned. However, an issue might arise when TCNs are occurring repeatedly with short intervals. The switches will constantly fast-age their forwarding tables so flooding will be nearly constant.

Whenever a port on a switch goes down or up there is a transition of the STP state to or from the forwarding state. If a port is flapping, it causes repetitive TCNs that can lead to flooding.

Ports with the STP portfast feature enabled will not cause TCNs when going to or from the forwarding state. Configuration of portfast on all end-device ports such as printers, PCs, servers, and so on, should limit TCNs to a low amount.

Another possible cause of flooding can be overflow of the switch forwarding table. In this case, new addresses cannot be learned and packets destined to such addresses are flooded. When space becomes available in the forwarding table, new addresses can then be learned. This is possible but rare, since most modern switches have large enough forwarding tables to accommodate MAC addresses for most designs.

Forwarding table exhaustion can also be caused by an attack on the network where one host starts generating frames each sourced with a different MAC address. This will tie up all the forwarding table resources. Once the forwarding tables become saturated, other traffic will be flooded because new learning cannot occur. This kind of attack can be detected by examining the switch forwarding table. Most of the MAC addresses will point to the same port or group of ports. Limiting the number of MAC addresses learned on untrusted ports by using the port security feature can prevent such attacks.

The configuration below enables this feature on an interface, and then specifies that the interface can have no more than 50 MAC addresses associated with it:

> Switch(config)#interface fastethernet 0/1
> Switch(config-if)#switchport port-security
> Switch(config-if)#switchport port-security maximum 50

The maximum number of MAC addresses that can be associated with an interface when port-security is active is 132.

In extremely rare cases, port VLAN information may be programmed incorrectly and the port may forward packets unexpectedly for the wrong VLAN. Such incorrect programming may result in packets from one VLAN being seen incorrectly on another VLAN.

A misplaced cable or VLAN assignment can lead to unicast frames crossing VLAN boundaries. Under normal conditions the switch should maintain VLAN boundaries, and a router such as an MSFC should route only those packets with source and destination IP addresses on different IP networks. If multicasts also cross VLAN boundaries, there is a good chance that Spanning Tree BPDUs are being flooded and therefore the STP tree of the two VLANs has merged. To validate this hypothesis, capture the show spanning-tree [vlan id] command for the two VLANs. Although the same physical switch may serve as the root for two VLANs, it uses unique root bridge IDs for each VLAN. If the output displays the same root bridge ID for both VLANs, the two VLANs are bridged somewhere in the network.

To isolate the cause of the problem, use the following steps:

- Select a PC on VLAN A.
- Select a PC on VLAN B.
- Assign the PC on VLAN B to an IP address on VLAN 2. This step effectively removes the router from the fault domain. If ping is successful between the devices, there is strong evidence to support a bridged connection, likely accidental, between the two VLANs.

At this point, the next steps are to trace the path of the packet between the two hosts. Capture the show mac-address-table command for both hosts. Is the MAC being learned on one VLAN or two? If so, what are the source ports?

Unfortunately, there is no special command to detect flooding on the switches. Flooding may be detected by capturing a trace of packets seen on a workstation during the time of slowdown or outage. Normally, unicast packets not involving the workstation should not be seen repeatedly on the port. If this is happening, chances are that there is flooding occurring. Packet traces may look different when there are various causes of flooding.

With asymmetric routing, there are likely to be packets to specific MAC addresses that will not stop flooding once the destination replies. With TCNs, the flooding will include many different addresses but should eventually stop then start again.

With Layer 2 forwarding table overflow, the same kind of flooding may be visible as with asymmetric routing. The difference is that there will likely be a high amount of strange packets, or normal packets in abnormal quantities with a different source MAC address.