Cisco 3750 crashes (Exception (0x2000)!)

Problem

Cisco 3750 switches drops or goes offline. Monitoring the switch shows random disconnects and log shows errors such as:

*Mar 1 00:01:21.528: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack
*Mar 1 00:01:23.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Mar 1 00:01:24.414: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
00:01:29: %SYS-5-CONFIG_I: Configured from memory by console
00:01:29: %STACKMGR-5-SWITCH_READY: Switch 1 is READY
00:01:29: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state DOWN
00:01:29: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN
00:01:29: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY
00:01:30: %SYS-5-RESTART: System restarted —
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE9, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Mon 03-Mar-14 22:44 by prod_rel_team
00:01:30: %SNMP-5-COLDSTART: SNMP agent on host switch-xyz is undergoing a cold start
00:01:30: %SSH-5-ENABLED: SSH 2.0 has been enabled
00:01:31: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
00:01:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
00:02:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
00:02:30: %PLATFORM-1-CRASHED: System previously crashed with the following message:
00:02:30: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(55)SE9, RELEASE SOFTWARE (fc1)
00:02:30: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport
00:02:30: %PLATFORM-1-CRASHED: Copyright (c) 1986-2014 by Cisco Systems, Inc.
00:02:30: %PLATFORM-1-CRASHED: Compiled Mon 03-Mar-14 22:44 by prod_rel_team
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: SRR0 = 0x0294E300 SRR1 = 0x00029230 SRR2 = 0x01BD0DF8 SRR3 = 0x00029230
00:02:30: %PLATFORM-1-CRASHED: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x10000000
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: CPU Register Context:
00:02:30: %PLATFORM-1-CRASHED: Vector = 0x00002000 PC = 0x01BD0DF8 MSR = 0x00029230 CR = 0x20000008
00:02:30: %PLATFORM-1-CRASHED: LR = 0x01BD0DF8 CTR = 0x02B486D8 XER = 0x6000005E
00:02:30: %PLATFORM-1-CRASHED: R0 = 0x01BD0DF8 R1 = 0x040B4528 R2 = 0x00000000 R3 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R4 = 0x0294E300 R5 = 0x00029230 R6 = 0x040A810C R7 = 0xBEEFCAFE
00:02:30: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x03C10000 R10 = 0x03B38F98 R11 = 0x03B38EC8
00:02:30: %PLATFORM-1-CRASHED: R12 = 0x001AD7B9 R13 = 0x00110000 R14 = 0x02941C28 R15 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
00:02:30: %PLATFORM-1-CRASHED: R28 = 0x00000000 R29 = 0x02BE87AC R30 = 0xD9C2CDD9 R31 = 0x00000003
00:02:30: %PLATFORM-1-CRASHED:
00:02:30: %PLATFORM-1-CRASHED: Stack trace:
00:02:30: %PLATFORM-1-CRASHED: PC = 0x01BD0DF8, SP = 0x040B4528
00:02:30: %PLATFORM-1-CRASHED: Frame 00: SP = 0x040B4538 PC = 0x01BD0DF8
00:02:30: %PLATFORM-1-CRASHED: Frame 01: SP = 0x040B4558 PC = 0x01BD40F4
00:02:30: %PLATFORM-1-CRASHED: Frame 02: SP = 0x040B4570 PC = 0x02941CE4
00:02:30: %PLATFORM-1-CRASHED: Frame 03: SP = 0x040B4578 PC = 0x01BD2DC8
00:02:30: %PLATFORM-1-CRASHED: Frame 04: SP = 0x00000000 PC = 0x01BC9840

 

Cause

Multiple suggested causes:

  1. slow memory leak in stacks of 3750 switches that
    have IP routing disabled
  2. Possible SNMP over polling of some OID’s that are not supported.

 

Solution

There are a number of recommended solutions such as those below

  1. IOS upgrade
  2. Enable ip routing and reload switch stack
  3. Disable SNMP polling

The solution that seemed to work for me was #3 disabling SNMP polling from our monitoring application to the affected switch.

 

Tested Platform

Cisco 3750 Switches
Cisco IOS C3750-IPBASEK9-M Version 12.2(55)SE9

Hits: 712

Leave a Reply