张文宁,某局点S12518的1框slot2板卡异常重启问题
组网及说明
/
告警信息
/
问题描述
接到某局点反馈，江西移动S12518的1框slot2板卡异常重启，数据库连接有告警，业务暂未受影响。
[17:27:33]LST2XP32REC1 1/2: uptime is 0 week, 0 day, 1 hour, 13 minutes
[17:27:33]Last reboot reason : Exception reboot
[17:27:33]1024 Mbytes SDRAM
[17:27:33]0 Kbytes NVRAM Memory
[17:27:33]Type : LST2XP32REC1
[17:27:33]Software : S12500-CMW520-R1829P04
[17:27:33]Patch : NONE
[17:27:33]PCB : Ver.A
[17:27:33]Board Cpu:
[17:27:33] Number of Cpld: 4
[17:27:33] Cpld 0:
[17:27:33] SoftWare : 001
[17:27:33] Cpld 1:
[17:27:33] SoftWare : 001
[17:27:33] Cpld 2:
[17:27:33] SoftWare : 001
[17:27:33] Cpld 3:
[17:27:33] SoftWare : 001
[17:27:33]PowChipA : 003
[17:27:33]PowChipB : 003
[17:27:33]PowChipC : 003
[17:27:33]CpuCard
[17:27:33] Type : LSR1CPAE
[17:27:33] PCB : Ver.C
[17:27:33] Number of Cpld: 1
[17:27:33] Cpld 0:
[17:27:33] SoftWare : 001
[17:27:33] BootRom : 1.15
[17:27:33]Mbus card
[17:27:33] Type : LSR1MBCB
[17:27:33] Software : 115
[17:27:33] PCB : Ver.B
[17:27:33]Daughter Card
[17:27:33] Type : LST2DXP16REC
[17:27:33] PCB : Ver.A
过程分析
根据反馈回来的诊断信息和diagfile信息，可以明显看到是二次电源故障导致的单板掉电重启：
[17:28:11]Slot 2:
[17:28:11]DEVICE_NAME : LST2XP32REC1
[17:28:11]DEVICE_SERIAL_NUMBER : 210231A0QBH126000003
[17:28:11]MAC_ADDRESS : NONE
[17:28:11]MANUFACTURING_DATE : 2012-10-26
[17:28:11]VENDOR_NAME : H3C
[17:41:33]%@4886921%Oct 15 16:12:27:637 2022 JX-HC-NCBOSS-S12518-IRF DEVD/3/BOARD REBOOT: Chassis 1 slot 2 need to be rebooted automatically!
[17:41:33]%@4886944%Oct 15 16:12:28:934 2022 JX-HC-NCBOSS-S12518-IRF DEVM/2/BOARD_STATE_FAULT: Board state changes to FAULT on Chassis 1 Slot 2, type is LST2XP32REC1.
[17:41:33]%@4886945%Oct 15 16:12:28:934 2022 JX-HC-NCBOSS-S12518-IRF STM/3/STM_LINK_STATUS_DOWN:
[17:41:33] IRF port 1 is down.
[17:41:33]%@4887007%Oct 15 16:13:12:772 2022 JX-HC-NCBOSS-S12518-IRF DEVM/4/BOARD_LOADING: Board is loading file on Chassis 1 Slot 2.
[17:41:33]%@4887009%Oct 15 16:13:14:396 2022 JX-HC-NCBOSS-S12518-IRF DEVM/5/LOAD_FINISHED: Board has finished loading file on Chassis 1 Slot 2.
[17:41:33]%@4887017%Oct 15 16:14:23:053 2022 JX-HC-NCBOSS-S12518-IRF DEVM/5/BOARD_STATE_NORMAL: Board state changes to NORMAL on Chassis 1 Slot 2, type is LST2XP32REC1.
[17:42:31]%@16868291 Oct 15 16:12:37:314 2022 JX-HC-NCBOSS-S12518-IRF DRIVER/3/DRV_DRIVER:0xcc011601 [3918]: Mbus 2 diag info:
[17:42:31]board power error!
[17:42:31][5/1864]: power off board.
[17:42:31]%@16868301 Oct 15 16:13:37:918 2022 JX-HC-NCBOSS-S12518-IRF DRIVER/3/DRV_DRIVER:0xcc011601 [3918]: Mbus 2 diag info:
[17:42:31][5/1804]: power on board.
[17:42:31]%@16868310 Oct 15 16:14:27:544 2022 JX-HC-NCBOSS-S12518-IRF Chassis=1-Slot=2; SYSM/3/DRV_SYSM_CMD:0xcc000000 [397]: Board has experienced a power down event.
同时因为堆叠口均在1框slot2上，并且没有配置IRF auto-merge自动合并命令，因此设备当下处于分裂状态，需要手工重启备框才能合入堆叠：
[17:27:55]%Oct 15 17:26:56:479 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
[17:27:55]%Oct 15 17:27:06:582 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
[17:27:55]%Oct 15 17:27:09:804 2022 JX-HC-NCBOSS-S12518-IRF SHELL/6/SHELL_CMD: -Task=vt0-IPAddr=10.183.235.236-User=aiuap; Command is dir
[17:27:55]%Oct 15 17:27:16:686 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
[17:27:55]%Oct 15 17:27:26:688 2022 JX-HC-NCBOSS-S12518-IRF SHELL/6/SHELL_CMD: -Task=vt0-IPAddr=10.183.235.236-User=aiuap; Command is dis diag
[17:27:55]%Oct 15 17:27:26:789 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
[17:27:55]%Oct 15 17:27:36:893 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
[17:27:55]%Oct 15 17:27:47:034 2022 JX-HC-NCBOSS-S12518-IRF STM/5/STM_MERGE:
[17:27:55] IRF merge occurs and the IRF system does not need to reboot.
解决方法
1框slot2硬件故障，返修处理。
