Found Bug: soft lockup - CPU #2 stuck for 61s! [kswapd0:75] on console.

Asked by Nick Belnap

Ubuntu 10.10
8 core machine with 16 GB RAM, Intel board with (2) 4 core Xeon's. RAID 6 array with 6 disks in software RAID (mdraid)

After large copy job to vbox virutal machine on this host found this on console:

Swap space is on RAID 6 array.

"[314082.372497] Bug: soft lockup - CPU #2 stuck for 61s! [kswapd0: 75]"
followed by some hex code.

This error was preceded by several errors like this:

Info: task kdmflush: 572 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
 message.

Info: task jbd2/md2-8:589 blocked for more than 120 seconds.
Info: task flush-9:2:639 blocked for more than 120 seconds.
Info: task jbd2/dm-0-8:1371 blocked for more than 120 seconds.
Info: task rsyslogd:7393 blocked for more than 120 seconds.
Info: task master:1569 blocked for more than 120 seconds.

The machine was not locked up and I have not yet attempted a reboot but this is very concerning.

Here's the soft lockup error from dmesg:

[314082.372497] BUG: soft lockup - CPU#2 stuck for 61s! [kswapd0:75]
[314082.372550] Modules linked in: vboxnetadp vboxnetflt vboxdrv lp ioatdma parp
ort joydev i7core_edac hed edac_core raid10 raid456 async_pq async_xor xor async
_memcpy async_raid6_recov usbhid hid igb dca raid6_pq async_tx raid1 raid0 multi
path linear
[314082.372576] CPU 2
[314082.372578] Modules linked in: vboxnetadp vboxnetflt vboxdrv lp ioatdma parp
ort joydev i7core_edac hed edac_core raid10 raid456 async_pq async_xor xor async
_memcpy async_raid6_recov usbhid hid igb dca raid6_pq async_tx raid1 raid0 multi
path linear
[314082.372605]
[314082.372609] Pid: 75, comm: kswapd0 Not tainted 2.6.35-28-server #49-Ubuntu S
5520HC/S5520HC
[314082.372612] RIP: 0010:[<ffffffff81118f70>] [<ffffffff81118f70>] zone_nr_fre
e_pages+0x0/0xc0
[314082.372620] RSP: 0018:ffff880265dade08 EFLAGS: 00000282
[314082.372624] RAX: 0000000000000020 RBX: ffff880265dade40 RCX: 000000000000000
0
[314082.372627] RDX: 0000000000000895 RSI: 0000000000000000 RDI: ffff880100000e0
0
[314082.372631] RBP: ffffffff8100aa8e R08: 0000000000000000 R09: 000000000000010
0
[314082.372634] R10: 0000000000000000 R11: 0000000000000003 R12: 000000000000000
0
[314082.372637] R13: ffff880265dade04 R14: ffff8802668144d0 R15: ffff880265daddb
0
[314082.372642] FS: 0000000000000000(0000) GS:ffff880001e20000(0000) knlGS:0000
000000000000
[314082.372646] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[314082.372649] CR2: 0000000001e003c0 CR3: 0000000001a2a000 CR4: 00000000000026e
0
[314082.372652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 000000000000000
0
[314082.372656] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 000000000000040
0
[314082.372660] Process kswapd0 (pid: 75, threadinfo ffff880265dac000, task ffff
8802668144d0)
[314082.372662] Stack:
[314082.372700] ffffffff8110700a ffff880265dade40 ffff880100000000 000000000000
0002
[314082.372705] <0> ffff8802668144d0 ffff880265dade70 ffff8801000040a8 ffff88026
5dadee0
[314082.372711] <0> ffffffff81111f6c ffff880265dadfd8 0000000000000000 000000006
5dadfd8
[314082.372717] Call Trace:
[314082.372760] [<ffffffff8110700a>] ? zone_watermark_ok+0x2a/0xf0
[314082.372765] [<ffffffff81111f6c>] ? kswapd+0x25c/0x300
[314082.372770] [<ffffffff8107fb10>] ? autoremove_wake_function+0x0/0x40
[314082.372775] [<ffffffff81111d10>] ? kswapd+0x0/0x300
[314082.372780] [<ffffffff8107f596>] ? kthread+0x96/0xa0
[314082.372785] [<ffffffff8100aee4>] ? kernel_thread_helper+0x4/0x10
[314082.372790] [<ffffffff8107f500>] ? kthread+0x0/0xa0
[314082.372794] [<ffffffff8100aee0>] ? kernel_thread_helper+0x0/0x10
[314082.372797] Code: 48 89 c2 e8 13 36 fe ff 89 df e8 6c e5 fd ff 48 8b 5d d8 4
c 8b 65 e0 4c 8b 6d e8 4c 8b 75 f0 4c 8b 7d f8 c9 c3 90 90 90 90 90 90 <55> 48 8
9 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 0f
[314082.373196] Call Trace:
[314082.373201] [<ffffffff8110700a>] ? zone_watermark_ok+0x2a/0xf0
[314082.373205] [<ffffffff81111f6c>] ? kswapd+0x25c/0x300
[314082.373210] [<ffffffff8107fb10>] ? autoremove_wake_function+0x0/0x40
[314082.373215] [<ffffffff81111d10>] ? kswapd+0x0/0x300
[314082.373219] [<ffffffff8107f596>] ? kthread+0x96/0xa0
[314082.373224] [<ffffffff8100aee4>] ? kernel_thread_helper+0x4/0x10
[314082.373229] [<ffffffff8107f500>] ? kthread+0x0/0xa0
[314082.373234] [<ffffffff8100aee0>] ? kernel_thread_helper+0x0/0x10

Question information

Language:
English Edit question
Status:
Open
For:
Ubuntu linux Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Jeruvy (jeruvy) said :
#1

You may actually want to bug this report and link it to here. Click on 'Create bug report'.

Can you help with this problem?

Provide an answer of your own, or ask Nick Belnap for more information if necessary.

To post a message you must log in.