[821488.722774] BUG: soft lockup - CPU#3 stuck for 11s! [kswapd0:330] [821488.796655] CPU 3: [821488.796656] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821488.796716] Pid: 330, comm: kswapd0 Not tainted 2.6.24-19-server #1 [821488.796718] RIP: 0010:[<ffffffff80470e65>] [<ffffffff80470e65>] _spin_lock_irqsave+0x15/0x30 [821488.796722] RSP: 0018:ffff810415823df8 EFLAGS: 00000286 [821488.796724] RAX: 0000000000000246 RBX: ffff81000003137d RCX: 0000000000000003 [821488.796725] RDX: 0000000000000001 RSI: ffff810415823ea0 RDI: ffff81000003137d [821488.796728] RBP: ffff810415823d60 R08: 0000000000000000 R09: 0000000000000000 [821488.796730] R10: 0000000000000000 R11: 000000000000057b R12: ffff810415823d60 [821488.796732] R13: ffffffff8028d11e R14: ffff810424ca6888 R15: ffff810424ca6850 [821488.796734] FS: 000000005b7bc950(0000) GS:ffff81041649c180(0000) knlGS:0000000000000000 [821488.796737] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [821488.796738] CR2: 00007f96e3dc8000 CR3: 0000000000201000 CR4: 00000000000006e0 [821488.796740] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821488.796742] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821488.796743] [821488.796744] Call Trace: [821488.796747] [<ffffffff802543f3>] prepare_to_wait+0x23/0x80 [821488.796750] [<ffffffff8029154a>] kswapd+0xfa/0x560 [821488.796756] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821488.796761] [<ffffffff80291450>] kswapd+0x0/0x560 [821488.796765] [<ffffffff80253e9b>] kthread+0x4b/0x80 [821488.796768] [<ffffffff8020d198>] child_rip+0xa/0x12 [821488.796774] [<ffffffff80253e50>] kthread+0x0/0x80 [821488.796777] [<ffffffff8020d18e>] child_rip+0x0/0x12 [821488.796779] [821488.833043] BUG: soft lockup - CPU#4 stuck for 11s! [mysqld:15920] [821488.907990] CPU 4: [821488.907991] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821488.908018] Pid: 15920, comm: mysqld Not tainted 2.6.24-19-server #1 [821488.908020] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821488.908024] RSP: 0018:ffff810428221b30 EFLAGS: 00000286 [821488.908027] RAX: 0000000000000282 RBX: ffff810428221b68 RCX: 0000000000000000 [821488.908028] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821488.908030] RBP: ffff81081815d300 R08: 0000000000000000 R09: 0000000000000000 [821488.908032] R10: 0000000000000000 R11: 0000000000000020 R12: ffffffff8028b62e [821488.908034] R13: ffff8108171c70a0 R14: 0000000000000001 R15: ffffffff80293052 [821488.908037] FS: 0000000056b00950(0063) GS:ffff8108171c7300(0000) knlGS:0000000000000000 [821488.908038] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821488.908041] CR2: 00007f96d2fff000 CR3: 0000000166d75000 CR4: 00000000000006e0 [821488.908042] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821488.908044] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821488.908046] [821488.908046] Call Trace: [821488.908048] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821488.908054] [<ffffffff882a2770>] :jfs:jfs_get_block+0x0/0x240 [821488.908059] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821488.908066] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821488.908073] [<ffffffff8028e577>] ondemand_readahead+0x117/0x1c0 [821488.908077] [<ffffffff80285dad>] do_generic_mapping_read+0x13d/0x3c0 [821488.908080] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821488.908085] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821488.908092] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821488.908097] [<ffffffff8020a88a>] __switch_to+0x1ba/0x310 [821488.908100] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821488.908104] [<ffffffff8046f130>] thread_return+0x3a/0x59a [821488.908111] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821488.908115] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821488.908119] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821488.908124] [821500.508361] BUG: soft lockup - CPU#6 stuck for 11s! [mysqld:15915] [821500.583310] CPU 6: [821500.583311] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821500.583336] Pid: 15915, comm: mysqld Not tainted 2.6.24-19-server #1 [821500.583338] RIP: 0010:[<ffffffff80470e65>] [<ffffffff80470e65>] _spin_lock_irqsave+0x15/0x30 [821500.583342] RSP: 0018:ffff8105a6a13b30 EFLAGS: 00000286 [821500.583344] RAX: 0000000000000282 RBX: ffff8105a6a13b68 RCX: 0000000000000000 [821500.583346] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821500.583348] RBP: ffff8104280b8ab0 R08: 0000000000000000 R09: 0000000000000000 [821500.583350] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8028b62e [821500.583352] R13: 0000000000000000 R14: ffff8104280b8a40 R15: 0000000600000000 [821500.583354] FS: 00000000542fb950(0063) GS:ffff8108171c7b00(0000) knlGS:0000000000000000 [821500.583357] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821500.583358] CR2: 00007f96c8332d78 CR3: 0000000166d75000 CR4: 00000000000006e0 [821500.583361] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821500.583363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821500.583365] [821500.583365] Call Trace: [821500.583367] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821500.583372] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821500.583380] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821500.583387] [<ffffffff8028e4fc>] ondemand_readahead+0x9c/0x1c0 [821500.583392] [<ffffffff80285e8b>] do_generic_mapping_read+0x21b/0x3c0 [821500.583396] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821500.583401] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821500.583407] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821500.583413] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821500.583417] [<ffffffff802326d4>] pick_next_task_fair+0x34/0x50 [821500.583421] [<ffffffff8046eff2>] schedule+0x162/0x266 [821500.583428] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821500.583432] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821500.583437] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821500.583442] [821500.587380] BUG: soft lockup - CPU#3 stuck for 11s! [kswapd0:330] [821500.661276] CPU 3: [821500.661277] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821500.661325] Pid: 330, comm: kswapd0 Not tainted 2.6.24-19-server #1 [821500.661327] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821500.661330] RSP: 0018:ffff810415823df8 EFLAGS: 00000286 [821500.661334] RAX: 0000000000000246 RBX: ffff81000003137d RCX: 0000000000000003 [821500.661335] RDX: 0000000000000001 RSI: ffff810415823ea0 RDI: ffff81000003137d [821500.661338] RBP: ffff810415823d60 R08: 0000000000000000 R09: 0000000000000000 [821500.661339] R10: 0000000000000000 R11: 000000000000057b R12: ffff810415823d60 [821500.661342] R13: ffffffff8028d11e R14: ffff810424ca6888 R15: ffff810424ca6850 [821500.661344] FS: 000000005b7bc950(0000) GS:ffff81041649c180(0000) knlGS:0000000000000000 [821500.661347] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [821500.661348] CR2: 00007f96e3dc8000 CR3: 0000000000201000 CR4: 00000000000006e0 [821500.661350] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821500.661352] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821500.661354] [821500.661355] Call Trace: [821500.661357] [<ffffffff802543f3>] prepare_to_wait+0x23/0x80 [821500.661361] [<ffffffff8029154a>] kswapd+0xfa/0x560 [821500.661369] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821500.661375] [<ffffffff80291450>] kswapd+0x0/0x560 [821500.661378] [<ffffffff80253e9b>] kthread+0x4b/0x80 [821500.661381] [<ffffffff8020d198>] child_rip+0xa/0x12 [821500.661387] [<ffffffff80253e50>] kthread+0x0/0x80 [821500.661390] [<ffffffff8020d18e>] child_rip+0x0/0x12 [821500.661392] [821500.697650] BUG: soft lockup - CPU#4 stuck for 11s! [mysqld:15920] [821500.772610] CPU 4: [821500.772611] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821500.772638] Pid: 15920, comm: mysqld Not tainted 2.6.24-19-server #1 [821500.772641] RIP: 0010:[<ffffffff80470e65>] [<ffffffff80470e65>] _spin_lock_irqsave+0x15/0x30 [821500.772645] RSP: 0018:ffff810428221b30 EFLAGS: 00000286 [821500.772647] RAX: 0000000000000282 RBX: ffff810428221b68 RCX: 0000000000000000 [821500.772649] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821500.772651] RBP: ffff81081815d300 R08: 0000000000000000 R09: 0000000000000000 [821500.772653] R10: 0000000000000000 R11: 0000000000000020 R12: ffffffff8028b62e [821500.772655] R13: ffff8108171c70a0 R14: 0000000000000001 R15: ffffffff80293052 [821500.772658] FS: 0000000056b00950(0063) GS:ffff8108171c7300(0000) knlGS:0000000000000000 [821500.772659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821500.772662] CR2: 00007f96d2fff000 CR3: 0000000166d75000 CR4: 00000000000006e0 [821500.772663] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821500.772666] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821500.772667] [821500.772668] Call Trace: [821500.772670] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821500.772676] [<ffffffff882a2770>] :jfs:jfs_get_block+0x0/0x240 [821500.772680] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821500.772687] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821500.772694] [<ffffffff8028e577>] ondemand_readahead+0x117/0x1c0 [821500.772698] [<ffffffff80285dad>] do_generic_mapping_read+0x13d/0x3c0 [821500.772702] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821500.772708] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821500.772714] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821500.772719] [<ffffffff8020a88a>] __switch_to+0x1ba/0x310 [821500.772722] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821500.772726] [<ffffffff8046f130>] thread_return+0x3a/0x59a [821500.772733] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821500.772737] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821500.772742] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821500.772747] [821512.372967] BUG: soft lockup - CPU#6 stuck for 11s! [mysqld:15915] [821512.447930] CPU 6: [821512.447931] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821512.447952] Pid: 15915, comm: mysqld Not tainted 2.6.24-19-server #1 [821512.447954] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821512.447957] RSP: 0018:ffff8105a6a13b30 EFLAGS: 00000286 [821512.447960] RAX: 0000000000000282 RBX: ffff8105a6a13b68 RCX: 0000000000000000 [821512.447961] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821512.447963] RBP: ffff8104280b8ab0 R08: 0000000000000000 R09: 0000000000000000 [821512.447965] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8028b62e [821512.447967] R13: 0000000000000000 R14: ffff8104280b8a40 R15: 0000000600000000 [821512.447969] FS: 00000000542fb950(0063) GS:ffff8108171c7b00(0000) knlGS:0000000000000000 [821512.447972] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821512.447973] CR2: 00007f96c8332d78 CR3: 0000000166d75000 CR4: 00000000000006e0 [821512.447976] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821512.447977] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821512.447980] [821512.447980] Call Trace: [821512.447982] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821512.447987] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821512.447995] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821512.448002] [<ffffffff8028e4fc>] ondemand_readahead+0x9c/0x1c0 [821512.448007] [<ffffffff80285e8b>] do_generic_mapping_read+0x21b/0x3c0 [821512.448010] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821512.448017] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821512.448023] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821512.448029] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821512.448032] [<ffffffff802326d4>] pick_next_task_fair+0x34/0x50 [821512.448035] [<ffffffff8046eff2>] schedule+0x162/0x266 [821512.448042] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821512.448046] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821512.448050] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821512.448056] [821512.451994] BUG: soft lockup - CPU#3 stuck for 11s! [kswapd0:330] [821512.525896] CPU 3: [821512.525897] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821512.525946] Pid: 330, comm: kswapd0 Not tainted 2.6.24-19-server #1 [821512.525948] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821512.525951] RSP: 0018:ffff810415823df8 EFLAGS: 00000286 [821512.525954] RAX: 0000000000000246 RBX: ffff81000003137d RCX: 0000000000000003 [821512.525955] RDX: 0000000000000001 RSI: ffff810415823ea0 RDI: ffff81000003137d [821512.525958] RBP: ffff810415823d60 R08: 0000000000000000 R09: 0000000000000000 [821512.525960] R10: 0000000000000000 R11: 000000000000057b R12: ffff810415823d60 [821512.525962] R13: ffffffff8028d11e R14: ffff810424ca6888 R15: ffff810424ca6850 [821512.525964] FS: 000000005b7bc950(0000) GS:ffff81041649c180(0000) knlGS:0000000000000000 [821512.525967] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [821512.525969] CR2: 00007f96e3dc8000 CR3: 0000000000201000 CR4: 00000000000006e0 [821512.525971] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821512.525973] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821512.525975] [821512.525975] Call Trace: [821512.525977] [<ffffffff802543f3>] prepare_to_wait+0x23/0x80 [821512.525982] [<ffffffff8029154a>] kswapd+0xfa/0x560 [821512.525989] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821512.525995] [<ffffffff80291450>] kswapd+0x0/0x560 [821512.525998] [<ffffffff80253e9b>] kthread+0x4b/0x80 [821512.526002] [<ffffffff8020d198>] child_rip+0xa/0x12 [821512.526008] [<ffffffff80253e50>] kthread+0x0/0x80 [821512.526011] [<ffffffff8020d18e>] child_rip+0x0/0x12 [821512.526013] [821512.572252] BUG: soft lockup - CPU#4 stuck for 11s! [mysqld:15920] [821512.647209] CPU 4: [821512.647210] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821512.647236] Pid: 15920, comm: mysqld Not tainted 2.6.24-19-server #1 [821512.647239] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821512.647243] RSP: 0018:ffff810428221b30 EFLAGS: 00000286 [821512.647244] RAX: 0000000000000282 RBX: ffff810428221b68 RCX: 0000000000000000 [821512.647247] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821512.647248] RBP: ffff81081815d300 R08: 0000000000000000 R09: 0000000000000000 [821512.647251] R10: 0000000000000000 R11: 0000000000000020 R12: ffffffff8028b62e [821512.647253] R13: ffff8108171c70a0 R14: 0000000000000001 R15: ffffffff80293052 [821512.647255] FS: 0000000056b00950(0063) GS:ffff8108171c7300(0000) knlGS:0000000000000000 [821512.647257] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821512.647260] CR2: 00007f96d2fff000 CR3: 0000000166d75000 CR4: 00000000000006e0 [821512.647261] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821512.647264] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821512.647265] [821512.647265] Call Trace: [821512.647268] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821512.647274] [<ffffffff882a2770>] :jfs:jfs_get_block+0x0/0x240 [821512.647278] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821512.647284] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821512.647292] [<ffffffff8028e577>] ondemand_readahead+0x117/0x1c0 [821512.647296] [<ffffffff80285dad>] do_generic_mapping_read+0x13d/0x3c0 [821512.647300] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821512.647306] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821512.647311] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821512.647316] [<ffffffff8020a88a>] __switch_to+0x1ba/0x310 [821512.647319] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821512.647323] [<ffffffff8046f130>] thread_return+0x3a/0x59a [821512.647331] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821512.647335] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821512.647339] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821512.647345] [821524.237574] BUG: soft lockup - CPU#6 stuck for 11s! [mysqld:15915] [821524.312445] CPU 6: [821524.312445] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821524.312467] Pid: 15915, comm: mysqld Not tainted 2.6.24-19-server #1 [821524.312470] RIP: 0010:[<ffffffff80470e65>] [<ffffffff80470e65>] _spin_lock_irqsave+0x15/0x30 [821524.312474] RSP: 0018:ffff8105a6a13b30 EFLAGS: 00000286 [821524.312476] RAX: 0000000000000282 RBX: ffff8105a6a13b68 RCX: 0000000000000000 [821524.312478] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821524.312480] RBP: ffff8104280b8ab0 R08: 0000000000000000 R09: 0000000000000000 [821524.312482] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8028b62e [821524.312484] R13: 0000000000000000 R14: ffff8104280b8a40 R15: 0000000600000000 [821524.312487] FS: 00000000542fb950(0063) GS:ffff8108171c7b00(0000) knlGS:0000000000000000 [821524.312488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821524.312491] CR2: 00007f96c8332d78 CR3: 0000000166d75000 CR4: 00000000000006e0 [821524.312492] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821524.312495] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821524.312496] [821524.312497] Call Trace: [821524.312499] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821524.312503] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821524.312511] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821524.312519] [<ffffffff8028e4fc>] ondemand_readahead+0x9c/0x1c0 [821524.312523] [<ffffffff80285e8b>] do_generic_mapping_read+0x21b/0x3c0 [821524.312527] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821524.312533] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821524.312538] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821524.312544] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821524.312547] [<ffffffff802326d4>] pick_next_task_fair+0x34/0x50 [821524.312551] [<ffffffff8046eff2>] schedule+0x162/0x266 [821524.312557] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821524.312560] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821524.312564] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821524.312570] [821524.316593] BUG: soft lockup - CPU#3 stuck for 11s! [kswapd0:330] [821524.390514] CPU 3: [821524.390515] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821524.390565] Pid: 330, comm: kswapd0 Not tainted 2.6.24-19-server #1 [821524.390567] RIP: 0010:[<ffffffff80470e62>] [<ffffffff80470e62>] _spin_lock_irqsave+0x12/0x30 [821524.390570] RSP: 0018:ffff810415823df8 EFLAGS: 00000286 [821524.390573] RAX: 0000000000000246 RBX: ffff81000003137d RCX: 0000000000000003 [821524.390574] RDX: 0000000000000001 RSI: ffff810415823ea0 RDI: ffff81000003137d [821524.390577] RBP: ffff810415823d60 R08: 0000000000000000 R09: 0000000000000000 [821524.390578] R10: 0000000000000000 R11: 000000000000057b R12: ffff810415823d60 [821524.390581] R13: ffffffff8028d11e R14: ffff810424ca6888 R15: ffff810424ca6850 [821524.390583] FS: 000000005b7bc950(0000) GS:ffff81041649c180(0000) knlGS:0000000000000000 [821524.390585] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [821524.390587] CR2: 00007f96e3dc8000 CR3: 0000000000201000 CR4: 00000000000006e0 [821524.390589] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821524.390591] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821524.390593] [821524.390594] Call Trace: [821524.390596] [<ffffffff802543f3>] prepare_to_wait+0x23/0x80 [821524.390600] [<ffffffff8029154a>] kswapd+0xfa/0x560 [821524.390608] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821524.390614] [<ffffffff80291450>] kswapd+0x0/0x560 [821524.390617] [<ffffffff80253e9b>] kthread+0x4b/0x80 [821524.390620] [<ffffffff8020d198>] child_rip+0xa/0x12 [821524.390627] [<ffffffff80253e50>] kthread+0x0/0x80 [821524.390629] [<ffffffff8020d18e>] child_rip+0x0/0x12 [821524.390632] [821524.436858] BUG: soft lockup - CPU#4 stuck for 11s! [mysqld:15920] [821524.511723] CPU 4: [821524.511724] Modules linked in: nfs lockd nfs_acl sunrpc ac dock sbs sbshc battery video output container iptable_filter ip_tables x_tables jfs sr_mod cdrom parport_pc lp parport loop joydev shpchp ipv6 evdev pcspkr pci_hotplug pata_acpi i2c_nforce2 i2c_core button usb_storage ext3 jbd mbcache libusual usbhid hid sg sd_mod sata_nv ata_generic ehci_hcd aacraid ohci_hcd libata forcedeth usbcore scsi_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse [821524.511750] Pid: 15920, comm: mysqld Not tainted 2.6.24-19-server #1 [821524.511752] RIP: 0010:[<ffffffff80470e65>] [<ffffffff80470e65>] _spin_lock_irqsave+0x15/0x30 [821524.511757] RSP: 0018:ffff810428221b30 EFLAGS: 00000286 [821524.511758] RAX: 0000000000000282 RBX: ffff810428221b68 RCX: 0000000000000000 [821524.511761] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff81000003137d [821524.511763] RBP: ffff81081815d300 R08: 0000000000000000 R09: 0000000000000000 [821524.511765] R10: 0000000000000000 R11: 0000000000000020 R12: ffffffff8028b62e [821524.511767] R13: ffff8108171c70a0 R14: 0000000000000001 R15: ffffffff80293052 [821524.511770] FS: 0000000056b00950(0063) GS:ffff8108171c7300(0000) knlGS:0000000000000000 [821524.511771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [821524.511774] CR2: 00007f96d2fff000 CR3: 0000000166d75000 CR4: 00000000000006e0 [821524.511775] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [821524.511778] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [821524.511779] [821524.511780] Call Trace: [821524.511782] [<ffffffff8023459d>] __wake_up+0x2d/0x70 [821524.511788] [<ffffffff882a2770>] :jfs:jfs_get_block+0x0/0x240 [821524.511792] [<ffffffff8028bec4>] __alloc_pages+0xd4/0x3d0 [821524.511799] [<ffffffff8028e1e0>] __do_page_cache_readahead+0xe0/0x210 [821524.511807] [<ffffffff8028e577>] ondemand_readahead+0x117/0x1c0 [821524.511811] [<ffffffff80285dad>] do_generic_mapping_read+0x13d/0x3c0 [821524.511815] [<ffffffff80285250>] file_read_actor+0x0/0x160 [821524.511820] [<ffffffff802878bf>] generic_file_aio_read+0xff/0x1b0 [821524.511826] [<ffffffff802b4d69>] do_sync_read+0xd9/0x120 [821524.511831] [<ffffffff8020a88a>] __switch_to+0x1ba/0x310 [821524.511834] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 [821524.511838] [<ffffffff8046f130>] thread_return+0x3a/0x59a [821524.511845] [<ffffffff802b571d>] vfs_read+0xed/0x190 [821524.511849] [<ffffffff802b5d54>] sys_pread64+0x84/0xa0 [821524.511854] [<ffffffff8020c37e>] system_call+0x7e/0x83 [821524.511860]