BUG: spinlock cpu recursion on CPU#0, mysqld/2906 (Not tainted) lock: ffff810276e0c860, .magic: dead4ead, .owner: mysqld/2992, .owner_cpu: 0 Call Trace: <ffffffff80213b35>{spin_bug+176} <ffffffff80207643>{_raw_spin_lock+88} <ffffffff880084c9>{:scsi_mod:scsi_request_fn+442} <ffffffff80229975>{sync_page+0} <ffffffff8025ee87>{generic_unplug_device+24} <ffffffff80261857>{blk_backing_dev_unplug+82} <ffffffff802299af>{sync_page+58} <ffffffff8026823a>{__wait_on_bit_lock+55} <ffffffff802429e1>{__lock_page+94} <ffffffff8029ee5f>{wake_bit_function+0} <ffffffff8020ba74>{do_generic_mapping_read+506} <ffffffff8020ca31>{file_read_actor+0} <ffffffff8020bdcb>{__generic_file_aio_read+343} <ffffffff802b4c9c>{generic_file_read+172} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff802b2244>{audit_syscall_entry+310} <ffffffff8020accd>{vfs_read+203} <ffffffff80213a65>{sys_pread64+80} <ffffffff80262425>{tracesys+113} <ffffffff80262485>{tracesys+209} Kernel panic - not syncing: bad locking NMI Watchdog detected LOCKUP on CPU 1 CPU 1 Modules linked in: ipv6 autofs4 nfs lockd nfs_acl rfcomm l2cap bluetooth sunrpc jfs dm_mod video button battery acpi_memhotplug ac ohci_hcd i2c_amd8111 i2c_amd756 i2c_core tg3 floppy ext3 jbd aacraid sd_mod scsi_mod Pid: 2894, comm: mysqld Not tainted 2.6.17-1.2142_FC4smp #1 RIP: 0010:[<ffffffff8020765a>] <ffffffff8020765a>{_raw_spin_lock+111} RSP: 0018:ffff8101f5595ba8 EFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff810276e0c860 RCX: 0000000000000001 RDX: 00000000014144e4 RSI: ffff810275c43328 RDI: ffff810276e0c860 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff81000d768540 R10: ffff81022c133ac0 R11: 0000000000000000 R12: ffff810276c89800 R13: 0000000000000002 R14: ffff810276e0c800 R15: ffff81027676e100 FS: 0000000049db6960(0063) GS:ffff81027706bf40(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055555566373c CR3: 0000000477a1d000 CR4: 00000000000006e0 Process mysqld (pid: 2894, threadinfo ffff8101f5594000, task ffff8101f5584860) Stack: ffff81000d768540 ffffffff880084c9 ffff810275c43328 ffff81027676e100 0000000000000000 ffff81028005d250 0000000000000002 ffffffff80229975 0000000000000000 ffffffff8025ee87 Call Trace: <ffffffff880084c9>{:scsi_mod:scsi_request_fn+442} <ffffffff80229975>{sync_page+0} <ffffffff8025ee87>{generic_unplug_device+24} <ffffffff80261857>{blk_backing_dev_unplug+82} <ffffffff802299af>{sync_page+58} <ffffffff8026823a>{__wait_on_bit_lock+55} <ffffffff802429e1>{__lock_page+94} <ffffffff8029ee5f>{wake_bit_function+0} <ffffffff8020ba74>{do_generic_mapping_read+506} <ffffffff8020ca31>{file_read_actor+0} <ffffffff8020bdcb>{__generic_file_aio_read+343} <ffffffff802b4c9c>{generic_file_read+172} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff802b2244>{audit_syscall_entry+310} <ffffffff8020accd>{vfs_read+203} <ffffffff80213a65>{sys_pread64+80} <ffffffff80262425>{tracesys+113} <ffffffff80262485>{tracesys+209} Code: 7f 60 48 ff c2 48 69 05 3e 4d 34 00 fa 00 00 00 48 39 c2 72 console shuts up ... NMI Watchdog detected LOCKUP on CPU 3 CPU 3 Modules linked in: ipv6 autofs4 nfs lockd nfs_acl rfcomm l2cap bluetooth sunrpc jfs dm_mod video button battery acpi_memhotplug ac ohci_hcd i2c_amd8111 i2c_amd756 i2c_core tg3 floppy ext3 jbd aacraid sd_mod scsi_mod Pid: 2978, comm: mysqld Not tainted 2.6.17-1.2142_FC4smp #1 RIP: 0010:[<ffffffff8020765a>] <ffffffff8020765a>{_raw_spin_lock+111} RSP: 0018:ffff8101fd1ddba8 EFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff810276e0c860 RCX: 0000000000000001 RDX: 000000000cd80746 RSI: ffff810477b7bba8 RDI: ffff810276e0c860 RBP: 0000000000000000 R08: 0000000000000000 R09: ffff810477cb9600 R10: ffff81046f411cc0 R11: 0000000000000000 R12: ffff810276c89800 R13: 0000000000000002 R14: ffff810276e0c800 R15: ffff81027676e100 FS: 000000004adca960(0063) GS:ffff8102801bc140(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aacadd8b000 CR3: 0000000477a1d000 CR4: 00000000000006e0 Process mysqld (pid: 2978, threadinfo ffff8101fd1dc000, task ffff8101fd1c0820) Stack: ffff810477cb9600 ffffffff880084c9 ffff810477b7bba8 ffff81027676e100 0000000000000000 ffff81028006d1f0 0000000000000002 ffffffff80229975 0000000000000000 ffffffff8025ee87 Call Trace: <ffffffff880084c9>{:scsi_mod:scsi_request_fn+442} <ffffffff80229975>{sync_page+0} <ffffffff8025ee87>{generic_unplug_device+24} <ffffffff80261857>{blk_backing_dev_unplug+82} <ffffffff802299af>{sync_page+58} <ffffffff8026823a>{__wait_on_bit_lock+55} <ffffffff802429e1>{__lock_page+94} <ffffffff8029ee5f>{wake_bit_function+0} <ffffffff8020ba74>{do_generic_mapping_read+506} <ffffffff8020ca31>{file_read_actor+0} <ffffffff8020bdcb>{__generic_file_aio_read+343} <ffffffff802b4c9c>{generic_file_read+172} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff802b2244>{audit_syscall_entry+310} <ffffffff8020accd>{vfs_read+203} <ffffffff80213a65>{sys_pread64+80} <ffffffff80262425>{tracesys+113} <ffffffff80262485>{tracesys+209} Code: 7f 60 48 ff c2 48 69 05 3e 4d 34 00 fa 00 00 00 48 39 c2 72 console shuts up ... NMI Watchdog detected LOCKUP on CPU 0 CPU 0 Modules linked in: ipv6 autofs4 nfs lockd nfs_acl rfcomm l2cap bluetooth sunrpc jfs dm_mod video button battery acpi_memhotplug ac ohci_hcd i2c_amd8111 i2c_amd756 i2c_core tg3 floppy ext3 jbd aacraid sd_mod scsi_mod Pid: 2906, comm: mysqld Not tainted 2.6.17-1.2142_FC4smp #1 RIP: 0010:[<ffffffff80278682>] <ffffffff80278682>{__smp_call_function+100} RSP: 0018:ffff8101f6dffa38 EFLAGS: 00000097 RAX: 0000000000000002 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 00000000000000ff RSI: 0000000000000003 RDI: 00000000000000c0 RBP: ffffffff8027847e R08: 0000000000000004 R09: 0000000000000004 R10: ffff8101f6dff998 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000003 R14: ffff810276e0c800 R15: ffff81027676e100 FS: 000000004a002960(0063) GS:ffffffff806aa000(0000) knlGS:00000000f7fdf6c0 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaaacdca70 CR3: 0000000477a1d000 CR4: 00000000000006e0 Process mysqld (pid: 2906, threadinfo ffff8101f6dfe000, task ffff8101f6ddc7e0) Stack: ffffffff8027847e 0000000000000000 0000000000000002 ffffffff00000000 ffffffff80480c5a 0000000000000001 ffff810276e0c860 ffffffff80480ced 0000000000000002 ffffffff802786d3 Call Trace: <ffffffff8027847e>{smp_really_stop_cpu+0} <ffffffff802786d3>{smp_send_stop+42} <ffffffff8028f652>{panic+148} <ffffffff802a413d>{module_text_address+49} <ffffffff8029d2ce>{kernel_text_address+26} <ffffffff8026fab4>{show_trace+451} <ffffffff8026fb19>{dump_stack+14} <ffffffff80213b5e>{spin_bug+217} <ffffffff80207643>{_raw_spin_lock+88} <ffffffff880084c9>{:scsi_mod:scsi_request_fn+442} <ffffffff80229975>{sync_page+0} <ffffffff8025ee87>{generic_unplug_device+24} <ffffffff80261857>{blk_backing_dev_unplug+82} <ffffffff802299af>{sync_page+58} <ffffffff8026823a>{__wait_on_bit_lock+55} <ffffffff802429e1>{__lock_page+94} <ffffffff8029ee5f>{wake_bit_function+0} <ffffffff8020ba74>{do_generic_mapping_read+506} <ffffffff8020ca31>{file_read_actor+0} <ffffffff8020bdcb>{__generic_file_aio_read+343} <ffffffff802b4c9c>{generic_file_read+172} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff802b2244>{audit_syscall_entry+310} <ffffffff8020accd>{vfs_read+203} <ffffffff80213a65>{sys_pread64+80} <ffffffff80262425>{tracesys+113} <ffffffff80262485>{tracesys+209} Code: f3 90 8b 44 24 10 44 39 e8 75 f5 85 db 75 04 eb 0b f3 90 8b console shuts up ... NMI Watchdog detected LOCKUP on CPU 2 CPU 2 Modules linked in: ipv6 autofs4 nfs lockd nfs_acl rfcomm l2cap bluetooth sunrpc jfs dm_mod video button battery acpi_memhotplug ac ohci_hcd i2c_amd8111 i2c_amd756 i2c_core tg3 floppy ext3 jbd aacraid sd_mod scsi_mod Pid: 0, comm: swapper Not tainted 2.6.17-1.2142_FC4smp #1 RIP: 0010:[<ffffffff8020765a>] <ffffffff8020765a>{_raw_spin_lock+111} RSP: 0018:ffff81028016bec8 EFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff810276e0c860 RCX: 0000000000000001 RDX: 000000005763a216 RSI: 0000000000000004 RDI: ffff810276e0c860 RBP: ffff810276c89800 R08: ffff810277231070 R09: 0000000000000004 R10: ffff8102767e2c90 R11: ffffffff8800275e R12: ffff810276c89800 R13: 0000000000000002 R14: ffff8102770a5f18 R15: 0000000000000246 FS: 000000004b10b960(0000) GS:ffff810280158240(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00002aacadd87000 CR3: 0000000477a1d000 CR4: 00000000000006e0 Process swapper (pid: 0, threadinfo ffff8102770a4000, task ffff810280159820) Stack: 0000000000000246 ffffffff80269629 ffff810276e0c800 ffffffff88008926 ffff810276e0c800 ffff8103c5352640 ffff810276c89800 ffffffff88002489 0000000000000001 ffffffff806c1fc0 Call Trace: <IRQ> <ffffffff80269629>{_spin_lock_irqsave+9} <ffffffff88008926>{:scsi_mod:scsi_device_unbusy+19} <ffffffff88002489>{:scsi_mod:scsi_finish_command+23} <ffffffff8023a096>{blk_done_softirq+113} <ffffffff8021190c>{__do_softirq+87} <ffffffff802636f2>{call_softirq+30} <ffffffff80270bfa>{do_softirq+44} <ffffffff80270c3f>{do_IRQ+65} <ffffffff8026f632>{default_idle+0} <ffffffff8026282e>{ret_from_intr+0} <EOI> <ffffffff8026f632>{default_idle+0} <ffffffff8026f65f>{default_idle+45} <ffffffff8024c24a>{cpu_idle+151} <ffffffff80279769>{start_secondary+1140} Code: 7f 60 48 ff c2 48 69 05 3e 4d 34 00 fa 00 00 00 48 39 c2 72 console shuts up ... <0>BUG: spinlock lockup on CPU#0, mysqld/2935, ffff810276e0c860 (Not tainted) Call Trace: <ffffffff802076b8>{_raw_spin_lock+205} <ffffffff880084c9>{:scsi_mod:scsi_request_fn+442} <ffffffff80229975>{sync_page+0} <ffffffff8025ee87>{generic_unplug_device+24} <ffffffff80261857>{blk_backing_dev_unplug+82} <ffffffff802299af>{sync_page+58} <ffffffff8026823a>{__wait_on_bit_lock+55} <ffffffff802429e1>{__lock_page+94} <ffffffff8029ee5f>{wake_bit_function+0} <ffffffff8020ba74>{do_generic_mapping_read+506} <ffffffff8020ca31>{file_read_actor+0} <ffffffff8020bdcb>{__generic_file_aio_read+343} <ffffffff802b4c9c>{generic_file_read+172} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff8029ee31>{autoremove_wake_function+0} <ffffffff80269683>{_spin_unlock_irq+6} <ffffffff802674c4>{thread_return+94} <ffffffff802b2244>{audit_syscall_entry+310} <ffffffff8020accd>{vfs_read+203} <ffffffff80213a65>{sys_pread64+80} <ffffffff80262425>{tracesys+113} <ffffffff80262485>{tracesys+209}