Jun 11 11:45:41 debian8 kernel: [250981.461697] ------------[ cut here ]------------ Jun 11 11:45:41 debian8 kernel: [250981.461709] WARNING: CPU: 10 PID: 0 at /build/linux-Xbe5gu/linux-4.0.2/kernel/watchdog.c:290 watchdog_overflow_callback+0x92/0xc0() Jun 11 11:45:41 debian8 kernel: [250981.461710] Watchdog detected hard LOCKUP on cpu 10 Jun 11 11:45:41 debian8 kernel: [250981.461712] Modules linked in: tcm_fc libfc ib_srpt ib_cm ib_sa ib_mad ib_core ib_addr iscsi_target_mod tcm_qla2xxx tcm_loop vhost_scsi vhost target_core_file target_core_iblock target_core_pscsi target_core_mod configfs ses enclosure mpt2sas dm_round_robin dm_multipath scsi_dh nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc bonding zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 iTCO_wdt lrw gf128mul drm iTCO_vendor_support glue_helper dm_mod joydev evdev ablk_helper cryptd pcspkr sb_edac edac_core mei_me lpc_ich mei i2c_i801 ipmi_si mfd_core ioatdma 8250_fintek ipmi_msghandler tpm_tis processor shpchp tpm thermal_sys wmi button autofs4 ext4 crc16 mbcache jbd2 sg sd_mod hid_generic usbhid hid isci crc32c_intel ahci ehci_pci libsas libahci qla2xxx ehci_hcd mpt3sas igb raid_class scsi_transport_fc libata i2c_algo_bit scsi_transport_sas i2c_core usbcore dca ptp usb_common pps_core scsi_mod Jun 11 11:45:41 debian8 kernel: [250981.461772] CPU: 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.0-1-amd64 #1 Debian 4.0.2-1 Jun 11 11:45:41 debian8 kernel: [250981.461773] Hardware name: Supermicro SSG-2027R-AR24NV/X9DRH-IF-NV, BIOS 3.0 12/11/2013 Jun 11 11:45:41 debian8 kernel: [250981.461775] 0000000000000000 ffffffff8172ddf0 ffffffff8155df6b ffff88203f005b20 Jun 11 11:45:41 debian8 kernel: [250981.461777] ffffffff8106ce91 ffff882038741800 0000000000000000 ffff88203f005c40 Jun 11 11:45:41 debian8 kernel: [250981.461778] ffff88203f005ef8 0000000000000000 ffffffff8106cf0a ffffffff8172ddc8 Jun 11 11:45:41 debian8 kernel: [250981.461780] Call Trace: Jun 11 11:45:41 debian8 kernel: [250981.461782] [] ? dump_stack+0x40/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461793] [] ? warn_slowpath_common+0x81/0xb0 Jun 11 11:45:41 debian8 kernel: [250981.461794] [] ? warn_slowpath_fmt+0x4a/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461796] [] ? watchdog_overflow_callback+0x92/0xc0 Jun 11 11:45:41 debian8 kernel: [250981.461801] [] ? __perf_event_overflow+0x86/0x230 Jun 11 11:45:41 debian8 kernel: [250981.461806] [] ? x86_perf_event_set_period+0xc9/0x170 Jun 11 11:45:41 debian8 kernel: [250981.461809] [] ? intel_pmu_handle_irq+0x1ba/0x3a0 Jun 11 11:45:41 debian8 kernel: [250981.461816] [] ? ioremap_page_range+0x21a/0x320 Jun 11 11:45:41 debian8 kernel: [250981.461821] [] ? ghes_copy_tofrom_phys+0x129/0x2c0 Jun 11 11:45:41 debian8 kernel: [250981.461822] [] ? ghes_read_estatus+0x71/0x150 Jun 11 11:45:41 debian8 kernel: [250981.461824] [] ? perf_event_nmi_handler+0x2a/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461829] [] ? native_sched_clock+0x26/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461830] [] ? nmi_handle+0x86/0x120 Jun 11 11:45:41 debian8 kernel: [250981.461832] [] ? default_do_nmi+0x44/0x130 Jun 11 11:45:41 debian8 kernel: [250981.461833] [] ? do_nmi+0x88/0xd0 Jun 11 11:45:41 debian8 kernel: [250981.461838] [] ? end_repeat_nmi+0x1e/0x2e Jun 11 11:45:41 debian8 kernel: [250981.461841] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461842] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461843] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461844] <> [] ? qlt_fc_port_deleted+0x52/0xb0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461861] [] ? qla2x00_mark_device_lost+0x153/0x2e0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461867] [] ? qla2x00_async_event+0xd6a/0x17f0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461873] [] ? qla24xx_msix_default+0x1ea/0x250 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461878] [] ? handle_irq_event_percpu+0x75/0x190 Jun 11 11:45:41 debian8 kernel: [250981.461880] [] ? read_tsc+0x5/0x10 Jun 11 11:45:41 debian8 kernel: [250981.461881] [] ? handle_irq_event+0x3e/0x60 Jun 11 11:45:41 debian8 kernel: [250981.461884] [] ? handle_edge_irq+0x81/0x140 Jun 11 11:45:41 debian8 kernel: [250981.461887] [] ? handle_irq+0x1d/0x30 Jun 11 11:45:41 debian8 kernel: [250981.461891] [] ? do_IRQ+0x46/0xe0 Jun 11 11:45:41 debian8 kernel: [250981.461893] [] ? common_interrupt+0x6d/0x6d Jun 11 11:45:41 debian8 kernel: [250981.461894] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:45:41 debian8 kernel: [250981.461902] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:45:41 debian8 kernel: [250981.461905] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:45:41 debian8 kernel: [250981.461908] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:45:41 debian8 kernel: [250981.461911] [] ? start_secondary+0x199/0x1c0 Jun 11 11:45:41 debian8 kernel: [250981.461912] ---[ end trace 82cffa7a20869247 ]--- Jun 11 11:45:41 debian8 kernel: [250991.901597] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 12, t=5252 jiffies, g=3003262, c=3003261, q=18101) Jun 11 11:45:41 debian8 kernel: [250991.901674] Task dump for CPU 10: Jun 11 11:45:41 debian8 kernel: [250991.901676] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:45:41 debian8 kernel: [250991.901678] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:45:41 debian8 kernel: [250991.901679] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:45:41 debian8 kernel: [250991.901681] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:45:41 debian8 kernel: [250991.901683] Call Trace: Jun 11 11:45:41 debian8 kernel: [250991.901687] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:45:41 debian8 kernel: [250991.901689] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:45:41 debian8 kernel: [250991.901691] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:45:41 debian8 kernel: [250991.901693] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:45:41 debian8 kernel: [250991.901695] [] ? start_secondary+0x199/0x1c0 Jun 11 11:45:53 debian8 kernel: [251002.954745] sd 14:0:47:0: attempting task abort! scmd(ffff88146bfee540) Jun 11 11:45:53 debian8 kernel: [251002.954751] sd 14:0:47:0: [sdau] CDB: Write(16) 8a 00 00 00 00 00 3a 81 c5 4b 00 00 00 09 00 00 Jun 11 11:45:53 debian8 kernel: [251002.954758] scsi target14:0:47: handle(0x003d), sas_address(0x5000cca027576842), phy(30) Jun 11 11:45:53 debian8 kernel: [251002.954760] scsi target14:0:47: enclosure_logical_id(0x50030480015ce63f), slot(18) Jun 11 11:45:53 debian8 kernel: [251002.954866] sd 15:0:36:0: attempting task abort! scmd(ffff8814b3274200) Jun 11 11:45:53 debian8 kernel: [251002.954870] sd 15:0:36:0: [sdck] CDB: Write(16) 8a 00 00 00 00 00 20 11 e9 b4 00 00 00 04 00 00 Jun 11 11:45:53 debian8 kernel: [251002.954875] scsi target15:0:36: handle(0x0033), sas_address(0x5000cca0273ab5a9), phy(15) Jun 11 11:45:53 debian8 kernel: [251002.954876] scsi target15:0:36: enclosure_logical_id(0x50030480015ce63f), slot(3) Jun 11 11:45:53 debian8 kernel: [251002.956319] sd 14:0:47:0: task abort: SUCCESS scmd(ffff88146bfee540) Jun 11 11:45:53 debian8 kernel: [251002.956667] sd 15:0:36:0: task abort: SUCCESS scmd(ffff8814b3274200) Jun 11 11:46:01 debian8 CRON[15759]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:46:01 debian8 CRON[15760]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:46:45 debian8 kernel: [251054.859700] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 6, t=21007 jiffies, g=3003262, c=3003261, q=59964) Jun 11 11:46:45 debian8 kernel: [251054.859787] Task dump for CPU 10: Jun 11 11:46:45 debian8 kernel: [251054.859791] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:46:45 debian8 kernel: [251054.859796] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:46:45 debian8 kernel: [251054.859800] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:46:45 debian8 kernel: [251054.859804] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:46:45 debian8 kernel: [251054.859808] Call Trace: Jun 11 11:46:45 debian8 kernel: [251054.859821] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:46:45 debian8 kernel: [251054.859826] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:46:45 debian8 kernel: [251054.859832] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:46:45 debian8 kernel: [251054.859837] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:46:45 debian8 kernel: [251054.859843] [] ? start_secondary+0x199/0x1c0 Jun 11 11:47:01 debian8 CRON[16368]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:47:01 debian8 CRON[16369]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:47:48 debian8 kernel: [251117.817778] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=36762 jiffies, g=3003262, c=3003261, q=95499) Jun 11 11:47:48 debian8 kernel: [251117.817859] Task dump for CPU 10: Jun 11 11:47:48 debian8 kernel: [251117.817862] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:47:48 debian8 kernel: [251117.817867] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:47:48 debian8 kernel: [251117.817870] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:47:48 debian8 kernel: [251117.817872] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:47:48 debian8 kernel: [251117.817875] Call Trace: Jun 11 11:47:48 debian8 kernel: [251117.817888] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:47:48 debian8 kernel: [251117.817892] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:47:48 debian8 kernel: [251117.817897] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:47:48 debian8 kernel: [251117.817902] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:47:48 debian8 kernel: [251117.817908] [] ? start_secondary+0x199/0x1c0 Jun 11 11:48:02 debian8 CRON[16877]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:48:02 debian8 CRON[16878]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:48:24 debian8 kernel: [251154.054372] INFO: task txg_sync:32359 blocked for more than 120 seconds. Jun 11 11:48:24 debian8 kernel: [251154.054409] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:48:24 debian8 kernel: [251154.054436] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:48:24 debian8 kernel: [251154.054482] txg_sync D ffff88203f214440 0 32359 2 0x00000000 Jun 11 11:48:24 debian8 kernel: [251154.054488] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0 Jun 11 11:48:24 debian8 kernel: [251154.054492] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10 Jun 11 11:48:24 debian8 kernel: [251154.054495] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:48:24 debian8 kernel: [251154.054499] Call Trace: Jun 11 11:48:24 debian8 kernel: [251154.054512] [] ? schedule+0x2f/0x80 Jun 11 11:48:24 debian8 kernel: [251154.054518] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:48:24 debian8 kernel: [251154.054526] [] ? __wake_up_common+0x54/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054533] [] ? read_tsc+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054537] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:48:24 debian8 kernel: [251154.054556] [] ? cv_wait_common+0xa5/0x120 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054561] [] ? wait_woken+0x90/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054589] [] ? zio_wait+0x11b/0x1f0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054612] [] ? dsl_pool_sync+0xb2/0x460 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054635] [] ? spa_sync+0x466/0xbb0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054640] [] ? __wake_up+0x34/0x50 Jun 11 11:48:24 debian8 kernel: [251154.054664] [] ? txg_sync_thread+0x373/0x5c0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054668] [] ? update_curr+0xd7/0x120 Jun 11 11:48:24 debian8 kernel: [251154.054690] [] ? txg_fini+0x270/0x270 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054697] [] ? thread_generic_wrapper+0x79/0x90 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054703] [] ? __thread_exit+0x20/0x20 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054707] [] ? kthread+0xc1/0xe0 Jun 11 11:48:24 debian8 kernel: [251154.054711] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054715] [] ? ret_from_fork+0x58/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054719] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054728] INFO: task kworker/10:1:5969 blocked for more than 120 seconds. Jun 11 11:48:24 debian8 kernel: [251154.054758] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:48:24 debian8 kernel: [251154.054785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:48:24 debian8 kernel: [251154.054829] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000 Jun 11 11:48:24 debian8 kernel: [251154.054839] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx] Jun 11 11:48:24 debian8 kernel: [251154.054841] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003 Jun 11 11:48:24 debian8 kernel: [251154.054844] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430 Jun 11 11:48:24 debian8 kernel: [251154.054847] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:48:24 debian8 kernel: [251154.054851] Call Trace: Jun 11 11:48:24 debian8 kernel: [251154.054855] [] ? schedule+0x2f/0x80 Jun 11 11:48:24 debian8 kernel: [251154.054859] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:48:24 debian8 kernel: [251154.054866] [] ? generic_make_request_checks+0x1df/0x3c0 Jun 11 11:48:24 debian8 kernel: [251154.054870] [] ? sched_clock+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054874] [] ? read_tsc+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054878] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:48:24 debian8 kernel: [251154.054882] [] ? wait_for_completion_io+0xad/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054888] [] ? wake_up_state+0x20/0x20 Jun 11 11:48:24 debian8 kernel: [251154.054892] [] ? blkdev_issue_discard+0x20b/0x270 Jun 11 11:48:24 debian8 kernel: [251154.054899] [] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock] Jun 11 11:48:24 debian8 kernel: [251154.054903] [] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock] Jun 11 11:48:24 debian8 kernel: [251154.054914] [] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod] Jun 11 11:48:24 debian8 kernel: [251154.054919] [] ? __switch_to+0x14b/0x5d0 Jun 11 11:48:24 debian8 kernel: [251154.054927] [] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod] Jun 11 11:48:24 debian8 kernel: [251154.054931] [] ? process_one_work+0x152/0x420 Jun 11 11:48:24 debian8 kernel: [251154.054934] [] ? worker_thread+0x53/0x540 Jun 11 11:48:24 debian8 kernel: [251154.054937] [] ? rescuer_thread+0x3a0/0x3a0 Jun 11 11:48:24 debian8 kernel: [251154.054940] [] ? kthread+0xc1/0xe0 Jun 11 11:48:24 debian8 kernel: [251154.054944] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054948] [] ? ret_from_fork+0x58/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054951] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:51 debian8 kernel: [251180.787862] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 31, t=52520 jiffies, g=3003262, c=3003261, q=130454) Jun 11 11:48:51 debian8 kernel: [251180.787941] Task dump for CPU 10: Jun 11 11:48:51 debian8 kernel: [251180.787943] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:48:51 debian8 kernel: [251180.787946] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:48:51 debian8 kernel: [251180.787948] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:48:51 debian8 kernel: [251180.787949] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:48:51 debian8 kernel: [251180.787951] Call Trace: Jun 11 11:48:51 debian8 kernel: [251180.787962] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:48:51 debian8 kernel: [251180.787964] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:48:51 debian8 kernel: [251180.787969] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:48:51 debian8 kernel: [251180.787972] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:48:51 debian8 kernel: [251180.787977] [] ? start_secondary+0x199/0x1c0 root@debian8:/sys/block/zd0/queue# grep -A 500 '11:45' /var/log/syslog Jun 11 11:45:01 debian8 CRON[15171]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:45:01 debian8 CRON[15170]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:45:01 debian8 CRON[15172]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jun 11 11:45:41 debian8 kernel: [250981.461697] ------------[ cut here ]------------ Jun 11 11:45:41 debian8 kernel: [250981.461709] WARNING: CPU: 10 PID: 0 at /build/linux-Xbe5gu/linux-4.0.2/kernel/watchdog.c:290 watchdog_overflow_callback+0x92/0xc0() Jun 11 11:45:41 debian8 kernel: [250981.461710] Watchdog detected hard LOCKUP on cpu 10 Jun 11 11:45:41 debian8 kernel: [250981.461712] Modules linked in: tcm_fc libfc ib_srpt ib_cm ib_sa ib_mad ib_core ib_addr iscsi_target_mod tcm_qla2xxx tcm_loop vhost_scsi vhost target_core_file target_core_iblock target_core_pscsi target_core_mod configfs ses enclosure mpt2sas dm_round_robin dm_multipath scsi_dh nfsd auth_rpcgss oid_registry nfs_acl nfs lockd grace fscache sunrpc bonding zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel ttm aesni_intel drm_kms_helper aes_x86_64 iTCO_wdt lrw gf128mul drm iTCO_vendor_support glue_helper dm_mod joydev evdev ablk_helper cryptd pcspkr sb_edac edac_core mei_me lpc_ich mei i2c_i801 ipmi_si mfd_core ioatdma 8250_fintek ipmi_msghandler tpm_tis processor shpchp tpm thermal_sys wmi button autofs4 ext4 crc16 mbcache jbd2 sg sd_mod hid_generic usbhid hid isci crc32c_intel ahci ehci_pci libsas libahci qla2xxx ehci_hcd mpt3sas igb raid_class scsi_transport_fc libata i2c_algo_bit scsi_transport_sas i2c_core usbcore dca ptp usb_common pps_core scsi_mod Jun 11 11:45:41 debian8 kernel: [250981.461772] CPU: 10 PID: 0 Comm: swapper/10 Tainted: P O 4.0.0-1-amd64 #1 Debian 4.0.2-1 Jun 11 11:45:41 debian8 kernel: [250981.461773] Hardware name: Supermicro SSG-2027R-AR24NV/X9DRH-IF-NV, BIOS 3.0 12/11/2013 Jun 11 11:45:41 debian8 kernel: [250981.461775] 0000000000000000 ffffffff8172ddf0 ffffffff8155df6b ffff88203f005b20 Jun 11 11:45:41 debian8 kernel: [250981.461777] ffffffff8106ce91 ffff882038741800 0000000000000000 ffff88203f005c40 Jun 11 11:45:41 debian8 kernel: [250981.461778] ffff88203f005ef8 0000000000000000 ffffffff8106cf0a ffffffff8172ddc8 Jun 11 11:45:41 debian8 kernel: [250981.461780] Call Trace: Jun 11 11:45:41 debian8 kernel: [250981.461782] [] ? dump_stack+0x40/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461793] [] ? warn_slowpath_common+0x81/0xb0 Jun 11 11:45:41 debian8 kernel: [250981.461794] [] ? warn_slowpath_fmt+0x4a/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461796] [] ? watchdog_overflow_callback+0x92/0xc0 Jun 11 11:45:41 debian8 kernel: [250981.461801] [] ? __perf_event_overflow+0x86/0x230 Jun 11 11:45:41 debian8 kernel: [250981.461806] [] ? x86_perf_event_set_period+0xc9/0x170 Jun 11 11:45:41 debian8 kernel: [250981.461809] [] ? intel_pmu_handle_irq+0x1ba/0x3a0 Jun 11 11:45:41 debian8 kernel: [250981.461816] [] ? ioremap_page_range+0x21a/0x320 Jun 11 11:45:41 debian8 kernel: [250981.461821] [] ? ghes_copy_tofrom_phys+0x129/0x2c0 Jun 11 11:45:41 debian8 kernel: [250981.461822] [] ? ghes_read_estatus+0x71/0x150 Jun 11 11:45:41 debian8 kernel: [250981.461824] [] ? perf_event_nmi_handler+0x2a/0x50 Jun 11 11:45:41 debian8 kernel: [250981.461829] [] ? native_sched_clock+0x26/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461830] [] ? nmi_handle+0x86/0x120 Jun 11 11:45:41 debian8 kernel: [250981.461832] [] ? default_do_nmi+0x44/0x130 Jun 11 11:45:41 debian8 kernel: [250981.461833] [] ? do_nmi+0x88/0xd0 Jun 11 11:45:41 debian8 kernel: [250981.461838] [] ? end_repeat_nmi+0x1e/0x2e Jun 11 11:45:41 debian8 kernel: [250981.461841] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461842] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461843] [] ? _raw_spin_lock_irqsave+0x40/0x90 Jun 11 11:45:41 debian8 kernel: [250981.461844] <> [] ? qlt_fc_port_deleted+0x52/0xb0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461861] [] ? qla2x00_mark_device_lost+0x153/0x2e0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461867] [] ? qla2x00_async_event+0xd6a/0x17f0 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461873] [] ? qla24xx_msix_default+0x1ea/0x250 [qla2xxx] Jun 11 11:45:41 debian8 kernel: [250981.461878] [] ? handle_irq_event_percpu+0x75/0x190 Jun 11 11:45:41 debian8 kernel: [250981.461880] [] ? read_tsc+0x5/0x10 Jun 11 11:45:41 debian8 kernel: [250981.461881] [] ? handle_irq_event+0x3e/0x60 Jun 11 11:45:41 debian8 kernel: [250981.461884] [] ? handle_edge_irq+0x81/0x140 Jun 11 11:45:41 debian8 kernel: [250981.461887] [] ? handle_irq+0x1d/0x30 Jun 11 11:45:41 debian8 kernel: [250981.461891] [] ? do_IRQ+0x46/0xe0 Jun 11 11:45:41 debian8 kernel: [250981.461893] [] ? common_interrupt+0x6d/0x6d Jun 11 11:45:41 debian8 kernel: [250981.461894] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:45:41 debian8 kernel: [250981.461902] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:45:41 debian8 kernel: [250981.461905] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:45:41 debian8 kernel: [250981.461908] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:45:41 debian8 kernel: [250981.461911] [] ? start_secondary+0x199/0x1c0 Jun 11 11:45:41 debian8 kernel: [250981.461912] ---[ end trace 82cffa7a20869247 ]--- Jun 11 11:45:41 debian8 kernel: [250991.901597] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 12, t=5252 jiffies, g=3003262, c=3003261, q=18101) Jun 11 11:45:41 debian8 kernel: [250991.901674] Task dump for CPU 10: Jun 11 11:45:41 debian8 kernel: [250991.901676] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:45:41 debian8 kernel: [250991.901678] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:45:41 debian8 kernel: [250991.901679] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:45:41 debian8 kernel: [250991.901681] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:45:41 debian8 kernel: [250991.901683] Call Trace: Jun 11 11:45:41 debian8 kernel: [250991.901687] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:45:41 debian8 kernel: [250991.901689] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:45:41 debian8 kernel: [250991.901691] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:45:41 debian8 kernel: [250991.901693] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:45:41 debian8 kernel: [250991.901695] [] ? start_secondary+0x199/0x1c0 Jun 11 11:45:53 debian8 kernel: [251002.954745] sd 14:0:47:0: attempting task abort! scmd(ffff88146bfee540) Jun 11 11:45:53 debian8 kernel: [251002.954751] sd 14:0:47:0: [sdau] CDB: Write(16) 8a 00 00 00 00 00 3a 81 c5 4b 00 00 00 09 00 00 Jun 11 11:45:53 debian8 kernel: [251002.954758] scsi target14:0:47: handle(0x003d), sas_address(0x5000cca027576842), phy(30) Jun 11 11:45:53 debian8 kernel: [251002.954760] scsi target14:0:47: enclosure_logical_id(0x50030480015ce63f), slot(18) Jun 11 11:45:53 debian8 kernel: [251002.954866] sd 15:0:36:0: attempting task abort! scmd(ffff8814b3274200) Jun 11 11:45:53 debian8 kernel: [251002.954870] sd 15:0:36:0: [sdck] CDB: Write(16) 8a 00 00 00 00 00 20 11 e9 b4 00 00 00 04 00 00 Jun 11 11:45:53 debian8 kernel: [251002.954875] scsi target15:0:36: handle(0x0033), sas_address(0x5000cca0273ab5a9), phy(15) Jun 11 11:45:53 debian8 kernel: [251002.954876] scsi target15:0:36: enclosure_logical_id(0x50030480015ce63f), slot(3) Jun 11 11:45:53 debian8 kernel: [251002.956319] sd 14:0:47:0: task abort: SUCCESS scmd(ffff88146bfee540) Jun 11 11:45:53 debian8 kernel: [251002.956667] sd 15:0:36:0: task abort: SUCCESS scmd(ffff8814b3274200) Jun 11 11:46:01 debian8 CRON[15759]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:46:01 debian8 CRON[15760]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:46:45 debian8 kernel: [251054.859700] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 6, t=21007 jiffies, g=3003262, c=3003261, q=59964) Jun 11 11:46:45 debian8 kernel: [251054.859787] Task dump for CPU 10: Jun 11 11:46:45 debian8 kernel: [251054.859791] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:46:45 debian8 kernel: [251054.859796] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:46:45 debian8 kernel: [251054.859800] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:46:45 debian8 kernel: [251054.859804] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:46:45 debian8 kernel: [251054.859808] Call Trace: Jun 11 11:46:45 debian8 kernel: [251054.859821] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:46:45 debian8 kernel: [251054.859826] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:46:45 debian8 kernel: [251054.859832] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:46:45 debian8 kernel: [251054.859837] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:46:45 debian8 kernel: [251054.859843] [] ? start_secondary+0x199/0x1c0 Jun 11 11:47:01 debian8 CRON[16368]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:47:01 debian8 CRON[16369]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:47:48 debian8 kernel: [251117.817778] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=36762 jiffies, g=3003262, c=3003261, q=95499) Jun 11 11:47:48 debian8 kernel: [251117.817859] Task dump for CPU 10: Jun 11 11:47:48 debian8 kernel: [251117.817862] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:47:48 debian8 kernel: [251117.817867] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:47:48 debian8 kernel: [251117.817870] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:47:48 debian8 kernel: [251117.817872] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:47:48 debian8 kernel: [251117.817875] Call Trace: Jun 11 11:47:48 debian8 kernel: [251117.817888] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:47:48 debian8 kernel: [251117.817892] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:47:48 debian8 kernel: [251117.817897] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:47:48 debian8 kernel: [251117.817902] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:47:48 debian8 kernel: [251117.817908] [] ? start_secondary+0x199/0x1c0 Jun 11 11:48:02 debian8 CRON[16877]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:48:02 debian8 CRON[16878]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:48:24 debian8 kernel: [251154.054372] INFO: task txg_sync:32359 blocked for more than 120 seconds. Jun 11 11:48:24 debian8 kernel: [251154.054409] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:48:24 debian8 kernel: [251154.054436] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:48:24 debian8 kernel: [251154.054482] txg_sync D ffff88203f214440 0 32359 2 0x00000000 Jun 11 11:48:24 debian8 kernel: [251154.054488] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0 Jun 11 11:48:24 debian8 kernel: [251154.054492] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10 Jun 11 11:48:24 debian8 kernel: [251154.054495] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:48:24 debian8 kernel: [251154.054499] Call Trace: Jun 11 11:48:24 debian8 kernel: [251154.054512] [] ? schedule+0x2f/0x80 Jun 11 11:48:24 debian8 kernel: [251154.054518] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:48:24 debian8 kernel: [251154.054526] [] ? __wake_up_common+0x54/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054533] [] ? read_tsc+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054537] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:48:24 debian8 kernel: [251154.054556] [] ? cv_wait_common+0xa5/0x120 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054561] [] ? wait_woken+0x90/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054589] [] ? zio_wait+0x11b/0x1f0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054612] [] ? dsl_pool_sync+0xb2/0x460 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054635] [] ? spa_sync+0x466/0xbb0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054640] [] ? __wake_up+0x34/0x50 Jun 11 11:48:24 debian8 kernel: [251154.054664] [] ? txg_sync_thread+0x373/0x5c0 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054668] [] ? update_curr+0xd7/0x120 Jun 11 11:48:24 debian8 kernel: [251154.054690] [] ? txg_fini+0x270/0x270 [zfs] Jun 11 11:48:24 debian8 kernel: [251154.054697] [] ? thread_generic_wrapper+0x79/0x90 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054703] [] ? __thread_exit+0x20/0x20 [spl] Jun 11 11:48:24 debian8 kernel: [251154.054707] [] ? kthread+0xc1/0xe0 Jun 11 11:48:24 debian8 kernel: [251154.054711] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054715] [] ? ret_from_fork+0x58/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054719] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054728] INFO: task kworker/10:1:5969 blocked for more than 120 seconds. Jun 11 11:48:24 debian8 kernel: [251154.054758] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:48:24 debian8 kernel: [251154.054785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:48:24 debian8 kernel: [251154.054829] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000 Jun 11 11:48:24 debian8 kernel: [251154.054839] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx] Jun 11 11:48:24 debian8 kernel: [251154.054841] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003 Jun 11 11:48:24 debian8 kernel: [251154.054844] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430 Jun 11 11:48:24 debian8 kernel: [251154.054847] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:48:24 debian8 kernel: [251154.054851] Call Trace: Jun 11 11:48:24 debian8 kernel: [251154.054855] [] ? schedule+0x2f/0x80 Jun 11 11:48:24 debian8 kernel: [251154.054859] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:48:24 debian8 kernel: [251154.054866] [] ? generic_make_request_checks+0x1df/0x3c0 Jun 11 11:48:24 debian8 kernel: [251154.054870] [] ? sched_clock+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054874] [] ? read_tsc+0x5/0x10 Jun 11 11:48:24 debian8 kernel: [251154.054878] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:48:24 debian8 kernel: [251154.054882] [] ? wait_for_completion_io+0xad/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054888] [] ? wake_up_state+0x20/0x20 Jun 11 11:48:24 debian8 kernel: [251154.054892] [] ? blkdev_issue_discard+0x20b/0x270 Jun 11 11:48:24 debian8 kernel: [251154.054899] [] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock] Jun 11 11:48:24 debian8 kernel: [251154.054903] [] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock] Jun 11 11:48:24 debian8 kernel: [251154.054914] [] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod] Jun 11 11:48:24 debian8 kernel: [251154.054919] [] ? __switch_to+0x14b/0x5d0 Jun 11 11:48:24 debian8 kernel: [251154.054927] [] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod] Jun 11 11:48:24 debian8 kernel: [251154.054931] [] ? process_one_work+0x152/0x420 Jun 11 11:48:24 debian8 kernel: [251154.054934] [] ? worker_thread+0x53/0x540 Jun 11 11:48:24 debian8 kernel: [251154.054937] [] ? rescuer_thread+0x3a0/0x3a0 Jun 11 11:48:24 debian8 kernel: [251154.054940] [] ? kthread+0xc1/0xe0 Jun 11 11:48:24 debian8 kernel: [251154.054944] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:24 debian8 kernel: [251154.054948] [] ? ret_from_fork+0x58/0x90 Jun 11 11:48:24 debian8 kernel: [251154.054951] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:48:51 debian8 kernel: [251180.787862] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 31, t=52520 jiffies, g=3003262, c=3003261, q=130454) Jun 11 11:48:51 debian8 kernel: [251180.787941] Task dump for CPU 10: Jun 11 11:48:51 debian8 kernel: [251180.787943] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:48:51 debian8 kernel: [251180.787946] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:48:51 debian8 kernel: [251180.787948] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:48:51 debian8 kernel: [251180.787949] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:48:51 debian8 kernel: [251180.787951] Call Trace: Jun 11 11:48:51 debian8 kernel: [251180.787962] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:48:51 debian8 kernel: [251180.787964] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:48:51 debian8 kernel: [251180.787969] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:48:51 debian8 kernel: [251180.787972] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:48:51 debian8 kernel: [251180.787977] [] ? start_secondary+0x199/0x1c0 Jun 11 11:49:01 debian8 CRON[17352]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:49:01 debian8 CRON[17353]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:49:54 debian8 kernel: [251243.749946] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 14, t=68276 jiffies, g=3003262, c=3003261, q=171658) Jun 11 11:49:54 debian8 kernel: [251243.750025] Task dump for CPU 10: Jun 11 11:49:54 debian8 kernel: [251243.750028] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:49:54 debian8 kernel: [251243.750031] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:49:54 debian8 kernel: [251243.750033] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:49:54 debian8 kernel: [251243.750034] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:49:54 debian8 kernel: [251243.750036] Call Trace: Jun 11 11:49:54 debian8 kernel: [251243.750047] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:49:54 debian8 kernel: [251243.750049] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:49:54 debian8 kernel: [251243.750053] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:49:54 debian8 kernel: [251243.750057] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:49:54 debian8 kernel: [251243.750061] [] ? start_secondary+0x199/0x1c0 Jun 11 11:50:01 debian8 CRON[17909]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:50:01 debian8 CRON[17910]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:50:24 debian8 kernel: [251273.936547] INFO: task txg_sync:32359 blocked for more than 120 seconds. Jun 11 11:50:24 debian8 kernel: [251273.936585] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:50:24 debian8 kernel: [251273.936612] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:50:24 debian8 kernel: [251273.936659] txg_sync D ffff88203f214440 0 32359 2 0x00000000 Jun 11 11:50:24 debian8 kernel: [251273.936666] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0 Jun 11 11:50:24 debian8 kernel: [251273.936670] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10 Jun 11 11:50:24 debian8 kernel: [251273.936674] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:50:24 debian8 kernel: [251273.936678] Call Trace: Jun 11 11:50:24 debian8 kernel: [251273.936693] [] ? schedule+0x2f/0x80 Jun 11 11:50:24 debian8 kernel: [251273.936699] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:50:24 debian8 kernel: [251273.936708] [] ? __wake_up_common+0x54/0x90 Jun 11 11:50:24 debian8 kernel: [251273.936716] [] ? read_tsc+0x5/0x10 Jun 11 11:50:24 debian8 kernel: [251273.936721] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:50:24 debian8 kernel: [251273.936743] [] ? cv_wait_common+0xa5/0x120 [spl] Jun 11 11:50:24 debian8 kernel: [251273.936749] [] ? wait_woken+0x90/0x90 Jun 11 11:50:24 debian8 kernel: [251273.936780] [] ? zio_wait+0x11b/0x1f0 [zfs] Jun 11 11:50:24 debian8 kernel: [251273.936806] [] ? dsl_pool_sync+0xb2/0x460 [zfs] Jun 11 11:50:24 debian8 kernel: [251273.936834] [] ? spa_sync+0x466/0xbb0 [zfs] Jun 11 11:50:24 debian8 kernel: [251273.936840] [] ? __wake_up+0x34/0x50 Jun 11 11:50:24 debian8 kernel: [251273.936867] [] ? txg_sync_thread+0x373/0x5c0 [zfs] Jun 11 11:50:24 debian8 kernel: [251273.936872] [] ? update_curr+0xd7/0x120 Jun 11 11:50:24 debian8 kernel: [251273.936898] [] ? txg_fini+0x270/0x270 [zfs] Jun 11 11:50:24 debian8 kernel: [251273.936906] [] ? thread_generic_wrapper+0x79/0x90 [spl] Jun 11 11:50:24 debian8 kernel: [251273.936913] [] ? __thread_exit+0x20/0x20 [spl] Jun 11 11:50:24 debian8 kernel: [251273.936919] [] ? kthread+0xc1/0xe0 Jun 11 11:50:24 debian8 kernel: [251273.936924] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:50:24 debian8 kernel: [251273.936928] [] ? ret_from_fork+0x58/0x90 Jun 11 11:50:24 debian8 kernel: [251273.936932] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:50:24 debian8 kernel: [251273.936942] INFO: task kworker/10:1:5969 blocked for more than 120 seconds. Jun 11 11:50:24 debian8 kernel: [251273.936973] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:50:24 debian8 kernel: [251273.937000] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:50:24 debian8 kernel: [251273.937045] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000 Jun 11 11:50:24 debian8 kernel: [251273.937056] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx] Jun 11 11:50:24 debian8 kernel: [251273.937058] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003 Jun 11 11:50:24 debian8 kernel: [251273.937062] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430 Jun 11 11:50:24 debian8 kernel: [251273.937066] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:50:24 debian8 kernel: [251273.937069] Call Trace: Jun 11 11:50:24 debian8 kernel: [251273.937075] [] ? schedule+0x2f/0x80 Jun 11 11:50:24 debian8 kernel: [251273.937080] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:50:24 debian8 kernel: [251273.937087] [] ? generic_make_request_checks+0x1df/0x3c0 Jun 11 11:50:24 debian8 kernel: [251273.937092] [] ? sched_clock+0x5/0x10 Jun 11 11:50:24 debian8 kernel: [251273.937096] [] ? read_tsc+0x5/0x10 Jun 11 11:50:24 debian8 kernel: [251273.937101] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:50:24 debian8 kernel: [251273.937106] [] ? wait_for_completion_io+0xad/0x180 Jun 11 11:50:24 debian8 kernel: [251273.937113] [] ? wake_up_state+0x20/0x20 Jun 11 11:50:24 debian8 kernel: [251273.937118] [] ? blkdev_issue_discard+0x20b/0x270 Jun 11 11:50:24 debian8 kernel: [251273.937126] [] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock] Jun 11 11:50:24 debian8 kernel: [251273.937131] [] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock] Jun 11 11:50:24 debian8 kernel: [251273.937143] [] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod] Jun 11 11:50:24 debian8 kernel: [251273.937149] [] ? __switch_to+0x14b/0x5d0 Jun 11 11:50:24 debian8 kernel: [251273.937158] [] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod] Jun 11 11:50:24 debian8 kernel: [251273.937162] [] ? process_one_work+0x152/0x420 Jun 11 11:50:24 debian8 kernel: [251273.937166] [] ? worker_thread+0x53/0x540 Jun 11 11:50:24 debian8 kernel: [251273.937170] [] ? rescuer_thread+0x3a0/0x3a0 Jun 11 11:50:24 debian8 kernel: [251273.937174] [] ? kthread+0xc1/0xe0 Jun 11 11:50:24 debian8 kernel: [251273.937178] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:50:24 debian8 kernel: [251273.937182] [] ? ret_from_fork+0x58/0x90 Jun 11 11:50:24 debian8 kernel: [251273.937186] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:50:34 debian8 kernel: [251283.830751] sd 9:1:0:0: attempting task abort! scmd(ffff8820356ed540) Jun 11 11:50:34 debian8 kernel: [251283.830762] sd 9:1:0:0: [sda] CDB: Write(10) 2a 00 05 c7 f5 20 00 00 98 00 Jun 11 11:50:34 debian8 kernel: [251283.830767] scsi target9:1:0: volume handle(0x0143), volume wwid(0x0e964964d5e3b6fa) Jun 11 11:50:34 debian8 kernel: [251283.830772] sd 9:1:0:0: task abort: FAILED scmd(ffff8820356ed540) Jun 11 11:50:34 debian8 kernel: [251283.830832] sd 9:1:0:0: attempting device reset! scmd(ffff8820356ed540) Jun 11 11:50:34 debian8 kernel: [251283.830839] sd 9:1:0:0: [sda] CDB: Write(10) 2a 00 05 c7 f5 20 00 00 98 00 Jun 11 11:50:34 debian8 kernel: [251283.830843] scsi target9:1:0: volume handle(0x0143), volume wwid(0x0e964964d5e3b6fa) Jun 11 11:50:34 debian8 kernel: [251283.880685] sd 9:1:0:0: device reset: SUCCESS scmd(ffff8820356ed540) Jun 11 11:50:57 debian8 kernel: [251306.712033] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 13, t=84032 jiffies, g=3003262, c=3003261, q=200277) Jun 11 11:50:57 debian8 kernel: [251306.712102] Task dump for CPU 10: Jun 11 11:50:57 debian8 kernel: [251306.712104] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:50:57 debian8 kernel: [251306.712107] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:50:57 debian8 kernel: [251306.712109] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:50:57 debian8 kernel: [251306.712110] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:50:57 debian8 kernel: [251306.712112] Call Trace: Jun 11 11:50:57 debian8 kernel: [251306.712123] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:50:57 debian8 kernel: [251306.712125] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:50:57 debian8 kernel: [251306.712129] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:50:57 debian8 kernel: [251306.712133] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:50:57 debian8 kernel: [251306.712138] [] ? start_secondary+0x199/0x1c0 Jun 11 11:51:01 debian8 CRON[18232]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:51:01 debian8 CRON[18233]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:52:00 debian8 kernel: [251369.674117] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 30, t=99788 jiffies, g=3003262, c=3003261, q=229197) Jun 11 11:52:00 debian8 kernel: [251369.674196] Task dump for CPU 10: Jun 11 11:52:00 debian8 kernel: [251369.674197] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:52:00 debian8 kernel: [251369.674201] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:52:00 debian8 kernel: [251369.674204] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:52:00 debian8 kernel: [251369.674207] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:52:00 debian8 kernel: [251369.674209] Call Trace: Jun 11 11:52:00 debian8 kernel: [251369.674219] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:52:00 debian8 kernel: [251369.674222] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:52:00 debian8 kernel: [251369.674227] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:52:00 debian8 kernel: [251369.674231] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:52:00 debian8 kernel: [251369.674236] [] ? start_secondary+0x199/0x1c0 Jun 11 11:52:01 debian8 CRON[18549]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:52:01 debian8 CRON[18550]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:52:24 debian8 kernel: [251393.818642] INFO: task txg_sync:32359 blocked for more than 120 seconds. Jun 11 11:52:24 debian8 kernel: [251393.818680] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:52:24 debian8 kernel: [251393.818708] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:52:24 debian8 kernel: [251393.818754] txg_sync D ffff88203f214440 0 32359 2 0x00000000 Jun 11 11:52:24 debian8 kernel: [251393.818761] ffff88102f8ec3d0 ffff881024ed89a0 ffff881038ca6ba0 ffff881024ed89a0 Jun 11 11:52:24 debian8 kernel: [251393.818766] ffff8810281dffd8 ffff88203f214440 7fffffffffffffff ffff882006098c10 Jun 11 11:52:24 debian8 kernel: [251393.818769] 0000000000000001 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:52:24 debian8 kernel: [251393.818773] Call Trace: Jun 11 11:52:24 debian8 kernel: [251393.818788] [] ? schedule+0x2f/0x80 Jun 11 11:52:24 debian8 kernel: [251393.818795] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:52:24 debian8 kernel: [251393.818804] [] ? __wake_up_common+0x54/0x90 Jun 11 11:52:24 debian8 kernel: [251393.818812] [] ? read_tsc+0x5/0x10 Jun 11 11:52:24 debian8 kernel: [251393.818817] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:52:24 debian8 kernel: [251393.818838] [] ? cv_wait_common+0xa5/0x120 [spl] Jun 11 11:52:24 debian8 kernel: [251393.818844] [] ? wait_woken+0x90/0x90 Jun 11 11:52:24 debian8 kernel: [251393.818874] [] ? zio_wait+0x11b/0x1f0 [zfs] Jun 11 11:52:24 debian8 kernel: [251393.818900] [] ? dsl_pool_sync+0xb2/0x460 [zfs] Jun 11 11:52:24 debian8 kernel: [251393.818928] [] ? spa_sync+0x466/0xbb0 [zfs] Jun 11 11:52:24 debian8 kernel: [251393.818934] [] ? __wake_up+0x34/0x50 Jun 11 11:52:24 debian8 kernel: [251393.818962] [] ? txg_sync_thread+0x373/0x5c0 [zfs] Jun 11 11:52:24 debian8 kernel: [251393.818966] [] ? update_curr+0xd7/0x120 Jun 11 11:52:24 debian8 kernel: [251393.818992] [] ? txg_fini+0x270/0x270 [zfs] Jun 11 11:52:24 debian8 kernel: [251393.819001] [] ? thread_generic_wrapper+0x79/0x90 [spl] Jun 11 11:52:24 debian8 kernel: [251393.819007] [] ? __thread_exit+0x20/0x20 [spl] Jun 11 11:52:24 debian8 kernel: [251393.819013] [] ? kthread+0xc1/0xe0 Jun 11 11:52:24 debian8 kernel: [251393.819018] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:52:24 debian8 kernel: [251393.819022] [] ? ret_from_fork+0x58/0x90 Jun 11 11:52:24 debian8 kernel: [251393.819026] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:52:24 debian8 kernel: [251393.819035] INFO: task kworker/10:1:5969 blocked for more than 120 seconds. Jun 11 11:52:24 debian8 kernel: [251393.819066] Tainted: P W O 4.0.0-1-amd64 #1 Jun 11 11:52:24 debian8 kernel: [251393.819093] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jun 11 11:52:24 debian8 kernel: [251393.819138] kworker/10:1 D ffff88203f014440 0 5969 2 0x00000000 Jun 11 11:52:24 debian8 kernel: [251393.819148] Workqueue: tcm_qla2xxx_free tcm_qla2xxx_handle_data_work [tcm_qla2xxx] Jun 11 11:52:24 debian8 kernel: [251393.819150] ffff88200733b430 ffff881c590afbf8 ffff881038c7a190 0000000000000003 Jun 11 11:52:24 debian8 kernel: [251393.819154] ffff881c590affd8 ffff88203f014440 7fffffffffffffff ffff88200733b430 Jun 11 11:52:24 debian8 kernel: [251393.819158] ffff881c590afcd0 0000000000000000 ffffffff8155fb5f 0000000000000000 Jun 11 11:52:24 debian8 kernel: [251393.819162] Call Trace: Jun 11 11:52:24 debian8 kernel: [251393.819167] [] ? schedule+0x2f/0x80 Jun 11 11:52:24 debian8 kernel: [251393.819172] [] ? schedule_timeout+0x21a/0x290 Jun 11 11:52:24 debian8 kernel: [251393.819180] [] ? generic_make_request_checks+0x1df/0x3c0 Jun 11 11:52:24 debian8 kernel: [251393.819184] [] ? sched_clock+0x5/0x10 Jun 11 11:52:24 debian8 kernel: [251393.819188] [] ? read_tsc+0x5/0x10 Jun 11 11:52:24 debian8 kernel: [251393.819193] [] ? io_schedule_timeout+0xa3/0x130 Jun 11 11:52:24 debian8 kernel: [251393.819199] [] ? wait_for_completion_io+0xad/0x180 Jun 11 11:52:24 debian8 kernel: [251393.819205] [] ? wake_up_state+0x20/0x20 Jun 11 11:52:24 debian8 kernel: [251393.819210] [] ? blkdev_issue_discard+0x20b/0x270 Jun 11 11:52:24 debian8 kernel: [251393.819219] [] ? iblock_execute_unmap+0x30/0x30 [target_core_iblock] Jun 11 11:52:24 debian8 kernel: [251393.819223] [] ? iblock_do_unmap+0x1f/0x50 [target_core_iblock] Jun 11 11:52:24 debian8 kernel: [251393.819235] [] ? sbc_execute_unmap+0x12c/0x270 [target_core_mod] Jun 11 11:52:24 debian8 kernel: [251393.819241] [] ? __switch_to+0x14b/0x5d0 Jun 11 11:52:24 debian8 kernel: [251393.819251] [] ? __target_execute_cmd+0x1a/0xa0 [target_core_mod] Jun 11 11:52:24 debian8 kernel: [251393.819255] [] ? process_one_work+0x152/0x420 Jun 11 11:52:24 debian8 kernel: [251393.819258] [] ? worker_thread+0x53/0x540 Jun 11 11:52:24 debian8 kernel: [251393.819262] [] ? rescuer_thread+0x3a0/0x3a0 Jun 11 11:52:24 debian8 kernel: [251393.819266] [] ? kthread+0xc1/0xe0 Jun 11 11:52:24 debian8 kernel: [251393.819271] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:52:24 debian8 kernel: [251393.819274] [] ? ret_from_fork+0x58/0x90 Jun 11 11:52:24 debian8 kernel: [251393.819279] [] ? kthread_create_on_node+0x180/0x180 Jun 11 11:53:01 debian8 CRON[18898]: (root) CMD (/sbin/zpool iostat 50 2 > /tmp/zpool_iostat_out.tmp && mv -f /tmp/zpool_iostat_out.tmp /tmp/zpool_iostat_out) Jun 11 11:53:01 debian8 CRON[18899]: (root) CMD (/usr/bin/iostat -p `ls /dev/zd*` -x -m -y 50 1 > /tmp/iostat_out.tmp && mv -f /tmp/iostat_out.tmp /tmp/iostat_out) Jun 11 11:53:03 debian8 kernel: [251432.636209] INFO: rcu_sched detected stalls on CPUs/tasks: { 10} (detected by 36, t=115544 jiffies, g=3003262, c=3003261, q=267550) Jun 11 11:53:03 debian8 kernel: [251432.636287] Task dump for CPU 10: Jun 11 11:53:03 debian8 kernel: [251432.636289] swapper/10 R running task 0 0 1 0x00000008 Jun 11 11:53:03 debian8 kernel: [251432.636292] ffffffffffffff11 ffffffff81432f51 0000000000000010 0000000000000207 Jun 11 11:53:03 debian8 kernel: [251432.636294] ffff881038c93e88 0000000000000018 ffffffff81432f3e 0000000000000092 Jun 11 11:53:03 debian8 kernel: [251432.636295] ffffffff818f4bc0 ffffe8ffff80a1d0 ffffffff818a5fc0 ffff881038c90000 Jun 11 11:53:03 debian8 kernel: [251432.636297] Call Trace: Jun 11 11:53:03 debian8 kernel: [251432.636307] [] ? cpuidle_enter_state+0x61/0x160 Jun 11 11:53:03 debian8 kernel: [251432.636309] [] ? cpuidle_enter_state+0x4e/0x160 Jun 11 11:53:03 debian8 kernel: [251432.636313] [] ? cpu_startup_entry+0x36f/0x420 Jun 11 11:53:03 debian8 kernel: [251432.636317] [] ? tick_check_new_device+0xe0/0x110 Jun 11 11:53:03 debian8 kernel: [251432.636322] [] ? start_secondary+0x199/0x1c0