I presented a new LUN (and no others) with emulate-caw set to 0 via
the following command:
echo 0 > /sys/kernel/config/target/core/iblock_1/test/attrib/emulate_caw
Upon starting the first ESXi host after configuring the new LUN, I
started getting the following on the console every few seconds:
Message from syslogd@labsan2 at Apr 7 23:00:29 ...
kernel:[17424.117000] NMI watchdog: BUG: soft lockup - CPU#2 stuck
for 22s! [systemd-udevd:1782]
after a few minutes of this (and while trying to get more logs) I
started getting the following call trace on the console:
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff811d3d03>]
handle_mm_fault+0xe43/0x1850
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff812686ae>] ?
ep_scan_ready_list+0x20e/0x220
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff81268930>] ?
ep_poll+0x240/0x440
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff81064d13>]
__do_page_fault+0x193/0x450
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff81065001>]
do_page_fault+0x31/0x70
Apr 7 23:11:25 labsan2 kernel: [18080.117000] [<ffffffff817887c8>]
page_fault+0x28/0x30
Apr 7 23:11:25 labsan2 kernel: [18080.117000] Code: 00 48 03 14 c5 c0
ff d0 81 48 89 de 48 89 df e8 6d 69 29 00 84 c0 74 13 44 89 e7 ff 15
50 9e b0 00 eb 08 66 0f 1f 44 00 00 f3 90 <f6> 43 18 01 75 f8 31 c0 48
8b 7d e8 65 48 33 3c 25 28 00 00 00
Apr 7 23:11:26 labsan2 kernel: [18081.031504] ABORT_TASK: Found
referenced qla2xxx task_tag: 1139688
Apr 7 23:11:26 labsan2 kernel: [18081.031506] ABORT_TASK: ref_tag:
1139688 already complete, skipping
Apr 7 23:11:26 labsan2 kernel: [18081.031508] ABORT_TASK: Sending
TMR_TASK_DOES_NOT_EXIST for ref_tag: 1139688
Apr 7 23:11:53 labsan2 kernel: [18108.117000] NMI watchdog: BUG: soft
lockup - CPU#2 stuck for 22s! [systemd-udevd:1782]
Apr 7 23:11:53 labsan2 kernel: [18108.117000] Modules linked in:
tcm_qla2xxx target_core_user uio target_core_pscsi target_core_file
target_core_iblock iscsi_target_mod target_core_mod ip6t_rpfilter
ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute
bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6
nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security
ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle
iptable_security iptable_raw iTCO_wdt iTCO_vendor_support gpio_ich
coretemp lpc_ich mfd_core kvm_intel kvm ipmi_devintf ipmi_ssif
serio_raw i5000_edac ioatdma dca shpchp edac_core ipmi_si i5k_amb
ipmi_msghandler acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace
sunrpc xfs libcrc32c radeon i2c_algo_bit drm_kms_helper qla2xxx ttm
mptsas drm scsi_transport_sas mptscsih ata_generic pata_acpi mptbase
bnx2 scsi_transport_fc
Apr 7 23:11:53 labsan2 kernel: [18108.117000] CPU: 2 PID: 1782 Comm:
systemd-udevd Tainted: G W L 4.0.0-0.rc2.git0.1.fc22.x86_64
#1
Apr 7 23:11:53 labsan2 kernel: [18108.117000] Hardware name: IBM IBM
eServer BladeCenter HS21 -[8853L6U]-/Server Blade, BIOS
-[BCE142BUS-1.18]- 06/17/2009
Apr 7 23:11:53 labsan2 kernel: [18108.117000] task: ffff8800c9f84f00
ti: ffff88012a344000 task.ti: ffff88012a344000
Apr 7 23:11:53 labsan2 kernel: [18108.117000] RIP:
0010:[<ffffffff8111b5aa>] [<ffffffff8111b5aa>]
generic_exec_single+0xea/0x1a0
Apr 7 23:11:53 labsan2 kernel: [18108.117000] RSP:
0000:ffff88012a347be8 EFLAGS: 00000202
Apr 7 23:11:53 labsan2 kernel: [18108.117000] RAX: 00000000000008fb
RBX: 0000000000000141 RCX: 0000000000000000
Apr 7 23:11:53 labsan2 kernel: [18108.117000] RDX: 00000000000008fb
RSI: 00000000000000fb RDI: 0000000000000282
Apr 7 23:11:53 labsan2 kernel: [18108.117000] RBP: ffff88012a347c28
R08: 0000000000000001 R09: ffff880000000e18
Apr 7 23:11:53 labsan2 kernel: [18108.117000] R10: ffff880000000000
R11: 0000000000130000 R12: 000200da00000001
Apr 7 23:11:53 labsan2 kernel: [18108.117000] R13: 0000000000000000
R14: ffff88012ffe8d80 R15: 0000010000000000
Apr 7 23:11:53 labsan2 kernel: [18108.117000] FS:
00007f41505c5880(0000) GS:ffff88012fd00000(0000)
knlGS:0000000000000000
Apr 7 23:11:53 labsan2 kernel: [18108.117000] CS: 0010 DS: 0000 ES:
0000 CR0: 0000000080050033
Apr 7 23:11:53 labsan2 kernel: [18108.117000] CR2: 00007ffcc75c3fa8
CR3: 000000012a324000 CR4: 00000000000007e0
Apr 7 23:11:53 labsan2 kernel: [18108.117000] Stack:
Apr 7 23:11:53 labsan2 kernel: [18108.117000] ffff88012a347c08
0000000000000000 ffffffff8106be30 ffff88012a347cb8
Apr 7 23:11:53 labsan2 kernel: [18108.117000] 0000000000000003
00000000cfa942cb 0000000000000001 ffffffff8106be30
Apr 7 23:11:53 labsan2 kernel: [18108.117000] ffff88012a347c58
ffffffff8111b6c0 ffff88012a347c58 ffffffff8139ec2b
Apr 7 23:11:53 labsan2 kernel: [18108.117000] Call Trace:
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8106be30>] ?
do_flush_tlb_all+0x50/0x50
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8106be30>] ?
do_flush_tlb_all+0x50/0x50
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8111b6c0>]
smp_call_function_single+0x60/0xa0
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8139ec2b>] ?
cpumask_next_and+0x3b/0x50
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8106be30>] ?
do_flush_tlb_all+0x50/0x50
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8111bce3>]
smp_call_function_many+0x233/0x280
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8106c048>]
native_flush_tlb_others+0xb8/0xc0
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff8106c2e4>]
flush_tlb_page+0x54/0x90
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff811e556e>]
ptep_clear_flush+0x4e/0x60
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff811d2333>]
do_wp_page+0x333/0x940
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff811d3d03>]
handle_mm_fault+0xe43/0x1850
Apr 7 23:11:53 labsan2 kernel: [18108.117000] [<ffffffff812686ae>] ?
ep_scan_ready_list+0x20e/0x220
Thanks again
On Tue, Apr 7, 2015 at 9:30 PM, Nicholas A. Bellinger
<nab(a)linux-iscsi.org> wrote:
On Tue, 2015-04-07 at 20:41 -0400, Dan Lane wrote:
> Initiators are the same as the crashy storage, QLA2462. Here's what
> lspci shows for them if it helps:
>
> 0d:01.0 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to
> PCI-X HBA (rev 02)
> 0d:01.1 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to
> PCI-X HBA (rev 02)
>
> I will work on getting storage presented from one of the problematic
> systems with the emulate_caw=0 setting, can you be more specific on
> where this should be set?
emulate_caw is a backend device attribute. It can be changed for each
backend device within the configuration file before restarting the
target service, or in /backstores/iblock/$DEVICE using targetcli.
Note that this will need to happen before the ESX FC initiator connects,
and performs the LUN SCAN to query if COMPARE_AND_WRITE is supported or
not.
--nab