Change in vdsm[master]: iscsi: Set node.session.timeo.replacement_timeout when creat...

nsoffer at redhat.com nsoffer at redhat.com
Wed Sep 3 11:21:44 UTC 2014


Nir Soffer has uploaded a new change for review.

Change subject: iscsi: Set node.session.timeo.replacement_timeout when creating a node
......................................................................

iscsi: Set node.session.timeo.replacement_timeout when creating a node

ISCSI multipath failover takes too much time, causing storage operations
timeouts in engine, stuck lvm operations in vdsm, and guests marked as
down. This issue is the root cause of many bugs.

The root cause is bad default in iscsi.conf:

    node.session.timeo.replacement_timeout = 120

Using this timeout, multipath will wait 120 seconds for SCSI commands
before trying the next path, which may cause another 120 seconds delay
if the device is faulty.

In device-mapper-multipath version 0.4.9-76 and later, multipath
overrides iscsi.conf's default timeout with multipath.conf's timeout,
eliminating this issue. This version is not available on current
supported platforms.

This patch fixes the issue in a simple backward compatible way by
modifying the timeout in the node database when we add a node. This
change affect only devices used by vdsm, leaving the system iscsi
configuration unchanged.

This does not handle the timeout for FC devices, which should be handled
in a different way.

When we require device-mapper-multipath > 0.4.9-76, we can revert this
change.

Change-Id: I021c6f72eb5608cd61772d549eb13a229abb6c0d
Bug-Url: https://bugzilla.redhat.com/980139
Signed-off-by: Nir Soffer <nsoffer at redhat.com>
---
M lib/vdsm/constants.py.in
M vdsm/storage/iscsi.py
M vdsm/storage/multipath.py
3 files changed, 20 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/63/32363/1

diff --git a/lib/vdsm/constants.py.in b/lib/vdsm/constants.py.in
index ee379d6..a795afe 100644
--- a/lib/vdsm/constants.py.in
+++ b/lib/vdsm/constants.py.in
@@ -68,6 +68,9 @@
 
 MEGAB = 2 ** 20  # = 1024 ** 2 = 1 MiB
 
+# Number of seconds the scsi layer will wait before failing IO on a device.
+FAST_IO_FAIL_TIMEOUT = 5
+
 #
 # Path definitions
 #
diff --git a/vdsm/storage/iscsi.py b/vdsm/storage/iscsi.py
index 2975776..e69d0cf 100644
--- a/vdsm/storage/iscsi.py
+++ b/vdsm/storage/iscsi.py
@@ -31,6 +31,7 @@
 from collections import namedtuple
 
 import misc
+from vdsm import constants
 from vdsm.config import config
 import devicemapper
 from threading import RLock
@@ -165,6 +166,18 @@
 
             iscsiadm.node_update(iface.name, portalStr, target.iqn,
                                  "node.startup", "manual")
+
+            # In multipath versions before 0.4.9-76, recovery_tmo iscsi sysfs
+            # parameter used iscsi.conf's replacement_timeout (120 seconds)
+            # instead of multipath.conf's fail_io_fast_tmo (5 seconds). This
+            # cause lvm operation to get stuck for multiple minutes when
+            # accessing faulty devices.
+            # See https://bugzilla.redhat.com/980139
+            #
+            # TODO: Remove when device-mapper-multipath > 0.4.9-76
+            iscsiadm.node_update(iface.name, portalStr, target.iqn,
+                                 "node.session.timeo.replacement_timeout",
+                                 "%d" % constants.FAST_IO_FAIL_TIMEOUT)
         except:
             removeIscsiNode(iface, target)
             raise
diff --git a/vdsm/storage/multipath.py b/vdsm/storage/multipath.py
index ba98866..a77aec4 100644
--- a/vdsm/storage/multipath.py
+++ b/vdsm/storage/multipath.py
@@ -64,7 +64,7 @@
     "    no_path_retry           fail\n"
     "    user_friendly_names     no\n"
     "    flush_on_last_del       yes\n"
-    "    fast_io_fail_tmo        5\n"
+    "    fast_io_fail_tmo        %(fast_io_fail_tmo)d\n"
     "    dev_loss_tmo            30\n"
     "    max_fds                 4096\n"
     "}\n"
@@ -163,7 +163,9 @@
             os.path.basename(MPATH_CONF), MAX_CONF_COPIES,
             cp=True, persist=True)
     with tempfile.NamedTemporaryFile() as f:
-        f.write(MPATH_CONF_TEMPLATE % {'scsi_id_path': _scsi_id.cmd})
+        f.write(MPATH_CONF_TEMPLATE % {
+            'scsi_id_path': _scsi_id.cmd,
+            'fast_io_fail_tmo': constants.FAST_IO_FAIL_TIMEOUT})
         f.flush()
         cmd = [constants.EXT_CP, f.name, MPATH_CONF]
         rc = misc.execCmd(cmd, sudo=True)[0]


-- 
To view, visit http://gerrit.ovirt.org/32363
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I021c6f72eb5608cd61772d549eb13a229abb6c0d
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Nir Soffer <nsoffer at redhat.com>


More information about the vdsm-patches mailing list