Change in vdsm[master]: itmap starts threads for each argument causes too many oop o...
ybronhei at redhat.com
ybronhei at redhat.com
Tue Oct 23 13:23:28 UTC 2012
Yaniv Bronhaim has uploaded a new change for review.
Change subject: itmap starts threads for each argument causes too many oop operations
......................................................................
itmap starts threads for each argument causes too many oop operations
This patch starts the threads until we reach to maxthreads, than we
wait for one thread to end and move on until we loaded all threads.
Change-Id: If2877c28b916475aead16d47d37522c2bf44cc16
Bug-Id: https://bugzilla.redhat.com/show_bug.cgi?id=523011
Signed-off-by: Yaniv Bronhaim <ybronhei at redhat.com>
---
M vdsm/storage/fileSD.py
M vdsm/storage/misc.py
2 files changed, 22 insertions(+), 3 deletions(-)
git pull ssh://gerrit.ovirt.org:29418/vdsm refs/changes/45/8745/1
diff --git a/vdsm/storage/fileSD.py b/vdsm/storage/fileSD.py
index 054fadb..5e50dc7 100644
--- a/vdsm/storage/fileSD.py
+++ b/vdsm/storage/fileSD.py
@@ -34,6 +34,7 @@
from remoteFileHandler import Timeout
from persistentDict import PersistentDict, DictValidator
from vdsm import constants
+from vdsm.config import config
import time
import supervdsm
import mount
@@ -486,7 +487,8 @@
log.warn("Could not collect metadata file for domain path %s",
possibleDomain, exc_info=True)
- for res in misc.itmap(collectMetaFiles, mntList):
+ for res in misc.itmap(collectMetaFiles, mntList,
+ config.getint("irs", "process_pool_max_slots_per_domain")):
if res is None:
continue
diff --git a/vdsm/storage/misc.py b/vdsm/storage/misc.py
index 8641312..67433e9 100644
--- a/vdsm/storage/misc.py
+++ b/vdsm/storage/misc.py
@@ -1250,7 +1250,7 @@
raise exception
-def itmap(func, iterable):
+def itmap(func, iterable, maxthreads):
"""
Make an iterator that computes the function using
arguments from the iterable. It works similar to tmap
@@ -1267,12 +1267,29 @@
except Exception, e:
respQueue.put(e)
+ def nextArg(iterable):
+ for arg in iterable:
+ yield arg
+
n = 0
- for arg in iterable:
+ if maxthreads < 1:
+ raise Exception("wrong input to function itmap: %s", maxthreads)
+
+ for arg in nextArg(iterable):
+ if not maxthreads:
+ if n:
+ yield respQueue.get()
+ # if yield returns one thread stopped, so we can run another
+ # simultaneosly thread
+ maxthreads += 1
+ n -= 1
+
t = threading.Thread(target=wrapper, args=(arg,))
t.start()
n += 1
+ maxthreads -= 1
+ # waiting for rest threads to end
for i in xrange(n):
yield respQueue.get()
--
To view, visit http://gerrit.ovirt.org/8745
To unsubscribe, visit http://gerrit.ovirt.org/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If2877c28b916475aead16d47d37522c2bf44cc16
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Yaniv Bronhaim <ybronhei at redhat.com>
More information about the vdsm-patches
mailing list