----- Original Message -----
From: "Dan Kenigsberg" danken@redhat.com To: "Yaniv Bronheim" ybronhei@redhat.com Cc: "VDSM Project Development" vdsm-devel@lists.fedorahosted.org, "Saggi Mizrahi" smizrahi@redhat.com Sent: Tuesday, February 4, 2014 12:20:52 PM Subject: Re: suggested patch for python-pthreading
On Tue, Feb 04, 2014 at 04:04:37AM -0500, Yaniv Bronheim wrote:
according to coredumps we found in the scope of the bug [1] we opened [2] that suggested to override python's implementation of thread.allocate_lock in each coredump we saw few threads stuck with the bt:
#16 0x00007fcb69288c93 in PyEval_CallObjectWithKeywords (func=0x2527820, arg=0x7fcb6972f050, kw=<value optimized out>) at Python/ceval.c:3663 #17 0x00007fcb692ba7ba in t_bootstrap (boot_raw=0x250a820) at Modules/threadmodule.c:428 #18 0x00007fcb68fa3851 in start_thread (arg=0x7fcb1bfff700) at pthread_create.c:301 #19 0x00007fcb6866694d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
in pystack the threads were stuck in /usr/lib64/python2.6/threading.py (513): __bootstrap_inner
in bootstrap_inner we use thread.allocate_lock which python-pthreading does not override.
we suggest the following commit:
From 9d89e9be1a379b3d93b23dd54a381b9ca0973ebc Mon Sep 17 00:00:00 2001
From: Yaniv Bronhaim ybronhei@redhat.com Date: Mon, 3 Feb 2014 19:24:30 +0200 Subject: [PATCH] Mocking thread.allocate_lock with Lock imp
Signed-off-by: Yaniv Bronhaim ybronhei@redhat.com
pthreading.py | 4 ++++ 1 file changed, 4 insertions(+)
diff --git a/pthreading.py b/pthreading.py index 916ca7f..96df42c 100644 --- a/pthreading.py +++ b/pthreading.py @@ -132,6 +132,10 @@ def monkey_patch(): Thus, Queue and SocketServer can easily enjoy them. """
import thread
thread.allocate_lock = Lock
import threading
threading.Condition = Condition
-- 1.8.3.1
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1022036 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1060749
It makes sense to use pthreading.Lock for thread.allocate_lock instead of the standard threading.Lock CPU hog. However, I do not understand its relevance to the deadlock sited above: pthreading.Lock fixes performance issues, but not correctness issues, of threading.Lock.
Would you explain, in the commit message of the pthreading patch, why you believe that the implementation of thread.allocate_lock() is buggy? Do you know if the bug is fixed in Python 3?
Regards, Dan.
We actually don't have concrete proof as we can't reproduce the bug so we can't test this. We are shooting in the dark hoping something hits. We assume it's there since all of our cordumps have a thread stuck acquiring the limbo lock. Since mixing lock implementations is probably a bad idea we assume that overriding this a thing we should do anyway we thought we'll give it a go. If VDSM gets stuck again we will have another coredump that we could compare to the others.