I have a python program that is making use of the vdscli and storage python modules. I noticed that my program was creating a bunch of sub-processes that I did not request. After some investigation I determined the chain of imports that is causing these threads to be created:
Traceback (most recent call last): File "./vdsmi", line 7, in <module> from backend.Backend import Backend File "/home/aglitke/src/vdsm-interface/backend/Backend.py", line 5, in <module> from storage.sd import LOCALFS_DOMAIN, DATA_DOMAIN File "/usr/share/vdsm/storage/sd.py", line 30, in <module> import resourceFactories File "/usr/share/vdsm/storage/resourceFactories.py", line 29, in <module> import image File "/usr/share/vdsm/storage/image.py", line 27, in <module> import volume File "/usr/share/vdsm/storage/volume.py", line 33, in <module> import task File "/usr/share/vdsm/storage/task.py", line 58, in <module> import outOfProcess as oop File "/usr/share/vdsm/storage/outOfProcess.py", line 38, in <module> _globalPool = ProcessPool(MAX_HELPERS, GRACE_PERIOD, DEFAULT_TIMEOUT)
outOfProcess.py is initializing a global ProcessPool at module import which is causing the unwanted threads. In order to prevent this, I believe we need to either clean up this mess of cross-dependencies or require that users of outOfProcess call a global processPoolInit() function if the ProcessPool is needed. I pulled in an awful lot of code in an effort to load two integer constants :)
On Sat, Sep 10, 2011 at 11:03:56AM -0500, Adam Litke wrote:
I have a python program that is making use of the vdscli and storage python modules. I noticed that my program was creating a bunch of sub-processes that I did not request. After some investigation I determined the chain of imports that is causing these threads to be created:
Traceback (most recent call last): File "./vdsmi", line 7, in <module> from backend.Backend import Backend File "/home/aglitke/src/vdsm-interface/backend/Backend.py", line 5, in <module> from storage.sd import LOCALFS_DOMAIN, DATA_DOMAIN File "/usr/share/vdsm/storage/sd.py", line 30, in <module> import resourceFactories File "/usr/share/vdsm/storage/resourceFactories.py", line 29, in <module> import image File "/usr/share/vdsm/storage/image.py", line 27, in <module> import volume File "/usr/share/vdsm/storage/volume.py", line 33, in <module> import task File "/usr/share/vdsm/storage/task.py", line 58, in <module> import outOfProcess as oop File "/usr/share/vdsm/storage/outOfProcess.py", line 38, in <module> _globalPool = ProcessPool(MAX_HELPERS, GRACE_PERIOD, DEFAULT_TIMEOUT)
outOfProcess.py is initializing a global ProcessPool at module import which is causing the unwanted threads. In order to prevent this, I believe we need to either clean up this mess of cross-dependencies or require that users of outOfProcess call a global processPoolInit() function if the ProcessPool is needed. I pulled in an awful lot of code in an effort to load two integer constants :)
Yes, this has become more painful with
commit 1c487d9007a7235404bcb467daf71d335bbbc26a Author: Saggi Mizrahi smizrahi@redhat.com Date: Mon Jul 25 16:18:36 2011 +0300
BZ#725967 - one process pool to rule them all
Instead of having a process pool per domain use one pool but limit the amount of workers a domain can hold at any given time.
that creates gazillion processes on startup. Unfortunately, this is not the worst effect of this patch; in many senses master is currently broken. Hopefully https://fedorahosted.org/pipermail/vdsm-patches/2011-September/001768.html would fix it soon.
Dan.
On Sun, Sep 11, 2011 at 10:22:04AM +0300, Dan Kenigsberg wrote:
On Sat, Sep 10, 2011 at 11:03:56AM -0500, Adam Litke wrote:
outOfProcess.py is initializing a global ProcessPool at module import which is causing the unwanted threads. In order to prevent this, I believe we need to either clean up this mess of cross-dependencies or require that users of outOfProcess call a global processPoolInit() function if the ProcessPool is needed. I pulled in an awful lot of code in an effort to load two integer constants :)
Yes, this has become more painful with
commit 1c487d9007a7235404bcb467daf71d335bbbc26a Author: Saggi Mizrahi smizrahi@redhat.com Date: Mon Jul 25 16:18:36 2011 +0300
BZ#725967 - one process pool to rule them all Instead of having a process pool per domain use one pool but limit the amount of workers a domain can hold at any given time.
that creates gazillion processes on startup. Unfortunately, this is not the worst effect of this patch; in many senses master is currently broken. Hopefully https://fedorahosted.org/pipermail/vdsm-patches/2011-September/001768.html would fix it soon.
Excellent, thanks Dan.
vdsm-devel@lists.fedorahosted.org