OK, I tested it more thoroughly and I was right all along (as usual) memory is shared and RSS includes the shared pages. I attached a python script I wrote to compare actual pages between processes.
fork() acts as expected even in python. The problem is that after the fork VDSM keeps changing the memory and the children keep a copy of the memory. This means that even though they'll never use it a copy will exists. Because of how we spawn processes. child processes can have different versions of VDSM thus actually ending up taking all the memory not because they rewrote it but because VDSM did.
Comparing OOPs that spawned at almost the same time they share a lot of memory 194 (776 KB) only in A, 2146 (8584 KB) shared, 194 (776 KB) only in B
OOP processes that spawned a few hours apart have a lot less in common 2336 (9344 KB) only in A, 4 (16 KB) shared, 2521 (10084 KB) only in B
So spawning all the processes at once instead of gradually would solve this memory problem.
I added the script if you want to test it yourselves on your hosts and test progs
On 07/21/2011 09:52 AM, Dan Kenigsberg wrote:
On Wed, Jul 20, 2011 at 11:41:33AM -0400, David Malcolm wrote:
On Wed, 2011-07-20 at 13:39 +0300, Moran Goldboim wrote:
Hi Dave,
We are suffering high memory usage on vdsm as a result of using threading in python, child process is taking most of the father memory. for example:
import multiprocessing import time l = [1000] * 2**20 def run(): time.sleep(10000) multiprocessing.Process(target=run).start()
Multiprocessing is one of my least favorite modules. It doesn't use threads so much as it forks child processes, without calling exec: each child process within multiprocessing has an entire copy of the parent, which gradually leads to copy-on-write differences, even for supposedly read-only operations, given that python uses reference counting (since all of the refcounts are constantly going through inc/dec operations, so pages rapidly get written to).
Ouch. Though I'm not sure this explains the issue with the example script: the child process does nothing, so it states should not change with time. It appeared to me that the pages became dirty instantaneously. Am I missing smething?
grep Shared_Dirty /proc/22986/smaps | awk '{s=s+$2} END {print s}' 11272
top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 22985 mgoldboi 20 0 184m 14m 2908 S 0.0 0.4 0:00.04 python 22986 mgoldboi 20 0 184m 12m 616 S 0.0 0.3 0:00.00 python
is it a known issue? can we do something about it?
How much do you need the child to inherit state from the parent? How long-lived is the child? In theory, I guess you could start "del"-ing things within the child, and run a garbage collection first. But if you merely need a subprocess, could you use the "subprocess" module instead? (which does an "exec" after the fork, so you're starting with a clean slate).
We used to have our own inter-process messaging with Python 2.4, and where very happy to ditch it in favor of multiprocessing... I am not sure we can go back to that. Saggi, can you find a way around this?
FWIW I wrote a tool to analyse python memory usage: gdb-heap; see http://fedoraproject.org/wiki/Features/MemoryDebuggingTools However, it is unpolished and (ironically) very memory-hungry itself.
Hope this is helpful
It is, in an unhappy way.
Thanks, Dan.