From: Ondrej Lichtner <olichtne(a)redhat.com>
File descriptors opened for communication with a Job process on the
slave were not being closed and cleaned up properly. This leads to file
descriptor ulimit exhaustion very quickly and only via launching recipe
jobs in a sequence. This basically made any Recipe "time" limited to a
certain number of jobs that it can execute.
Closing the multiprocessing.Pipe-s (unix sockets) and removing the Job
from the JobContext to clean up the sentinel pipe resolves this issue.
The file descriptor limit can still be hit in case enough parallel Jobs
are run, but that is intended and a problem that is out-of-scope for
LNST.
Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com>
---
lnst/Slave/Job.py | 7 +++++++
lnst/Slave/NetTestSlave.py | 3 +++
2 files changed, 10 insertions(+)
diff --git a/lnst/Slave/Job.py b/lnst/Slave/Job.py
index 004ef02..b4cdac4 100644
--- a/lnst/Slave/Job.py
+++ b/lnst/Slave/Job.py
@@ -95,6 +95,8 @@ class Job(object):
return True
def _run(self):
+ self._parent_pipe.close()
+
os.setpgrp()
signal.signal(signal.SIGHUP, signal.SIG_DFL)
signal.signal(signal.SIGINT, signal.SIG_DFL)
@@ -150,6 +152,11 @@ class Job(object):
self._finished = True
self._result = result
+ self._parent_pipe.close()
+ self._child_pipe.close()
+ self._parent_pipe = None
+ self._child_pipe = None
+
def get_result(self):
return self._result
diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py
index e2e8d6d..710dbce 100644
--- a/lnst/Slave/NetTestSlave.py
+++ b/lnst/Slave/NetTestSlave.py
@@ -1032,6 +1032,9 @@ class NetTestSlave:
job.set_finished(msg["result"])
self._server_handler.send_data_to_ctl(msg)
+
+ self._job_context.del_job(job)
+
elif msg["type"] == "from_netns":
self._server_handler.send_data_to_ctl(msg["data"])
elif msg["type"] == "to_netns":
--
2.26.0