LNST-developers March 2013

lnst-developers@lists.fedorahosted.org

7 participants
68 discussions

[PATCH 0/5] Transmitting exceptions from slave to ctl
by Ondrej Lichtner 29 Mar '13

29 Mar '13

From: Ondrej Lichtner <olichtne(a)redhat.com> This patch series implements transmission of exceptions over our communication protocol. The exceptions are logged on the slaves and the logs are transmitted to the controller. Additionaly the slave sends a message of type "exception" to notify the controller that an exception has occured. When this happens the controller stops the interpretation of the recipe and performs cleanup on slaves. Ondrej Lichtner (5): NetTestSlave: add handling for exception messages NetTestSlave: catch exceptions from methods NetTestController: add handling for exception messages NetTestSlave: add kill_cmds method and cleanup NetTestController: fix slave cleanup lnst/Common/NetTestCommand.py | 1 + lnst/Controller/NetTestController.py | 30 ++++++++++++++++++++++++----- lnst/Slave/NetTestSlave.py | 37 +++++++++++++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 6 deletions(-) -- 1.7.11.7

2 6

[lnst] NetTestSlave: add handling for exception messages
by Jiří Pírko 29 Mar '13

29 Mar '13

commit d65e7193d347dd46fcecd7de15f457c6b59a78a8 Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 14:03:18 2013 +0100 NetTestSlave: add handling for exception messages If an exception is raised in a command a message of type "exception" is sent to the slave server. This commit adds support for correct handling of such messages. I also added an additional field to the exception message that can be used to identify the command from which the message arrived. This is used both in the log message created and to correctly remove the command after it's failed. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Common/NetTestCommand.py | 1 + lnst/Slave/NetTestSlave.py | 9 +++++++++ 2 files changed, 10 insertions(+), 0 deletions(-) --- diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index 146401e..2ce9b67 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -126,6 +126,7 @@ class NetTestCommand: except: type, value, tb = sys.exc_info() result = {"type": "exception", + "cmd_id": self._id, "Exception": ''.join(traceback.format_exception(type, value, tb))} send_data(self._write_pipe, result) diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index aad7074..677fe6b 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -377,6 +377,15 @@ class NetTestSlave: if not self._server_handler.send_data_to_ctl(msg): self._log_ctl.cancel_connection() elif msg["type"] == "exception": + if msg["cmd_id"] != None: + logging.debug("Recieved an exception from command with id: %s" + % msg["cmd_id"]) + else: + logging.debug("Recieved an exception from foreground command") + logging.error(msg["Exception"]) + cmd = self._cmd_context.get_cmd(msg["cmd_id"]) + cmd.join() + self._cmd_context.del_cmd(cmd) if not self._server_handler.send_data_to_ctl(msg): self._log_ctl.cancel_connection() elif msg["type"] == "result":

1 0

[lnst] NetTestSlave: adjust response handling to fit NetTestCommand
by Jiří Pírko 29 Mar '13

29 Mar '13

commit 48dfa9512867d77d02952ab84a13317bc47f53ac Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 13:55:04 2013 +0100 NetTestSlave: adjust response handling to fit NetTestCommand This patch reiplements the method run_command to reflect the changes of NetTestCommand class. The method returns the result of the run() method of the NetTestCommand class which is None for most cases but in case of wait/intr/kill commands can be the result of the test command running in background. And in the case of running a background command in the traditional sense we still return a notification that yes the command was forked. Because of this the server also had to be adjusted a little. Currently if the server recieves a return value from the run_command method it will forward it to the controller. However if None is returned, nothing is send to the controller which causes the controller to wait until a response arrives. This way I simulate the wait for the end of a foreground test command while also being able to recieve messages (such as logs) from other slaves. Now if the server recieves a result message this could mean 3 things: * a foreground command finished and the server forwards the result to the controller * a background command that wasn't killed/interrupted/waited on has finished- the server stores the result of this command and will forward it to the controller when the kill/intr/wait command is recieved * a background command that was already killed/interrupted/waited on has finished- the server forwards the result to the controller. This way the controller is waiting for a result message for every command that he sends and at the same time he will be able to continue when a background command has been launched. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Slave/NetTestSlave.py | 39 +++++++++++++++++++++++++++++++++------ 1 files changed, 33 insertions(+), 6 deletions(-) --- diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index 253a75a..aad7074 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -159,9 +159,14 @@ class SlaveMethods: def run_command(self, command): try: - cmd_cls = NetTestCommand(self._command_context, command, + cmd = NetTestCommand(self._command_context, command, self._resource_table, self._log_ctl) - return cmd_cls.run() + self._command_context.add_cmd(cmd) + + res = cmd.run() + if not cmd.forked(): + self._command_context.del_cmd(cmd) + return res except: log_exc_traceback() cmd_type = command["type"] @@ -351,26 +356,48 @@ class NetTestSlave: if self._server_handler.get_ctl_sock() == None: self._log_ctl.cancel_connection() + self._cmd_context.cleanup() + def _process_msg(self, msg): if msg["type"] == "command": method = getattr(self._methods, msg["method_name"], None) if method != None: result = method(*msg["args"]) - response = {"type": "result", "result": result} - if not self._server_handler.send_data_to_ctl(response): - self._log_ctl.cancel_connection() + + if result != None: + response = {"type": "result", "result": result} + if not self._server_handler.send_data_to_ctl(response): + self._log_ctl.cancel_connection() else: err = "Method not found: %s" % msg["method_name"] response = {"type": "error", "err": err} if not self._server_handler.send_data_to_ctl(response): self._log_ctl.cancel_connection() - elif msg["type"] == "log": if not self._server_handler.send_data_to_ctl(msg): self._log_ctl.cancel_connection() elif msg["type"] == "exception": if not self._server_handler.send_data_to_ctl(msg): self._log_ctl.cancel_connection() + elif msg["type"] == "result": + if msg["cmd_id"] == None: + del msg["cmd_id"] + if not self._server_handler.send_data_to_ctl(msg): + self._log_ctl.cancel_connection() + cmd = self._cmd_context.get_cmd(None) + cmd.join() + self._cmd_context.del_cmd(cmd) + else: + cmd = self._cmd_context.get_cmd(msg["cmd_id"]) + cmd.join() + del msg["cmd_id"] + + if cmd.finished(): + if not self._server_handler.send_data_to_ctl(msg): + self._log_ctl.cancel_connection() + self._cmd_context.del_cmd(cmd) + else: + cmd.set_result(msg["result"]) else: raise Exception("Recieved unknown command")

1 0

[lnst] NetTestCommand: reimplement the NetTestCommand classes
by Jiří Pírko 29 Mar '13

29 Mar '13

commit 0438c8330b1659792038da945e2e55a5ceefdcee Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 13:55:03 2013 +0100 NetTestCommand: reimplement the NetTestCommand classes These classes previously worked only with background processes and needed to be changed to reflect this. The kill/intr/wait test commands are the only exception to the "all test commands will run in the background" and will be run in the main process since they need access to the child processes. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Common/NetTestCommand.py | 24 +++++++++++++++--------- 1 files changed, 15 insertions(+), 9 deletions(-) --- diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index 3fd6ac4..146401e 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -326,29 +326,35 @@ class NetTestCommandControl(NetTestCommandGeneric): class NetTestCommandWait(NetTestCommandControl): def run(self): bg_id = self._command["value"] - bg_cmd = self._command_context.get_bg_cmd(bg_id) + bg_cmd = self._command_context.get_cmd(bg_id) bg_cmd.wait_for() result = bg_cmd.get_result() - self._command_context.del_bg_cmd(bg_cmd) - self.set_result(result) + if result != None: + bg_cmd.join() + self._command_context.del_cmd(bg_cmd) + return result class NetTestCommandIntr(NetTestCommandControl): def run(self): bg_id = self._command["value"] - bg_cmd = self._command_context.get_bg_cmd(bg_id) + bg_cmd = self._command_context.get_cmd(bg_id) bg_cmd.interrupt() result = bg_cmd.get_result() - self._command_context.del_bg_cmd(bg_cmd) - self.set_result(result) + if result != None: + bg_cmd.join() + self._command_context.del_cmd(bg_cmd) + return result class NetTestCommandKill(NetTestCommandControl): def run(self): bg_id = self._command["value"] - bg_cmd = self._command_context.get_bg_cmd(bg_id) + bg_cmd = self._command_context.get_cmd(bg_id) bg_cmd.kill() result = bg_cmd.get_result() - self._command_context.del_bg_cmd(bg_cmd) - self.set_result({"passed": True}) + if result != None: + bg_cmd.join() + self._command_context.del_cmd(bg_cmd) + return result def get_command_class(command_context, command, resource_table): cmd_type = command["type"]

1 0

[lnst] NetTestCommand: renaming in NetTestCommandContext
by Jiří Pírko 29 Mar '13

29 Mar '13

commit 295fccf92a4b21e1f5eec9dd5d5b592a793944c8 Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 13:55:02 2013 +0100 NetTestCommand: renaming in NetTestCommandContext This commit changes the names in the class NetTestCommandContext. Previously the class was only used for background commands. But since all of our test will now run in the background I renamed the methods to something more appropriate. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Common/NetTestCommand.py | 24 +++++++++++++----------- 1 files changed, 13 insertions(+), 11 deletions(-) --- diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index a88c76f..3fd6ac4 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -172,27 +172,29 @@ class NetTestCommandContext: def __init__(self): self._dict = {} - def add_bg_cmd(self, bg_cmd): - self._dict[bg_cmd.get_bg_id()] = bg_cmd + def add_cmd(self, cmd): + self._dict[cmd.get_id()] = cmd - def del_bg_cmd(self, bg_cmd): - del self._dict[bg_cmd.get_bg_id()] + def del_cmd(self, cmd): + del self._dict[cmd.get_id()] - def get_bg_cmd(self, bg_id): - return self._dict[bg_id] + def get_cmd(self, id): + return self._dict[id] - def _kill_all_bg_cmds(self): - for bg_id in self._dict: - self._dict[bg_id].kill() + def _kill_all_cmds(self): + for id in self._dict: + self._dict[id].kill() def cleanup(self): - self._kill_all_bg_cmds() + self._kill_all_cmds() self._dict = {} def get_read_pipes(self): pipes = {} for key in self._dict: - pipes[key] = self._dict[key].get_connection_pipe() + pipe = self._dict[key].get_connection_pipe() + if pipe != None: + pipes[key] = pipe return pipes def NetTestCommandTest(command, resource_table):

1 0

[lnst] NetTestCommand: reimplementation of NetTestCommand
by Jiří Pírko 29 Mar '13

29 Mar '13

commit fe264cae967640f6c6da1345ceee07901942741e Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 13:55:01 2013 +0100 NetTestCommand: reimplementation of NetTestCommand This commit merges the classes BgCommand and NetTestCommand into one single NetTestCommand class. The reason behind this is that we want all test commands to be launched in the background because of how our new ctl<->slave communication works and also for the cases where something goes wrong in the test execution. The new NetTestCommand class therefore handles the creation of the forked processes and also the execution of the commands. With this change in classes I also reimplemented our way of creating a process in the background to use the multiprocessing module. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Common/NetTestCommand.py | 147 ++++++++++++++++++++++++----------------- 1 files changed, 87 insertions(+), 60 deletions(-) --- diff --git a/lnst/Common/NetTestCommand.py b/lnst/Common/NetTestCommand.py index 05a7ca3..a88c76f 100644 --- a/lnst/Common/NetTestCommand.py +++ b/lnst/Common/NetTestCommand.py @@ -18,6 +18,7 @@ import imp import pickle, traceback import multiprocessing from lnst.Common.ExecCmd import exec_cmd, ExecCmdFail +from lnst.Common.ConnectionHandler import recv_data, send_data def str_command(command): out = ("type (%s), machine_id (%s), value (%s)" @@ -46,79 +47,124 @@ class BgCommandException(Exception): def __str__(self): return "BgCommandError: " + self._str -class BgCommand: - def __init__(self, bg_id, cmd_cls, log_ctl): - self._bg_id = bg_id - self._cmd_cls = cmd_cls - self._pid = None + +class NetTestCommand: + def __init__(self, command_context, command, resource_table, log_ctl): + self._cmd_cls = get_command_class(command_context, command, + resource_table) + self._command_context = command_context + self._command = command + + self._process = None + self._id = None self._read_pipe = None + self._write_pipe = None + self._connection_pipe = None self._killed = False + self._finished = False + self._result = None self._log_ctl = log_ctl - def get_bg_id(self): - return self._bg_id + if "bg_id" not in self._command: + self._id = None + else: + self._id = self._command["bg_id"] + + def get_id(self): + return self._id + + def forked(self): + return self._process != None + + def finished(self): + return self._finished def run(self): - read_pipe, write_pipe = os.pipe() #TODO multiprocessing - con_read_pipe, con_write_pipe = multiprocessing.Pipe() - self._pid = os.fork() - if self._pid: - os.close(write_pipe) + if isinstance(self._cmd_cls, NetTestCommandControl): + return self._cmd_cls.run() + + self._read_pipe, self._write_pipe = multiprocessing.Pipe() + self._process = multiprocessing.Process(target=self._run) + + self._process.daemon = False + self._process.start() + self._pid = self._process.pid + + self._connection_pipe = self._read_pipe + + if not self._id: + logging.debug("Running command with" + " pid \"%d\"" % (self._pid)) + return None + else: logging.debug("Running in background with" - " id \"%s\", pid \"%d\"" % (self._bg_id, self._pid)) - self._read_pipe = read_pipe - self._connection_pipe = con_read_pipe - return {"passed": True, "pipe": self._read_pipe} - os.close(read_pipe) + " bg_id \"%s\" pid \"%d\"" % (self._id, self._pid)) + return {"passed": True} + + def _run(self): os.setpgrp() self._cmd_cls.set_handle_intr() - self._connection_pipe = con_write_pipe + + self._connection_pipe = self._write_pipe self._log_ctl.unset_recipe() self._log_ctl.cancel_connection() - self._log_ctl.set_connection(self._connection_pipe) + self._log_ctl.set_connection(self._write_pipe) + result = {} try: self._cmd_cls.run() - result = self._cmd_cls.get_result() + res_data = self._cmd_cls.get_result() + result["type"] = "result" + result["cmd_id"] = self._id + result["result"] = res_data except KeyboardInterrupt: - result = self._cmd_cls.get_result() + res_data = self._cmd_cls.get_result() + result["type"] = "result" + result["cmd_id"] = self._id + result["result"] = res_data except: type, value, tb = sys.exc_info() - result = {"Exception": ''.join(traceback.format_exception(type, value, tb))} - tmp = pickle.dumps(result) - os.write(write_pipe, tmp) - os.close(write_pipe) - os._exit(0) + result = {"type": "exception", + "Exception": ''.join(traceback.format_exception(type, + value, tb))} + send_data(self._write_pipe, result) + self._write_pipe.close() + + def join(self): + self._process.join() def wait_for(self): - logging.debug("Waiting for background command with id \"%s\", pid \"%d\"" % (self._bg_id, self._pid)) - os.waitpid(self._pid, 0) + logging.debug("Waiting for background command with id \"%s\", pid \"%d\"" % (self._id, self._pid)) + self._finished = True def interrupt(self): - logging.debug("Interrupting background command with id \"%s\", pid \"%d\"" % (self._bg_id, self._pid)) - os.killpg(os.getpgid(self._pid), signal.SIGINT) - os.waitpid(self._pid, 0) + self._finished = True + if os.path.exists("/proc/%d" % self._pid): + logging.debug("Interrupting background command with id \"%s\", pid \"%d\"" % (self._id, self._pid)) + os.killpg(os.getpgid(self._pid), signal.SIGINT) + self._process.join() def kill(self): - logging.debug("Killing background command with id \"%s\", pid \"%d\"" % (self._bg_id, self._pid)) - self._killed = True - os.killpg(os.getpgid(self._pid), signal.SIGKILL) + if os.path.exists("/proc/%d" % self._pid): + logging.debug("Killing background command with id \"%s\", pid \"%d\"" % (self._id, self._pid)) + self._killed = True + os.killpg(os.getpgid(self._pid), signal.SIGKILL) + self._process.join() def get_result(self): - result = {} if self._killed: - result["logs"] = [] + result = {} + result["type"] = "result" result["passed"] = True else: - tmp = os.read(self._read_pipe, 4096*10) - result = pickle.loads(tmp) - if "Exception" in result: - raise BgCommandException(result["Exception"]) + result = self._result - os.close(self._read_pipe) return result + def set_result(self, result): + self._result = result + def get_connection_pipe(self): return self._connection_pipe @@ -322,22 +368,3 @@ def get_command_class(command_context, command, resource_table): cmd_cls.set_resource_table(resource_table) return cmd_cls - -class NetTestCommand: - def __init__(self, command_context, command, resource_table, log_ctl): - self._log_ctl = log_ctl - self._command_class = get_command_class(command_context, command, - resource_table) - self._command_context = command_context - self._command = command - - def run(self): - cmd_cls = self._command_class - if "bg_id" in self._command: - bg_id = self._command["bg_id"] - bg_cmd = BgCommand(bg_id, cmd_cls, self._log_ctl) - self._command_context.add_bg_cmd(bg_cmd) - return bg_cmd.run() - else: - cmd_cls.run() - return cmd_cls.get_result()

1 0

[lnst] NetTestSlave: bug fix for the ServerHandler
by Jiří Pírko 29 Mar '13

29 Mar '13

commit b7cb8fcf319f3c9ded16bcae9c57d93c7ca8b98c Author: Ondrej Lichtner <olichtne(a)redhat.com> Date: Fri Mar 29 13:55:00 2013 +0100 NetTestSlave: bug fix for the ServerHandler There was a bug in the ServerHandler class, with passing parameters in the remove_connection method. This patch fixes it. Signed-off-by: Ondrej Lichtner <olichtne(a)redhat.com> lnst/Slave/NetTestSlave.py | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-) --- diff --git a/lnst/Slave/NetTestSlave.py b/lnst/Slave/NetTestSlave.py index 5a00b0d..253a75a 100644 --- a/lnst/Slave/NetTestSlave.py +++ b/lnst/Slave/NetTestSlave.py @@ -304,14 +304,16 @@ class ServerHandler(object): def add_connection(self, id, connection): self._connection_handler.add_connection(id, connection) - def remove_connection(self, id): - self._connection_handler.remove_connection(id, connection) + def remove_connection(self, key): + connection = self._connection_handler.get_connection(key) + self._connection_handler.remove_connection(connection) def clear_connections(self): self._connection_handler.clear_connections() def update_connections(self, connections): for key, connection in connections.iteritems(): + self.remove_connection(key) self.add_connection(key, connection) class NetTestSlave:

1 0

[PATCH 0/5] Reimplementation of NetTestCommand
by Ondrej Lichtner 29 Mar '13

29 Mar '13

From: Ondrej Lichtner <olichtne(a)redhat.com> The following patch series reimplements the NetTestCommand class used on the slave. In the new implementation we run all commands in a sepparate process to unify the design. This change was also necessary to make the SlaveServer fully non-blocking. Ondrej Lichtner (5): NetTestSlave: bug fix for the ServerHandler NetTestCommand: reimplementation of NetTestCommand NetTestCommand: renaming in NetTestCommandContext NetTestCommand: reimplement the NetTestCommand classes NetTestSlave: adjust response handling to fit NetTestCommand lnst/Common/NetTestCommand.py | 195 +++++++++++++++++++++++++----------------- lnst/Slave/NetTestSlave.py | 45 ++++++++-- 2 files changed, 152 insertions(+), 88 deletions(-) -- 1.7.11.7

1 5

[lnst] Removing flags from sub
by Jiří Pírko 29 Mar '13

29 Mar '13

commit 087fda7efa313aba7eaad03d16205f13864f432e Author: Jan Tluka <jtluka(a)redhat.com> Date: Thu Mar 28 17:47:16 2013 +0100 Removing flags from sub Using flags parameter in re.sub() was added in Python 2.7. I'd like to be able use LNST on RHEL6 that's coming with Python 2.6. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> lnst/Controller/SlavePool.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) --- diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py index e430bbe..8470190 100644 --- a/lnst/Controller/SlavePool.py +++ b/lnst/Controller/SlavePool.py @@ -58,7 +58,7 @@ class SlavePool: parser.disable_events() machine = {"params": {}, "netdevices": {}} - machine_id = re.sub("\.xml$", "", basename, flags=re.I) + machine_id = re.sub("\.[xX][mM][lL]$", "", basename) parser.set_machine(machine_id, machine) slavemachine = dom.getElementsByTagName("slavemachine")[0]

1 0

[PATCH] Removing flags from sub
by Jan Tluka 28 Mar '13

28 Mar '13

Using flags parameter in re.sub() was added in Python 2.7. I'd like to be able use LNST on RHEL6 that's coming with Python 2.6. Signed-off-by: Jan Tluka <jtluka(a)redhat.com> --- lnst/Controller/SlavePool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lnst/Controller/SlavePool.py b/lnst/Controller/SlavePool.py index e430bbe..8470190 100644 --- a/lnst/Controller/SlavePool.py +++ b/lnst/Controller/SlavePool.py @@ -58,7 +58,7 @@ class SlavePool: parser.disable_events() machine = {"params": {}, "netdevices": {}} - machine_id = re.sub("\.xml$", "", basename, flags=re.I) + machine_id = re.sub("\.[xX][mM][lL]$", "", basename) parser.set_machine(machine_id, machine) slavemachine = dom.getElementsByTagName("slavemachine")[0] -- 1.7.11.7

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

LNST-developers March 2013