Hi everyone,
I've got a new version of the API spec file.
what changed: * I've renamed some stuff - Task -> Recipe (since the class handles both requirements and the test execution, calling it task makes no sense to me * commented out the "def test(m1, m2, m3...)" method of the BaseRecipe class, based on upstream discussion this might be confusing, hard to implement and if we can always decide later to add it (removing would be more problematic) * Added 'Controller' to the Tester facing API - provided by the LNST library to enable a tester to use the LNST controller from his own executable script * Host - added some discussion about the name "params" - removed 'tool' argument from run(), kept the 'path' argument because i didn't know if jpirko meant to remove them both or just the second one * Job - expanded basic description - how all Jobs will be technically in background, and fg/bg handling will be on the Controller. * added proposal for changing how the Result summary will look like
I still haven't started working on the Device API due to reworking how Hosts will be matched and allocated (this is related to properly creating the Device objects...)
Attaching the doc here:
1. test modules
class BaseTestModule: def __init__(self, **kwargs): #by defaults loads the params into self.params - no checks pseudocode: for x in vars(self): if isinstance(x, Param): param_class = self.getattr(x) try: val = kwargs[x] except KeyError: if param_class.is_mandatory(): raise TestModuleError("Option x is mandatory") self.setattr(x.params, param_class.construct(val)) del kwargs[x] for x in kwargs.keys(): log.error("Undefined parameter x") if len(kwargs): raise TestModuleError("Undefined TestModule parameters")
#check mandatory parameters for name, param in self.params: if param.mandatory and not param.set: raise TestModuleException("Parameter {} is mandatory".format(name))
def run(): #needs to be over-ridden - throw an exception to notify the test developer
class MyTest(BaseTestModule): param = ParamType(mandatory=True) param2 = ParamType2() param3 = Multiparam(ParamType())
#optional __init__ #def __init__(self, **kwargs): #super(MyTest).__init__(kwargs) #additional tester defined checks
def run(): #do my test #parameters available in self.params
#all imports used by the test module need to happen HERE, reason is #that the object is instantiated on the Controller and THEN sent to the #slave -> stuff imported on the Controller is not available on the Slave
#in Task: import lnst #module lnst.tests will dynamically look for module classes in configured #locations, similar to how we do it now #or you can import BaseTestmodule and define your test module directly in your recipe
ping = lnst.modules.Ping(dst=m2.if1.ip[0], count=100, interval=0.1)
m1.run(ping)
================================================
2. Recipe: Renamed Task to Recipe... since the class represents the "whole" LNST workflow of setting requirements and defining the test
class BaseRecipe(object): def __init__(self): #initialize instance specific requirements self.requirements = Requirements() for x in dir(self): val = getattr(self, x) if instance(val, Requirement): setattr(self.requirements, x, val)
# check types of parameters, the Param class hierarchy is the same as # the one used by BaseTestModule self.params = object() for x in dir(self): val = getattr(self, x) if instance(val, Param): setattr(self.params, x, val)
# check mandatory parameters for name, param in self.params: if param.mandatory and not param.set: raise RecipeException("Parameter {} is mandatory".format(name))
def test(): raise Exception("Method test MUST be defined.")
class MyRecipe(lnst.BaseRecipe): #class-wide definition of requirements m1 = HostReq(arch="x86_64", ...) m1.if1 = IfaceReq(label="xyz", driver="mlx", ...)
#m1.params = IfaceReq() #m1.devs = IfaceReq() #raise Exception("param/devs name is a reserved keyword")
m2 = HostReq(param="val", ...) m2.if1 = IfaceReq(label="xyz", param="val", ...)
# idosch requested the ability to "blacklist" parameters, jpirko's # suggestion was to use regular expressions in the mapper. This should be # easy to do, though it's not entirely "pretty" since regexes are not very # good at doing negative conditions... maybe we can think of something # better?
#class-wide definition of Task parameters, with type checking, #possibly even type conversions (e.g. string to int), using the same #*Type classes as with Module parameters mtu = IntType(default=1500)
def __init__(self, **kwargs): super(self, lnst.BaseTask).__init__()
#do something with kwargs interface_driver = kwargs["driver"] self.reqs.m1.if1.driver = interface_driver #adjust instance specific requirements self.reqs.m3 = HostReq(...)
def test(self): self.matched.m1.run(Module) self.matched.m1.run("command")
# optionally we can add support for something like this, but at the moment # we think it could be confusing and/or complicated to do right #def test(m1, m2, m3, m4,...): # m1.run(Module) # m2.run("command") # m1.params.arch == "x86_64"
================================================
3. Running Recipes:
my_test_script.py: #!/bin/python from MyRecipes import MyRecipe from lnst import Controller
recipe_instance = MyRecipe(mtu="5000")
ctl = Controller(config="/etc/lnst-ctl.conf") ctl.run(recipe_instance)
chmod +x my_test_script.py ./my_test_script.py args
OR
lnst-ctl -d run MyRecipe.py -- mtu=8000 # looks for NAME class in the NAME.py file (MyRecipe in this case for which # the condition "isinstance(NAME, BaseRecipe)" must be True
# could also run for all classes in the file where "isinstance(x, BaseRecipe)" # is True. with the option to restrict to specific task class... lnst-ctl # rewritten to do the same as manually running the task from it's own python # script
Aliases lose meaning - they're parameters passed to the MyTask __init__, when using the lnst-ctl CLI, use "-- task_params"?? might not work for multiple tasks,
================================================
4. Tester facing API:
#!!!! breakpoints!!!
class Controller: # This is the class which serves as an entry point to using LNST in a # library-like way to run a controller - as in running a Recipe from it's # own executable python script" # It's partially based on the old NetTestController class
def __init__(self, debug=0, mapper=MachineMapper, pools=[], pool_checks=True): # defines the logging level self._debug = debug self._mac_pool = MacPool() # class controlling logging self._log_ctl = LoggingCtl(debug, ...) # dictionary of dynamically created networks (libvirt) self._network_bridges = {}
# a ConnectionHandler for communicating with slaves self._msg_dispatcher = MessageDispatcher(self._log_ctl)
# a Mapper class - handling the matching of requirements to the Hosts # available in pools, you can define your own class that supports the # API (specified later) to do your own matching API, the basic idea is # for the Mapper to get requirements in a certain format and the pools # in a certain format and to generate mappings between these two self._mapper = mapper()
# pool loading - from a config file, and optionaly restricted from the # pools argument, functionality copied from the NetTestController, I'm # thinking this could change (pools default value is grabbed from # config, so that the user can override them completely?) select_pools = {} conf_pools = ctl_config.get_pools() if len(pools) > 0: for pool_name in pools: if pool_name in conf_pools: select_pools[pool_name] = conf_pools[pool_name] elif len(pools) == 1 and os.path.isdir(pool_name): select_pools = {"cmd_line_pool": pool_name} else: raise NetTestError("Pool %s does not exist!" % pool_name) else: select_pools = conf_pools
# a PoolManager, that loads the slave description XML files and checks # the availability of test hosts. Defines an API to access the available # pools usable by the Mapper to match against requirements. I'm thinking # this could also be made an argument of the __init__ method when the # API is defined, the users could then replace both the Mapper and the # PoolManager... although I don't think it would be used very much... self._pools = SlavePoolManager(select_pools, pool_checks)
def run(self, recipe, **mapper_kwargs): # this method implements the main loop of the recipe execution, the # mapper_kwargs are arguments passed to the mapper to influence the # mapping process, such as to try all matches, or to enable virtual # matches and so on
for match in self._mapper.matches(**mapper_kwargs): matched = True # uses the current match to map the Machine objects from the # PoolManager to create Host objects for the Recipe self._map_match(match) self._print_match_description(match) try: recipe.test() except Exception as exc: logging.error("Recipe execution terminated by unexpected exception") raise finally: for machine in self._machines.values(): machine.restore_system_config() self._cleanup_slaves()
Host objects available in self.matched.selector_name:
class Host: #name can change #attributes:
# dynamically filled object of Host attributes such as architecture and # so on. Use example in test() would look like this: # if host.params.arch == "x86": # I separated this into the "params" object so I can overwrite its # __getattr__ method and return None/UnknownParam exception for unknown # parameters, and to avoid name conflicts with other attributes # they should also be iterable by: # for param in host.params: params = object()
!!! params sounds odd? other possibilities 'property', 'description'? !!! m1.property.arch == "x86" ? !!! m1.description.arch == "x86" ? !!! we've also discussed that it would probably be easiest to just have a !!! predefined static set of these attributes that are checked on !!! lnst-slave startup and are sent to the Controller on connection.
# dynamically filled object of NetDevice objects accessible directly as the # object attributes: # host.eth0.set_ip(...) # I separated this into the "ifaces" object to avoid name conflicts with # other attributes # creation of new NetDevices should be possible through simple assignement: # m1.team0 = TeamDevice(...) # assignement of an incompatible Type or to an existing Device object will # return an exception # to deconfigure+remove call m1.team0.remove() __devs = object()
# device iterator for the Host object: # for dev in host:
def run(what, bg=False, fail=False, timeout=60, path="", json=False, netns, desc) # will run "what" on the remote host # "what" is either a Module object, or a string command that will be # executed as a bash command # "bg" when True, runs "what" on background - the run() call # immediately returns, and "timeout" is ignored, the background # process can be controlled through the returned Job object # "fail" if True then the Job is expected to fail, and will be reported # as PASSed if it does # "timeout" in seconds, determines how long to block test execution for # before killing the Job. Only when running in foreground # "path" changes the current working directory to the specified path # before "what" is executed and changes back after execution is # finished. # IGNORE FOR NOW # "json" if True will attempt to parse the returned stdout of the Job # as json into a dictionary # "netns" Job will be run in the specified network namespace # "desc" is a description message that will show up in logs and the # result summary # Returns a Job object # !! Exceptions?!?!
# !!!THINK ABOUT THESE... def __set(path, value) # copied from old API, provides a shortcut for "echo $value # >/proc/or/sys/path" # and returns the original value when the test is finished
def sysfs_set(path, value): # check if path starts with "/sys"? self.__set(path, value)
def procfs_set(path, value): self.__set("/proc/"+path, value)
def send_file(ctl_path="", slave_path="", recursive=False) # copies the specified file from the controller to the specified # destination path, if recursive == True and srcpath refers to a # directory it copies the entire directory
def recv_file(recv_path="", ctl_path="", recursive=False)
def {enable, disable}_service(service) # copied from old API, enables or disables the specified service
def del_device(name) # removes the specified device, probably easier (more logical?) to do # this then "devs.name = None" and "del devs.name" would be unreliable # !!! REMOVE
class Device: #DeviceAPI, InterfaceAPI? name can change... # attributes:
# dynamically created Device attributes such as driver and so on. Use # example in test() would look like this: # if host.devs.eth0.driver == "ixgbe": # achieved through rewriting of the __getattr__ method of the Device class # should return None or throw UnknownParam exception for unknown parameters # this should directly mirror the Device objects that are managed by the # InterfaceManager on the Slave # !!! predefine the params, assignment will set them, sync on set - wait # !!! on the slave for the set to finish and return the current status of # !!! everything else as well # eg: driver = something mtu = something ips = [IpAddress, ...]
class Job: #ProcessAPI? name can change... # The way Jobs are launched is different from how we handled them with XML # recipes -> on Slaves, ALL jobs are run in a separate process ("in # background"), without a timeout, and they all have an id (previously # foreground commands had id == None) # # The distinction of foreground/background Jobs is handled by the Hosts # "run" method -> for background Jobs, the method immediately returns with # the Job handle (instance of this class), for foreground Jobs, the method # calls the wait method of this class (default timeout) and if it times out, # it calls the kill method of this class (SIGKILL) # # For now, I'm just considering Module and "Shell command" Jobs, system # configs will probably be handled in a different way... # # Job results can be picked up at any time the execution is handed over to # the LNST library - the message carrying results from the slave will be # handled by the connection handler and sent to the correspoding Job # object. This is relevant for background Jobs only as foreground Jobs will # always be in a finished state after the Host run method returns
#attributes:
# True if the Job finished, False if it's still running in the background finished = bool
# contains the result data returned by the Job result = object
# contain the stdout and stderr generated by the job, None for Module Jobs stdout = "" stderr = ""
# simple True/False value indicating success/failure of the Job passed = bool
def wait(timeout=0): # for background jobs, will wait until the job finished # "timeout" in seconds, determines how long to wait for. After timeout # reached, nothing happens, status of the job can be checked with the # "finished" attribute. If timeout=0, then wait forever.
def kill(signalnum=signal.SIGKILL): # sends the specified signal to the process of the Job running in # background # in case the Job finished while this method call is executed then the # job just finishes normally and the signal isn't sent # "signalnum" the signal to be sent, default is SIGKILL
def __str__(self): # for easy print m1.run(what) return stdout+stderr # what about Modules?
================================================
5. Result summary proposal
Since I've changed how Job execution is handled, I've also wrote down a proposal to change how we log Recipe results - the RESULTS SUMMARY logs at the end of a recipe run. I haven't started working on it yet, I've just wrote an example on paper which I'm copying here. Any comments are appreciated.
RESULTS SUMMARY: Host m1 Job 1 XYZ PASS/FAIL Formatted results: ... Host m2 Job 1 XYZ started Host m1 Job 3 XYZ PASS/FAIL Formatted results: ... Host m2 Job 1 XYZ PASS/FAIL Formatted results: ... Custom summary record.... (optional PASS/FAIL) ... optional additional data ... i still need to figure out how this will look like
The main difference to the old results summary is that Jobs have numerical ids that are unique per host, and you ALWAYS see the id (previously only background commands had ids). Since all Jobs "run in background" this will make matching "started" "finished" logs easier. There also won't be any more "kill cmd" "intr cmd" logs here since these commands don't exist anymore.
Since "all Jobs are in background" it means that in reality all of them generate a "started" and "finished" log, however, if these are in a direct sequence after each other they get shortened to just the PASS/FAIL log. This will also be true for background commands if there were no results to report between their start and finish.