Beaker-devel June 2017

beaker-devel@lists.fedorahosted.org

4 participants
7 discussions

Future directions for the Beaker test harness
by Nick Coghlan 12 Mar '18

12 Mar '18

A few weeks back, Amit put together a proof of concept for running the test harness in a container, rather than directly on the host (http://gerrit.beaker-project.org/#/c/3199) That proof of concept relies on restraint, the new reference harness, that is intended to eventually replace beah (https://beaker-project.org/dev/proposals/reference-harness.html) At the same time, I don't think restraint is currently getting the level of review and testing that it needs to mature into a plausible replacement for beah as the default harness. I think Amit's proposed patch provides a possible way forward: 1. Accept the initial approach where restraint is the *only* supported harness when running inside a container. Specifying both "contained_harness" and "harness" as ks_meta variables should be an error at this point (side note: 'harness' should also be documented along with all the other ks_meta variables, with a link to https://beaker-project.org/docs/alternative-harnesses/) 2. Recommend publishing both beah *and* restraint in the harness repos for Beaker installations. This will not only make restraint available for container based testing, but also make it readily available via "harness=restraint" for normal testing, without needing to add a custom repo definition. 3. Once we have container based testing working reliably with restraint, drop the restriction against using alternative harnesses in containers. The priority at the moment though is to get something working that can run on an Atomic Host, and still provide a relatively normal execution environment for the executed tasks. Supporting alternative harnesses *inside* containers is a nice-to-have that can wait until later - by flatly disallowing it, we ensure we don't have to spend any time working on container related issues that don't impact restraint. For the initial iteration, we can also ignore the question of choosing the base image used to run the harness, as well as being able to start and stop other containers on the host. I've filed an RFE for 0.18 on that basis: https://bugzilla.redhat.com/show_bug.cgi?id=1131388 As part of this, we may also want to move restraint from Bill's personal account on GitHub to the main Beaker project account, but I don't think that's particularly urgent at this point. Regards, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect

5 10

Adding system firmware to system details
by Don Zickus 26 Jul '17

26 Jul '17

Hi Dan, Shawn is working with Jon and myself to add system firmware/bios info to the beaker database to be displayed on the system details page. The idea was to make it easy to find how old a bios is on our lab machines and see which ones need an update. Shawn made all the front end changes in the inventory job, but the backend work is becoming a challenge. It seems like we are going to have to add a table entry to the database in server/model/inventory.py? Which would probably cause a database migration event. For now, we are using numa_nodes as our template to copy the output of lshw (from the inventory script) into the database and onto the webpage. Are we going in the right direction? Is this more work than we thought? Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a generic key/value table to dynamically support new fields without having to migrate the whole database all the time? And then to prevent random junk from being added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries. This is just us trying to understand the architecture a little more and trying to see where we can add some value to make it easier to maintain our tests. Cheers, Don

6 16

webdriver cache?
by Shawn Doherty 28 Jun '17

28 Jun '17

Hello all. I have a question regarding WebDriver testing. I was working on creating an integration test. When I would change code in the widgets.py file, my integration test was not responsive to my changes, and seemed to be holding on to information from previous versions. Is there a cache in the webdriver or something that needs to be updated when we make a change? Thanks, Shawn -- Shawn Doherty Software Engineer, Kernel-HW Red Hat <https://www.redhat.com> 314 Littleton Rd Westford, MA 01886 sdoherty(a)redhat.com T: 19785891080 INTERNAL:-8131080 IM: sdoherty <https://red.ht/sig>

2 1

beaker server: backend state is changed but front end UI doesn't get a push update?
by Jonathan Toppins 26 Jun '17

26 Jun '17

Anyone else think it is odd that the backend server state for a machine entry can change, example schedule setting, and the front end UI doesn't either get a push update or a 409 conflict error? Is there a bug filed for this? -Jon

2 1

Re: Fwd: Change in beaker[develop]: Add ExcludeAll button to Excluded Families
by Don Zickus 23 Jun '17

23 Jun '17

Hi Dan, How do we move Gerrit changes forward? Cheers, Don On Thu, Jun 22, 2017 at 02:16:44PM -0400, Shawn Doherty wrote: > ---------- Forwarded message ---------- > From: Jonathan Toppins (Code Review) <gerrit(a)beaker-project.org> > Date: Thu, Jun 15, 2017 at 2:27 PM > Subject: Change in beaker[develop]: Add ExcludeAll button to Excluded > Families > To: Dan Callaghan <dcallagh(a)redhat.com> > Cc: sdoherty(a)redhat.com > > > Jonathan Toppins would like Dan Callaghan to *review* this change. > > View Change <https://gerrit.beaker-project.org/5712> > > Add ExcludeAll button to Excluded Families > > List of excluded families is large in some cases. Added a button to the > exclude page that will toggle checkbox of Major releases. > > Change-Id: I3e86fbf4421023eb591d838ed5259825f5feb03a > Signed-off-by: Shawn Doherty <sdoherty(a)redhat.com> > --- > M Server/bkr/server/static/javascript/util.js > M Server/bkr/server/widgets.py > 2 files changed, 9 insertions(+), 0 deletions(-) > > git pull ssh://gerrit.beaker-project.org:29418/beaker refs/changes/12/5712/1 > > To view, visit change 5712 <https://gerrit.beaker-project.org/5712>. To > unsubscribe, visit settings <https://gerrit.beaker-project.org/settings>. > Gerrit-Project: beaker > Gerrit-Branch: develop > Gerrit-MessageType: newchange > Gerrit-Change-Id: I3e86fbf4421023eb591d838ed5259825f5feb03a > Gerrit-Change-Number: 5712 > Gerrit-PatchSet: 1 > Gerrit-Owner: sdoherty(a)redhat.com > Gerrit-Reviewer: Dan Callaghan <dcallagh(a)redhat.com> > > > > -- > > Shawn Doherty > > Software Engineer, Kernel-HW > > Red Hat > > <https://www.redhat.com> > > 314 Littleton Rd > > Westford, MA 01886 > > sdoherty(a)redhat.com T: 19785891080 INTERNAL:-8131080 IM: sdoherty > <https://red.ht/sig>

2 2

converting CSV_import_export to flask and Backbone.js
by Don Zickus 06 Jun '17

06 Jun '17

Hi Dan, I thought instead of describing what I am doing, I would just show you the patch so you can point at the problem quicker. As I mentioned before, I decided to just randomly grab a page like CSV_import_export and try to convert it to flask. I started by mimicing reserve_workflow thinking it had a similar display and POST frontend. However, I found out, I can get the page to render the title but that is about it. Even on the reserve_workflow page, I don't get the pretty Distro and other forms (without my changes applied too). So I can't tell if my patch below is technically wrong (I am sure it is missing lots of pieces) or my devel env is not quite configured correctly (because reserve_workflow doesn't quite work either). Thoughts? Help? Cheers, Don diff --git a/Server/assets/csv-export.js b/Server/assets/csv-export.js new file mode 100644 index 0000000..aed9db9 --- /dev/null +++ b/Server/assets/csv-export.js @@ -0,0 +1,35 @@ + +// This program is free software; you can redistribute it and/or modify +// it under the terms of the GNU General Public License as published by +// the Free Software Foundation; either version 2 of the License, or +// (at your option) any later version. + +;(function () { + +var CSVExportSelection = Backbone.Model.extend({ +}); + +window.CSVExport = Backbone.View.extend({ + template: JST['csv-export'], + events: { + 'submit form': 'submit', + }, + initialize: function (options) { + this.csv_types = options.options['csv_types'] + this.render(); + }, + render: function () { + this.$el.html(this.template(this)); + }, + submit: function (evt) { + evt.preventDefault(); + var xhr = $.ajax({ + url: 'action_export', + type: 'POST', + data: this.selection.attributes, + traditional: true, + }); + }, +}); + +})(); diff --git a/Server/assets/jst/csv-export.html b/Server/assets/jst/csv-export.html new file mode 100644 index 0000000..eec875e --- /dev/null +++ b/Server/assets/jst/csv-export.html @@ -0,0 +1,19 @@ +<form class="form-horizontal"> + <fieldset> + <div class="control-group"> + <label class="control-label">CSV TYPE</label> + <div class="controls"> + <% _.each(csv_types, function (csv) { %> + <label class="radio"> + <input type="radio" name="csv_type" value="<%- csv.toLowerCase() %>" + <% if (csv == 'System' ) { %>checked<% } %> + /> + <%- csv %> + </label> + </div> + </div> + <div class="form-actions"> + <button class="btn btn-primary" type="submit">Export CSV</button> + </div> + </fieldset> +</form> diff --git a/Server/bkr/server/CSV_import_export.py b/Server/bkr/server/CSV_import_export.py index 41fcb26..94bb00f 100644 --- a/Server/bkr/server/CSV_import_export.py +++ b/Server/bkr/server/CSV_import_export.py @@ -10,7 +10,11 @@ from bkr.server import identity from bkr.server.xmlrpccontroller import RPCRoot from tempfile import NamedTemporaryFile -from cherrypy.lib.cptools import serve_file +#from cherrypy.lib.cptools import serve_file +from flask import send_file, request +from bkr.server.flask_util import render_tg_template + +from bkr.server.app import app from bkr.server.model import (System, SystemType, Activity, SystemActivity, User, Group, LabController, LabInfo, OSMajor, OSVersion, @@ -70,6 +74,64 @@ def line_num(self): def fieldnames(self): return self.reader.fieldnames +# For XMLRPC methods in this class. +exposed = False + +export_help_text = XML(u'<span>Refer to the <a href="http://beaker-project.org/docs/' + 'admin-guide/interface.html#export" target="_blank">' + 'documentation</a> to learn more about the exported data.</span>').expand() +import_help_text = XML(u'<span>Refer to the <a href="http://beaker-project.org/docs/' + 'admin-guide/interface.html#import" target="_blank">' + 'documentation</a> for details about the supported CSV format.</span>').expand() + +upload = widgets.FileField(name='csv_file', label='Import CSV', \ + help_text = import_help_text) +download = RadioButtonList(name='csv_type', label='CSV Type', + options=[('system', 'Systems'), + ('system_id', 'Systems (for modification)'), + ('labinfo', 'System LabInfo'), + ('power', 'System Power'), + ('exclude', 'System Excluded Families'), + ('install', 'System Install Options'), + ('keyvalue', 'System Key/Values'), + ('system_pool', 'System Pools'), + ('user_group', 'User Groups')], + default='system', + help_text = export_help_text) +exportform = HorizontalForm( + 'export', + action = 'export data', + submit_text = _(u'Export CSV'), +) + +(a)app.route('/csv', methods=['GET']) +(a)identity.require(identity.not_anonymous()) +def index(): + options = {} + options['csv_types'] = ('system', 'system_id', 'labinfo', 'power', + 'exclude', 'install', 'keyvalue', 'system_pool', + 'user_group') + #return render_template('assets.jst.csv_export.html', options=options) + return render_tg_template('bkr.server.templates.csv_export', { + 'title' : u'CSV Export Don', + 'options' : options + }) + +(a)app.route('/csv/action_export', methods=['POST']) +(a)identity.require(identity.not_anonymous()) +def action_export(): + csv_type = request.form.get('csv_type') + logger.debug("DON: request: %s" % csv_type()) + + file = NamedTemporaryFile() + log = self.to_csv(file, csv_type) + file.seek(0) + + logger.debug('DON CSV export with send_file type: %s', csv_type) + return send_file(file.name, mimetype="text/csv", + as_attachment=True, + attachment_filename="%s.csv" % csv_type) + class CSV(RPCRoot): # For XMLRPC methods in this class. exposed = False @@ -139,6 +201,7 @@ def action_export(self, csv_type, *args, **kw): log = self.to_csv(file, csv_type) file.seek(0) + logger.debug('DON CSV export with send_file type: %s', csv_type) return serve_file(file.name, contentType="text/csv", disposition="attachment", name="%s.csv" % csv_type) diff --git a/Server/bkr/server/assets.py b/Server/bkr/server/assets.py index 0c5a117..71747f4 100644 --- a/Server/bkr/server/assets.py +++ b/Server/bkr/server/assets.py @@ -90,6 +90,7 @@ def _create_env(source_dir, output_dir, **kwargs): 'pools.js', 'query-builder.js', 'reserve-workflow.js', + 'csv-export.js', 'installation-model.js', 'task-library-model.js', 'scheduler-model.js', diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..936cdc2 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -146,7 +146,7 @@ class Root(RPCRoot): users = Users() arches = Arches() auth = Auth() - csv = CSV() + #csv = CSV() jobs = Jobs() recipesets = RecipeSets() recipes = Recipes() diff --git a/Server/bkr/server/templates/csv_export.kid b/Server/bkr/server/templates/csv_export.kid new file mode 100644 index 0000000..848cfff --- /dev/null +++ b/Server/bkr/server/templates/csv_export.kid @@ -0,0 +1,22 @@ +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml" xmlns:py="http://purl.org/kid/ns#" + py:extends="'master.kid'"> +<head> +<title>$title</title> +</head> +<body> +<div class="page-header"> + <h1>$title</h1> +</div> +<div class="csv_export"></div> +<script type="text/javascript"> +var csvexport = new CSVExport( +$(function () { + new CSVExport({ + el: '.csv_export', + options: ${tg.to_json(options)}, + }); +}); +</script> +</body> +</html>

2 4

beaker server migration
by Don Zickus 06 Jun '17

06 Jun '17

Hi Dan, A recurring theme we seem to encounter when working through various features we want to implement in beaker, is the challenges the front-end poses to add code. It isn't just a straight-forward add code (once you understand the technology), but there is an added bonus of understanding the strange quirks the legacy code is laying upon the current code. :-) But you already know that. :-) I wanted to reach out and understand what the initial thoughts were to migrate away from the legacy stuff. Obviously your team made an initial effort and stopped. And there is still plenty of work to be done. Was a process to move forward ever documented? Granted things are a little bit better now that we understand how to run the server in developer mode better and have the corresponding lengthy dependencies installed. But I still struggle just to convert one page to strictly flask and still involves using kid files to pass back to cherrypy??? With all the changing technology out there, it would be nice to have beaker be updated easily to match our needs. So I was trying to wrap my head around an easy process to migrate things to a better place. Cheers, Don

2 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Beaker-devel June 2017