Future directions for the Beaker test harness
by Nick Coghlan
A few weeks back, Amit put together a proof of concept for running the
test harness in a container, rather than directly on the host
(http://gerrit.beaker-project.org/#/c/3199).
That proof of concept relies on restraint, the new reference harness,
that is intended to eventually replace beah
(https://beaker-project.org/dev/proposals/reference-harness.html)
At the same time, I don't think restraint is currently getting the level
of review and testing that it needs to mature into a plausible
replacement for beah as the default harness.
I think Amit's proposed patch provides a possible way forward:
1. Accept the initial approach where restraint is the *only* supported
harness when running inside a container. Specifying both
"contained_harness" and "harness" as ks_meta variables should be an
error at this point (side note: 'harness' should also be documented
along with all the other ks_meta variables, with a link to
https://beaker-project.org/docs/alternative-harnesses/).
2. Recommend publishing both beah *and* restraint in the harness repos
for Beaker installations. This will not only make restraint available
for container based testing, but also make it readily available via
"harness=restraint" for normal testing, without needing to add a custom
repo definition.
3. Once we have container based testing working reliably with restraint,
drop the restriction against using alternative harnesses in containers.
The priority at the moment though is to get something working that can
run on an Atomic Host, and still provide a relatively normal execution
environment for the executed tasks. Supporting alternative harnesses
*inside* containers is a nice-to-have that can wait until later - by
flatly disallowing it, we ensure we don't have to spend any time working
on container related issues that don't impact restraint. For the initial
iteration, we can also ignore the question of choosing the base image
used to run the harness, as well as being able to start and stop other
containers on the host.
I've filed an RFE for 0.18 on that basis:
https://bugzilla.redhat.com/show_bug.cgi?id=1131388
As part of this, we may also want to move restraint from Bill's personal
account on GitHub to the main Beaker project account, but I don't think
that's particularly urgent at this point.
Regards,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
5 years, 8 months
Adding system firmware to system details
by Don Zickus
Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to the
beaker database to be displayed on the system details page.
The idea was to make it easy to find how old a bios is on our lab machines
and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the backend
work is becoming a challenge.
It seems like we are going to have to add a table entry to the database in
server/model/inventory.py? Which would probably cause a database migration
event.
For now, we are using numa_nodes as our template to copy the output of lshw
(from the inventory script) into the database and onto the webpage.
Are we going in the right direction? Is this more work than we thought?
Then as a follow-on, we are assuming the database has to be updated to add
this entry, would it make sense to convert the whole database to a generic
key/value table to dynamically support new fields without having to migrate
the whole database all the time? And then to prevent random junk from being
added by the inventory script (or other script), have a whitelist filter
that only allows certain keys to be added/updated. Maintaining the
whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and
trying to see where we can add some value to make it easier to maintain our
tests.
Cheers,
Don
6 years, 4 months
converting CSV_import_export to flask and Backbone.js
by Don Zickus
Hi Dan,
I thought instead of describing what I am doing, I would just show you the
patch so you can point at the problem quicker.
As I mentioned before, I decided to just randomly grab a page like
CSV_import_export and try to convert it to flask. I started by mimicing
reserve_workflow thinking it had a similar display and POST frontend.
However, I found out, I can get the page to render the title but that is
about it.
Even on the reserve_workflow page, I don't get the pretty Distro and other
forms (without my changes applied too).
So I can't tell if my patch below is technically wrong (I am sure it is
missing lots of pieces) or my devel env is not quite configured correctly
(because reserve_workflow doesn't quite work either).
Thoughts? Help?
Cheers,
Don
diff --git a/Server/assets/csv-export.js b/Server/assets/csv-export.js
new file mode 100644
index 0000000..aed9db9
--- /dev/null
+++ b/Server/assets/csv-export.js
@@ -0,0 +1,35 @@
+
+// This program is free software; you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation; either version 2 of the License, or
+// (at your option) any later version.
+
+;(function () {
+
+var CSVExportSelection = Backbone.Model.extend({
+});
+
+window.CSVExport = Backbone.View.extend({
+ template: JST['csv-export'],
+ events: {
+ 'submit form': 'submit',
+ },
+ initialize: function (options) {
+ this.csv_types = options.options['csv_types']
+ this.render();
+ },
+ render: function () {
+ this.$el.html(this.template(this));
+ },
+ submit: function (evt) {
+ evt.preventDefault();
+ var xhr = $.ajax({
+ url: 'action_export',
+ type: 'POST',
+ data: this.selection.attributes,
+ traditional: true,
+ });
+ },
+});
+
+})();
diff --git a/Server/assets/jst/csv-export.html b/Server/assets/jst/csv-export.html
new file mode 100644
index 0000000..eec875e
--- /dev/null
+++ b/Server/assets/jst/csv-export.html
@@ -0,0 +1,19 @@
+<form class="form-horizontal">
+ <fieldset>
+ <div class="control-group">
+ <label class="control-label">CSV TYPE</label>
+ <div class="controls">
+ <% _.each(csv_types, function (csv) { %>
+ <label class="radio">
+ <input type="radio" name="csv_type" value="<%- csv.toLowerCase() %>"
+ <% if (csv == 'System' ) { %>checked<% } %>
+ />
+ <%- csv %>
+ </label>
+ </div>
+ </div>
+ <div class="form-actions">
+ <button class="btn btn-primary" type="submit">Export CSV</button>
+ </div>
+ </fieldset>
+</form>
diff --git a/Server/bkr/server/CSV_import_export.py b/Server/bkr/server/CSV_import_export.py
index 41fcb26..94bb00f 100644
--- a/Server/bkr/server/CSV_import_export.py
+++ b/Server/bkr/server/CSV_import_export.py
@@ -10,7 +10,11 @@
from bkr.server import identity
from bkr.server.xmlrpccontroller import RPCRoot
from tempfile import NamedTemporaryFile
-from cherrypy.lib.cptools import serve_file
+#from cherrypy.lib.cptools import serve_file
+from flask import send_file, request
+from bkr.server.flask_util import render_tg_template
+
+from bkr.server.app import app
from bkr.server.model import (System, SystemType, Activity, SystemActivity,
User, Group, LabController, LabInfo,
OSMajor, OSVersion,
@@ -70,6 +74,64 @@ def line_num(self):
def fieldnames(self):
return self.reader.fieldnames
+# For XMLRPC methods in this class.
+exposed = False
+
+export_help_text = XML(u'<span>Refer to the <a href="http://beaker-project.org/docs/'
+ 'admin-guide/interface.html#export" target="_blank">'
+ 'documentation</a> to learn more about the exported data.</span>').expand()
+import_help_text = XML(u'<span>Refer to the <a href="http://beaker-project.org/docs/'
+ 'admin-guide/interface.html#import" target="_blank">'
+ 'documentation</a> for details about the supported CSV format.</span>').expand()
+
+upload = widgets.FileField(name='csv_file', label='Import CSV', \
+ help_text = import_help_text)
+download = RadioButtonList(name='csv_type', label='CSV Type',
+ options=[('system', 'Systems'),
+ ('system_id', 'Systems (for modification)'),
+ ('labinfo', 'System LabInfo'),
+ ('power', 'System Power'),
+ ('exclude', 'System Excluded Families'),
+ ('install', 'System Install Options'),
+ ('keyvalue', 'System Key/Values'),
+ ('system_pool', 'System Pools'),
+ ('user_group', 'User Groups')],
+ default='system',
+ help_text = export_help_text)
+exportform = HorizontalForm(
+ 'export',
+ action = 'export data',
+ submit_text = _(u'Export CSV'),
+)
+
+(a)app.route('/csv', methods=['GET'])
+(a)identity.require(identity.not_anonymous())
+def index():
+ options = {}
+ options['csv_types'] = ('system', 'system_id', 'labinfo', 'power',
+ 'exclude', 'install', 'keyvalue', 'system_pool',
+ 'user_group')
+ #return render_template('assets.jst.csv_export.html', options=options)
+ return render_tg_template('bkr.server.templates.csv_export', {
+ 'title' : u'CSV Export Don',
+ 'options' : options
+ })
+
+(a)app.route('/csv/action_export', methods=['POST'])
+(a)identity.require(identity.not_anonymous())
+def action_export():
+ csv_type = request.form.get('csv_type')
+ logger.debug("DON: request: %s" % csv_type())
+
+ file = NamedTemporaryFile()
+ log = self.to_csv(file, csv_type)
+ file.seek(0)
+
+ logger.debug('DON CSV export with send_file type: %s', csv_type)
+ return send_file(file.name, mimetype="text/csv",
+ as_attachment=True,
+ attachment_filename="%s.csv" % csv_type)
+
class CSV(RPCRoot):
# For XMLRPC methods in this class.
exposed = False
@@ -139,6 +201,7 @@ def action_export(self, csv_type, *args, **kw):
log = self.to_csv(file, csv_type)
file.seek(0)
+ logger.debug('DON CSV export with send_file type: %s', csv_type)
return serve_file(file.name, contentType="text/csv",
disposition="attachment",
name="%s.csv" % csv_type)
diff --git a/Server/bkr/server/assets.py b/Server/bkr/server/assets.py
index 0c5a117..71747f4 100644
--- a/Server/bkr/server/assets.py
+++ b/Server/bkr/server/assets.py
@@ -90,6 +90,7 @@ def _create_env(source_dir, output_dir, **kwargs):
'pools.js',
'query-builder.js',
'reserve-workflow.js',
+ 'csv-export.js',
'installation-model.js',
'task-library-model.js',
'scheduler-model.js',
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py
index cece8db..936cdc2 100644
--- a/Server/bkr/server/controllers.py
+++ b/Server/bkr/server/controllers.py
@@ -146,7 +146,7 @@ class Root(RPCRoot):
users = Users()
arches = Arches()
auth = Auth()
- csv = CSV()
+ #csv = CSV()
jobs = Jobs()
recipesets = RecipeSets()
recipes = Recipes()
diff --git a/Server/bkr/server/templates/csv_export.kid b/Server/bkr/server/templates/csv_export.kid
new file mode 100644
index 0000000..848cfff
--- /dev/null
+++ b/Server/bkr/server/templates/csv_export.kid
@@ -0,0 +1,22 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml" xmlns:py="http://purl.org/kid/ns#"
+ py:extends="'master.kid'">
+<head>
+<title>$title</title>
+</head>
+<body>
+<div class="page-header">
+ <h1>$title</h1>
+</div>
+<div class="csv_export"></div>
+<script type="text/javascript">
+var csvexport = new CSVExport(
+$(function () {
+ new CSVExport({
+ el: '.csv_export',
+ options: ${tg.to_json(options)},
+ });
+});
+</script>
+</body>
+</html>
6 years, 6 months
beaker server migration
by Don Zickus
Hi Dan,
A recurring theme we seem to encounter when working through various features
we want to implement in beaker, is the challenges the front-end poses to add
code. It isn't just a straight-forward add code (once you understand the
technology), but there is an added bonus of understanding the strange quirks
the legacy code is laying upon the current code. :-) But you already know
that. :-)
I wanted to reach out and understand what the initial thoughts were to
migrate away from the legacy stuff. Obviously your team made an initial
effort and stopped. And there is still plenty of work to be done. Was a
process to move forward ever documented?
Granted things are a little bit better now that we understand how to run the
server in developer mode better and have the corresponding lengthy
dependencies installed. But I still struggle just to convert one page to
strictly flask and still involves using kid files to pass back to
cherrypy???
With all the changing technology out there, it would be nice to have beaker
be updated easily to match our needs. So I was trying to wrap my head
around an easy process to migrate things to a better place.
Cheers,
Don
6 years, 6 months
RHEL-7.4-EA2 ARM64 import
by Ed Gasiorowski
Just tried to import RHEL-7.4-EA2 for arm64 and received the following
error.. any ideas?
[root@beaker ~]# beaker-repo-update
2017-05-26 10:15:42,332 bkr.server.tools.repo_update WARNING failure:
repodata/r epomd.xml from
http:--beaker-project.org-yum-harness-RedHatEnterpriseLinuxPegas7
-: [Errno 256] No more mirrors to try.
http://beaker-project.org/yum/harness/RedHatEnterpriseLinuxPegas7/repodat...
d.xml: [Errno 14] HTTPS Error 404 - Not Found
Pegas is the codename for RHEL-7.4.... what file in the distro does beaker
use to generate this http: url... media.repo?
[InstallMedia]
name=Red Hat Enterprise Linux Pegas 7.4
mediaid=1493121908.775686
metadata_expire=-1
gpgcheck=0
cost=500
checked .treeinfo and .diskinfo and both are Redhat Enterprise Linux
7.3... thats a different issue
6 years, 6 months
Getting a copy of a database to play with
by Don Zickus
Hi Dan,
Is there a way we can get a copy of a database that has some real world data
in it to play with some settings? You mentioned that for development it is
recommended. I wasn't sure where one gets a copy.
Cheers,
Don
6 years, 6 months
front end help?
by Shawn Doherty
Hello.
I'm trying to get my feet wet in adding some features to Beaker and running
into some hurdles. I was hoping to get some pointers please.
It would be helpful to have the ability to exclude all distros for a
system. I have placed 2 buttons on the SystemExclude form but it is VERY
crudely done. Part of the issue is that I'm not sure that I'm linking the
util.js file correctly, When I debug in a browser I can see the util file
is available but none of my appends to it are available(setup
build/install/restart httpd did not correct). I placed my functions inline
in the template but am very limited and have not been able to get arch
variables going correctly.
Any helpful information on how I can do better with widgets in Beaker?
Thanks, Shawn
index 9246918..3a9b65f 100644
--- a/Server/bkr/server/widgets.py
+++ b/Server/bkr/server/widgets.py
@@ -41,7 +41,7 @@ class
AutoCompleteTextField(widgets.AutoCompleteTextField):
template="""
<span xmlns:py="http://purl.org/kid/ns#" class="${field_class}">
- <script type="text/javascript">
+ <script type="text/javascript" src='/static/javascript/util.js'>
AutoCompleteManager${field_id} = new
AutoCompleteManager('${field_id}', '${field_id}', null,
'${search_controller}', '${search_param}', '${result_name}',
${str(only_suggest).lower()},
'${show_spinner and
tg.url('/tg_widgets/turbogears.widgets/spinner.gif') or None}',
@@ -951,9 +951,31 @@ class SystemExclude(Form):
method="${method}" width="100%">
${display_field_for("id")}
${display_field_for("excluded_families")}
- <a py:if="not readonly" class="btn btn-primary"
href="javascript:document.${name}.submit();">Save Exclude Changes</a>
+
+ <!-- separate checkall from source-->
+ <span class="checkAll">
+ <a py:if="not readonly" class="btn btn-secondary btn-sm" role="button"
onclick='javascript:checkMajor()'>Exclude All</a>
+ <!-- Todo toggle capability & proper variable for arch -->
+ <script type="text/javascript">
+ function checkMajor(){
+ $('[name="excluded_families.x86_64"]').prop("checked", true);
+ $('[name="excluded_families.i386"]').prop("checked", true);
+ }
+ </script>
+ <a py:if="not readonly" class="btn btn-secondary btn-sm" role="button"
onclick='javascript:uncheckMajor()'>Exclude None</a>
+ <script type="text/javascript">
+ function uncheckMajor(){
+ $('[name="excluded_families.x86_64"]').prop("checked", false);
+ $('[name="excluded_families.i386"]').prop("checked", false);
+ }
+ </script>
+ </span>
+
+ <a py:if="not readonly" class="btn btn-primary"
href="javascript:document.${name}.submit();">Save Exclude Changes</a>
</form>
"""
+ #not seeing js link in debug
+ javascript = [LocalJSLink('bkr', '/static/javascript/util.js')]
member_widgets = ["id", "excluded_families"]
params = ['options', 'readonly']
params_doc = {}
--- a/Server/bkr/server/static/javascript/util.js
+++ b/Server/bkr/server/static/javascript/util.js
@@ -109,3 +109,19 @@ function system_action_remote_form_request(form,
options, action) {
remoteRequest(form, action, null, query, options);
return true;
}
+
+
+function checkMajorCheckboxes(){
+ var majorDistro =
document.getElementsByName("excluded_families.x86_64");
+ for (var i = 0; i < majorDistro.length; i++){
+ majorDistro[i].checked = true;
+ }
+}
+
+function toggleMajorCheckboxes(){
+ var majorDistro =
document.getElementsByName("excluded_families.x86_64");
+ for (var i = 0; i < majorDistro.length; i++){
+ majorDistro[i].checked = !majorDistro[i].checked;
+ }
+}
+
--
Shawn Doherty
Software Engineer, Kernel-HW
Red Hat
<https://www.redhat.com>
314 Littleton Rd
Westford, MA 01886
sdoherty(a)redhat.com T: 19785891080 INTERNAL:-8131080 IM: sdoherty
<https://red.ht/sig>
6 years, 6 months
Poking at removing cherrypy
by Don Zickus
Hi Dan,
I was trying to poke through some of the server code and I was struggling to
fall the flow of the code (lots of hidden magic with turbogears and
cherrypy). Of course, with the turbogears->flask and cherrypy->flask
wrappers, it makes things more challenging. :-)
So I thought if I start peeling away some of the cherrypy stuff it would
help me understand some of the code better. Using a lot of ignorance, I
pulled out this patch inside beaker-in-a-box:
diff --git a/Server/bkr/server/CSV_import_export.py b/Server/bkr/server/CSV_import_export.py
index 41fcb26..8c98d68 100644
--- a/Server/bkr/server/CSV_import_export.py
+++ b/Server/bkr/server/CSV_import_export.py
@@ -10,7 +10,8 @@
from bkr.server import identity
from bkr.server.xmlrpccontroller import RPCRoot
from tempfile import NamedTemporaryFile
-from cherrypy.lib.cptools import serve_file
+#from cherrypy.lib.cptools import serve_file
+from flask import send_file
from bkr.server.model import (System, SystemType, Activity, SystemActivity,
User, Group, LabController, LabInfo,
OSMajor, OSVersion,
@@ -139,9 +140,9 @@ def action_export(self, csv_type, *args, **kw):
log = self.to_csv(file, csv_type)
file.seek(0)
- return serve_file(file.name, contentType="text/csv",
- disposition="attachment",
- name="%s.csv" % csv_type)
+ return send_file(file.name, mimetype="text/csv",
+ as_attachment=True,
+ attachment_filename="%s.csv" % csv_type)
def _import_row(self, data, log):
if data['csv_type'] in system_types and ('fqdn' in data or 'id' in data):
Restarting the httpd service and trying to export a CSV led to a 500
failure, with the beaker debug logs spitting out 'cherrypy can not iterate
through the response' failures. Not surprised.
I am sure I am untangling spaghetti here, but I guess I was hoping I would
eventually hit one of your cherrypy->flask wrappers.
Is there a good direction to go here or am I in for a long ride of
unhappiness?
Also for some reason when I am in IntegrationTests and run ./run-tests.sh, I
get the following error:
======================================================================
ERROR: test suite for <module 'bkr.inttest' from
'/root/git/beaker/IntegrationTests/src/bkr/inttest/__init__.pyc'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/nose/suite.py", line 209, in run
self.setUp()
File "/usr/lib/python2.6/site-packages/nose/suite.py", line 292, in setUp
self.setupContext(ancestor)
File "/usr/lib/python2.6/site-packages/nose/suite.py", line 315, in setupContext
try_run(context, names)
File "/usr/lib/python2.6/site-packages/nose/util.py", line 471, in try_run
return func()
File "/root/git/beaker/IntegrationTests/src/bkr/inttest/__init__.py", line 351, in setup_package
data_setup.setup_model()
File "/usr/lib/python2.6/site-packages/beaker_server-24.2-py2.6.egg/bkr/server/tests/data_setup.py", line 51, in setup_model
connection = engine.connect()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1779, in connect
return self._connection_cls(self, **kwargs)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 60, in __init__
self.__connection = connection or engine.raw_connection()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1848, in raw_connection
return self.pool.unique_connection()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 280, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 641, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 440, in checkout
rec = pool._do_get()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 961, in _do_get
return self._create_connection()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 285, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 411, in __init__
self.connection = self.__connect()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 537, in __connect
connection = self.__pool._creator()
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 96, in connect
connection_invalidated=invalidated
File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 90, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 377, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (OperationalError) (1044, "Access denied for user 'beaker'@'localhost' to database 'beaker_test'") None None
-------------------- >> begin captured logging << --------------------
Thoughts?
Cheers,
Don
6 years, 6 months