Modernising Beaker's web UI
by Dan Callaghan
I've just uploaded an experimental patch series to Gerrit under the
topic "webui-modernisation":
http://gerrit.beaker-project.org/#/q/topic:webui-modernisation,n,z
I have been working on it on-and-off in my spare time for a while now.
I see this as the most viable way forward for Beaker's web UI, to start
replacing some of the most problematic parts of our TG1 stack without
a wholesale rewrite.
The most interesting patches are:
http://gerrit.beaker-project.org/2086
which lets us escape from TurboGears Visit/Identity in favour of our own
simpler and more flexible implementation, and:
http://gerrit.beaker-project.org/2087
which lets us eventually replace CherryPy's object mapping and
TurboGears' @expose decorator with Flask.
The third big TurboGears pain point for us is TG widgets. Given we are
already at the point where JavaScript is required for some parts of the
web UI to work, we might as well go whole hog and embrace some of the
modern JS widget libraries or even "single-page application" frameworks.
But I am still new to this modern JS world and this patch using Backbone
is just an experiment with one possible way of doing things:
http://gerrit.beaker-project.org/2095
I wasn't particularly happy with how messy and verbose it came out. So
I'm not sure this is the final answer.
What's still missing:
* Patches to update our HTML for Bootstrap on all other pages (only the
group page is done so far). Unfortunately this really has to be done
wholesale in one go, otherwise you end up with some pages with the old
Fedora-ish style and others in the new Bootstrap style. The good news
is that there's not *too* much work involved in this, assuming that
you don't fall into the trap of trying to clean up or improve the
existing layouts or UI too much (which is a trap I had to keep myself
from continually falling into).
Some unresolved questions:
* JS libraries: Backbone seems like a good fit, but it seems very
barebones and so we would need to (re-)invent a lot of things
ourselves. There are some promising-looking extension libraries for
it, such as Epoxy.js, which might be useful. Alternatives are Ember.js
and AngularJS, but I have not experimented with those yet. In any
case, it makes sense to settle on a single preferred approach and
start working towards using that consistently and eliminating older
pieces of JS (including MochiKit).
* Building assets: right now the compiled versions of the CSS and JS are
committed to git, to avoid depending on LESS and Uglifyjs during the
build. But that will very quickly get annoying to maintain I think.
Can we reasonably pull in those tools (and the entire NodeJS
dependency chain) during the build?
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 9 months
Cleaning up hostRequires and system searching
by Nick Coghlan
Bill raised the point that we still have the legacy key-value system
hanging around, despite having replaced most of that subsystem with
dedicated entries in the database. This gives opportunities for stale
data to accumulate as we change the way the inventory system works.
However, simply dropping key-value support isn't acceptable, since it
may break usage of existing Beaker instances.
One possible answer to that (and my currently preferred approach) is to
define a mapping from particular "key" settings to their more data model
friendly equivalents:
For example, when we see: <key_value key="Vendor" op="!=" value="SGI"/>
We would automatically replace it with: <system><vendor op="!="
value="SGI"/></system>
If we find existing "key" settings that we can't map, well, that's a
sign that there's a missing piece in the system data model :)
Creating this mapping would also make it clear how to ensure the legacy
data is migrating correctly in an existing installation: scan the
database for systems that haven't been reinventoried since the inventory
task was last updated and the key/value query and the remapped query
give different answers.
Once we have those in place to provide a clean transition plan, then we
can move forward with simply dropping the key/value store from the
system data model.
As a related change (to avoid needing to do the key/value remapping in
two places), we should look into redesigning how the query system works,
creating an internal representation used by both the XML and web
interfaces to invoke a common query engine.
Anyone interested in writing the design proposal(s) for this? (Amit, it
may be somewhat related to the work you're already doing on the
inventory subsystem)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months
No more bundled JS in Fedora packages
by Dan Callaghan
Fedora has (finally) announced packaging guidelines for "static assets":
that is, JS and other stuff served up to browsers. This is to avoid
every web app bundling its own copy of jQuery and the like.
https://fedoraproject.org/wiki/Changes/Web_Assets
This doesn't affect Beaker immediately but it's something for us to work
towards adhering to.
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 9 months
Tool for Inventory gathering
by Amit Saha
Hi all,
I spent some time on Friday to see how the lshw based inventory task is doing by running on one of our test systems - both in terms of what it gets
and how does it compare to the existing smolt based task.
I didn't go far with the comparison. Why? Because, lshw's parsing script broke due to the XML being different for the particular CPU I was running on
and what the parsing script was expecting. I could fix it (like I have done earlier) to accommodate for this formatting change..and do the same for
similar errors when I come across one.
But, here is what I am thinking:
Why don't we just get rid of the smolt and lshw tasks and instead query directly from the /proc and /sys (procfs and sysfs)?
I have done some exploration in this area [1] and [2]. In [1], I am reading directly off /proc and using Python's in built text
processing methods to get the information I want. Today I explored using pyudev (Python interface to libudev for accessing the device
attributes from /sys) which you can see in [2].
>From what I can see these are the information we usually look to get from our inventory task and how
we could get them using one/both of the approaches (/proc and/or /sys):
- CPU details (easily done by reading /proc/cpuinfo)
- Memory details (easily done by reading /proc/memenfo)
- Disk details (we could continue using libparted - we will need to deal with the shared library version choices that we currently have).
The total size is easily gettable from /sys/block, but I don't see a way to get the geometry details.
- Devices (Gettable from /sys/devices)
What do you all think?
Here are the links:
[1] http://amitsaha.github.io/site/notes/articles/python_linux/article.html
[2] http://nbviewer.ipython.org/5946601
Best, Amit.
--
Amit Saha <http://echorand.me>
Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 9 months
Exposing the the host's hostname to the guest
by Raymond Mancy
We're currently considering a feature[1] request that would expose the hypervisor's
host name to it's guest recipe via environment variables.
We currently expose many guest recipe details via the to_xml() (which is what produces
the output of the /recipes/<recipe_id> harness API call). To include the host's FQDN
in this details would seem to be consistent with existing behaviour and would take
minimal code changes (as opposed to creating a new harness API call, controller, and model
methods).
1 https://bugzilla.redhat.com/show_bug.cgi?id=887760
10 years, 9 months
Re: [Beaker-devel] fedmsg-download
by Amit Saha
Hi Bill,
(Moving to the public devel list)
----- Original Message -----
> From: "Bill Peck" <bpeck(a)redhat.com>
> To: beaker-user-list(a)redhat.com
> Sent: Thursday, November 1, 2012 2:54:31 AM
> Subject: fedmsg-download
>
>
> I created a new package called fedmsg-download which will make it easy
> to mirror the latest fedora development branches. When a message is
> received over the message bus stating that the tree has finished
> rsyncing to dl.fedoraproject.org it will rsync the .composeinfo file.
> From that file It will read the name and use it as the directory to
> rsync to. After the initial rsync it will use the previous tree so that
> only diffs are rsynced. The default config will delete the previous
> tree keeping only one active copy, you can disable this but then you are
> responsible for cleaning up old releases. A symlink pointing to the
> latest will be put in the same directory. It will end up looking like this:
>
> drwxrwxr-x. 4 fedmsg fedmsg 4096 Oct 31 08:56 Fedora-18-20121031
> lrwxrwxrwx. 1 fedmsg fedmsg 18 Oct 31 09:20 latest-18 ->
> Fedora-18-20121031
>
<snip>
> *Git repo Home*
> https://github.com/p3ck/fedmsg-download
Just wanted to let you know that I tried this yesterday (cloning off your repo
and installing from the created RPMs) and I see that it has created a rawhide tree
locally. I will next give it a shot to run 'beaker-import' after this in my
local dev. env.
Thanks,
Amit.
--
Amit Saha <http://echorand.me>
Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 9 months
Implicit retries considered useful?
by Nick Coghlan
To help narrow down a problem that sometimes occurs with lab controller
<-> main server communications, Beaker 0.13.2 is currently scheduled to
completely remove the implicit retries for the affected XML-RPC calls
(see https://bugzilla.redhat.com/show_bug.cgi?id=974352).
However, it occurs to me that there's a potentially big downside to
doing this: currently, we can usually restart the main web service (e.g.
for a maintenance update that doesn't include any database changes)
without significant ill effects, because it will just trip the automatic
retries in the lab controllers for any in progress calls and those calls
will generally succeed on the second attempt.
By turning the implicit retries off *completely* for the lab controller
daemons (rather then just reducing the number of attempts and logging
each failed attempt properly), any calls back to the server when the
server shuts down will actually *fail* completely.
So while our initial assessment was that the implicit retries were doing
more harm than good, I think this may be a case where we actually need
them. If I'm right, then 0.13.2 isn't shippable in its current form (we
need to either revert that change to the lab controllers, or update it
to still retry at least once, and log the first failure properly).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months
Tweaking the merge criteria a bit
by Nick Coghlan
Our test suite has an existing unit test/integration test split that I
think we can use to make for saner merge criteria for Gerrit.
Specifically, I think we can set the *must pass* level at:
* all unit tests
* the integration tests for components being altered
Whether to require the full integration test suite before granting +1
Verified is then a judgement call on the part of the developer
responsible for the patch. If the change *only* altered integration
tests, without adding new unit tests, then that's a sign a full test run
may be needed (or else more unit tests).
Something we should also consider is adding dependencies on unittest2
(for RHEL) and mock (for both RHEL and Fedora). mock greatly simplifies
the task of writing good unit tests (by letting you stub out things you
don't have available, like the database), while unittest2 improves the
available set of assertions so you can get improved error reporting from
the test suite (you get some of that automatically just by running on
2.7, but other aspects require changes that will break on Python 2.6).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
10 years, 9 months