As a heads up, all QA services will be affected by the infra outages
scheduled over the next 3 days: 2014-12-15, 2014-12-16 and 2014-12-17.
Taskotron and Blockerbugs will be affected by all three outages but
Beaker will be less affected due to how it's deployed as a
We'll keep Taskotron up and running as best we can during the downtime
and will be keeping track of missed jobs that need to be run after the
downtime is complete. Blockerbugs will be synced after the outage is
complete and outside of not being available during some of the outage
times, should not be affected.
# Fedora QA Devel Meeting
# Date: 2014-12-15
# Time: 15:00 UTC
# Location: #fedora-meeting-1 on irc.freenode.net
Now that F21 is out the door, we can start working a bit more on
tooling. I'd like to spend most of the time discussing the various
planning/proposal threads that I've sent out to qa-devel@ in the last
couple weeks. Hopefully we'll be able to make good progress on the
higher priority bits before F22 starts ramping up but I'm not sure how
much that'll happen before January (depending on vacations etc.).
If you have agenda items to add, feel free to respond to this thread
or bring them up during open floor.
- Disposable Clients
- Fedmsg Integration
- Disposable Clients
Upcoming Potential Stuff
- Who's going to be on vacation when?
While there are a lot of things which folks want to use Taskotron for,
most of those things are predicated on our support for disposable
clients and that's still the next big thing that I'd like to get
working. I'm ignoring the other things that we'd do with disposable
clients for the scope of this thread because there's a lot of work here
Assuming that we go forward with my proposal for no-cloud disposable
clients, there are several things which still need to be done:
- define a VM spec for the recipes
- figure out how to manage images and image metadata so that we can
find and retrieve images on the buildslaves.
* Mike was talking about looking into using Glance from OpenStack for
* Another option would be to integrate it into trigger/ExecDb
- Write any required code for the image management and retrieval system
- deploy the system for image management and retrieval
- re-deploy the buildslaves so that they work better with the new
image-based approach. ie - run them on bare metal so that we're not
slowing stuff down with virt-in-virt
- get the code working
* we can start by specifying local paths to VM images instead of any
tags or metadata
* stealing bits from Mike's testCloud may be a good option
* not sure if we want to use much from Beaker here, I don't like the
idea of using expect scripts unless there are no other options
- Get a basic ExecDB working
* Josef has started on this, still waiting for a demo instance to
start working out the details and kinks
- Write tasks to generate updated images to use for tasks
* This could be manual at first but I'd like to see it happen on a
regularly scheduled basis.
Blockerbugs has been around for a while and while there's been talk
about putting some effort into improving it, that's not gotten a whole
lot farther than just talk.
The list of the smaller things that I'd like to see fixed/improved:
- build for el7 (in progress)
- fix the admin interface (can't change data through it right now)
- get the group fix merged and deployed
- Finish the build-grabbing script that I started writing for releng
to be able to get the list of builds beyond stable that are
available to fix blockers and FEs
There are some larger things that we could tackle if it's a high enough
- Improve the blocker proposal interface to allow changing blocker/FE
proposals instead of just creating new proposals.
- Listen for changes from the bugzilla-fedmsg gateway instead of
syncing every 30 minutes.
- Add a "historical" view to allow folks to see blocker/fe bugs that
have been closed
- Improve the spin request interface so folks are willing to use it
A lot of different projects have come up as of late, many of them in
meetings or other semi-informal situations and I wanted to start
discussing some of them in a way that's easy for others to follow.
I've started three new threads to discuss the three biggest areas that
I think are the most important in the short/medium term. With the folks
who are usually around for qadevel work, I don't think we can get all
of them done before F22 so we're going to have to set some priorities.
There are still more projects/areas that have been proposed/discussed
(graphical testing for gnome, reworking resultsdb, dist-git-ish setup
for tasks, automated integration testing etc.) but those are either
dependent on the three areas I'm proposing for focus (almost all of
them) or are of a lower priority for now (mostly resultsdb).
There will be an opportunity to discuss the different areas at the
qadevel meeting on Monday but I wanted to get the information out there
so folks have a chance to read up, comment and ask questions before
I've built new packages for phabricator and deployed them to staging
with a fresh db snapshot from production.
I'd appreciate it if folks took a bit of time to poke at it to make
sure there aren't problems before I deploy the new version in
production. This has been the most involved upgrade (in terms of
changes to config) since we started using Phabricator and I'd like to
be more certain that I didn't break anything before deploying it to
Biggest Changes for Us (that I've found so far):
- ACLs on Wiki Pages (we can have wiki pages without login now)
- Dashboards (looks cool but I think they're missing some
functionality before they're usable for us)
This all got started with an idea that Mike pitched earlier today and
I'm not sure how much this has diverged from what he was originally
thinking and it's still a rough concept but the more I think about
this, the more I like the idea.
One of the big blockers that we have right now is support for
disposable clients. Most of the exploration work that we've done so far
has involved some sort of cloud system (openstack, open nebula etc.)
but I'm wondering if we would be better served by avoiding cloud
We've (or maybe just me) been assuming that the tasks have be executed
directly on buildslaves like they currently are and any disposable
client would have to be prepared and booted before we start task
What if we turned that on it's head a bit and put VM spawning into the
task itself - spawn a vm local to the buildslave that is then
responsible for the actual work in the task instead of doing all the
work inside the buildslave process? This would allow for image and vm
requirements to be described in the task without weird multiple
parsings and could avoid most if not all of the problems/complexity that
we've started seeing in getting buildbot to work with latent
I've written up a brief article in the phab wiki:
Any thoughts on whether this is worth pursuing as an alternative to
openstack et. al? If there's anything that isn't clear, please ask for
A couple of the conversations I've had this week have brought up an
important question that I don't think we've gotten into specifics on
yet: what functionality will disposable clients need?
Stepping aside from specific cloud/virt systems for the moment, I put
together a list of Requirements, Possible Requirements (stuff we'd need
in order to support certain use-cases) and Nice to Haves for disposable
clients and wanted to see if this makes sense to other folks.
I've tried to abstract this to just what the client VMs would have to
do, so let's stay away from image management/generation methods and
other related topics for this thread.
Thoughts on this list? Did I miss anything or is there something on
this list that shouldn't be there?
* Use anything from a relatively arbitrary list of images
* Destroy-able after every task
* Fast boot time (say, 30 sec at most)
* The same images are available in identical form to all clients
* Ability to log remotely (syslog, taskotron.log, maybe others?)
* Some capability for graphical connection (required for T377 and
* Allow users to ssh into VMs for triage purposes
- Find a way to use FAS ssh key for auth? (possible security issues
- Something other than some global username/password
* capable of building images?
* capable of configuring itself or another host with something like
* capable of working with multi-host tests (spawning one or more
machines during task)
I used the blockerbugs application at https://qa.fedoraproject.org/blockerbugs/propose_bug but found that this is restricted to the primary arch releases.
Could it be possible to have it improved to support secondary arch releases (ppc64, s390, ...) ?