Excerpts from Don Zickus's message of 2017-04-20 15:21 -04:00:
Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to the
beaker database to be displayed on the system details page.
The idea was to make it easy to find how old a bios is on our lab machines
and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the backend
work is becoming a challenge.
It seems like we are going to have to add a table entry to the database in
server/model/inventory.py? Which would probably cause a database migration
event.
For now, we are using numa_nodes as our template to copy the output of lshw
(from the inventory script) into the database and onto the webpage.
Are we going in the right direction?
Yes, it's important to settle on a good db structure for this now
because it is quite costly to put it in place and to change it in
future.
The NUMA stuff is not a great example necessarily because it was always
intended that we would flesh it out to store more info about NUMA
topology, but that never happened. Right now it is just a single table
1-to-1 with "system" with a single column. It could have just been a new
column on the system table.
It also depends exactly what info you want to store about the firmware.
Just an opaque version string? Or also a release date? Assuming the
latter, the options I can see would be:
* Add a new table "firmware" which is 1-to-1 with system, rows are
created for a system when the value is first populated. This is how
Cpu, Numa, Power, and others work right now.
* Add new columns to the "system" table for firmware version and
firmware release date. NULLable with default NULL, and populated by
the inventory script if possible.
* Represent the system firmware as a device in the existing devices
table (in fact it might already be in there from lshw?). You could add
new columns to the "device" table for firmware version and firmware
release date. Indeed this might make more sense, if for example we
want to record details about *other* firmware inside the system
besides the actual BIOS (system firmware). Think hard disk firmware,
or attached USB devices, or similar.
It might help to discuss examples. You said lshw is already extracting
the info you want (which is good). Can you paste a piece of the XML
showing it? Which XML element does lshw put it into?
I have been wanting to gradually move Beaker's device model to more
closely match lshw's tree of devices so this might be a good opportunity
to bring them into closer alignment.
Is this more work than we thought?
Probably. :-)
The good news is that you don't need to be *too* scared of adding new db
structures. You would have found the existing setup we have with Alembic
migrations, and there is an extensive test suite that goes with them.
The only downside we have is that they require an outage for deployment.
Then as a follow-on, we are assuming the database has to be updated
to
add
this entry, would it make sense to convert the whole database to a generic
key/value table to dynamically support new fields without having to migrate
the whole database all the time? And then to prevent random junk from being
added by the inventory script (or other script), have a whitelist filter
that only allows certain keys to be added/updated. Maintaining the
whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and
trying to see where we can add some value to make it easier to maintain our
tests.
As Nick mentioned, Beaker *does* already have a key-value store for
inventory, based on the original design from RHTS, but we are trying to
get away from it.
Its biggest deficiency is types. Keys can be either strings or numbers,
but that is not really structured enough. For example how do you
represent multiple disks, each of which has various properties like size
and model etc. The existing key-value system has no good way to store
and query that kind of info. And the key names tended to be just
invented by Beaker admins randomly and then not used consistently.
So we prefer instead to build proper db structures to represent the
hardware info. (As we did for disks, for example.)
The other problem with the existing key-value system is performance in
MySQL's query planner, this proved to be quite a nightmarish bug:
https://bugzilla.redhat.com/show_bug.cgi?id=590723
For pie-in-the-sky kind of future goals, there are indeed some modern
alternatives like JSON columns in Postgres like Nick suggested. But that
is pretty far off. We are stuck on MySQL 5.1 for the foreseeable future
and migrating MySQL->Postgres is an even bigger task.
--
Dan Callaghan <dcallagh(a)redhat.com>
Senior Software Engineer, Products & Technologies Operations
Red Hat