Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to the beaker database to be displayed on the system details page.
The idea was to make it easy to find how old a bios is on our lab machines and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the backend work is becoming a challenge.
It seems like we are going to have to add a table entry to the database in server/model/inventory.py? Which would probably cause a database migration event.
For now, we are using numa_nodes as our template to copy the output of lshw (from the inventory script) into the database and onto the webpage.
Are we going in the right direction? Is this more work than we thought?
Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a generic key/value table to dynamically support new fields without having to migrate the whole database all the time? And then to prevent random junk from being added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and trying to see where we can add some value to make it easier to maintain our tests.
Cheers, Don
On Fri, Apr 21, 2017 at 5:21 AM, Don Zickus dzickus@redhat.com wrote:
Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a generic key/value table to dynamically support new fields without having to migrate the whole database all the time? And then to prevent random junk from being added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries.
Beaker used to have a generic key-value store as a MySQL table, but it caused all sorts of performance problems on the display and filtering side of things, so it wouldn't be a good idea to go back down that specific road.
That said, things have also moved on significantly on the data storage side of things since then, with "Dynamic Columns" in MariaDB 10+, and support for hstore and JSON columns in PostgreSQL. That means there should be options available that allow the inventory system to move back to a schemaless storage model without encountering the same kinds of problems it did historically, and without requiring the addition of a completely separate document store to all Beaker deployments.
Cheers, Nick.
On Thu, Apr 20, 2017 at 03:21:05PM -0400, Don Zickus wrote:
Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to the beaker database to be displayed on the system details page.
Ping Dan?
The idea was to make it easy to find how old a bios is on our lab machines and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the backend work is becoming a challenge.
It seems like we are going to have to add a table entry to the database in server/model/inventory.py? Which would probably cause a database migration event.
For now, we are using numa_nodes as our template to copy the output of lshw (from the inventory script) into the database and onto the webpage.
Are we going in the right direction? Is this more work than we thought?
Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a generic key/value table to dynamically support new fields without having to migrate the whole database all the time? And then to prevent random junk from being added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and trying to see where we can add some value to make it easier to maintain our tests.
Hi Nick,
Thanks for the response. I wasn't subscribed to the list, but I had Jeff forward it to me. It is good to know the historical perspective. A shame that the original approach had performance issues.
Cheers, Don
Excerpts from Don Zickus's message of 2017-04-20 15:21 -04:00:
Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to the beaker database to be displayed on the system details page.
The idea was to make it easy to find how old a bios is on our lab machines and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the backend work is becoming a challenge.
It seems like we are going to have to add a table entry to the database in server/model/inventory.py? Which would probably cause a database migration event.
For now, we are using numa_nodes as our template to copy the output of lshw (from the inventory script) into the database and onto the webpage.
Are we going in the right direction?
Yes, it's important to settle on a good db structure for this now because it is quite costly to put it in place and to change it in future.
The NUMA stuff is not a great example necessarily because it was always intended that we would flesh it out to store more info about NUMA topology, but that never happened. Right now it is just a single table 1-to-1 with "system" with a single column. It could have just been a new column on the system table.
It also depends exactly what info you want to store about the firmware. Just an opaque version string? Or also a release date? Assuming the latter, the options I can see would be:
* Add a new table "firmware" which is 1-to-1 with system, rows are created for a system when the value is first populated. This is how Cpu, Numa, Power, and others work right now.
* Add new columns to the "system" table for firmware version and firmware release date. NULLable with default NULL, and populated by the inventory script if possible.
* Represent the system firmware as a device in the existing devices table (in fact it might already be in there from lshw?). You could add new columns to the "device" table for firmware version and firmware release date. Indeed this might make more sense, if for example we want to record details about *other* firmware inside the system besides the actual BIOS (system firmware). Think hard disk firmware, or attached USB devices, or similar.
It might help to discuss examples. You said lshw is already extracting the info you want (which is good). Can you paste a piece of the XML showing it? Which XML element does lshw put it into?
I have been wanting to gradually move Beaker's device model to more closely match lshw's tree of devices so this might be a good opportunity to bring them into closer alignment.
Is this more work than we thought?
Probably. :-)
The good news is that you don't need to be *too* scared of adding new db structures. You would have found the existing setup we have with Alembic migrations, and there is an extensive test suite that goes with them. The only downside we have is that they require an outage for deployment.
Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a generic key/value table to dynamically support new fields without having to migrate the whole database all the time? And then to prevent random junk from being added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and trying to see where we can add some value to make it easier to maintain our tests.
As Nick mentioned, Beaker *does* already have a key-value store for inventory, based on the original design from RHTS, but we are trying to get away from it.
Its biggest deficiency is types. Keys can be either strings or numbers, but that is not really structured enough. For example how do you represent multiple disks, each of which has various properties like size and model etc. The existing key-value system has no good way to store and query that kind of info. And the key names tended to be just invented by Beaker admins randomly and then not used consistently.
So we prefer instead to build proper db structures to represent the hardware info. (As we did for disks, for example.)
The other problem with the existing key-value system is performance in MySQL's query planner, this proved to be quite a nightmarish bug:
https://bugzilla.redhat.com/show_bug.cgi?id=590723
For pie-in-the-sky kind of future goals, there are indeed some modern alternatives like JSON columns in Postgres like Nick suggested. But that is pretty far off. We are stuck on MySQL 5.1 for the foreseeable future and migrating MySQL->Postgres is an even bigger task.
It might help to discuss examples. You said lshw is already extracting
the info you want (which is good). Can you paste a piece of the XML showing it? Which XML element does lshw put it into?
using my laptop as an example. $ lshw -xml
the node info for firmware is
</node> <node id="firmware" claimed="true" class="memory" handle=""> <description>BIOS</description> <vendor>LENOVO</vendor> <physid>31</physid> <version>N14ET26W (1.04 )</version> <date>01/23/2015</date> <size units="bytes">131072</size> <capacity units="bytes">16711680</capacity> <capabilities> <capability id="pci" >PCI bus</capability> <capability id="pnp" >Plug-and-Play</capability> <capability id="upgrade" >BIOS EEPROM can be upgraded</capability> <capability id="shadowing" >BIOS shadowing</capability> <capability id="cdboot" >Booting from CD-ROM/DVD</capability> <capability id="bootselect" >Selectable boot path</capability> <capability id="acpi" >ACPI</capability> <capability id="usb" >USB legacy emulation</capability> <capability id="biosbootspecification" >BIOS boot specification</capability> <capability id="uefi" >UEFI specification is supported</capability> </capabilities> </node>
I was just pulling from the version and date tags.
from lxml import etree inventory = etree.parse('test.xml')
sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0] sysfw = dict(version = sysfwinfo.findtext('version'), date = sysfwinfo.findtext('date'))
hope that helps. Shawn
Hello all.. apologies if zombie thread is inappropriate but I'd like to revisit this conversation regarding firmware information being included.
We have started a patch to display firmware and date and are hoping to get some input on progressing. What we were last discussing was to update the device table. I'm also curious about a migration script. Here is what we have so far.
Thanks for looking. Shawn.
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..32dc0d4 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -123,9 +123,11 @@ def default(self, *args, **kw): widgets.PaginateDataGrid.Column(name='driver', getter=lambda x: x.driver, title='Driver', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='vendor_id', getter=lambda x: x.vendor_id, title='Vendor ID', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='device_id', getter=lambda x: x.device_id, title='Device ID', options=dict(sortable=True)), - widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)), + widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.subsys_device_id, title='Subsys Device ID', options=dict(sortable=True)), - ]) + widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_version, title='Firmware Version', options=dict(sortable=True)), ++ widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_date, title='Firmware Date', options=dict(sortable=True)), + ]) return dict(title="Devices", grid = devices_grid, search_bar=None, diff --git a/Server/bkr/server/model/inventory.py b/Server/bkr/server/model/inventory.py index ecbc073..88301ae 100644 --- a/Server/bkr/server/model/inventory.py +++ b/Server/bkr/server/model/inventory.py @@ -297,6 +297,8 @@ class System(DeclarativeMappedObject, ActivityMixin): kernel_type_id = Column(Integer, ForeignKey('kernel_type.id'), default=select([KernelType.id], limit=1).where(KernelType.kernel_type==u'default').correlate(None), nullable=False) + fw_version = Column(String(32)) + fw_date = Column(DateTime, default=None) kernel_type = relationship('KernelType') devices = relationship('Device', secondary=system_device_map, back_populates='systems') @@ -1246,6 +1248,13 @@ def update(self, inventory): self.date_modified = datetime.utcnow() return 0
+ def updateSystemFirmware(self, firmware): + # TODO finish this method + if firmware + + self.fw_version = firmware['version'] + self.fw_date = firmware['date'] + def updateHypervisor(self, hypervisor): if hypervisor: hvisor = Hypervisor.by_name(hypervisor) @@ -2362,6 +2371,8 @@ class Device(DeclarativeMappedObject): device_class = relationship(DeviceClass) date_added = Column(DateTime, default=datetime.utcnow, nullable=False) systems = relationship(System, secondary=system_device_map, back_populates='devices') + fw_version = Column(String(32)) + fw_date = Column(DateTime, default=None)
Index('ix_device_pciid', Device.vendor_id, Device.device_id)
diff --git a/Server/bkr/server/templates/system_details.kid b/Server/bkr/server/templates/system_details.kid index 19ba196..fd6f9d1 100644 --- a/Server/bkr/server/templates/system_details.kid +++ b/Server/bkr/server/templates/system_details.kid @@ -156,6 +156,12 @@ <td> ${device.subsys_device_id} </td> + <td> + ${device.fw_revision} + </td> + <td> + ${device.fw_date} + </td> </tr> </tbody> </table>
diff --git a/systemscan/main.py b/systemscan/main.py index 64b9312..41895f2 100755 --- a/systemscan/main.py +++ b/systemscan/main.py @@ -286,6 +286,14 @@ def read_inventory(inventory, arch = None, proc_cpuinfo='/proc/cpuinfo'):
#Break the xml into the relevant sets of data cpuinfo = inventory.xpath(".//node[@class='processor']")[0] + + #system firmware info + sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0] + #create dictionary with key-values for version and date + SystemFirmware = dict(version = sysfwinfo.findtext('version'), date = sysfwinfo.findtext('date')) + #add to data + data['SystemFirmware'] = SystemFirmware + memory_elems = inventory.xpath(".//node[@class='memory']") memoryinfo = None for m in memory_elems:
On Thu, Apr 27, 2017 at 3:13 AM, Dan Callaghan dcallagh@redhat.com wrote:
Excerpts from Don Zickus's message of 2017-04-20 15:21 -04:00:
Hi Dan,
Shawn is working with Jon and myself to add system firmware/bios info to
the
beaker database to be displayed on the system details page.
The idea was to make it easy to find how old a bios is on our lab
machines
and see which ones need an update.
Shawn made all the front end changes in the inventory job, but the
backend
work is becoming a challenge.
It seems like we are going to have to add a table entry to the database
in
server/model/inventory.py? Which would probably cause a database
migration
event.
For now, we are using numa_nodes as our template to copy the output of
lshw
(from the inventory script) into the database and onto the webpage.
Are we going in the right direction?
Yes, it's important to settle on a good db structure for this now because it is quite costly to put it in place and to change it in future.
The NUMA stuff is not a great example necessarily because it was always intended that we would flesh it out to store more info about NUMA topology, but that never happened. Right now it is just a single table 1-to-1 with "system" with a single column. It could have just been a new column on the system table.
It also depends exactly what info you want to store about the firmware. Just an opaque version string? Or also a release date? Assuming the latter, the options I can see would be:
Add a new table "firmware" which is 1-to-1 with system, rows are created for a system when the value is first populated. This is how Cpu, Numa, Power, and others work right now.
Add new columns to the "system" table for firmware version and firmware release date. NULLable with default NULL, and populated by the inventory script if possible.
Represent the system firmware as a device in the existing devices table (in fact it might already be in there from lshw?). You could add new columns to the "device" table for firmware version and firmware release date. Indeed this might make more sense, if for example we want to record details about *other* firmware inside the system besides the actual BIOS (system firmware). Think hard disk firmware, or attached USB devices, or similar.
It might help to discuss examples. You said lshw is already extracting the info you want (which is good). Can you paste a piece of the XML showing it? Which XML element does lshw put it into?
I have been wanting to gradually move Beaker's device model to more closely match lshw's tree of devices so this might be a good opportunity to bring them into closer alignment.
Is this more work than we thought?
Probably. :-)
The good news is that you don't need to be *too* scared of adding new db structures. You would have found the existing setup we have with Alembic migrations, and there is an extensive test suite that goes with them. The only downside we have is that they require an outage for deployment.
Then as a follow-on, we are assuming the database has to be updated to add this entry, would it make sense to convert the whole database to a
generic
key/value table to dynamically support new fields without having to
migrate
the whole database all the time? And then to prevent random junk from
being
added by the inventory script (or other script), have a whitelist filter that only allows certain keys to be added/updated. Maintaining the whitelist would be easier on the database then adding table entries.
This is just us trying to understand the architecture a little more and trying to see where we can add some value to make it easier to maintain
our
tests.
As Nick mentioned, Beaker *does* already have a key-value store for inventory, based on the original design from RHTS, but we are trying to get away from it.
Its biggest deficiency is types. Keys can be either strings or numbers, but that is not really structured enough. For example how do you represent multiple disks, each of which has various properties like size and model etc. The existing key-value system has no good way to store and query that kind of info. And the key names tended to be just invented by Beaker admins randomly and then not used consistently.
So we prefer instead to build proper db structures to represent the hardware info. (As we did for disks, for example.)
The other problem with the existing key-value system is performance in MySQL's query planner, this proved to be quite a nightmarish bug:
https://bugzilla.redhat.com/show_bug.cgi?id=590723
For pie-in-the-sky kind of future goals, there are indeed some modern alternatives like JSON columns in Postgres like Nick suggested. But that is pretty far off. We are stuck on MySQL 5.1 for the foreseeable future and migrating MySQL->Postgres is an even bigger task.
-- Dan Callaghan dcallagh@redhat.com Senior Software Engineer, Products & Technologies Operations Red Hat
Excerpts from Shawn Doherty's message of 2017-07-14 09:17 -04:00:
Hello all.. apologies if zombie thread is inappropriate but I'd like to revisit this conversation regarding firmware information being included.
We have started a patch to display firmware and date and are hoping to get some input on progressing. What we were last discussing was to update the device table. I'm also curious about a migration script. Here is what we have so far.
Thanks for looking. Shawn.
Might be easier to read this if you post it to Gerrit. There is no harm in posting a WIP or draft patch to Gerrit and asking for reviews on there.
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..32dc0d4 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -123,9 +123,11 @@ def default(self, *args, **kw): widgets.PaginateDataGrid.Column(name='driver', getter=lambda x: x.driver, title='Driver', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='vendor_id', getter=lambda x: x.vendor_id, title='Vendor ID', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='device_id', getter=lambda x: x.device_id, title='Device ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id',getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.subsys_device_id, title='Subsys Device ID', options=dict(sortable=True)),
])
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_version, title='Firmware Version', options=dict(sortable=True)), ++ widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_date, title='Firmware Date', options=dict(sortable=True)),
])
You'll want to fix the column name and title on these. Name can be anything but by convention it should match the db column. It's used for sorting inside the widget.
return dict(title="Devices", grid = devices_grid, search_bar=None,diff --git a/Server/bkr/server/model/inventory.py b/Server/bkr/server/model/inventory.py index ecbc073..88301ae 100644 --- a/Server/bkr/server/model/inventory.py +++ b/Server/bkr/server/model/inventory.py @@ -297,6 +297,8 @@ class System(DeclarativeMappedObject, ActivityMixin): kernel_type_id = Column(Integer, ForeignKey('kernel_type.id'), default=select([KernelType.id], limit=1).where(KernelType.kernel_type==u'default').correlate(None), nullable=False)
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None) kernel_type = relationship('KernelType') devices = relationship('Device', secondary=system_device_map, back_populates='systems')
It looks like these are new columns on System, but they should actually be new columns on Device, right?
@@ -2362,6 +2371,8 @@ class Device(DeclarativeMappedObject): device_class = relationship(DeviceClass) date_added = Column(DateTime, default=datetime.utcnow, nullable=False) systems = relationship(System, secondary=system_device_map, back_populates='devices')
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None)
... oh never mind, they are here too. :-) So I guess just remove the ones above on System.
We probably want Date not Datetime as the type for the fw_date column (I assume no firmware reports its build date down to the second).
Is 32 chars enough for the version string? We can always expand it later. Does lshw itself have any length limits/expectations for the firmware version?
diff --git a/systemscan/main.py b/systemscan/main.py index 64b9312..41895f2 100755 --- a/systemscan/main.py +++ b/systemscan/main.py @@ -286,6 +286,14 @@ def read_inventory(inventory, arch = None, proc_cpuinfo='/proc/cpuinfo'):
#Break the xml into the relevant sets of data cpuinfo = inventory.xpath(".//node[@class='processor']")[0]
- #system firmware info
- sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0]
- #create dictionary with key-values for version and date
- SystemFirmware = dict(version = sysfwinfo.findtext('version'), date =
sysfwinfo.findtext('date'))
- #add to data
- data['SystemFirmware'] = SystemFirmware
I guess this is not quite the right place... There is a loop starting line 492:
for device in devices:
where it compiles the list of devices with the properties for each one. I guess you would just fill in fw_version and fw_date there, whenever the values exist.
On 07/17/2017 06:36 PM, Dan Callaghan wrote:
Excerpts from Shawn Doherty's message of 2017-07-14 09:17 -04:00:
Hello all.. apologies if zombie thread is inappropriate but I'd like to revisit this conversation regarding firmware information being included.
We have started a patch to display firmware and date and are hoping to get some input on progressing. What we were last discussing was to update the device table. I'm also curious about a migration script. Here is what we have so far.
Thanks for looking. Shawn.
Might be easier to read this if you post it to Gerrit. There is no harm in posting a WIP or draft patch to Gerrit and asking for reviews on there.
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..32dc0d4 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -123,9 +123,11 @@ def default(self, *args, **kw): widgets.PaginateDataGrid.Column(name='driver', getter=lambda x: x.driver, title='Driver', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='vendor_id', getter=lambda x: x.vendor_id, title='Vendor ID', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='device_id', getter=lambda x: x.device_id, title='Device ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id',getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.subsys_device_id, title='Subsys Device ID', options=dict(sortable=True)),
])
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_version, title='Firmware Version', options=dict(sortable=True)), ++ widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_date, title='Firmware Date', options=dict(sortable=True)),
])You'll want to fix the column name and title on these. Name can be anything but by convention it should match the db column. It's used for sorting inside the widget.
return dict(title="Devices", grid = devices_grid, search_bar=None,diff --git a/Server/bkr/server/model/inventory.py b/Server/bkr/server/model/inventory.py index ecbc073..88301ae 100644 --- a/Server/bkr/server/model/inventory.py +++ b/Server/bkr/server/model/inventory.py @@ -297,6 +297,8 @@ class System(DeclarativeMappedObject, ActivityMixin): kernel_type_id = Column(Integer, ForeignKey('kernel_type.id'), default=select([KernelType.id], limit=1).where(KernelType.kernel_type==u'default').correlate(None), nullable=False)
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None) kernel_type = relationship('KernelType') devices = relationship('Device', secondary=system_device_map, back_populates='systems')
It looks like these are new columns on System, but they should actually be new columns on Device, right?
We need two sets of firmware info, system level info to track what version of BIOS0 is installed on a machine, these entries here. And device level info to track what firmware, if any, is on a given PCI connected device, the additions below. It is currently our understanding that the layout is something like
system_entry -> device_entry device_entry
A system_entry exists in a different table from device_entries hence why we decided to use two different sets of data, thus the two changes. Maybe I misunderstood the DB schema and device entries are reused for a singular machine entry?
The catch for the system level firmware is it must be directly associated with a given FQDN, i.e. it is a property of an FQDN.
Hopefully this makes a little more sense as to why there are two sets of firmware entries, it was intentional - maybe not needed :-D
@@ -2362,6 +2371,8 @@ class Device(DeclarativeMappedObject): device_class = relationship(DeviceClass) date_added = Column(DateTime, default=datetime.utcnow, nullable=False) systems = relationship(System, secondary=system_device_map, back_populates='devices')
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None)
... oh never mind, they are here too. :-) So I guess just remove the ones above on System.
We probably want Date not Datetime as the type for the fw_date column (I assume no firmware reports its build date down to the second).
Is 32 chars enough for the version string? We can always expand it later. Does lshw itself have any length limits/expectations for the firmware version?
Good question, lshw has a char array of 32 for network device firmware, network devices were the initial set of devices we had considered:
in core/network.cc: 82 struct ethtool_drvinfo 83 { 84 u32 cmd; 85 char driver[32]; /* driver short name, "tulip", "eepro100" */ 86 char version[32]; /* driver version string */ 87 char fw_version[32]; /* firmware version string, if applicable */ 88 char bus_info[ETHTOOL_BUSINFO_LEN]; /* Bus info for this IF. */
diff --git a/systemscan/main.py b/systemscan/main.py index 64b9312..41895f2 100755 --- a/systemscan/main.py +++ b/systemscan/main.py @@ -286,6 +286,14 @@ def read_inventory(inventory, arch = None, proc_cpuinfo='/proc/cpuinfo'):
#Break the xml into the relevant sets of data cpuinfo = inventory.xpath(".//node[@class='processor']")[0]
- #system firmware info
- sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0]
- #create dictionary with key-values for version and date
- SystemFirmware = dict(version = sysfwinfo.findtext('version'), date =
sysfwinfo.findtext('date'))
- #add to data
- data['SystemFirmware'] = SystemFirmware
I guess this is not quite the right place... There is a loop starting line 492:
for device in devices:where it compiles the list of devices with the properties for each one. I guess you would just fill in fw_version and fw_date there, whenever the values exist.
Beaker-devel mailing list -- beaker-devel@lists.fedorahosted.org To unsubscribe send an email to beaker-devel-leave@lists.fedorahosted.org
What happens with something like the SGI UV "composable" systems?
On Jul 17, 2017 20:26, "Jonathan Toppins" jtoppins@redhat.com wrote:
On 07/17/2017 06:36 PM, Dan Callaghan wrote:
Excerpts from Shawn Doherty's message of 2017-07-14 09:17 -04:00:
Hello all.. apologies if zombie thread is inappropriate but I'd like to revisit this conversation regarding firmware information being included.
We have started a patch to display firmware and date and are hoping to
get
some input on progressing. What we were last discussing was to update
the
device table. I'm also curious about a migration script. Here is
what we
have so far.
Thanks for looking. Shawn.
Might be easier to read this if you post it to Gerrit. There is no harm in posting a WIP or draft patch to Gerrit and asking for reviews on there.
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..32dc0d4 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -123,9 +123,11 @@ def default(self, *args, **kw): widgets.PaginateDataGrid.Column(name='driver', getter=lambda x: x.driver, title='Driver', options=dict(sortable=True)), widgets.PaginateDataGrid.
Column(name='vendor_id',
getter=lambda x: x.vendor_id, title='Vendor ID', options=dict(sortable=True)), widgets.PaginateDataGrid.
Column(name='device_id',
getter=lambda x: x.device_id, title='Device ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda
x:
x.subsys_vendor_id, title='Subsys Vendor ID',
options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id',getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda
x:
x.subsys_device_id, title='Subsys Device ID',
options=dict(sortable=True)),
])
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda
x:
x.fw_version, title='Firmware Version', options=dict(sortable=True)), ++ widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda
x:
x.fw_date, title='Firmware Date', options=dict(sortable=True)),
])You'll want to fix the column name and title on these. Name can be anything but by convention it should match the db column. It's used for sorting inside the widget.
return dict(title="Devices", grid = devices_grid, search_bar=None,diff --git a/Server/bkr/server/model/inventory.py b/Server/bkr/server/model/inventory.py index ecbc073..88301ae 100644 --- a/Server/bkr/server/model/inventory.py +++ b/Server/bkr/server/model/inventory.py @@ -297,6 +297,8 @@ class System(DeclarativeMappedObject,
ActivityMixin):
kernel_type_id = Column(Integer, ForeignKey('kernel_type.id'), default=select([KernelType.id],limit=1).where(KernelType.kernel_type==u'default').correlate(None), nullable=False)
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None) kernel_type = relationship('KernelType') devices = relationship('Device', secondary=system_device_map, back_populates='systems')
It looks like these are new columns on System, but they should actually be new columns on Device, right?
We need two sets of firmware info, system level info to track what version of BIOS0 is installed on a machine, these entries here. And device level info to track what firmware, if any, is on a given PCI connected device, the additions below. It is currently our understanding that the layout is something like
system_entry -> device_entry device_entry
A system_entry exists in a different table from device_entries hence why we decided to use two different sets of data, thus the two changes. Maybe I misunderstood the DB schema and device entries are reused for a singular machine entry?
The catch for the system level firmware is it must be directly associated with a given FQDN, i.e. it is a property of an FQDN.
Hopefully this makes a little more sense as to why there are two sets of firmware entries, it was intentional - maybe not needed :-D
@@ -2362,6 +2371,8 @@ class Device(DeclarativeMappedObject): device_class = relationship(DeviceClass) date_added = Column(DateTime, default=datetime.utcnow,
nullable=False)
systems = relationship(System, secondary=system_device_map,back_populates='devices')
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None)
... oh never mind, they are here too. :-) So I guess just remove the ones above on System.
We probably want Date not Datetime as the type for the fw_date column (I assume no firmware reports its build date down to the second).
Is 32 chars enough for the version string? We can always expand it later. Does lshw itself have any length limits/expectations for the firmware version?
Good question, lshw has a char array of 32 for network device firmware, network devices were the initial set of devices we had considered:
in core/network.cc: 82 struct ethtool_drvinfo 83 { 84 u32 cmd; 85 char driver[32]; /* driver short name, "tulip", "eepro100" */ 86 char version[32]; /* driver version string */ 87 char fw_version[32]; /* firmware version string, if applicable */ 88 char bus_info[ETHTOOL_BUSINFO_LEN]; /* Bus info for this IF. */
diff --git a/systemscan/main.py b/systemscan/main.py index 64b9312..41895f2 100755 --- a/systemscan/main.py +++ b/systemscan/main.py @@ -286,6 +286,14 @@ def read_inventory(inventory, arch = None, proc_cpuinfo='/proc/cpuinfo'):
#Break the xml into the relevant sets of data cpuinfo = inventory.xpath(".//node[@class='processor']")[0]
- #system firmware info
- sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0]
- #create dictionary with key-values for version and date
- SystemFirmware = dict(version = sysfwinfo.findtext('version'),
date =
sysfwinfo.findtext('date'))
- #add to data
- data['SystemFirmware'] = SystemFirmware
I guess this is not quite the right place... There is a loop starting line 492:
for device in devices:where it compiles the list of devices with the properties for each one. I guess you would just fill in fw_version and fw_date there, whenever the values exist.
Beaker-devel mailing list -- beaker-devel@lists.fedorahosted.org To unsubscribe send an email to beaker-devel-leave@lists.
fedorahosted.org
Beaker-devel mailing list -- beaker-devel@lists.fedorahosted.org To unsubscribe send an email to beaker-devel-leave@lists.fedorahosted.org
On Mon, Jul 17, 2017 at 08:26:17PM -0400, Jonathan Toppins wrote:
On 07/17/2017 06:36 PM, Dan Callaghan wrote:
Excerpts from Shawn Doherty's message of 2017-07-14 09:17 -04:00:
Hello all.. apologies if zombie thread is inappropriate but I'd like to revisit this conversation regarding firmware information being included.
We have started a patch to display firmware and date and are hoping to get some input on progressing. What we were last discussing was to update the device table. I'm also curious about a migration script. Here is what we have so far.
Thanks for looking. Shawn.
Hi Dan,
Any followup on Jon's email? Also any tips on how to write database migration scripts for the changes we want to do?
Cheers, Don
Might be easier to read this if you post it to Gerrit. There is no harm in posting a WIP or draft patch to Gerrit and asking for reviews on there.
diff --git a/Server/bkr/server/controllers.py b/Server/bkr/server/controllers.py index cece8db..32dc0d4 100644 --- a/Server/bkr/server/controllers.py +++ b/Server/bkr/server/controllers.py @@ -123,9 +123,11 @@ def default(self, *args, **kw): widgets.PaginateDataGrid.Column(name='driver', getter=lambda x: x.driver, title='Driver', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='vendor_id', getter=lambda x: x.vendor_id, title='Vendor ID', options=dict(sortable=True)), widgets.PaginateDataGrid.Column(name='device_id', getter=lambda x: x.device_id, title='Device ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id', getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_vendor_id',getter=lambda x: x.subsys_vendor_id, title='Subsys Vendor ID', options=dict(sortable=True)),
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.subsys_device_id, title='Subsys Device ID', options=dict(sortable=True)),
])
widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_version, title='Firmware Version', options=dict(sortable=True)), ++ widgets.PaginateDataGrid.Column(name='subsys_device_id', getter=lambda x: x.fw_date, title='Firmware Date', options=dict(sortable=True)),
])You'll want to fix the column name and title on these. Name can be anything but by convention it should match the db column. It's used for sorting inside the widget.
return dict(title="Devices", grid = devices_grid, search_bar=None,diff --git a/Server/bkr/server/model/inventory.py b/Server/bkr/server/model/inventory.py index ecbc073..88301ae 100644 --- a/Server/bkr/server/model/inventory.py +++ b/Server/bkr/server/model/inventory.py @@ -297,6 +297,8 @@ class System(DeclarativeMappedObject, ActivityMixin): kernel_type_id = Column(Integer, ForeignKey('kernel_type.id'), default=select([KernelType.id], limit=1).where(KernelType.kernel_type==u'default').correlate(None), nullable=False)
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None) kernel_type = relationship('KernelType') devices = relationship('Device', secondary=system_device_map, back_populates='systems')
It looks like these are new columns on System, but they should actually be new columns on Device, right?
We need two sets of firmware info, system level info to track what version of BIOS0 is installed on a machine, these entries here. And device level info to track what firmware, if any, is on a given PCI connected device, the additions below. It is currently our understanding that the layout is something like
system_entry -> device_entry device_entry
A system_entry exists in a different table from device_entries hence why we decided to use two different sets of data, thus the two changes. Maybe I misunderstood the DB schema and device entries are reused for a singular machine entry?
The catch for the system level firmware is it must be directly associated with a given FQDN, i.e. it is a property of an FQDN.
Hopefully this makes a little more sense as to why there are two sets of firmware entries, it was intentional - maybe not needed :-D
@@ -2362,6 +2371,8 @@ class Device(DeclarativeMappedObject): device_class = relationship(DeviceClass) date_added = Column(DateTime, default=datetime.utcnow, nullable=False) systems = relationship(System, secondary=system_device_map, back_populates='devices')
- fw_version = Column(String(32))
- fw_date = Column(DateTime, default=None)
... oh never mind, they are here too. :-) So I guess just remove the ones above on System.
We probably want Date not Datetime as the type for the fw_date column (I assume no firmware reports its build date down to the second).
Is 32 chars enough for the version string? We can always expand it later. Does lshw itself have any length limits/expectations for the firmware version?
Good question, lshw has a char array of 32 for network device firmware, network devices were the initial set of devices we had considered:
in core/network.cc: 82 struct ethtool_drvinfo 83 { 84 u32 cmd; 85 char driver[32]; /* driver short name, "tulip", "eepro100" */ 86 char version[32]; /* driver version string */ 87 char fw_version[32]; /* firmware version string, if applicable */ 88 char bus_info[ETHTOOL_BUSINFO_LEN]; /* Bus info for this IF. */
diff --git a/systemscan/main.py b/systemscan/main.py index 64b9312..41895f2 100755 --- a/systemscan/main.py +++ b/systemscan/main.py @@ -286,6 +286,14 @@ def read_inventory(inventory, arch = None, proc_cpuinfo='/proc/cpuinfo'):
#Break the xml into the relevant sets of data cpuinfo = inventory.xpath(".//node[@class='processor']")[0]
- #system firmware info
- sysfwinfo = inventory.xpath('.//node[@id="firmware"]')[0]
- #create dictionary with key-values for version and date
- SystemFirmware = dict(version = sysfwinfo.findtext('version'), date =
sysfwinfo.findtext('date'))
- #add to data
- data['SystemFirmware'] = SystemFirmware
I guess this is not quite the right place... There is a loop starting line 492:
for device in devices:where it compiles the list of devices with the properties for each one. I guess you would just fill in fw_version and fw_date there, whenever the values exist.
Beaker-devel mailing list -- beaker-devel@lists.fedorahosted.org To unsubscribe send an email to beaker-devel-leave@lists.fedorahosted.org
Beaker-devel mailing list -- beaker-devel@lists.fedorahosted.org To unsubscribe send an email to beaker-devel-leave@lists.fedorahosted.org
Excerpts from Don Zickus's message of 2017-07-20 11:12 -04:00:
Hi Dan,
Any followup on Jon's email? Also any tips on how to write database migration scripts for the changes we want to do?
In general you need an Alembic migration script, with upgrade and downgrade steps. Running "alembic migrate" in the Server directory should create a new empty migration with random id and the necessary boilerplate, which you can fill in.
If the migration includes some step to migrate/transform data to a new format then you would also include a test case that proves the transformation does the right thing -- but in this case, just adding some new columns with no values, there is no data transformation to be tested.
Alembic can generate the migrations automatically by comparing an existing database to the model definition, when you give it the --autogenerate option... *but* for our database it typically does not give usable results, because of deficiencies in how MySQL mangles the schema. You are better off writing the upgrade/downgrade operations by hand.
There are plenty of examples of past migrations in the git history, for example this is probably a good one:
https://git.beaker-project.org/cgit/beaker/commit/?id=da355153e4583da4e3147a...
op.add_column / op.drop_column are the ones you will want here too.
On Mon, Jul 24, 2017 at 03:06:10PM +1000, Dan Callaghan wrote:
Excerpts from Don Zickus's message of 2017-07-20 11:12 -04:00:
Hi Dan,
Any followup on Jon's email? Also any tips on how to write database migration scripts for the changes we want to do?
In general you need an Alembic migration script, with upgrade and downgrade steps. Running "alembic migrate" in the Server directory should create a new empty migration with random id and the necessary boilerplate, which you can fill in.
If the migration includes some step to migrate/transform data to a new format then you would also include a test case that proves the transformation does the right thing -- but in this case, just adding some new columns with no values, there is no data transformation to be tested.
Alembic can generate the migrations automatically by comparing an existing database to the model definition, when you give it the --autogenerate option... *but* for our database it typically does not give usable results, because of deficiencies in how MySQL mangles the schema. You are better off writing the upgrade/downgrade operations by hand.
There are plenty of examples of past migrations in the git history, for example this is probably a good one:
https://git.beaker-project.org/cgit/beaker/commit/?id=da355153e4583da4e3147a...
Thanks Dan! We did something like that last week, but we were using 'alembic upgrade/downgrade' instead of 'migration'. The upgrade kept failing because of 'alembic no handlers found for logger alembic.migration' or something. Thoughts there?
Cheers, Don
Excerpts from Don Zickus's message of 2017-07-24 09:54 -04:00:
Thanks Dan! We did something like that last week, but we were using 'alembic upgrade/downgrade' instead of 'migration'. The upgrade kept failing because of 'alembic no handlers found for logger alembic.migration' or something. Thoughts there?
Sorry I actually meant to say "alembic revision". It is the command to create a new migration script in the source tree based on the template. I always get that command name wrong.
The "no handlers found" message is just pointless spew from the Python logging subsystem, not an actual error. However the "alembic upgrade" and "downgrade" commands won't work with Beaker, due to the hackery needed to glue Alembic into all the existing database code which is inherited from TurboGears.
The normal way to upgrade/downgrade the database schema on a Beaker installation is using beaker-init (to upgrade) or beaker-init --downgrade=<version> (to downgrade).
If you're running from a git checkout with a database configured in dev.cfg, you can just do something like:
cd Server PYTHONPATH=../Common:. python bkr/server/tools/init.py --debug
On Mon, Jul 24, 2017 at 11:59 PM, Dan Callaghan dcallagh@redhat.com wrote:
Excerpts from Don Zickus's message of 2017-07-24 09:54 -04:00:
Thanks Dan! We did something like that last week, but we were using 'alembic upgrade/downgrade' instead of 'migration'. The upgrade kept failing because of 'alembic no handlers found for logger alembic.migration' or something. Thoughts there?
Sorry I actually meant to say "alembic revision". It is the command to create a new migration script in the source tree based on the template. I always get that command name wrong.
The "no handlers found" message is just pointless spew from the Python logging subsystem, not an actual error. However the "alembic upgrade" and "downgrade" commands won't work with Beaker, due to the hackery needed to glue Alembic into all the existing database code which is inherited from TurboGears.
The normal way to upgrade/downgrade the database schema on a Beaker installation is using beaker-init (to upgrade) or beaker-init --downgrade=<version> (to downgrade).
If you're running from a git checkout with a database configured in dev.cfg, you can just do something like:
cd Server PYTHONPATH=../Common:. python bkr/server/tools/init.py --debug
Thanks for the description
something flukey here for me. I have auto generated 3dd091c46d50 as head using alembic revision, but db version stays at f18df089261. Using methods described above, including beaker-init (--downgrade)have no affect on the db version. I can interact with the db with the alembic stamp command but only to maniuplate the version_number. Other db changes do not seem to be affected.
Thoughts?
alembic history Rev: 3dd091c46d50 (head) Parent: f18df089261 Path: bkr/server/alembic/versions/3dd091c46d50_.py
empty message
Revision ID: 3dd091c46d50 Revises: f18df089261 Create Date: 2017-07-25 18:03:04.320470
Rev: f18df089261 Parent: 4f7a4316565f Path: bkr/server/alembic/versions/f18df089261_associate_a_floating_ip_to_each_openstack_instance.py
associate a floating ip to each OpenStack instance
Revision ID: f18df089261 Revises: 4f7a4316565f Create Date: 2016-12-05 05:59:28.858413
root@beaker-server-lc Server]# PYTHONPATH=../Common:. python bkr/server/tools/init.py --debug 2017-07-25 18:06:40,981 __main__ INFO Upgrading schema to head revision 2017-07-25 18:06:40,981 alembic.migration INFO Context impl MySQLImpl. 2017-07-25 18:06:40,981 alembic.migration INFO Will assume non-transactional DDL. 2017-07-25 18:06:40,987 __main__ INFO Upgrade completed 2017-07-25 18:06:40,987 __main__ INFO Populating tables with pre-defined values if necessary 2017-07-25 18:06:41,225 __main__ INFO Pre-defined values populated 2017-07-25 18:06:41,226 __main__ INFO Exiting
SELECT * FROM alembic_version; +-------------+ | version_num | +-------------+ | f18df089261 | +-------------+
-- Dan Callaghan dcallagh@redhat.com Senior Software Engineer, Products & Technologies Operations Red Hat
Excerpts from Shawn Doherty's message of 2017-07-25 14:35 -04:00:
Thanks for the description
something flukey here for me. I have auto generated 3dd091c46d50 as head using alembic revision, but db version stays at f18df089261. Using methods described above, including beaker-init (--downgrade)have no affect on the db version. I can interact with the db with the alembic stamp command but only to maniuplate the version_number. Other db changes do not seem to be affected.
Thoughts?
Are you possibly running this on a system where you have your git checkout, but you *also* have a copy of Beaker installed system-wide in /usr/lib/python2.6/site-packages from the beaker-server RPM?
If so, beaker-init might be loading the migrations from site-packages instead of from the source checkout. It uses pkg_resources which obeys sys.path, sometimes '/usr/lib/python2.6/site-packages' sneaks itself onto the front of sys.path, usually due to hackery with *.pth files from some package or other.
The alembic command is looking in its .ini file to find the migrations, which would be why it sees the ones in the git checkout as you expect. Whereas beaker-init uses pkg_resources to find them, so that it works even when it is installed as an RPM package.
alembic history Rev: 3dd091c46d50 (head) Parent: f18df089261 Path: bkr/server/alembic/versions/3dd091c46d50_.py
empty message Revision ID: 3dd091c46d50 Revises: f18df089261 Create Date: 2017-07-25 18:03:04.320470Rev: f18df089261 Parent: 4f7a4316565f Path: bkr/server/alembic/versions/f18df089261_associate_a_floating_ip_to_each_openstack_instance.py
associate a floating ip to each OpenStack instance Revision ID: f18df089261 Revises: 4f7a4316565f Create Date: 2016-12-05 05:59:28.858413root@beaker-server-lc Server]# PYTHONPATH=../Common:. python bkr/server/tools/init.py --debug 2017-07-25 18:06:40,981 __main__ INFO Upgrading schema to head revision 2017-07-25 18:06:40,981 alembic.migration INFO Context impl MySQLImpl. 2017-07-25 18:06:40,981 alembic.migration INFO Will assume non-transactional DDL. 2017-07-25 18:06:40,987 __main__ INFO Upgrade completed 2017-07-25 18:06:40,987 __main__ INFO Populating tables with pre-defined values if necessary 2017-07-25 18:06:41,225 __main__ INFO Pre-defined values populated 2017-07-25 18:06:41,226 __main__ INFO Exiting
SELECT * FROM alembic_version; +-------------+ | version_num | +-------------+ | f18df089261 | +-------------+
-- Dan Callaghan dcallagh@redhat.com Senior Software Engineer, Products & Technologies Operations Red Hat
Hi Dan.
Yes, That sounds like that is what is happening.
Thanks for the explanation. Shawn
On Tue, Jul 25, 2017 at 7:12 PM, Dan Callaghan dcallagh@redhat.com wrote:
Excerpts from Shawn Doherty's message of 2017-07-25 14:35 -04:00:
Thanks for the description
something flukey here for me. I have auto generated 3dd091c46d50 as head using alembic revision, but db version stays at f18df089261. Using methods described above, including beaker-init (--downgrade)have no affect on the db version. I can interact with the db with the alembic stamp command but only to maniuplate the version_number. Other db changes do not seem to be affected.
Thoughts?
Are you possibly running this on a system where you have your git checkout, but you *also* have a copy of Beaker installed system-wide in /usr/lib/python2.6/site-packages from the beaker-server RPM?
If so, beaker-init might be loading the migrations from site-packages instead of from the source checkout. It uses pkg_resources which obeys sys.path, sometimes '/usr/lib/python2.6/site-packages' sneaks itself onto the front of sys.path, usually due to hackery with *.pth files from some package or other.
The alembic command is looking in its .ini file to find the migrations, which would be why it sees the ones in the git checkout as you expect. Whereas beaker-init uses pkg_resources to find them, so that it works even when it is installed as an RPM package.
alembic history Rev: 3dd091c46d50 (head) Parent: f18df089261 Path: bkr/server/alembic/versions/3dd091c46d50_.py
empty message Revision ID: 3dd091c46d50 Revises: f18df089261 Create Date: 2017-07-25 18:03:04.320470Rev: f18df089261 Parent: 4f7a4316565f Path: bkr/server/alembic/versions/f18df089261_associate_a_floating_ip_to_each_openstack_instance.py
associate a floating ip to each OpenStack instance Revision ID: f18df089261 Revises: 4f7a4316565f Create Date: 2016-12-05 05:59:28.858413root@beaker-server-lc Server]# PYTHONPATH=../Common:. python bkr/server/tools/init.py --debug 2017-07-25 18:06:40,981 __main__ INFO Upgrading schema to head revision 2017-07-25 18:06:40,981 alembic.migration INFO Context impl MySQLImpl. 2017-07-25 18:06:40,981 alembic.migration INFO Will assume non-transactional DDL. 2017-07-25 18:06:40,987 __main__ INFO Upgrade completed 2017-07-25 18:06:40,987 __main__ INFO Populating tables with pre-defined values if necessary 2017-07-25 18:06:41,225 __main__ INFO Pre-defined values populated 2017-07-25 18:06:41,226 __main__ INFO Exiting
SELECT * FROM alembic_version; +-------------+ | version_num | +-------------+ | f18df089261 | +-------------+
-- Dan Callaghan dcallagh@redhat.com Senior Software Engineer, Products & Technologies Operations Red Hat
-- Dan Callaghan dcallagh@redhat.com Senior Software Engineer, Products & Technologies Operations Red Hat
Excerpts from Jonathan Toppins's message of 2017-07-17 20:26 -04:00:
We need two sets of firmware info, system level info to track what version of BIOS0 is installed on a machine, these entries here. And device level info to track what firmware, if any, is on a given PCI connected device, the additions below. It is currently our understanding that the layout is something like
system_entry -> device_entry device_entry
A system_entry exists in a different table from device_entries hence why we decided to use two different sets of data, thus the two changes. Maybe I misunderstood the DB schema and device entries are reused for a singular machine entry?
The catch for the system level firmware is it must be directly associated with a given FQDN, i.e. it is a property of an FQDN.
Hopefully this makes a little more sense as to why there are two sets of firmware entries, it was intentional - maybe not needed :-D
Okay I gotcha. Didn't realise that's what you were going for.
So in lshw's output, the "system firmware" is actually just represented as another node in the tree of devices (at the top, as a direct child of the "core" node at the root). Which is a neat idea, because it makes the data model simpler, you don't need any special case for "the system itself", it just appears similar to any other device. And that gives you other stuff about the firmware too like the vendor name, which is not necessarily the same as the vendor of the system itself.
So right now the lshw node for system firmware is just ignored by beaker-system-scan. But maybe we can make it include it in the list of devices? Perhaps with some special value for the "type" or "bus" fields.
That means in Beaker we would only need to add fw_version and fw_date fields in one place, and only add one set of new search filters instead of two, etc.
Ultimately I would like to evolve Beaker's idea of "devices" to be closer to the lshw tree model because it seems to be a good way of generically describing a lot of different hardware. Not saying we need to go full speed down this route right now -- but anything that brings Beaker's model of devices closer to lshw's seems worthwhile to me.
beaker-devel@lists.fedorahosted.org
-
Dan Callaghan -
Don Zickus -
Jonathan Toppins -
Nick Coghlan -
Shawn Doherty -
Subhendu Ghosh