Hi,
This is not a formal proposal, this is for discussion and identifying liabilities. This email has an x86 GRUB bias only because that's the bootloader regime I'm most familiar with. I think it should apply to other archs as well, i.e. their bootloaders shouldn't be permitted to become stale.
Short version: Fedora should take responsibility for the bootloader being up to date, by updating it during major version upgrades. This is already the case on UEFI with conventional installations. I'd like to make sure it always happens on major version upgrades for BIOS installations. What's the problem? This would step on any custom bootloader configuration a user has for multiboot. There is no reasonable mechanism on BIOS systems to determine if the bootloader is a Fedora bootloader, and somehow only update a Fedora bootloader and not touch any other bootloader.
Fedora should be responsible for keeping the bootloader it installs in ~80% of the use cases up-to-date; and ignore the fallout from the ~20% who have a custom setup that would be stepped on by a forced bootloader update. The former is a feature and security risk, by allowing the bootloader to go stale over time. The latter is an inconvenience.
Longer version:
terms:
bootloader - this is the pre-boot bootloader binaries; on BIOS it's embedded in the MBR, and the MBR gap or BIOS Boot partition, and modules found in /boot/grub2 on Fedora. On UEFI, this is the /EFI/fedora/shimx64.efi and /EFI/fedora/grubx64.efi which in the typical installation are built and signed by the Fedora build system. Other bootloaders have different names but generally follow a similar convention, the point here is to distinguish between the "bootloader" which runs in the pre-boot environment, and the installed package containing user space tools that are used to install (or contain) the bootloader.
Fedora bootloader - this is a bootloader that is derived from Fedora packages and effort; as contrasted to merely upstream GRUB, or Ubuntu's GRUB, etc. as these derivatives can actually be substantially different from each other.
Discussion:
Both gnome-software (pk-offline-update), and dnf system-upgrade, on BIOS firmware x86 computers, do not update the bootloader. That is, we do not run 'grub2-install' either before starting an upgrade, or as a post install operation following an upgrade. Therefore the bootloader will become stale if the user does not manually run 'grub2-install' periodically.
On conventional Fedora installations on UEFI computers, the bootloader is included in the shim and grub2 packages as a file, and is therefore updated. This happens sporadically within a Fedora release, not only at major upgrade time. This updating of the bootloader files doesn't happen on Silverblue on UEFI, so it ends up being subject to the same problem under discussion.
In the most recent release, Fedora 30, we saw quite a lot of people run into a problem we knew about before release, directly related to the bootloader becoming stale. And it only affected BIOS systems (and Silverblue on both firmware types). https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1652806
In the bug, you can find some dups as well but also quite a bit of user aggravation. I'd like to avoid this going forward.
Windows and macOS never involved users in this stuff. They always asserted complete and total domain over the bootloader, there wasn't even an option to avoid updating it. And I think Fedora needs to do the same. Just update it. That benefits most users. It's the responsible thing to do. And those who need custom setups, can still do that. It would only get stepped on at major upgrade time. And it's decently likely we can warn them in advance.
There are some gotchas I'm already thinking of: MBR gap is too small, there are multiple drives and it's ambiguous which one should get the bootloader, and so on. I think it's sane to have a test for reasonable certainty we can and should update the bootloader, and warn and not update it in the cases that fail that test.
Thoughts?
On Wed, Jun 26, 2019 at 14:19:26 -0600, Chris Murphy lists@colorremedies.com wrote:
Short version: Fedora should take responsibility for the bootloader being up to date, by updating it during major version upgrades. This is already the case on UEFI with conventional installations. I'd like to make sure it always happens on major version upgrades for BIOS installations. What's the problem? This would step on any custom bootloader configuration a user has for multiboot. There is no reasonable mechanism on BIOS systems to determine if the bootloader is a Fedora bootloader, and somehow only update a Fedora bootloader and not touch any other bootloader.
How do you envision this working for rawhide? I had a problem where grub1 configs were no longer updated with kernel updates, where I finally needed to upgrade to grub2.
On Thu, Jun 27, 2019 at 9:06 AM Bruno Wolff III bruno@wolff.to wrote:
On Wed, Jun 26, 2019 at 14:19:26 -0600, Chris Murphy lists@colorremedies.com wrote:
Short version: Fedora should take responsibility for the bootloader being up to date, by updating it during major version upgrades. This is already the case on UEFI with conventional installations. I'd like to make sure it always happens on major version upgrades for BIOS installations. What's the problem? This would step on any custom bootloader configuration a user has for multiboot. There is no reasonable mechanism on BIOS systems to determine if the bootloader is a Fedora bootloader, and somehow only update a Fedora bootloader and not touch any other bootloader.
How do you envision this working for rawhide? I had a problem where grub1 configs were no longer updated with kernel updates, where I finally needed to upgrade to grub2.
Yep, small problem. And I'm not even sure how a 'grub2-install' on BIOS systems would be initiated only at major upgrade time. But even Fedora Rawhide does change versions when fcXX becomes fcXX+1, but is that a sane trigger? I'm not sure what the alternatives are.
And could there be a policy setting in /etc/default/grub for this, where the user can opt out? And should such an opt out exist? There is no such opt out for UEFI. Anyway, the how to do this is a different question than whether to do it, and whether the policy is the same across all Fedora editions, spins, images, etc.
I think it's uncontroversial to say on IoT the bootloader must be updated, I imagine no reasonable justification for making this user domain on this class of device. That is probably also true for cloud/virtual, and baremetal servers. It's really desktops that there's an open question of what/who owns the bootloader. And there I'd say the default really ought to be, maybe even must be, update the bootloader with some regularity. And if that's true and agreed upon, then fill in the gaps with some needed details: how often, how to do it, should there be an opt out, and how to implement that, etc.
Le jeudi 27 juin 2019 à 12:26 -0600, Chris Murphy a écrit :
On Thu, Jun 27, 2019 at 9:06 AM Bruno Wolff III bruno@wolff.to wrote:
On Wed, Jun 26, 2019 at 14:19:26 -0600, Chris Murphy lists@colorremedies.com wrote:
Short version: Fedora should take responsibility for the bootloader being up to date, by updating it during major version upgrades. This is already the case on UEFI with conventional installations. I'd like to make sure it always happens on major version upgrades for BIOS installations. What's the problem? This would step on any custom bootloader configuration a user has for multiboot. There is no reasonable mechanism on BIOS systems to determine if the bootloader is a Fedora bootloader, and somehow only update a Fedora bootloader and not touch any other bootloader.
How do you envision this working for rawhide? I had a problem where grub1 configs were no longer updated with kernel updates, where I finally needed to upgrade to grub2.
Yep, small problem. And I'm not even sure how a 'grub2-install' on BIOS systems would be initiated only at major upgrade time. But even Fedora Rawhide does change versions when fcXX becomes fcXX+1, but is that a sane trigger?
It's not, such magic “it's FCX time” break all the time for users that only do dnf update, or use rawhide (which branches before the next version logic is deployed), etc
I'm not sure what the alternatives are.
That's, easy,
1. add a generic bootctl install command that knows the different variants of bootloader used in Fedora, how to install them, how to identify which variant is appropriate for a system (make grub and other bootloaders packages install the corresponding info in a directory read by this command)
2. make it write the id, generation, and deployment options used in a lockfile every time it (re)installs the bootloader
3. add a config file with a variable to inhibit auto-redeployment
4. add a scriptlet to the bootctl package that calls bootctl install and – checks the variant in the lockfile is appropriate for the hardware (in case of disk mode or copy) – checks if the generation is current or unknown (unknown = future, do not touch) – and if any of those is false, reinstalls the bootloader, unless the inhibit variable is set
5. add a --force switch to bootctl to allow the operator to force a bootloader rewrite every time somethins else broke it
On Fri, 2019-06-28 at 09:18 +0200, Nicolas Mailhot via devel wrote:
That's, easy,
- add a generic bootctl install command that knows the different
variants of bootloader used in Fedora, how to install them, how to identify which variant is appropriate for a system (make grub and other bootloaders packages install the corresponding info in a directory read by this command)
I am not sure you quite understand the meaning of the word "easy". :P
Le vendredi 28 juin 2019 à 09:06 -0700, Adam Williamson a écrit :
On Fri, 2019-06-28 at 09:18 +0200, Nicolas Mailhot via devel wrote:
That's, easy,
- add a generic bootctl install command that knows the different
variants of bootloader used in Fedora, how to install them, how to identify which variant is appropriate for a system (make grub and other bootloaders packages install the corresponding info in a directory read by this command)
I am not sure you quite understand the meaning of the word "easy". :P
It's the same core logic that already exists in anaconda, and in the envisionned dist upgrade, except packaged in a sane reusable easy to test unit.
Packaging it sanely does not make the core logic simpler sure, but it removes the mass of problems caused by bootloader installations happening in multiple badly defined in-between gray dimensions, without any tracking of what was effectively done in the past.
And the core logic need to be maintained in any case.
So yes, I do believe that quick and dirty is more difficult in the long run.
On Fri, Jun 28, 2019, 1:19 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le jeudi 27 juin 2019 à 12:26 -0600, Chris Murphy a écrit :
Yep, small problem. And I'm not even sure how a 'grub2-install' on BIOS systems would be initiated only at major upgrade time. But even Fedora Rawhide does change versions when fcXX becomes fcXX+1, but is that a sane trigger?
It's not, such magic “it's FCX time” break all the time for users that only do dnf update, or use rawhide (which branches before the next version logic is deployed), etc
Twice a year for Rawhide. And upon initiation of major upgrade on release versions. It wouldn't be all the time.
I'm not sure what the alternatives are.
That's, easy,
- add a generic bootctl install command that knows the different
variants of bootloader used in Fedora, how to install them, how to identify which variant is appropriate for a system (make grub and other bootloaders packages install the corresponding info in a directory read by this command)
- make it write the id, generation, and deployment options used in a
lockfile every time it (re)installs the bootloader
add a config file with a variable to inhibit auto-redeployment
add a scriptlet to the bootctl package that calls bootctl install
and – checks the variant in the lockfile is appropriate for the hardware (in case of disk mode or copy) – checks if the generation is current or unknown (unknown = future, do not touch) – and if any of those is false, reinstalls the bootloader, unless the inhibit variable is set
- add a --force switch to bootctl to allow the operator to force a
bootloader rewrite every time somethins else broke it
I think that's completely out of scope. It's inappropriate to wait 10 months let alone 10 years to resolve this problem. And it's an overly complicated solution. It sounds like the bootloader install equivalent of grubby that was just deprecated in Fedora 30, not least of which it was so overly complicated it was difficult getting people to contribute to or maintain it. Yet another one command to rule them all? No thanks.
It might be sane for Anaconda to write out a file, or modify /etc/default/grub, to indicate the user chose to prevent the Fedora bootloader from being installed (as far as I'm aware this only actually works on x86 BIOS computers even though it's presented in the UI for all other firmwares and archs) when Fedora was installed. And from that information, infer that they do not want the Fedora bootloader (re)installed.
-- Chris Murphy
Le vendredi 28 juin 2019 à 18:49 -0600, Chris Murphy a écrit :
On Fri, Jun 28, 2019, 1:19 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le jeudi 27 juin 2019 à 12:26 -0600, Chris Murphy a écrit :
Yep, small problem. And I'm not even sure how a 'grub2-install' on BIOS systems would be initiated only at major upgrade time. But even Fedora Rawhide does change versions when fcXX becomes fcXX+1, but is that a sane trigger?
It's not, such magic “it's FCX time” break all the time for users that only do dnf update, or use rawhide (which branches before the next version logic is deployed), etc
Twice a year for Rawhide. And upon initiation of major upgrade on release versions. It wouldn't be all the time.
I'm not sure what the alternatives are.
That's, easy,
- add a generic bootctl install command that knows the different
variants of bootloader used in Fedora, how to install them, how to identify which variant is appropriate for a system (make grub and other bootloaders packages install the corresponding info in a directory read by this command)
- make it write the id, generation, and deployment options used in
a lockfile every time it (re)installs the bootloader
add a config file with a variable to inhibit auto-redeployment
add a scriptlet to the bootctl package that calls bootctl
install and – checks the variant in the lockfile is appropriate for the hardware (in case of disk mode or copy) – checks if the generation is current or unknown (unknown = future, do not touch) – and if any of those is false, reinstalls the bootloader, unless the inhibit variable is set
- add a --force switch to bootctl to allow the operator to force a
bootloader rewrite every time somethins else broke it
I think that's completely out of scope. It's inappropriate to wait 10 months let alone 10 years to resolve this problem. And it's an overly complicated solution.
It's not an overcomplicated solution it's just putting a single stable unchanging installation command in front of whatever anaconda does. Which grubby was not since it was grub-specific with lots of options.
Not making the effort to put this single command in place, is the reason why Fedora boot problems in 2019, are the same than RHL boot problems in 2000:
1. no one knows exactly what to call to reinstall the bootloader except bootloader people (it changes and is badly documented),
2. therefore no one can test the result on the scale needed for the variety of hardware out there
3. when someone hits a bootloader problem, and tries to fix his system with whatever lies on the internet (because the documentation is not here, and the documentation is needed, because the actual commands to type change), it's never exactly what bootloader people think he should do, so the result is invalid (positive or negative), and not used to improve the way Fedora does things
4. because there is no master command, and the actual commands change over time, there is no incentive to do something in new sets of commands, that salvages whatever was done in the previous set of commands
5. it ends up in new full system installs just to get anaconda to the point it does this part (and RHL supported in-place updates, while Fedora will insist on blowing up the previous install)
6. and everyone involved insists it's "too complex" to set up an easy to document master install command
7. something should be put in place "now" and not wait 10 months or 10 years
Well Fedora and RHL have spent 20 years avoiding cleaning up this mess and it's still the same original mess so at one point 10 months seems awfully short period to get it done.
Regards,
I don't have the technical expertise in this area to comment on the details of the implementation, but from a dumb user point of view, could there be a simple or no-payload package that says "Fedora owns the bootloader" that's installed by default and power users that wish to maintain it themselves (or use other bootloaders) and remove the package?
Thanks, Richard
On Sat, Jun 29, 2019 at 5:24 AM Richard Shaw hobbes1069@gmail.com wrote:
I don't have the technical expertise in this area to comment on the details of the implementation, but from a dumb user point of view, could there be a simple or no-payload package that says "Fedora owns the bootloader" that's installed by default and power users that wish to maintain it themselves (or use other bootloaders) and remove the package?
I like this idea. Would bootloader projects be willing to use a single canonical location and format for this indication? On the other hand, if I don't want to use the Fedora GRUB bootloader, why do I have that package still installed? If I were using the Ubuntu bootloader, or systemd-boot, I would uninstall all the GRUB stuff to ensure it's not being used, accidentally or otherwise.
Nevertheless, I'm OK with the idea that BIOS GRUB not be automatically reinstalled everytime there's a package update, but rather postpone it for major Fedora upgrades.
On Sat, Jun 29, 2019 at 1:05 AM Nicolas Mailhot via devel devel@lists.fedoraproject.org wrote:
Le vendredi 28 juin 2019 à 18:49 -0600, Chris Murphy a écrit :
I think that's completely out of scope. It's inappropriate to wait 10 months let alone 10 years to resolve this problem. And it's an overly complicated solution.
It's not an overcomplicated solution it's just putting a single stable unchanging installation command in front of whatever anaconda does. Which grubby was not since it was grub-specific with lots of options.
grubby isn't GRUB specific. It supports LILO, ELILO, SILO, ZIPL, extlinux, GRUB legacy, GRUB2, yaboot, and something else I'm forgetting no doubt.
This also seems applicable https://xkcd.com/927/
Not making the effort to put this single command in place, is the reason why Fedora boot problems in 2019, are the same than RHL boot problems in 2000:
a. Near as I can tell, the only system under discussion, x86 + BIOS + multiboot, means exactly one bootloader: BIOS GRUB. That's the only bootloader Fedora supports in that configuration.
b. Fedora only narrowly supports multiboot: Fedora + Windows, and Fedora + macOS. As in, only the dual boot variety, and not any Linux distros. Not even Fedora + Fedora is supported. By that I mean, there is no release criteria that forms a basis for blocking the release if something related to creating such a setup during installation is found to be broken.
I just don't see how a single command to rule them all solves a single problem. It's like playing musical chairs, except no seat has been removed.
- no one knows exactly what to call to reinstall the bootloader except
bootloader people (it changes and is badly documented),
Fedora x86 BIOS, it's been 'grub2-install' since circa Fedora 14. And on UEFI since early days it's reinstall shim and grub packages, also hasn't changed.
- therefore no one can test the result on the scale needed for the
variety of hardware out there
- when someone hits a bootloader problem, and tries to fix his system
with whatever lies on the internet (because the documentation is not here, and the documentation is needed, because the actual commands to type change), it's never exactly what bootloader people think he should do, so the result is invalid (positive or negative), and not used to improve the way Fedora does things
But in effect you're proposing the inevitable day when the user still invokes grub2-install, and the Fedora oracles still say "yeah that's deprecated on Fedora, you ran the wrong command, that's why your system is now hosed"
By the way, bootctl is already taken by systemd-boot.
-- Chris Murphy
Hello Chris,
Thanks for bringing this topic and sorry for the late reply.
On Wed, Jun 26, 2019 at 10:20 PM Chris Murphy lists@colorremedies.com wrote:
Hi,
This is not a formal proposal, this is for discussion and identifying liabilities. This email has an x86 GRUB bias only because that's the bootloader regime I'm most familiar with. I think it should apply to other archs as well, i.e. their bootloaders shouldn't be permitted to become stale.
I agree with you on this. I've commented my thoughts in the fedora-workstation issue you created:
https://pagure.io/fedora-workstation/issue/97#comment-580794
Best regards, Javier