Two feature requests, comments please.
1. EFI System partition is being mounted persistently at /boot/efi, and I'd like to put an end to that because there's no good reason to do it. None of the binaries on it are regularly being updated, and if they are, the volume should be mounted on demand rather than persistently. The grub.cfg on each ESP should be a simple static grub.cfg like this: # cat /boot/efi/EFI/fedora/grub.cfg # # DO NOT EDIT THIS FILE # THIS FORWARDS TO THE REAL GRUB.CFG # { search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt1 --hint-efi=hd1,gpt1 10044fb4-8f90-4ca6-888c-98a0c88169e2 configfile /boot/grub2/grub.cfg }
And then we update /boot/grub2/grub.cfg just like on BIOS systems, on a file system that can tolerate some abuse, unlike FAT. It also means there's some chance of resiliently bootable raid on UEFI systems which right now isn't assured because we don't create ESP's on each system disk chosen, let alone have a way of syncing the grub.cfg's on each of those ESPs. Nor a way to mount them all at the same time anyway.
Last, Windows and OS X don't keep their ESP or boot volumes persistently mounted, I don't know what good reason we have for doing this.
2. Right now we aren't doing an fsck on the single ESP we do mount. If we're going to continue to mount it on every boot despite the arguments against doing so, shouldn't /etc/fstab fspassno be set to 1 for /boot/efi? I have experienced ESP file system corruption and /boot/efi wouldn't mount as a result, which hangs the system and eventually drops it to an obscure emergency shell.
Affected by this change are anaconda to set fspassno to 1; dracut to include mkfs.msdos initramfs; and possibly to fsck so that it runs mkfs.msdos with -a which currently it doesn't do, instead it runs it interactively which isn't suitable for boot time fixing. If we don't have the will to be checking and fixing a file system critical for booting, maybe we shouldn't be persistently mounting it rw (back to 1 above)?
3. Currently Fedora anaconda installs result in a FAT16 ESP which also isn't done on Windows or OS X, except for removable media. My reading of the UEFI spec says that on system drives (non-removables), it should be FAT32. I don't know if it's really a problem that we're using FAT16, but figured I'd point it out here in case someone has two cents to add on this. A bug has been filed already. https://bugzilla.redhat.com/show_bug.cgi?id=1046577
Chris Murphy
On 18 March 2014 07:02, Chris Murphy lists@colorremedies.com wrote:
<snip>
- EFI System partition is being mounted persistently at /boot/efi, and
I'd like to put an end to that because there's no good reason to do it. None of the binaries on it are regularly being updated, and if they are, the volume should be mounted on demand rather than persistently.
</snip>
Exactly; we get frequent power cuts where I live, all my filesystems can recover well (ext4 with journaling) except the ESP, which works, yes, but keeps spewing an error about not being unmounted properly. As a result I have changed the ESP mount options to add "noauto,x-systemd.automount", I usually don't need to mount it at all.
On Mon, 17.03.14 23:02, Chris Murphy (lists@colorremedies.com) wrote:
- EFI System partition is being mounted persistently at /boot/efi,
and I'd like to put an end to that because there's no good reason to do it. None of the binaries on it are regularly being updated, and if they are, the volume should be mounted on demand rather than persistently.
So, systemd actually contains a logic that will automatically create an autofs mount point on /boot for the ESP that has been used for booting. This logic is automatically disabled if /boot is non-empty or if another partition is listed for /boot in /etc/fstab. This setup makes things simple, and avoids mucking with the ESP if it is not actually accessed, but still makes it available without any manual mounting.
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
I think mounting the ESP into a subdir of the boot partition is an awkward choice. Among other things this makes it impossible to make both /boot and /boot/efi autofs file systems. I wished Fedora wouldn't nest these mounts...
- Right now we aren't doing an fsck on the single ESP we do mount. If
we're going to continue to mount it on every boot despite the arguments against doing so, shouldn't /etc/fstab fspassno be set to 1 for /boot/efi? I have experienced ESP file system corruption and /boot/efi wouldn't mount as a result, which hangs the system and eventually drops it to an obscure emergency shell.
Affected by this change are anaconda to set fspassno to 1; dracut to include mkfs.msdos initramfs; and possibly to fsck so that it runs mkfs.msdos with -a which currently it doesn't do, instead it runs it interactively which isn't suitable for boot time fixing. If we don't have the will to be checking and fixing a file system critical for booting, maybe we shouldn't be persistently mounting it rw (back to 1 above)?
Yeah, Fedora really should set passno to 1 for all physical file systems it mounts, and that certainly includes the ESP.
(The ESP logic in systemd mentioned above, of course does pull in an fsck by default for the ESP).
Lennart
On Mar 18, 2014, at 10:13 AM, Lennart Poettering mzerqung@0pointer.de wrote:
On Mon, 17.03.14 23:02, Chris Murphy (lists@colorremedies.com) wrote:
- EFI System partition is being mounted persistently at /boot/efi,
and I'd like to put an end to that because there's no good reason to do it. None of the binaries on it are regularly being updated, and if they are, the volume should be mounted on demand rather than persistently.
So, systemd actually contains a logic that will automatically create an autofs mount point on /boot for the ESP that has been used for booting. This logic is automatically disabled if /boot is non-empty or if another partition is listed for /boot in /etc/fstab. This setup makes things simple, and avoids mucking with the ESP if it is not actually accessed, but still makes it available without any manual mounting.
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
It looks like the existence of the /boot/efi line in fstab causes boot-efi.mount job to be created; and x-systemd.automount adds an .automount job while not removing .mount job. The .mount job causes it to be mounted anyway. If this is unintended I can post to systemd list or file a bug.
Yeah, Fedora really should set passno to 1 for all physical file systems it mounts, and that certainly includes the ESP.
I just tested setting it for /boot/efi, marking the volume dirty, and rebooting. It does automatically get fixed so the only change here after all is anaconda to set fspassno to 1. So I filed this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1077917
Strictly speaking, per the fstab man page, fspassno should be 0 for XFS and Btrfs since those file systems don't have an fsck, certainly not a non-interactive one designed to be used at boot time. Systemd is smart enough not to try to run fsck on Btrfs. While xfsprogs installs a "do nothing, successfully" fsck.xfs that gets used at boot.
Chris Murphy
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
Lennart
On Tue, Mar 18, 2014 at 5:19 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
It seems like 'ls /boot/efi' shouldn't be enough to trigger a mount -- the poi nt is that /boot/efi should stay unmounted unless there's a genuine need to mount it. So just plain noauto might be good enough here (i.e. without the automount).
I'm usually a fan of giving mountpoints mode 000 to avoid accidentally using them when unmounted, but that doesn't really do anything for root.
--Andy
On Tue, 2014-03-18 at 17:27 -0700, Andrew Lutomirski wrote:
On Tue, Mar 18, 2014 at 5:19 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
It seems like 'ls /boot/efi' shouldn't be enough to trigger a mount -- the poi nt is that /boot/efi should stay unmounted unless there's a genuine need to mount it. So just plain noauto might be good enough here (i.e. without the automount).
Um. What? How would it then get automounted?
You need 'noauto' to tell systemd not to mount it on boot, and 'x-systemd-automount' (when did that change from 'comment=systemd.automount'?) to tell systemd to automount it on access.
On Tue, Mar 18, 2014 at 6:07 PM, Adam Williamson awilliam@redhat.com wrote:
On Tue, 2014-03-18 at 17:27 -0700, Andrew Lutomirski wrote:
On Tue, Mar 18, 2014 at 5:19 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
It seems like 'ls /boot/efi' shouldn't be enough to trigger a mount -- the poi nt is that /boot/efi should stay unmounted unless there's a genuine need to mount it. So just plain noauto might be good enough here (i.e. without the automount).
Um. What? How would it then get automounted?
Exactly. What needs it to automount and why? Isn't Chris's main point that Fedora shouldn't neet to mount the ESP in the first place?
I can see an argument for (auto)mounting it readonly, though.
You need 'noauto' to tell systemd not to mount it on boot, and 'x-systemd-automount' (when did that change from 'comment=systemd.automount'?) to tell systemd to automount it on access. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
On Tue, 2014-03-18 at 20:52 -0700, Andrew Lutomirski wrote:
Exactly. What needs it to automount and why?
Kernel updates.
Isn't Chris's main point that Fedora shouldn't neet to mount the ESP in the first place?
In a perfect world, yes, but right now, the grub config file is on it. If you were saying 'in a perfect world we wouldn't have to mount it at all', sure.
On Mar 18, 2014, at 6:27 PM, Andrew Lutomirski luto@mit.edu wrote:
On Tue, Mar 18, 2014 at 5:19 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
It seems like 'ls /boot/efi' shouldn't be enough to trigger a mount -- the poi nt is that /boot/efi should stay unmounted unless there's a genuine need to mount it. So just plain noauto might be good enough here (i.e. without the automount).
I'm usually a fan of giving mountpoints mode 000 to avoid accidentally using them when unmounted, but that doesn't really do anything for root.
I agree with both points, however it takes additional work first. Otherwise if the ESP isn't (auto) mounted when doing a kernel update today, the updating of /boot/efi/EFI/fedora/grub.cfg will fail. And I believe gummiboot scripts also are stored on the ESP.
My motivation is to avoid complex and fragile configurations, (re)enable resiliently bootable multiple device installs, and not require the user to know esoteric things about UEFI's requirements or differences. UEFI comes with a lot more baggage that's inherently going to burden someone. And I'm arguing in favor of not shifting this burden onto the user.
Chris Murphy
On Mar 18, 2014, at 6:19 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Tue, 18.03.14 15:07, Chris Murphy (lists@colorremedies.com) wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
When I add x-systemd.automount to fstab for /boot/efi, it still gets mounted on every boot.
Ah, yeah sorry, forgot to mention, you need to also add "noauto" to the line. If it is "auto" we'll still wait for the mount unit to complete.
Basically, combining x-systemd.automount + auto is just a away to speed up boot by fscking in the bg while the mount point is already established. After boot the file system will be mounted as if x-systemd.automount hadn't been used.
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
That does work. It's automounted on any command I threw at it, including kernel install, update, or remove. It's a bit half-hearted of a solution for the problem, since it doesn't umount the volume. But since the usual case is that it wouldn't be accessed except for a kernel update (due to the overwriting of grub.cfg), after which a reboot follows, it's seems like a decent short term work around.
RFE: Do not persistently mount EFI System partition at /boot/efi https://bugzilla.redhat.com/show_bug.cgi?id=1077984
It's still better to remove the on-going writing of configuration files to the ESP, however. A simple one-time forwarding-configuration file pointing to the /boot volume UUID, permits configuration files to be written somewhere on /boot, which can then be md raid1 or btrfs raid1 based. Boot is made more resilient whether single or multiple disk. This works today on BIOS, but not on UEFI.
Chris Murphy
On Tue, 2014-03-18 at 21:39 -0600, Chris Murphy wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
RFE: Do not persistently mount EFI System partition at /boot/efi https://bugzilla.redhat.com/show_bug.cgi?id=1077984
It's still better to remove the on-going writing of configuration files to the ESP, however. A simple one-time forwarding-configuration file pointing to the /boot volume UUID, permits configuration files to be written somewhere on /boot, which can then be md raid1 or btrfs raid1 based. Boot is made more resilient whether single or multiple disk. This works today on BIOS, but not on UEFI.
Why not also extend this to /boot also? It's "rarely" used in day to day on a system, really only for yum updates that include a kernel.
[root@strawberry ~]# lsof | grep /boot [root@strawberry ~]#
On Mar 18, 2014, at 10:31 PM, William Brown william@firstyear.id.au wrote:
On Tue, 2014-03-18 at 21:39 -0600, Chris Murphy wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
RFE: Do not persistently mount EFI System partition at /boot/efi https://bugzilla.redhat.com/show_bug.cgi?id=1077984
It's still better to remove the on-going writing of configuration files to the ESP, however. A simple one-time forwarding-configuration file pointing to the /boot volume UUID, permits configuration files to be written somewhere on /boot, which can then be md raid1 or btrfs raid1 based. Boot is made more resilient whether single or multiple disk. This works today on BIOS, but not on UEFI.
Why not also extend this to /boot also? It's "rarely" used in day to day on a system, really only for yum updates that include a kernel.
Sure. I just prefer being charged with one heresy at a time, and that particular one is much bigger than /boot/efi.
Unfortunately it may be too late to change this for RHEL 7, which means an ESP always mounted at /boot/efi may become a glorified tradition in any case.
Chris Murphy
On Wed, 19.03.14 15:01, William Brown (william@firstyear.id.au) wrote:
Why not also extend this to /boot also? It's "rarely" used in day to day on a system, really only for yum updates that include a kernel.
[root@strawberry ~]# lsof | grep /boot [root@strawberry ~]#
To establish the automount point /boot/efi you need /boot mounted. You cannot have an automount point nested into an automount point.
It's one of the reasons why I really really dislike the invention of /boot/efi as the mount point for the ESP...
Lennart
On Mar 19, 2014, at 7:02 AM, Lennart Poettering mzerqung@0pointer.de wrote:
It's one of the reasons why I really really dislike the invention of /boot/efi as the mount point for the ESP…
I agree, although I go farther. The EFI System partition doesn't scale, isn't resilient, can neither be mirrored nor easily sync'd (multidevice boot). It should be considered a pre-boot and OS installer domain only.
Are bootloader updates necessary? On BIOS this is ignored, no updates effectively happen, even if the grub package is update, grub2-install isn't invoked so no update really occurs to the bootloader.
On UEFI it's the opposite. An updated grub-efi package causes grub<arch>.efi to be overwritten, and invoking grub2-install will break Secure Boot systems.
If UEFI bootloader updates are considered necessary then we need a better way than assuming there's only one ESP, by only updating the one at /boot/efi. Because that's not necessarily the one being executed by the firmware. The bootloader RPM would need to mount all ESP's on the system, replace-existing, unmount.
So whether yes/no to bootloader updates, /boot/efi either isn't needed or doesn't meet the requirements.
Chris Murphy
On Wed, 19.03.14 13:13, Chris Murphy (lists@colorremedies.com) wrote:
I agree, although I go farther. The EFI System partition doesn't scale, isn't resilient, can neither be mirrored nor easily sync'd (multidevice boot). It should be considered a pre-boot and OS installer domain only.
You know, the ESP is actually FAT, one of the simplest, best-understood, most used, and most stable file systems around. Sure, one can always break things, but it is simply misguided to believe that this is the part that will likely break and that we really really really need to stay away from, and not the later parts of the boot that involve grub and raid, and yuck.
You know, by creating a chain of many steps where you first go for the ESP, and then follow that by another boot partition and so on, you just make things more complex.
If you want things robust, make them simple. Sure, keep writing to the ESP at a minimum, but don't play games of just moving those writes to another boot partition that is a lot more fragile. You are not helping anyone with that...
Lennart
On Wed, Mar 19, 2014 at 3:53 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Wed, 19.03.14 13:13, Chris Murphy (lists@colorremedies.com) wrote:
I agree, although I go farther. The EFI System partition doesn't scale, isn't resilient, can neither be mirrored nor easily sync'd (multidevice boot). It should be considered a pre-boot and OS installer domain only.
You know, the ESP is actually FAT, one of the simplest, best-understood, most used, and most stable file systems around. Sure, one can always break things, but it is simply misguided to believe that this is the part that will likely break and that we really really really need to stay away from, and not the later parts of the boot that involve grub and raid, and yuck.
You know, by creating a chain of many steps where you first go for the ESP, and then follow that by another boot partition and so on, you just make things more complex.
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
Grub 1 on MD RAID actually just works, once you figure out the incantation to install it. It would be a shame if that ability got lost in the name of simplicity.
On Wed, 2014-03-19 at 15:57 -0700, Andrew Lutomirski wrote:
On Wed, Mar 19, 2014 at 3:53 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Wed, 19.03.14 13:13, Chris Murphy (lists@colorremedies.com) wrote:
I agree, although I go farther. The EFI System partition doesn't scale, isn't resilient, can neither be mirrored nor easily sync'd (multidevice boot). It should be considered a pre-boot and OS installer domain only.
You know, the ESP is actually FAT, one of the simplest, best-understood, most used, and most stable file systems around. Sure, one can always break things, but it is simply misguided to believe that this is the part that will likely break and that we really really really need to stay away from, and not the later parts of the boot that involve grub and raid, and yuck.
You know, by creating a chain of many steps where you first go for the ESP, and then follow that by another boot partition and so on, you just make things more complex.
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it. The anaconda devs have thought about this, and are planning to implement it. On the UEFI side you just write entries for each of the ESPs into the EFI boot manager. If one of them fails, then the firmware will boot from the next in line.
On Mar 19, 2014, at 7:51 PM, Adam Williamson awilliam@redhat.com wrote:
On Wed, 2014-03-19 at 15:57 -0700, Andrew Lutomirski wrote:
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it.
This shouldn't be done. It's a source of corruption to treat an md raid1 device as a plain volume, rather than as a degraded md device. If the plain partition volume is modified, its mdadm metadata won't be updated, so when the array is finally assembled, mdadm has no idea member devices are not sync'd, and also has no way to resolve the ambiguity.
Because of this, at least one raid stack kernel maintainer wants to do away with the use of mdadm metadata v 1.0 by anaconda. It's v1.0 metadata that permits this plain old partition treatment as a side effect.
v1.2 metadata is preferred since it compels the proper treatment of individual member devices as md devices not plain partitions. The metadata is offset 4KB by the start, so only something that will treat it as an md device (normally or degraded assembly) will be able to use it. And hence not useable by UEFI firmware.
Therefore I don't think software RAID is a solution. It's more complicated and problem prone than what I've suggested by obviating the on-going need to update the ESP in the first place. Instead we should treat the ESP like the MBR gap, and BIOS Boot: Write once, forget about it. (Short of some critical security update.)
On the UEFI side you just write entries for each of the ESPs into the EFI boot manager. If one of them fails, then the firmware will boot from the next in line.
The firmware itself will do a fallback even without specific NVRAM entries, and will eventually land on one of the ESP's /EFI/BOOT/BOOTX64.efi's.
Chris Murphy
On 03/20/2014 01:21 AM, Chris Murphy wrote:
On Mar 19, 2014, at 7:51 PM, Adam Williamson awilliam@redhat.com wrote:
On Wed, 2014-03-19 at 15:57 -0700, Andrew Lutomirski wrote:
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it.
This shouldn't be done. It's a source of corruption to treat an md raid1 device as a plain volume, rather than as a degraded md device. If the plain partition volume is modified, its mdadm metadata won't be updated, so when the array is finally assembled, mdadm has no idea member devices are not sync'd, and also has no way to resolve the ambiguity.
Right, but UEFI doesn't write to it----it just uses them to read the boot content. It would be better if UEFI could read software raid1 even if it was degraded due to disk failures, but apparently it can't, so Adam's scheme is the only possibility.
I agree with you that nothing else but UEFI should mount those devices as individual plain volumes, of course.
Because of this, at least one raid stack kernel maintainer wants to do away with the use of mdadm metadata v 1.0 by anaconda. It's v1.0 metadata that permits this plain old partition treatment as a side effect.
It is a darn useful side effect. How else can you implement redundant boot that is transparently updated? I could (and did) try multiple /boot partitions on separate drives by copying the partition content after updates, but it was fragile and I was never confident that it would work when I needed it. Adam's raid1 /boot just seems more reliable, especially if it became a designed feature.
On Thu, Mar 20, 2014 at 9:32 AM, Przemek Klosowski przemek.klosowski@nist.gov wrote:
On 03/20/2014 01:21 AM, Chris Murphy wrote:
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it.
This shouldn't be done. It's a source of corruption to treat an md raid1 device as a plain volume, rather than as a degraded md device. If the plain partition volume is modified, its mdadm metadata won't be updated, so when the array is finally assembled, mdadm has no idea member devices are not sync'd, and also has no way to resolve the ambiguity.
Right, but UEFI doesn't write to it----it just uses them to read the boot content. It would be better if UEFI could read software raid1 even if it was degraded due to disk failures, but apparently it can't, so Adam's scheme is the only possibility.
Um.
EFI_FILE_PROTOCOL's Write method sounds suspiciously like it writes to a filesystem.
Because of this, at least one raid stack kernel maintainer wants to do away with the use of mdadm metadata v 1.0 by anaconda. It's v1.0 metadata that permits this plain old partition treatment as a side effect.
It is a darn useful side effect. How else can you implement redundant boot that is transparently updated? I could (and did) try multiple /boot partitions on separate drives by copying the partition content after updates, but it was fragile and I was never confident that it would work when I needed it. Adam's raid1 /boot just seems more reliable, especially if it became a designed feature.
I think you may be confusing /boot and ESP (aka /boot/efi). They are not the same thing.
You get a transparently updated list of kernels by doing it exactly the way that Chris suggested. You have /boot on md raid with metadata 1.1 or 1.2. Or you have /boot on any other sensible RAID backing store, where sensible means that everything that mounts it knows that it's RAID.
Then you stick something *that never changes* into each copy of the ESP. And you don't even pretend that ESP is RAID -- it's simply a bunch of FAT filesystems that *are not mounted read/write* except for when the bootloader is reinstalled. And you do *not* reinstall the bootloader just because you have a new kernel.
All of the RAID devices are only ever accessed by RAID drivers. All of the ESP non-RAID filesystems are treated as the crappy FAT filesystems that they are and are almost never written by Linux userspace.
--Andy
On Thu, 2014-03-20 at 09:49 -0700, Andrew Lutomirski wrote:
Um.
EFI_FILE_PROTOCOL's Write method sounds suspiciously like it writes to a filesystem.
Sure, UEFI has the capability, but it's not going to be used when simply booting the system normally. All the firmware does in that case is mount the partition and execute the bootloader it finds there.
On Thu, 2014-03-20 at 12:32 -0400, Przemek Klosowski wrote:
Adam's scheme is the only possibility.
Adam's raid1 /boot just seems more reliable, especially if it became a designed feature.
It's not my plan, it's the anaconda developers'. I only described it. Actually it took them like 15 minutes to get it into my dumb brain. :)
On Thu, Mar 20, 2014 at 10:01 AM, Adam Williamson awilliam@redhat.com wrote:
On Thu, 2014-03-20 at 12:32 -0400, Przemek Klosowski wrote:
Adam's scheme is the only possibility.
Adam's raid1 /boot just seems more reliable, especially if it became a designed feature.
It's not my plan, it's the anaconda developers'. I only described it. Actually it took them like 15 minutes to get it into my dumb brain. :)
Can you all please try to make sure you're talking about the same thing?
I believe that Chris is suggesting /boot as raid1 and ESP not mounted.
Adam and Przemek seem to be talking about /boot on RAID1 (I've never heard anyone say that /boot as RAID is a bad idea) but I think something's very confused about my understanding of your opinions about how the ESP should or does work. Can one or both of you please describe, unambiguously, how you think the ESP should be created, when and if it should be mounted, what filesystem and/or raid config should be used to mount it, and what should happen when a kernel is updated?
Sure, UEFI has the capability, but it's not going to be used when simply booting the system normally. All the firmware does in that case is mount the partition and execute the bootloader it finds there.
Why not? It's completely safe when the OS is Mac OS or Windows. It's sometimes safe when the OS is Linux. It's even possibly useful: you might want to have an ESP bootloader, shim, or whatever that logs errors. I bet there's at least one UEFI firmware out there that backs up settings to ESP and/or backs up a whole firmware image to ESP.
It would be a shame for a Linux "RAID" install to corrupt the ESP just because you did something unusual in various UEFI menus.
--Andy
On Thu, Mar 20, 2014 at 10:07 AM, Andrew Lutomirski luto@mit.edu wrote:
On Thu, Mar 20, 2014 at 10:01 AM, Adam Williamson awilliam@redhat.com wrote:
Sure, UEFI has the capability, but it's not going to be used when simply booting the system normally. All the firmware does in that case is mount the partition and execute the bootloader it finds there.
Why not? It's completely safe when the OS is Mac OS or Windows. It's sometimes safe when the OS is Linux. It's even possibly useful: you might want to have an ESP bootloader, shim, or whatever that logs errors. I bet there's at least one UEFI firmware out there that backs up settings to ESP and/or backs up a whole firmware image to ESP.
It would be a shame for a Linux "RAID" install to corrupt the ESP just because you did something unusual in various UEFI menus.
The standard EFI Shell can do exactly this. It has commands like 'edit' and 'mkdir'. I think that using them should not cause filesystem corruption.
--Andy
On Thu, 2014-03-20 at 10:07 -0700, Andrew Lutomirski wrote:
On Thu, Mar 20, 2014 at 10:01 AM, Adam Williamson awilliam@redhat.com wrote:
On Thu, 2014-03-20 at 12:32 -0400, Przemek Klosowski wrote:
Adam's scheme is the only possibility.
Adam's raid1 /boot just seems more reliable, especially if it became a designed feature.
It's not my plan, it's the anaconda developers'. I only described it. Actually it took them like 15 minutes to get it into my dumb brain. :)
Can you all please try to make sure you're talking about the same thing?
I believe that Chris is suggesting /boot as raid1 and ESP not mounted.
He is also giving feedback on the ESP-as-RAID1 idea. He's both doing that *and* proposing an alternative, which may be what you're missing. I don't think anyone but you is confused about the alternative ideas here.
Adam and Przemek seem to be talking about /boot on RAID1 (I've never heard anyone say that /boot as RAID is a bad idea) but I think something's very confused about my understanding of your opinions about how the ESP should or does work. Can one or both of you please describe, unambiguously, how you think the ESP should be created, when and if it should be mounted, what filesystem and/or raid config should be used to mount it, and what should happen when a kernel is updated?
I thought I already did. Just go back and read my original mail again. And again, it's nothing about what "I think", I was simply describing a potential design which had been explained to me by someone else.
On Mar 20, 2014, at 10:32 AM, Przemek Klosowski przemek.klosowski@nist.gov wrote:
It is a darn useful side effect. How else can you implement redundant boot that is transparently updated?
For /boot, rather than /boot/efi, use metadata v1.2 and a bootloader that understands v1.2 metadata. GRUB2 has for some time.
Chris Murphy
On Wed, 2014-03-19 at 23:21 -0600, Chris Murphy wrote:
Therefore I don't think software RAID is a solution.
Well, this is new: I'm fairly sure you were the one who filed a bug requesting it in the first damn place.
On Mar 20, 2014, at 10:56 AM, Adam Williamson awilliam@redhat.com wrote:
On Wed, 2014-03-19 at 23:21 -0600, Chris Murphy wrote:
Therefore I don't think software RAID is a solution.
Well, this is new: I'm fairly sure you were the one who filed a bug requesting it in the first damn place.
Where? I'm fairly certain I haven't ever suggested putting the ESP on RAID as a sync mechanism, when it's just one file that's usually changing.
I have said anaconda should a.) always create the ESP for us, even in custom partitioning; and b.) always create ESPs on each device selected as part of an install.
RFE: always create required bootloader partitions in custom partitioning https://bugzilla.redhat.com/show_bug.cgi?id=1022316
RFE: on multiple disk installs, always create bootloader partitions on all selected devices (was:resiliently bootable raid1 on UEFI can't be created) https://bugzilla.redhat.com/show_bug.cgi?id=1060576
Chris Murphy
On Wed, 19.03.14 18:51, Adam Williamson (awilliam@redhat.com) wrote:
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it. The anaconda devs have thought about this, and are planning to implement it. On the UEFI side you just write entries for each of the ESPs into the EFI boot manager. If one of them fails, then the firmware will boot from the next in line.
I'd really stay away from doing anything RAID related with the ESP.
We really shouldn't have the same partition multiple times with the same uuid and stuff. That is a call for trouble.
This is a really awful idea. Really, things like the pre-kernel boot logic should be kept simple, and what you are suggesting there is just frickin' crazy.
Lennart
On Thu, 2014-03-20 at 20:20 +0100, Lennart Poettering wrote:
On Wed, 19.03.14 18:51, Adam Williamson (awilliam@redhat.com) wrote:
More complex than trying to mirror a FAT ESP partition across multiple boot disks, keeping it properly synchronized, because RAID isn't supported?
You can in theory just have a bunch of RAID-1 (mirrored) ESPs, because of how RAID-1 works; each individual member can also be mounted as if it was just a plain old partition, which is how the firmware will mount it. The anaconda devs have thought about this, and are planning to implement it. On the UEFI side you just write entries for each of the ESPs into the EFI boot manager. If one of them fails, then the firmware will boot from the next in line.
I'd really stay away from doing anything RAID related with the ESP.
We really shouldn't have the same partition multiple times with the same uuid and stuff. That is a call for trouble.
This is a really awful idea. Really, things like the pre-kernel boot logic should be kept simple, and what you are suggesting there is just frickin' crazy.
Sigh, for the fourth time, I'm not suggesting it. It's not my idea. I'm not going to defend it. I'm relaying a design that was outlined to me by the anaconda devs. The motivation is that there have been multiple requests to make it possible to have a redundant boot chain on native UEFI installs, as you can on native BIOS installs.
On Mar 20, 2014, at 1:39 PM, Adam Williamson awilliam@redhat.com wrote:
Sigh, for the fourth time, I'm not suggesting it. It's not my idea. I'm not going to defend it. I'm relaying a design that was outlined to me by the anaconda devs. The motivation is that there have been multiple requests to make it possible to have a redundant boot chain on native UEFI installs, as you can on native BIOS installs.
Sure but the burden is on whatever installs to or modifies the ESP. Anaconda needs to create the required ESPs on each selected device, install the bootloader to each ESP, and then either:
a. mount and call grub2-mkconfig on each ESP in turn b. write a basic forwarding grub.cfg on each ESP, and grub2-mkconfig in the normal location /boot/grub2.
For kernel updates:
a. means either the kernel package or grubby must be capable of finding and mounting each ESP, and updating their grub.cfg's. b. means doing nothing new. Grubby updates grub.cfg at /boot/grub2, which is made redundant by virtue of whatever the underlying scheme is for the boot volume.
The b. path is easier for both single device and multiple device installs than what we have now; and it essentially eliminate needing to update or sync ESPs, and therefore no need to auto or persistently mount it. It kills multiple birds with one stone. (Sorry birds.)
Chris Murphy
On Mar 19, 2014, at 4:53 PM, Lennart Poettering mzerqung@0pointer.de wrote:
On Wed, 19.03.14 13:13, Chris Murphy (lists@colorremedies.com) wrote:
I agree, although I go farther. The EFI System partition doesn't scale, isn't resilient, can neither be mirrored nor easily sync'd (multidevice boot). It should be considered a pre-boot and OS installer domain only.
You know, the ESP is actually FAT, one of the simplest, best-understood, most used, and most stable file systems around.
Designed for floppies. Used today because it meets the resource requirements of camera and USB media manufacturers, not because it's good for users or their data. Otherwise it's abandoned. Windows, OS X, iOS, Android bootloaders read their kernels from more modern file systems. Apple's firmware directly reads HFS+ so they're not even reading their bootloader from a FAT fs.
FAT is mainly used for write-mostly or read-mostly use cases, where resilience, permissions, extended attributes, etc. aren't needed, and a small footprint for fs code is.
Sure, one can always break things, but it is simply misguided to believe that this is the part that will likely break and that we really really really need to stay away from, and not the later parts of the boot that involve grub and raid, and yuck.
It actually has broken for me more times in 9 months than all other file system corruptions I've had combined. (The corruptions were caused by crashes and forced power offs, but users have crashes and unplanned power failures.)
In no case did the firmware refuse to load the bootloader, but in a few cases the kernel would not mount the ESP at /boot/efi and since /boot/efi mount failed, the startup failed and I was dropped to an emergency shell. So yes, this is in fact more fragile, it's not some misguided hypothetical.
Aside from the corruption concern, there are regressions in actual functionality by putting the configuration files on the ESP.
You know, by creating a chain of many steps where you first go for the ESP, and then follow that by another boot partition and so on, you just make things more complex.
A chain of many steps?
BIOS->boot.img->core.img->/boot/grub2/i386/normal.mod->grub.cfg
UEFI->grubx64.efi->grub.cfg->grub.cfg
That's fewer steps, even with the second grub.cfg. There's no additional boot partition from what's previously been used either.
If you want things robust, make them simple. Sure, keep writing to the ESP at a minimum, but don't play games of just moving those writes to another boot partition that is a lot more fragile. You are not helping anyone with that…
Huh? Those writes have been on ext3/4 on Fedora for what, 10 years, up until UEFI? And now they're on FAT16. I'm suggesting moving those writes back to /boot on ext4. How is going back to the way it's been done on BIOS a lot more fragile?
Chris Murphy
On Wed, Mar 19, 2014 at 03:01:27PM +1030, William Brown wrote:
On Tue, 2014-03-18 at 21:39 -0600, Chris Murphy wrote:
Fedora takes a different approach though, and will mount an explicit boot partition to /boot and the ESP to /boot/efi, and do so unconditionally without involving autofs. Fedora could add "x-systemd-automount" to the mount options of /boot/efi, and thus turning /boot/efi into an autofs too.
RFE: Do not persistently mount EFI System partition at /boot/efi https://bugzilla.redhat.com/show_bug.cgi?id=1077984
It's still better to remove the on-going writing of configuration files to the ESP, however. A simple one-time forwarding-configuration file pointing to the /boot volume UUID, permits configuration files to be written somewhere on /boot, which can then be md raid1 or btrfs raid1 based. Boot is made more resilient whether single or multiple disk. This works today on BIOS, but not on UEFI.
Why not also extend this to /boot also? It's "rarely" used in day to day on a system, really only for yum updates that include a kernel.
Speak for yourself. libguestfs uses /boot/vmlinuz, as do other packages, as does anyone wanting to see how the kernel was configured, or to look at or change grub configuration, and so on.
Rich.
2014-03-19 5:31 GMT+01:00 William Brown william@firstyear.id.au:
On Tue, 2014-03-18 at 21:39 -0600, Chris Murphy wrote:> RFE: Do not persistently mount EFI System partition at /boot/efi
https://bugzilla.redhat.com/show_bug.cgi?id=1077984
It's still better to remove the on-going writing of configuration files
to the ESP, however. A simple one-time forwarding-configuration file pointing to the /boot volume UUID, permits configuration files to be written somewhere on /boot, which can then be md raid1 or btrfs raid1 based. Boot is made more resilient whether single or multiple disk. This works today on BIOS, but not on UEFI.
Why not also extend this to /boot also? It's "rarely" used in day to day on a system, really only for yum updates that include a kernel.
Well, why not also extend this to any rarely-used directory like perhaps /usr/share/zoneinfo, or even any top-level directory?
- It doesn't make anything easier (well, it does in Chris' RFE under the assumption that the partition is frequently being corrupted, but that shouldn't be happening in the first place; unlike /boot/efi we do have the technology to minimize these occurrences for other partitions). - It makes the system more complex and more difficult to understand. - The auto-mounting also requires system resources, probably quite comparable to keeping the partition simply mounted. - Problems with the filesystem could go undetected for a long time. This is especially problematic for /boot, consider (admittedly the worst case) - The /boot metadata or journal becomes corrupted (by a disk error, or an errant write) - If the kernel used to boot isn't directly affected, the system will continue to boot - After a few weeks, the user initiates a kernel upgrade, triggering an auto mount and a journal replay that makes the metadata corruption visible and problematic, perhaps making it impossible to boot even the *old* kernel
Mirek
I think dynamically mounted /boot is a separate conversation. It's only relevant here in that if x-systemd.automount,noauto behavior is desired for /boot, then for sure that means we need "a better way" for the EFI System partition because we can't have nested automounts.
As for resources, the automount is super fast. As in, I can't tell it's not already mounted, even when fsck.fat is used at first mount time. All Fedora products use systemd, which is what implements this. So I think that's pretty minimal.
The two easy changes currently on the table for /boot/efi apply only to anaconda:
1. Do run fsck on the EFI System partition. We don't now. https://bugzilla.redhat.com/show_bug.cgi?id=1077917 2. Change EFI System partition mount option to include both x-systemd.automount,noauto. https://bugzilla.redhat.com/show_bug.cgi?id=1077984
These changes significantly reduce the likelihood of accumulating ESP corruption, and hopefully fixing it in time if it does occur. But this doesn't solve all the problems, only the corruption angle.
The real solution is to:
1. Anaconda always creates ESPs on each selected device for install. 2. Anaconda always installs bootloader binaries to each selected devices' ESP. 3. Anaconda writes a simple one-time static config, the only variable data are /boot volume UUID, and path prefix pointing to the updated configuration file(s), as EFI/fedora/grub.cfg. 4. Anaconda + grub2-mkconfig -o /boot/grub2/grub.cfg - grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
** By virtue of putting grub.cfg on /boot again, if /boot is on md or Btrfs raid, we get bootloader configuration file syncing for "free" across all disks. The disk ESP contains only generic information so it's also much easier to restore, simple copy is all that's needed. Something one day a DE could do when we ask it to rebuild a replacement raid member device.
5. Bootloader package change to mount each ESP and update them; if bootloader updatability is considered important.
6. Now we can get rid of /boot/efi and any need for persistently mounted ESP.
X. Set x-systemd.automount,noauto on /boot. This is not part of this RFC/RFE, but could be seen as making it easy to go down this road if we want since it only means making an fstab change.
Chris Murphy
On Tue, 18.03.14 21:39, Chris Murphy (lists@colorremedies.com) wrote:
Combining x-systemd.automount + noauto however is a way to establish a mount point and only lazily triggering it on access. And that's what you want to use here.
That does work. It's automounted on any command I threw at it, including kernel install, update, or remove. It's a bit half-hearted of a solution for the problem, since it doesn't umount the volume.
It has been on our TODO list for a while to implement expiration for .automount units. It's kinda messy however, since the protocol for that involves blocking ioctls and thus would require us to do threads for this from PID1. It's a really messy kernel API.
But anyway, this is certainly on our TODO list. In fact, this is really something we want to have in place for removable media too, one day, so that the file systems are quickly unmounted after use and hence highly likely to be in a clean state when removed without manual umounting.
Lennart