On Thu, Oct 11, 2018 at 09:24:08PM -0600, Chris Murphy wrote:
On Thu, Oct 11, 2018 at 6:37 PM, Marek Marczykowski-Górecki
<marmarek(a)invisiblethingslab.com> wrote:
> Hi all!
>
> I'm new on this list. I work on Qubes OS, where Fedora is used as a base
> distribution.
>
> While trying to build the installation image in reproducible manner[1],
> I found the current installation image have unusual layout. Quoting
> dracut.cmdline manual page:
>
> squashfs.img | Squashfs from LiveCD .iso downloaded via network
> !(mount)
> /LiveOS
> |- rootfs.img | Filesystem image to mount read-only
> !(mount)
> /bin | Live filesystem
> /boot |
> /dev |
> ... |
>
> This rootfs.img layer makes the image build very much unreproducible.
> Why is it even there? Bare squashfs.img layer should be enough. Then,
> mount overlayfs over it (I see there is even some partial support for it
> in dmsquash-live). Most other Live systems I've seen use just squashfs +
> overlayfs (or aufs if kernel is older), so it's commonly tested
> configuration. I *guess* it's there for historical reason, from before
> aufs/overlayfs being available. Is there any other reason for that?
I'm pretty sure the original reason was the default live install use
dd to block copy the root file system into the fedora-root LV, and
then resized the LV and ext4 file system.
How is it done now?
There have also been a
number of squashfs improvements since that decision so there might
have been limitations with squashfs that ext4 didn't have (I'm
thinking xattr were long supported in ext4 before squashfs, and maybe
capabilities?)
>
> If there is no other reason, I propose to drop this and have
> installer/live filesystem directly in squashfs.img. This have multiple
> benefits:
> - it's much easier to make the image build process reproducible (see
> below)
> - less complexity, both in the build and in the boot (the whole
> dmsquash-live dracut module can be replaced with <20 line
> function[2]
> - smaller initramfs (which is extremely important if needed to be
> included in efiboot.img, which can't be larger than 32MB)
> - slightly faster boot time (device-mapper is slow)
>
> What do you think?
Whatever we do should take into account the persistent root and
persistent home use cases, specifically:
https://github.com/livecd-tools/livecd-tools/blob/master/tools/livecd-iso...
--overlay-size-mb
--home-size-mb
A particular criticism of the device-mapper solution currently being
used is in that script: it blows up. Literally it's WORM, and deleting
files simply dereferences them, it doesn't free up pool space, so it
is inevitable that the pool will fill up, and when it does it blows up
the file system, and it can't be repaired. All you can do is reset the
overlay which means deleting all changes and starting over.
At least one of our spins, SOAS, depends on livecd-iso-to-disk for
creating their final installation because it's predicated on running
Fedora SOAS from a stick.
Why does efiboot.img have a 32MiB limit?
Because "32MB should be enough for everybody"...
Long story short, "El Torito" boot catalog structure have 16-bit field
for image size (expressed in 512-bytes sectors). For details see here:
https://wiki.osdev.org/El-Torito
https://web.archive.org/web/20180112220141/https://download.intel.com/sup...
(page 10)
Full story:
https://github.com/QubesOS/qubes-issues/issues/794#issuecomment-135988806
I've spent a lot of time debugging this, because mkisofs doesn't
complain about it, just silently overflow higher bits to adjacent field,
which results in weird results, depending on where you boot it. Adding
isohybrid to the picture doesn't make it easier (there, higher bits are
truncated, or actually not copied to the MBR partition table, as wasn't
part of the original field).
> As for the reproducibility, I've made changes to lorax
(including
> dropping rootfs.img layer), anaconda, pungi and createrepo and this all
> allows to build bit-by-bit identical image, given the same input (rpm
> packages, pungi configuration, $SOURCE_DATE_EPOCH variable[3]). Well,
> almost - there is an issue with efiboot.img, but I already have a
> solution, just not pushed it yet.
>
> You can find all the pull requests collected here:
>
https://github.com/QubesOS/qubes-installer-qubes-os/pull/26
>
> I'll work further to make the changes merged upstream.
>
> [1]
https://reproducible-builds.org/
> [2]
https://github.com/QubesOS/qubes-installer-qubes-os/pull/26/commits/332be...
> [3]
https://reproducible-builds.org/specs/source-date-epoch/
Cool! Well you've already done most of the work and if this has
support elsewhere already then I'm in favor of continuing in that
direction.
I did give all of these things some thought a long time ago when I ran
into a lorax hack by Will Woods who used Btrfs as the root.img file
system, I'm not sure why it was used. But it gave me the idea of using
a few features built into Btrfs specifically for this use case:
- seed/sprout feature can be used with zram block device for volatile
overlay; and used with a blank partition on the stick for persistent
overlay. Discovery is part of the btrfs kernel code.
- Since metadata and data is always checksummed on every read, we
wouldn't have to depend on the slow and transient ISO checksum
(rd.live.check which uses checkisomd5) which likewise breaks when
creating a stick with livecd-iso-to-disk.
- Btrfs supports zstd compression. I did some testing and squashfs is
still a bit more efficient because it compresses fs metadata, whereas
Btrfs only compresses data extents.
The gotcha here is the resulting image isn't going to be bit for bit
reproducible: UUIDs and time stamps are strewn throughout the file
system (similar to ext4 and XFS), but any sufficiently complex file
system is going to have this problem.
I wouldn't worry about _files_ timestamps that much - in most cases this is
solvable problem by elaborate enough find+touch[4]. But that's not all
obviously, there are various timestamps in superblock, and other
metadata. The most problematic part in "normal" filesystems, using
kernel driver is inode allocation, block allocation etc. This greatly
depends on timing, ordering, specific kernel version etc.
See [5] for details.
Off hand I'm not sure how
squashfs would get around it since it's going to draw from an ext4
source (not sure if the ephemeral root could be tmpfs and use it as
the source for mksquashfs?)
mksquashfs 5.0-rc1 have support for clamping mtime to $SOURCE_DATE_EPOCH
variable[3]. And the other metadata is reproducible already in mksquashfs
4.3 (I think files are sorted or similar approach is taken).
TBH, there is also a tool to build ext4 filesystem reproducible, not
using kernel driver. It's make_ext4 from OpenWRT projet. But I still
think it would be better to drop that layer anyway.
[4]
https://reproducible-builds.org/docs/archives/
[5]
https://reproducible-builds.org/docs/system-images/
--
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?