On Fri, Oct 12, 2018 at 03:44:38PM -0600, Chris Murphy wrote:
On Fri, Oct 12, 2018 at 4:30 AM, Marek Marczykowski-Górecki
> On Thu, Oct 11, 2018 at 09:24:08PM -0600, Chris Murphy wrote:
>> Why does efiboot.img have a 32MiB limit?
> Because "32MB should be enough for everybody"...
> Long story short, "El Torito" boot catalog structure have 16-bit field
> for image size (expressed in 512-bytes sectors). For details see here:
> (page 10)
OK. On Fedora 28 media, efiboot.img is ~9.2 MiB and does not contain
either the kernel or initramfs.
I know, this particular problem was specific to Qubes OS, where
kernel+initramfs needed to be on ESP, because of Xen+EFI limitation
(basically kernel needs to be loaded through through UEFI instead of
by grub, so it needs to live on something that UEFI understands). And
actually recent Xen version doesn't have this limitation anymore (at
least in theory...). This is just a bit of context from where it all
got here, much less relevant today.
> Full story:
> I've spent a lot of time debugging this, because mkisofs doesn't
> complain about it, just silently overflow higher bits to adjacent field,
> which results in weird results, depending on where you boot it. Adding
> isohybrid to the picture doesn't make it easier (there, higher bits are
> truncated, or actually not copied to the MBR partition table, as wasn't
> part of the original field).
I think we're stuck with isohybrid for a while. Having UEFI and BIOS
bootloaders, along with isohybrid supporting both as well as Macs, all
on one media image, that can be burned to optical media and written to
a USB stick - is hugely beneficial.
I have no problem with isohybrid alone. It's major hack, but definitely
The compose process takes about 12 hours. That every ISO for all the
editions, and the spins, and the VM images, for all archs. Even having
separate UEFI and BIOS images, or splitting out Macs with their own
image, it'll increase compose times and complexity across the board.
And also complexity for the users - which image to download. I totally
understand why it is beneficial.
>> I did give all of these things some thought a long time ago
when I ran
>> into a lorax hack by Will Woods who used Btrfs as the root.img file
>> system, I'm not sure why it was used. But it gave me the idea of using
>> a few features built into Btrfs specifically for this use case:
>> - seed/sprout feature can be used with zram block device for volatile
>> overlay; and used with a blank partition on the stick for persistent
>> overlay. Discovery is part of the btrfs kernel code.
>> - Since metadata and data is always checksummed on every read, we
>> wouldn't have to depend on the slow and transient ISO checksum
>> (rd.live.check which uses checkisomd5) which likewise breaks when
>> creating a stick with livecd-iso-to-disk.
>> - Btrfs supports zstd compression. I did some testing and squashfs is
>> still a bit more efficient because it compresses fs metadata, whereas
>> Btrfs only compresses data extents.
>> The gotcha here is the resulting image isn't going to be bit for bit
>> reproducible: UUIDs and time stamps are strewn throughout the file
>> system (similar to ext4 and XFS), but any sufficiently complex file
>> system is going to have this problem.
> I wouldn't worry about _files_ timestamps that much - in most cases this is
> solvable problem by elaborate enough find+touch. But that's not all
> obviously, there are various timestamps in superblock, and other
> metadata. The most problematic part in "normal" filesystems, using
> kernel driver is inode allocation, block allocation etc. This greatly
> depends on timing, ordering, specific kernel version etc.
> See  for details.
mkfs.btrfs has --rootdir and --shrink features to pre-allocate a
volume with files at mkfs time; I have no idea to what degree it
depends on kernel code.
Probably not at all, given it works as non-root user too.
I've tried to run it twice on the same directory (and with the same
--uuid) on 32MB of data and got different images (~2000 lines of hexdump
diff). Could be some timestamps, could be something else.
The main benefit with this is it's really easy
to implement full checksum matching for metadata and data on every
read, and user space ends up with EIO instead of corrupt data, and
super clear kernel complaints. And such corruption whether on optical
or USB sticks, is common. Even the more rare case of a stick that
passes md5 checksum, can later have transient and silent corruption
that ends up showing up in weird ways.
It's plausible squashfs could implement this, I think by default it
already checksums every file to look for duplicates, but it doesn't
retain the per file hash for integrity checking later on.
Indeed it looks that way. I'm able to make one-byte modification to the
image file resulting in different files (diff -r), but no read error. I
wonder if integrity checking is something on squashfs roadmap...
possible with dm-verity or dm-integrity but then that adds back the dm
Oh, please, no...
There are two almost separate aspects here:
- image layout (squashfs+ext4, squashfs alone, squashfs+btrfs)
- how copy-on-write is achieved (dm-snapshot, overlay fs)
For reproducibility, squashfs alone is the best option, but does not
improve integrity checking (but also doesn't make it worse).
For integrity checking, squashfs+btrfs may be better, but doesn't help
that much with reproducibility. Maybe even make it worse, because
mkfs.btrfs also make not reproducible result, while make_ext4 (do not
confuse with mkfs.ext4!) is reproducible. Not being packaged for Fedora
is only a small issue here.
As for copy-on-write, dm-snapshot is quite complex to setup and require
underlying FS to support write. Also, doesn't allow to write more data
than original image size (may be an issue for persistent partition
case). Overlay fs on the other hand works with any underlying fs, you
can write as much data as you want. And in case of persistent partition,
you can access that data even if base image (the lower layer) is
unavailable/broken. I think the only downside of overlay fs is when you
modify large file it gets copied in full to the upper layer. But I don't
think that's an issue in this use case.
For me, overlay fs is a clear winner here.
But as for image layout, it isn't that simple. For reproducibility,
squashfs alone is better. But if the goal of this change would be also
improving read errors detection, then it isn't that clear anymore. It
may be that it takes a simple mkfs.btrfs patch to make it reproducible,
but it isn't obvious for me at this stage. Also, keeping two layers
looks like unnecessary complexity.
What do you think about sidestepping this discussion a little and
replacing dm-snapshot with overlay fs regardless of other changes here?
That should be doable without any change to image format and will give
more flexibility there.
Then, it could be even made to support both 1-layer and 2-layer formats
at the same time (depending on rootfs.img presence). Something that
isn't possible with dm-snapshot right now.
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?