On Mon, 2014-07-28 at 09:37 -0500, Major Hayden wrote:
Hello there,
I'm working with F20 and CentOS 7 to create some live booted images.
I'm not looking to do live USB/CD media, but rather boot a server over
the network with a kernel, initramfs, and squashfs. It's working well
so far, but I have a filesystem issue that I can't seem to fix.
My build scripts create a 10GB sparse file and I fill that with an
ext4 filesystem. I package that up into a squashfs as specified in
the docs[1]. That boots just fine.
However, if I attempt to fill the filesystem, it fills and becomes
corrupt much earlier than I'd expect. I've put some log info into a
gist[2].
My expectation is that if I create a 10GB live filesystem and and the
system has > 10GB of RAM available, I'd expect that I could store
somewhere around 10GB of data in the live filesystem before I run into
a full filesystem. Is that expectation incorrect? Am I configuring
something incorrectly?
You're not quite right about how the overlay works.
The default in-memory overlay is just 512MB. And the device-mapper docs
note that "if it fills up the snapshot will become useless and be
disabled, returning errors."[1].
You should also note that the overlay is a block-level snapshot - so any
changes to existing files or filesystem metadata will cause data to be
written to the overlay. Furthermore, the default chunk size is 4kb - so
any change less than 4kb will take 4kb of space.
Basically, the dmsquash-live + overlay thing is a gross hack. It was
designed to solve the problem of fitting an entire desktop OS on a 650MB
CD-ROM, to be used as a demonstration, or for other short-lived systems
(e.g. the installer). It was not designed for long-term use.
I assume you're using filesystem images 'cuz you don't have a reliable
network connection (otherwise you'd probably be using NFS or iSCSI or
something)?
Since your systems have lots of RAM, why not just use a regular ext4
filesystem image as your root filesystem? Then you don't need to worry
about blowing up the overlay at all.
If you need compression to save bandwidth/download time: you could just
xz-compress the filesystem image and uncompress it after download?
If you need compression to save RAM: why not use a squashfs image
directly, and mount/bind a tmpfs to the places you'll be writing data?
You might even consider using btrfs, mounted with the "compress" or
"compress-force" option[2], which would give you the benefit of
compression *and* a normal read-write filesystem.
Is there a particular reason you need to use dmsquash-live, or is this
just a case of the hammer making all your problems look like nails?
-w
[1]
https://www.kernel.org/doc/Documentation/device-mapper/snapshot.txt
[2]
https://btrfs.wiki.kernel.org/index.php/Compression