Hi!
I haven't grasped the persistance overlay usage yet.
I usually create/update a personal spin every week and install it to a usb stick. Since F9, I've also been using the persistance flag. I don't use my liveusb everyday but I like to keep it updated and in my pocket.
Last weekend, I updated my spin and installed on the usb stick. Halfway through, I noticed I had forgotten the overlay flag in livecd-iso-to-disk command. I thought, "well, i guess it will be detected since it was created previously...".
Yesterday, I booted my usb stick at work and upon boot I can see a message saying no persistance overlay found. Err... my best guess was wrong.
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again without the overlay flag, previous persistance overlay is deleted?
- Persistance is created with livecd-iso-to-disk, further runs of livecd-iso-to-disk won't detect previous overlay?
I honestly looked for more info on the fedora wiki and man pages, but I couldn't find a wiki page or a man page about it. If these don't exist, I would be happy to help build one.
Thanks and best regards,
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again
without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that it is created for.
... and for the gory details. The way that persistence is implemented is that the persistence file is used as the backing store for the device mapper snapshot that we build on top of the "base" filesystem image. This means that it contains just changed blocks from the base image and that changes continue to build up over time, never reusing blocks from the snapshot.
So, not ideal for more long-term cases as if you're updating with yum, then you'll run out of space on your snapshot before you'd expect as we overwrite things multiple times[1].
Which brings me to what I've been working on this week -- persistence purely for /home[2]. So rather than using the persistence file for the snapshot backing store, we instead mount it as /home[3]. At this point, it's almost to where I'm happy enough with it to push, so stay tuned for more details :-)
Jeremy
[1] Plus, your kernel will never get updated [2] Actually, implemented such that you can have persistent changes for the "OS" as well as a persistent /home. Then your /home sticks around when you update the USB stick but any "OS" change disappear as with the present bits [3] Including support for encryption
Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
So, not ideal for more long-term cases as if you're updating with yum, then you'll run out of space on your snapshot before you'd expect as we overwrite things multiple times[1].
I update my personal spin just by building it again but there maybe some cases where yum maybe used on a livecd for quick and small stuff.
Which brings me to what I've been working on this week -- persistence purely for /home[2]. So rather than using the persistence file for the snapshot backing store, we instead mount it as /home[3]. At this point, it's almost to where I'm happy enough with it to push, so stay tuned for more details :-)
That's great news! :D
[2] Actually, implemented such that you can have persistent changes for the "OS" as well as a persistent /home. Then your /home sticks around when you update the USB stick but any "OS" change disappear as with the present bits
So, this means changes in /etc files (for example) are lost?
[3] Including support for encryption
Sweeeet!
Thanks for your info!
Best regards,
On Wed, 2008-06-04 at 20:03 +0100, Pedro Silva wrote:
Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
[2] Actually, implemented such that you can have persistent changes for the "OS" as well as a persistent /home. Then your /home sticks around when you update the USB stick but any "OS" change disappear as with the present bits
So, this means changes in /etc files (for example) are lost?
Right. There's just not any good way as it stands to preserve them. On the plus side, there should be less and less of this needing to be done as more and more gets controlled from the user session rather than at a system level. Stateless Linux here we come!
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Jeremy
Why not put the hook in the fedora-live init script?
Tim
Jeremy Katz wrote:
On Wed, 2008-06-04 at 20:03 +0100, Pedro Silva wrote:
Qua, 2008-06-04 às 14:46 -0400, Jeremy Katz escreveu:
[2] Actually, implemented such that you can have persistent changes for the "OS" as well as a persistent /home. Then your /home sticks around when you update the USB stick but any "OS" change disappear as with the present bits
So, this means changes in /etc files (for example) are lost?
Right. There's just not any good way as it stands to preserve them. On the plus side, there should be less and less of this needing to be done as more and more gets controlled from the user session rather than at a system level. Stateless Linux here we come!
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Jeremy
-- Fedora-livecd-list mailing list Fedora-livecd-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-livecd-list
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
Jeremy Katz wrote:
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script + livecd-iso-to-disk). Like I said, just not entirely sure if it's overkill to try to support or not
Jeremy
Qua, 2008-06-04 às 15:46 -0400, Jeremy Katz escreveu:
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
Jeremy Katz wrote:
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script + livecd-iso-to-disk). Like I said, just not entirely sure if it's overkill to try to support or not
I'm in favor of a solution that would include /home and other system folders. Since I have no real understanding of the inner workings of livecd-iso-to-disk, I don't know how hard that would be. I find it usefull to keep some changes in /etc files but there are other files too. Would this have to include a "blacklist" for stuff like /tmp, /var/cache ?
Regards,
On Wed, 2008-06-04 at 20:58 +0100, Pedro Silva wrote:
Qua, 2008-06-04 às 15:46 -0400, Jeremy Katz escreveu:
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
Jeremy Katz wrote:
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script + livecd-iso-to-disk). Like I said, just not entirely sure if it's overkill to try to support or not
I'm in favor of a solution that would include /home and other system folders. Since I have no real understanding of the inner workings of livecd-iso-to-disk, I don't know how hard that would be. I find it usefull to keep some changes in /etc files but there are other files too. Would this have to include a "blacklist" for stuff like /tmp, /var/cache ?
The problem is there's no good way to define what is a "system file". Not to mention that if you update a package, you may want to get its new config files. Perhaps at some point, we'll be able to use something other than dm-snapshot for our persistence, at which point, we'll have some more options. But for now, it's what we've got
Jeremy
I'm fond of using that script to wave a wand or three ;-)
if [[ -f /home/fedora/.init_script.sh ]]; then sh /home/fedora/.init_script.sh fi
Tim
Jeremy Katz wrote:
On Wed, 2008-06-04 at 13:28 -0600, Tim Wood wrote:
Jeremy Katz wrote:
But at the same time, I am considering adding some form of hook to run a script off of /home during the boot so that you could implement making any changes you wanted. Just not sure if it's overkill.
Why not put the hook in the fedora-live init script?
That's actually where all of this magic lives (well init script + livecd-iso-to-disk). Like I said, just not entirely sure if it's overkill to try to support or not
Jeremy
-- Fedora-livecd-list mailing list Fedora-livecd-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-livecd-list
Jeremy Katz wrote:
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again
without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that it is created for.
... and for the gory details. The way that persistence is implemented is that the persistence file is used as the backing store for the device mapper snapshot that we build on top of the "base" filesystem image. This means that it contains just changed blocks from the base image and that changes continue to build up over time, never reusing blocks from the snapshot.
So, not ideal for more long-term cases as if you're updating with yum, then you'll run out of space on your snapshot before you'd expect as we overwrite things multiple times[1].
Which brings me to what I've been working on this week -- persistence purely for /home[2]. So rather than using the persistence file for the snapshot backing store, we instead mount it as /home[3]. At this point, it's almost to where I'm happy enough with it to push, so stay tuned for more details :-)
Jeremy
[1] Plus, your kernel will never get updated [2] Actually, implemented such that you can have persistent changes for the "OS" as well as a persistent /home. Then your /home sticks around when you update the USB stick but any "OS" change disappear as with the present bits [3] Including support for encryption
Sounds good Jeremy, you know I've always supported this idea because it helps so many use cases.
FYI- the fixes to the current persistance implementation that I alluded to before are basically this-
Have a defragmentation process which can merge the snapshot overlay back. Even if it takes awhile for MarkMC's live kernel snapshot merging patches to go upstream (which wouldn't help for the squashfs case), there is actually a fairly straightforward alternative.
You can either have a simple boot choice to run a 'defragmentation/merge' pass while/before booting, or a more complex version that does it to a running system (using the rootfs swap trick my rebootless installer uses).
I.e. during the defrag/merge process, you just look at the combined device (that is in a readonly/unchanging state) and then run mksquashfs on it to build a new merged version. You'd have to have free space on the liveusb or disk or ram to handle the transition time where you need 2 copies, but that's not so bad.
-dmc
Jeremy Katz wrote:
On Wed, 2008-06-04 at 19:12 +0100, Pedro Silva wrote:
So, how does the persistance work:
- Persistance is created with livecd-iso-to-disk, if I run it again
without the overlay flag, previous persistance overlay is deleted?
Yes. Also, the overlay is specific to the *exact* filesystem image that it is created for.
... and for the gory details. The way that persistence is implemented is that the persistence file is used as the backing store for the device mapper snapshot that we build on top of the "base" filesystem image. This means that it contains just changed blocks from the base image and that changes continue to build up over time, never reusing blocks from the snapshot.
This is not actually true. It's not that it never reuses blocks from teh snapshot, it's that it doesn't try to.
If say you have a 256MB overlay, half used, and half of that wasted (e.g. you copy two 128MB files to /home, then delete one of them). Then currently there will be about 2G of free blocks on 4G of rootfs, and 128MB of those currently in a wasted state on the overlay. Assuming ext3 is roughly random in its choice of free block to use, that means that each new block written has about a 1/16 chance of actually reclaiming overlay space, instead of using more space. Theoretically if you have a 4G overlay, you'll never run out of space, though at some point you'll run into the issues of why you didn't just install fedora to a 4G partition on that device to start with.
-dmc
livecd@lists.fedoraproject.org