I realize that this is off-topic. Hopefully you will forgive me for imposing on you like this.
I'm new to *nix administration. I've used *nix installations for years in various incarnations (Xenix, Solaris, HPUX et al.) but not on the admin side. Backup is still something of a mystery to me. It seems that there are two schools of thought
cpio tar
It also seems that each side thinks the other side is nuts. It also seems that using links (soft or otherwise) is not well handled by either technique. It also seems that everyone agrees that using tape is the Way To Go(tm).
Can anyone tell me whether my impressions on this matter be correct? Is there a good tutorial which can give me relative pros and cons of cpio style vs. tar style backup? How about which directories actually need backing up? How about how does one actually recover when the worst happens? How about disc upgrades? I suppose that /etc/fstab needs to be new, but /etc/hosts needs to be restored. How does one go about doing these "partial" restores to get the machine back running again?
I also don't want to use a tape drive, being (as some are) on a restricted budget, both for time to learn new stuff and monetarily, being among the Great Telecom Layoff. There are very nice Windows programs which create initial/disaster recovery CDs which can completely rebuild a system to the way it was when initially created, and then do backups to CD after that. *nix seems not to have any such concept.
Anyway, thanks for you time.
Mike
On Fri, 2005-07-01 08:07, Mike McCarty wrote:
I realize that this is off-topic. Hopefully you will forgive me for imposing on you like this.
Sadly, that is not possible.
Backup is still something of a mystery to me.
man man man dump man restore
works great with DVD-RAM cartridges.
Angela Kahealani wrote:
On Fri, 2005-07-01 08:07, Mike McCarty wrote:
I realize that this is off-topic. Hopefully you will forgive me for imposing on you like this.
Sadly, that is not possible.
Well, that's too bad, then.
Backup is still something of a mystery to me.
man man man dump man restore
works great with DVD-RAM cartridges.
Hmm. I know how to use man. The documentation for dump does not tell me what files *nix needs saved. I suppose one could dump to a file, and use split to break it into CD size pieces, and XCDROAST them. What is a DVD-RAM cartridge?
I suppose that Fedora has some special areas that need attendance, as well, over and above what a generic *nix installation needs.
Mike
Mike McCarty wrote:
man man man dump man restore
works great with DVD-RAM cartridges.
Backup is still something of a mystery to me. Hmm. I know how to use man. The documentation for dump does not tell me what files *nix needs saved. I suppose one could dump to a file, and use split to break it into CD size pieces, and XCDROAST them. What is a DVD-RAM cartridge?
I suppose that Fedora has some special areas that need attendance, as well, over and above what a generic *nix installation needs.
Mike
I'd avoid to use dump/restore.. Use tar + gzip or tar + bz2. You'll get good compression rates and all permissions will be kept. If you use dump , you're copying *everything* from the disk, including the data structures used to store the data and permissions on disk, which is a waste of space.
And I think he meant DVD-RAM disc? Because my DVD burner works with DVD-RAM and it is a disc like any DVD, except that it's golden and works pretty much like a big UDF filesystem (like those packet writing apps that windows used on CD-RW)
-- Pedro Macedo
On Fri, 2005-07-01 13:26, Pedro Fernandes Macedo miswrote:
Mike McCarty wrote:
Angela Kahealani wrote:
man man man dump man restore
works great with DVD-RAM cartridges.
What is a DVD-RAM cartridge?
see below...
I'd avoid to use dump/restore.. Use tar + gzip or tar + bz2. You'll get good compression rates and all permissions will be kept. If you use dump , you're copying *everything* from the disk, including the data structures used to store the data and permissions on disk, which is a waste of space.
which, considering SE-GNU/Linux(TM-Richard Stallman / Linus Torvalds) is *bad*, why?
And I think he meant DVD-RAM disc?
I'm the one using DVD-RAM disks as tapes... yes, you can also partition them, format them, install filesystems on them, and mount them as hard disk partitions (which write *slowly*). I use the Matsushita (Panasonic) series mostly: hda: MATSHITA DVD-RAM LF-D311, ATAPI DVD-ROM DVD-R-RAM drive, 1024kB Cache, UDMA(33) hdb: MATSHITA PD-2 LF-D110, ATAPI DVD-ROM DVD-RAM drive, 2048kB Cache, DMA hdc: MATSHITA CR-585, ATAPI 24X CD-ROM drive, 128kB Cache, DMA hdd: ASUS CRW-5232AS, ATAPI 52X CD-ROM CD-R/RW drive, 2048kB Cache, UDMA(33) Uniform CD-ROM driver Revision: 3.20
Because my DVD burner works with DVD-RAM and it is a disc like any DVD, except that it's golden and works pretty much like a big UDF filesystem (like those packet writing apps that windows used on CD-RW) -- Pedro Macedo
Even the first generation "DVD-RAM" drive (hdb above) which writes single sided 2.6GB disks (format out about 2.3GB) or double sided 5.2GB disks (format out about 4.7GB) goes beyond the 2GB barrier of some old software/systems to handle either: files of length greater than 2GB file-systems larger than 2GB so depending upon how you partition, format, and which filesystem you use on a DVD-RAM drive (noatime recommended for speed), you may run into 2GB barriers which you don't see when you treat the cartridge as a "tape" of greater than 2GB length, which various versions of tar (e.g. s-tar (star)) and/or dump/restore use with no problems. This has "just worked" since Redhat9 (maybe earlier?)... I've not encountered anything that couldn't handle these cartridges as raw block devices (tapes).
Since dump/restore manages multiple volumes and detects end of tape without you telling it volume size, it's pretty much a no brainer... and DVD-RAM offers the unique advantage of protecting those scratch-prone fingerprint-prone dust-prone optical disks safely protected in a cartridge... your 5.25" Diskette of the modern day.
I've found .tgz's written to DVD-RAM are infinitely portable... and, oh, yeah, when you get to a drive that won't accept the cartridge, you take the single sided disk out of the cartridge and all drives treat it as a DVD-ROM you can restore from or tar -vxzf /dev/dvd-ram
WIth a write life of 100,000 versus {C/DV}D-RW's 10,000 cycles.
A backed-up computer user is a happy computer user :-)
--On Friday, July 01, 2005 8:26 PM -0300 Pedro Fernandes Macedo webmaster@margo.bijoux.nom.br wrote:
I'd avoid to use dump/restore.. Use tar + gzip or tar + bz2. You'll get good compression rates and all permissions will be kept. If you use dump , you're copying *everything* from the disk, including the data structures used to store the data and permissions on disk, which is a waste of space.
Oh? Dump records the files contiguously, so there's no need for any structures beyond the metadata in the inode (those same pesky things that tar saves). dump also handles sparse files (the gigantic ones that have no actual space allocated) correctly. (It may be that tar does that as well. I haven't checked recently.) dump accesses the raw disk, so it can back up files hidden by mount points. (OTOH, restore reads through the filesystem, so it can't verify files hidden by mount points.) The latest dump backs up extended attributes. Does tar do that? dump is capable of compression.
I backup up my system to a 2nd system weekly via dump over the network, and do nightly incrementals.
On Sat, 2005-07-02 at 10:08, Kenneth Porter wrote:
Oh? Dump records the files contiguously, so there's no need for any structures beyond the metadata in the inode (those same pesky things that tar saves). dump also handles sparse files (the gigantic ones that have no actual space allocated) correctly. (It may be that tar does that as well. I haven't checked recently.)
Gnutar handles them.
dump accesses the raw disk, so it can back up files hidden by mount points.
I think I remember seeing Linus say that the raw disk as seen by dump can be out of sync with the filesystem cache as seen by things reading and writing files. Is that still true?
(OTOH, restore reads through the filesystem, so it can't verify files hidden by mount points.) The latest dump backs up extended attributes. Does tar do that?
Star is supposed to do extended attributes, I don't thing gnutar does. However, you can extract files from tar/cpio archives onto just about any operating system and filesystem. With dump you can only put back the same filesystem type you dumped.
--On Saturday, July 02, 2005 4:37 PM -0500 Les Mikesell lesmikesell@gmail.com wrote:
I think I remember seeing Linus say that the raw disk as seen by dump can be out of sync with the filesystem cache as seen by things reading and writing files. Is that still true?
Here's the full scoop:
http://dump.sourceforge.net/isdumpdeprecated.html
Star is supposed to do extended attributes, I don't thing gnutar does. However, you can extract files from tar/cpio archives onto just about any operating system and filesystem. With dump you can only put back the same filesystem type you dumped.
restore, the counterpart to dump, only uses normal filesystem API's, not raw device I/O, so it can restore to any filesystem. Some information may be lost if the target filesystem doesn't support all the features of the dumped ext3 system, but that's not the fault of dump/restore nor ext3.
Kenneth Porter wrote:
--On Friday, July 01, 2005 8:26 PM -0300 Pedro Fernandes Macedo <> wrote:
I'd avoid to use dump/restore.. Use tar + gzip or tar + bz2. You'll get good compression rates and all permissions will be kept. If you use dump , you're copying *everything* from the disk, including the data structures used to store the data and permissions on disk, which is a waste of space.
Oh? Dump records the files contiguously, so there's no need for any structures beyond the metadata in the inode (those same pesky things that tar saves). dump also handles sparse files (the gigantic ones that have no actual space allocated) correctly. (It may be that tar does that as well. I haven't checked recently.) dump accesses the raw disk, so it can back up files hidden by mount points. (OTOH, restore reads through the filesystem, so it can't verify files hidden by mount points.) The latest dump backs up extended attributes. Does tar do that? dump is capable of compression.
Probably I made a wrong assumption then. But I still preffer to use tar for backups for its portability.. Try to restore a filesystem from a dump backup when you dont have a machine with the same OS anymore.. (or with an OS that has the same tools)... Not very fun... (I've had to restore a dump from a solaris machine and no linux machine worked for that)...
-- Pedro Macedo
I'm new to *nix administration. I've used *nix installations for years in various incarnations (Xenix, Solaris, HPUX et al.) but not on the admin side. Backup is still something of a mystery to me. It seems that there are two schools of thought
cpio tar
Try rsync. Check out this great article from Mike Rubel.
http://www.mikerubel.org/computers/rsync_snapshots/index.html
It will give you a new perspective on backups.
On Fri, 2005-07-01 at 13:07, Mike McCarty wrote:
It seems that there are two schools of thought
cpio tar
No, both those work only through the filesystem. Dump is the other alternative - it is excessively intimate with the disk structure. Straight dd images of the raw disk are even more extreme.
It also seems that each side thinks the other side is nuts. It also seems that using links (soft or otherwise) is not well handled by either technique.
Gnutar and star provide extensions to the original tar (which did omit support for some essential things).
It also seems that everyone agrees that using tape is the Way To Go(tm).
Tape is OK for archiving large amounts of data that you probably don't want to restore or keeping offsite for disaster recover, which again you don't really ever plan to do...
Can anyone tell me whether my impressions on this matter be correct? Is there a good tutorial which can give me relative pros and cons of cpio style vs. tar style backup?
Both of these copy files through the filesystem with extensions for special files. The only real difference is that tar knows how to recurse through directories itself, where cpio must be handed a list of files, generally by some contortions with 'find'. A side effect of this is that gnutar has a '--listed-incremental' mode that will catch renamed directories in an incremental and take the old files under them, where this is basically impossible with cpio.
I also don't want to use a tape drive, being (as some are) on a restricted budget, both for time to learn new stuff and monetarily, being among the Great Telecom Layoff. There are very nice Windows programs which create initial/disaster recovery CDs which can completely rebuild a system to the way it was when initially created, and then do backups to CD after that. *nix seems not to have any such concept.
There are at least three different reasons you might want to make backups and different systems work better for the different instances and there are ways to handle compromises. If your goal is to be able to restore an exactly identical system as fast as possible you might want mondo or a disk image solution or a 'warm spare' machine with periodic rsync to it's drives. Dump is also suitable for this, along with tar/cpio archives of the whole system as long as you know a few tricks about restoring using a bootable CD, making your own filesystems and making them bootable with grub.
Another goal might be to have long-term archives, in which case tar and cpio are likely to be the most portable and readable at some future date. These will also be a good option if you are rebuilding a new and better machine instead of restoring exactly the one that died, since you will have the option to restore onto different filesystem types.
The most likely reason to need the backup is that someone accidentally deleted some files or directories and you are restoring back to the same machine. In this case, something online instead of tapes is a big win. If you have more than one machine involved, look at backuppc (http://backuppc.sourceforge.net/) which uses a scheme of compression an linking to keep copies on disk in a web browsable form, plus it can generate tar images of any version of any filesystem within the range you are keeping. It is as handy as you can get for the individual file/directory restores and you can do the bare-iron restores with a small amount of work.