I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I: 1) mount /dev/sdb1 /mnt 2) cd /mnt/b 3) df -h . Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
5) mount /dev/sdb1 /mnt 6) cd /mnt 7) df -h . Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I:
- mount /dev/sdb1 /mnt
- cd /mnt/b
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
- mount /dev/sdb1 /mnt
- cd /mnt
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
On Fri, 2009-12-04 at 13:42 -0800, Daniel B. Thurman wrote:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
You seem to have a misconception about what dd is for. It just copies data from A to B with no interpretation at all. In this case dd copied the entire contents of Disk1 to Disk1, including all the system meta-information (superblock, inode list, free list etc.), but since partition table info is not part of that data (it's held on a different part of the disk), Disk2's partition layout didn't change. So Disk 2 now has a filesystem sitting on it that has a lot of physical free space at the end which it doesn't know about. It needs to be resized, e.g. with gparted.
poc
On Fri, Dec 4, 2009 at 3:57 PM, Patrick O'Callaghan pocallaghan@gmail.com wrote:
On Fri, 2009-12-04 at 13:42 -0800, Daniel B. Thurman wrote:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
You seem to have a misconception about what dd is for. It just copies data from A to B with no interpretation at all. In this case dd copied the entire contents of Disk1 to Disk1, including all the system meta-information (superblock, inode list, free list etc.), but since partition table info is not part of that data (it's held on a different part of the disk), Disk2's partition layout didn't change. So Disk 2 now has a filesystem sitting on it that has a lot of physical free space at the end which it doesn't know about. It needs to be resized, e.g. with gparted.
poc
Would it have been better to dd the entire disk (i.e. dd if=/dev/sda of=/dev/sdb) which should also copy the partition data and then use gparted or other tool to resize the partition?
Richard
On 12/04/2009 01:56 PM, Tom H wrote:
I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I:
- mount /dev/sdb1 /mnt
- cd /mnt/b
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
- mount /dev/sdb1 /mnt
- cd /mnt
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
Ok, I used gparted to shrink the size of the partition and grow the partition again. That did the trick.
I wonder if it would have been easier to just:
resize2fs /dev/sdb1
and be done with it?
On 12/04/2009 01:56 PM, Tom H wrote:
I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I:
- mount /dev/sdb1 /mnt
- cd /mnt/b
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
- mount /dev/sdb1 /mnt
- cd /mnt
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
The answer is, you have to rezise after dd and there are two ways
1) e2fsck -f /dev/sdb; resize2fs /dev/sdb 2) Use gparted: a) shrink it down minimally, then regrow
Step 1 is faster, or so it seems
On 12/04/2009 02:37 PM, Daniel B. Thurman wrote:
On 12/04/2009 01:56 PM, Tom H wrote:
I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I:
- mount /dev/sdb1 /mnt
- cd /mnt/b
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
- mount /dev/sdb1 /mnt
- cd /mnt
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
The answer is, you have to rezise after dd and there are two ways
- e2fsck -f /dev/sdb; resize2fs /dev/sdb
- Use gparted: a) shrink it down minimally, then regrow
Step 1 is faster, or so it seems
Dang... step 1 should be: /dev/sdbX where X is the partition desired.
I am copying a disk/partition to another disk/partition using dd, with the following:
BEFORE: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G (newly formatted)
I did a dd if=/dev/sda1 of=/dev/sdb1
AFTER: Disk1: partition#1: psize=100G, size=97.65G, used=91.23G, unused=6.42G Disk2: partition#1: psize=250G, size=244.14G, used=237.72G, unused=6.42G
WTF...
Then I:
- mount /dev/sdb1 /mnt
- cd /mnt/b
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 97G 90G 1.6G 99% /mnt/b 4) umount /mnt
- mount /dev/sdb1 /mnt
- cd /mnt
- df -h .
Filesystem Size Used Avail Use% Mounted on /dev/sda1 97G 90G 1.6G 99% /mnt/b 8) Umount /mnt
Hmm... this is not what I expected... I had assumed that somehow the free disk space would somehow be reclaimed...
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
The answer is, you have to rezise after dd and there are two ways
- e2fsck -f /dev/sdb; resize2fs /dev/sdb
- Use gparted: a) shrink it down minimally, then regrow
# parted (parted) unit cyl (parted) print (parted) resize <partition> <start> <end> (parted) quit
On Fri, 2009-12-04 at 14:07 -0800, Daniel B. Thurman wrote:
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
Ok, I used gparted to shrink the size of the partition and grow the partition again. That did the trick.
I wonder if it would have been easier to just:
resize2fs /dev/sdb1
The whole approach is wrong IMHO. Unless the two partitions are the same size there's no particular point in using dd for this. Much better to create (mkfs) a new filesystem on the new partition, then use tar, or rsync, or even "cp -r" to copy the contents. I doubt it's slower than dd +resize, doesn't copy anything except file blocks that are actually in use, and has the potential advantage of leaving you with a better layout at the end. You can also take advantage of the copy to change filesystem types, e.g. from ext3 to ext4.
poc
On 12/04/2009 03:44 PM, Patrick O'Callaghan wrote:
On Fri, 2009-12-04 at 14:07 -0800, Daniel B. Thurman wrote:
What am I doing wrong here and how do I get around this?
You have to resize Disk2 to 250Gb with resize2fs or parted.
Ok, I used gparted to shrink the size of the partition and grow the partition again. That did the trick.
I wonder if it would have been easier to just:
resize2fs /dev/sdb1
The whole approach is wrong IMHO. Unless the two partitions are the same size there's no particular point in using dd for this. Much better to create (mkfs) a new filesystem on the new partition, then use tar, or rsync, or even "cp -r" to copy the contents. I doubt it's slower than dd +resize, doesn't copy anything except file blocks that are actually in use, and has the potential advantage of leaving you with a better layout at the end. You can also take advantage of the copy to change filesystem types, e.g. from ext3 to ext4.
poc
The two partitions are not the same. And there is the problem of UUID and SeLinux. I used rsync, and I noticed certain types of problems due to the OS using UUID in various places. As for SeLinux, you are forced to relabel, otherwise you might get certain problems, one example is that you might not be able to log in.
The problem I have specifically with rsync (or tar, or cp) is that it does not save ACLs, file attributes too well, and so on that I gave up using it. Perhaps the problem in this case is not to use the -a option but to use the manual options to save everything about the files that you can.
Specifically: rsync manual: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -H, --hard-links preserve hard links -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes
I don't use UUID but labels and it helps me to keep track of mulit-boot partitions, exactly where I need them.
As for dd, I have resorted to using ddrescue in cases where there are partition errors. This saved my a$$, big time.
So far, I am a happy camper. I have many linux distros, other OSes, Windows - all that. The only "snag" now is dealing with Vista - a different animal that I hope to resolve soon enough.
FWIW, Dan
On Fri, 2009-12-04 at 16:13 -0800, Daniel B. Thurman wrote:
The problem I have specifically with rsync (or tar, or cp) is that it does not save ACLs, file attributes too well, and so on that I gave up using it. Perhaps the problem in this case is not to use the -a option but to use the manual options to save everything about the files that you can.
Specifically: rsync manual: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -H, --hard-links preserve hard links -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes
Do you mean that you've tried these (particularly -A and -X) and they don't work? If so, have you filed a bug report?
Also, have you looked at [1] wrt copying while maintaining SElinux contexts?
[1] http://docs.fedoraproject.org/selinux-user-guide/f10/en-US/sect-Security-Enh...
poc
Patrick O'Callaghan wrote:
On Fri, 2009-12-04 at 16:13 -0800, Daniel B. Thurman wrote:
The problem I have specifically with rsync (or tar, or cp) is that it does not save ACLs, file attributes too well, and so on that I gave up using it. Perhaps the problem in this case is not to use the -a option but to use the manual options to save everything about the files that you can.
Specifically: rsync manual: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -H, --hard-links preserve hard links -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes
Do you mean that you've tried these (particularly -A and -X) and they don't work? If so, have you filed a bug report?
I am not certain that it's a bug per-se, it's just that there are cases, or so it seems, where it is not clear how to deal with the issue on my part. For example, hard links (-H). How are hard links handled on one drive to be copied over to another and is it guaranteed to work? I could not get my mind around this one so I did not want to take a chance. As for ACLs (-A), what exactly is being handled here and does this work for ALL Oses concerned - do they follow the same "standard"? I am thinking about Vista in particular, so I did not want to experiment on this one either. As for extended attributes (-X), I have no clue exactly what this is. I guess I have to someday take the time to do more research before messing around with these rsync options. This is a particular reason why I am using dd/rescue & resizing - it works sans Vista, which I have yet to try.
Also, have you looked at [1] wrt copying while maintaining SElinux contexts?
[1] http://docs.fedoraproject.org/selinux-user-guide/f10/en-US/sect-Security-Enh...
Yes, I am familiar with SeLinux to a point of creating them and "permentizing" them. Symbolic links are one of the nasty ones that take some getting used to though.
Dan
On Sat, 2009-12-05 at 11:02 -0800, Daniel B. Thurman wrote:
Patrick O'Callaghan wrote:
On Fri, 2009-12-04 at 16:13 -0800, Daniel B. Thurman wrote:
The problem I have specifically with rsync (or tar, or cp) is that it does not save ACLs, file attributes too well, and so on that I gave up using it. Perhaps the problem in this case is not to use the -a option but to use the manual options to save everything about the files that you can.
Specifically: rsync manual: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -H, --hard-links preserve hard links -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes
Do you mean that you've tried these (particularly -A and -X) and they don't work? If so, have you filed a bug report?
I am not certain that it's a bug per-se, it's just that there are cases, or so it seems, where it is not clear how to deal with the issue on my part. For example, hard links (-H). How are hard links handled on one drive to be copied over to another and is it guaranteed to work? I could not get my mind around this one so I did not want to take a chance.
Well, you could try it. Since you're copying to a fresh drive anyway, there's no harm in doing it and checking the result.
Hard links are an easy case in fact, and trivial to check. Create file A and hard link B to it. Rsync A and B to a different filesystem, using the -H option. Use "ls -i" on the copied files to see that they both have the same inode. That's all a hard link is.
As for ACLs (-A), what exactly is being handled here and does this work for ALL Oses concerned - do they follow the same "standard"?
WTF? Of course not. I thought we were talking about Linux here. In fact there's been no indication in this thread of anything to the contrary. There is no "standard" for this stuff across OSes (I suppose Posix might be considered a standard but I'd be very careful about relying on it). We're doing low-level system maintenance here. Don't expect anything to be portable.
I am thinking about Vista in particular, so I did not want to experiment on this one either.
I wouldn't dream of doing this on a non-Unix system.
As for extended attributes (-X), I have no clue exactly what this is. I guess I have to someday take the time to do more research before messing around with these rsync options. This is a particular reason why I am using dd/rescue & resizing - it works sans Vista, which I have yet to try.
Again, rsync is a *Unix* utility, designed for *Unix* filesystems *only*. If you'd said at the beginning that you wanted to move a Vista partition we could have saved ourselves a lot of time.
poc
Patrick O'Callaghan wrote:
On Sat, 2009-12-05 at 11:02 -0800, Daniel B. Thurman wrote:
Patrick O'Callaghan wrote:
On Fri, 2009-12-04 at 16:13 -0800, Daniel B. Thurman wrote:
The problem I have specifically with rsync (or tar, or cp) is that it does not save ACLs, file attributes too well, and so on that I gave up using it. Perhaps the problem in this case is not to use the -a option but to use the manual options to save everything about the files that you can.
Specifically: rsync manual: -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X) -H, --hard-links preserve hard links -A, --acls preserve ACLs (implies -p) -X, --xattrs preserve extended attributes
Do you mean that you've tried these (particularly -A and -X) and they don't work? If so, have you filed a bug report?
I am not certain that it's a bug per-se, it's just that there are cases, or so it seems, where it is not clear how to deal with the issue on my part. For example, hard links (-H). How are hard links handled on one drive to be copied over to another and is it guaranteed to work? I could not get my mind around this one so I did not want to take a chance.
Well, you could try it. Since you're copying to a fresh drive anyway, there's no harm in doing it and checking the result.
Hard links are an easy case in fact, and trivial to check. Create file A and hard link B to it. Rsync A and B to a different filesystem, using the -H option. Use "ls -i" on the copied files to see that they both have the same inode. That's all a hard link is.
As for ACLs (-A), what exactly is being handled here and does this work for ALL Oses concerned - do they follow the same "standard"?
WTF? Of course not. I thought we were talking about Linux here. In fact there's been no indication in this thread of anything to the contrary. There is no "standard" for this stuff across OSes (I suppose Posix might be considered a standard but I'd be very careful about relying on it). We're doing low-level system maintenance here. Don't expect anything to be portable.
I am thinking about Vista in particular, so I did not want to experiment on this one either.
I wouldn't dream of doing this on a non-Unix system.
As for extended attributes (-X), I have no clue exactly what this is. I guess I have to someday take the time to do more research before messing around with these rsync options. This is a particular reason why I am using dd/rescue & resizing - it works sans Vista, which I have yet to try.
Again, rsync is a *Unix* utility, designed for *Unix* filesystems *only*. If you'd said at the beginning that you wanted to move a Vista partition we could have saved ourselves a lot of time.
poc
I was being general. The same issue applies to linux when it comes to rsync. If one relies on rsync to keep all the file atttibutes with -a option, there are surprises as I mentioned. I threw in Vista just to make a point that rsync does not work with Vista (or Xp), as you already know. It pays to know what these rsync options do and what OSes rsync works with, again which you already know, but it is written here for those that don't - me included. I learned the hard way by testing it all out so that I know what works and what doesn't. I found that rsync does copy the data on Xp or Vista, but sans the file attributes. At least with a failing drive most of the data was "saved" when in a hurry. It sure is fast.
Getting back on topic - it is dd/rescue that works for (some) OSes and so far it does. I discovered that it definitely works with XP so as long as the partition copied from source to destination is the same partition# - otherwise, one is forced to 'fixBoot' because somehow the partition data (boot.ini) and the partition "MBR" are not "in sync". Vista on the other hand does not work even if dd'ing source to destination, with the same partition#s.
I am fighting a battle on another system trying to get XP & Vista's boot partition to work where the partition to partition copy are not the same, local or to different drives.
FWIW, Dan
On Sun, 2009-12-06 at 12:17 -0800, Dan Thurman wrote:
Again, rsync is a *Unix* utility, designed for *Unix* filesystems *only*. If you'd said at the beginning that you wanted to move a
Vista
partition we could have saved ourselves a lot of time.
poc
I was being general. The same issue applies to linux when it comes to rsync. If one relies on rsync to keep all the file atttibutes with -a option, there are surprises as I mentioned.
But you still haven't said what the "surprises" are. Please be specific. You seem to be saying that an important Linux utility has a serious bug. You need to describe exactly what it is.
I threw in Vista just to make a point that rsync does not work with Vista (or Xp), as you already know. It pays to know what these rsync options do and what OSes rsync works with, again which you already know, but it is written here for those that don't - me included. I learned the hard way by testing it all out so that I know what works and what doesn't. I found that rsync does copy the data on Xp or Vista, but sans the file attributes. At least with a failing drive most of the data was "saved" when in a hurry. It sure is fast.
rsync doesn't actually run on any Windows system AFAIK. I think what you're trying to say is that it copies the data *from* XP or Vista filesystems mounted on Linux, *to* other filesystems also mounted on Linux (which may in turn be any fs Linux supports). Obviously it can do this, but as far as rsync is concerned, it's just a bunch of files and directories. For the same reason, it doesn't copy (say) NTFS attribute info since it knows nothing about it, but does copy Linux attribute info since that's what it's designed to do (and if it doesn't, we have a bug which needs to be reported).
Getting back on topic - it is dd/rescue that works for (some) OSes and so far it does. I discovered that it definitely works with XP so as long as the partition copied from source to destination is the same partition#
- otherwise, one is forced to 'fixBoot' because somehow the partition
data (boot.ini) and the partition "MBR" are not "in sync". Vista on the other hand does not work even if dd'ing source to destination, with the same partition#s.
dd simply copies data from A to B, with no interpretation of its structure, i.e. it's a lower-level function than rsync. If an NTFS filesystem copied this way needs to be fixed up to live on its new partition, that to me would seem to indicate brokenness in the design of NTFS (e.g. that it's not inherently relocatable without patching), but let's not go there.
poc