I have the f12 Beta dvd images run through "split -b 1G" on a vfat usb stick. For both the i36 and the x86_64 images when I do something like this:
cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
The sha256sum -c fails on the resulting file :-(. (The checksums checked OK on the original files I downloaded).
Then I do this:
cat Fedora-12-Beta-x86_64-DVD.isoaa > ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoab >> ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoac >> ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoad >> ~/Fedora-12-Beta-x86_64-DVD.iso
The resulting file is the same size as the one I got the 1st time, but this file passes the sha256sum -c.
Has bash started randomly ordering wildcard expansion or something?
Has my usb stick flaked out and only works right when I read the same set of files twice in a row?
I got the exact same results getting the i386 images off the usb stick as well.
What is going on?
On Tue, 2009-10-20 at 19:44 -0400, Tom Horsley wrote:
Has my usb stick flaked out and only works right when I read the same set of files twice in a row?
Are the split files still on the USB stick? What happens if you copy them to your hard drive and then do the cat?
poc
Tom Horsley wrote:
I have the f12 Beta dvd images run through "split -b 1G" on a vfat usb stick. For both the i36 and the x86_64 images when I do something like this:
cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
The sha256sum -c fails on the resulting file :-(. (The checksums checked OK on the original files I downloaded).
Then I do this:
cat Fedora-12-Beta-x86_64-DVD.isoaa > ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoab >> ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoac >> ~/Fedora-12-Beta-x86_64-DVD.iso cat Fedora-12-Beta-x86_64-DVD.isoad >> ~/Fedora-12-Beta-x86_64-DVD.iso
The resulting file is the same size as the one I got the 1st time, but this file passes the sha256sum -c.
Has bash started randomly ordering wildcard expansion or something?
I don't think bash has ever ordered the files. Instead, they're listed in the order matched.
Where order matters, something modelled on this: echo $(\ls Derf*) or echo $(ls -1 Der* | sort)
ls supports ordering on other criteria than filename, even for short lists.
That said, my testing on RHEL6-clone just now doesn't support my contention, so maybe I misremember something.
btw FAT32 does support files larger than 1 Gb, and you can format them ext2 if you wish. I'd not bother with journals in the ordinary course of events.
<after further research>
DESCRIPTION The scandir() function scans the directory dir, calling filter() on each directory entry. Entries for which filter() returns non-zero are stored in strings allocated via malloc(), sorted using qsort() with the comparison function compar(), and collected in array namelist which is allocated via malloc(). If filter is NULL, all entries are selected.
Why you appeared to have the files concatenated in the wrong order, I'm running out of ideas. This might help elucidate:
echo \ cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
Note I escaped the ">" so as to prevent its interpretation by your shell. This command will show you what bash thinks it's doing.
On Wed, 21 Oct 2009 08:33:49 +0800 John Summerfield wrote:
btw FAT32 does support files larger than 1 Gb
Yea, but not bigger than 2 gig, at least it has always refused in the past, so I just split them at 1 because that is well within any limit that might crop up.
echo \ cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
Yea, I tried doing an echo and they sure looked like the right order. My best guess at the moment is some sort of intermittent bit dropping in the usb stick, but it takes a long time to copy that much data, so I'm not really up for doing lots of different tests, at least I finally got the files on my system.
Tom Horsley wrote:
Yea, I tried doing an echo and they sure looked like the right order. My best guess at the moment is some sort of intermittent bit dropping in the usb stick, but it takes a long time to
Are USB sticks and such inherently that slow? I've got 20 Mbytes/sec and better off laptop drives in enclosures, but haven't used the sticks much as they're too small or too expensive. However, I bought a card reader and was dismayed to find ot takes 70 minutes and more to copy a full 4Gbyte CF disk (Sandisk Ultra II) supposedly capable of 15 Mbytes/sec.
copy that much data, so I'm not really up for doing lots of different tests, at least I finally got the files on my system.
Tom Horsley wrote:
On Wed, 21 Oct 2009 08:33:49 +0800 John Summerfield wrote:
btw FAT32 does support files larger than 1 Gb
Yea, but not bigger than 2 gig, at least it has always refused in the past, so I just split them at 1 because that is well within any limit that might crop up.
According to MS:
You cannot create a file larger than (2^32)-1 bytes (this is one byte less than 4 GB) on a FAT32 partition.
For other limitations, see http://support.microsoft.com/kb/314463/EN-US/
Once upon a time, John Summerfield debian@herakles.homelinux.org said:
Tom Horsley wrote:
Has bash started randomly ordering wildcard expansion or something?
I don't think bash has ever ordered the files. Instead, they're listed in the order matched.
Nope, per the standard, shell glob results "sorted according to the collating sequence in effect in the current locale" (from IEEE Std 1003.1-2008).
The only thing I can think of is a random bit-flip on your USB stick. The shell and "cat" are large-file safe, so even if you were running this on a 32 bit host, you should have no problems.
On Wednesday 21 of October 2009 01:44:13 Tom Horsley wrote:
I have the f12 Beta dvd images run through "split -b 1G" on a vfat usb stick. For both the i36 and the x86_64 images when I do something like this:
cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
The sha256sum -c fails on the resulting file :-(. (The checksums checked OK on the original files I downloaded).
something like that happened to me too ...
I needed to copy a really huge file via ssh over slow Internet connection, that got disconnected from time to time
so I had split the file in parts
I've run md5sum on the transferred parts to ensure everything was copied ok
still, after getting the pieces together, the checksum on the combined file failed
I've tried that several times (including splitting and transferring again) with no success
I wanted to file a bug, but eventually, I got to the other place with notebook and copied the data over local network, then I forgot about the case
so, please, try to reproduce the problem, and file the bug for me ;-)
(and no, I do not suspect my hardware - copying huge files is without any problem, no other checksum has ever failed on any of those two involved machines, so it would be really strange that checking all the pieces passes, and magically there creeps in some bad bit on each try to combine those pieces)
K.
Karel Voln wrote:
On Wednesday 21 of October 2009 01:44:13 Tom Horsley wrote:
I have the f12 Beta dvd images run through "split -b 1G" on a vfat usb stick. For both the i36 and the x86_64 images when I do something like this:
cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
The sha256sum -c fails on the resulting file :-(. (The checksums checked OK on the original files I downloaded).
something like that happened to me too ...
I needed to copy a really huge file via ssh over slow Internet connection, that got disconnected from time to time
I expect that if you can use ssh, you can also use rsync. With suitable options (and they are legion), rsync will happily resume and, if you exercise some skill, repair large files.
Don't use checksum on a public server you don't own, I was discussing this with Jason from planetmirror one time and apparently it can kill the server.
Or sftp, but I've not used that much to find out about its resume/recovery options.
On Thursday 22 of October 2009 03:44:49 John Summerfield wrote:
Karel Voln wrote:
On Wednesday 21 of October 2009 01:44:13 Tom Horsley wrote:
I have the f12 Beta dvd images run through "split -b 1G" on a vfat usb stick. For both the i36 and the x86_64 images when I do something like this:
cat Fedora-12-Beta-x86_64-DVD.isoa* > ~/Fedora-12-Beta-x86_64-DVD.iso
The sha256sum -c fails on the resulting file :-(. (The checksums checked OK on the original files I downloaded).
something like that happened to me too ...
I needed to copy a really huge file via ssh over slow Internet connection, that got disconnected from time to time
I expect that if you can use ssh, you can also use rsync. With suitable options (and they are legion), rsync will happily resume and, if you exercise some skill, repair large files.
don't know ... can it run over ssh tunnel?
oh forget the question (I can RTFM anyway) ... I've resolved the issue in another way
I just wanted to share the story so that Tom knows he's not alone with such a problem and he does not need to suspect his hardware but rather split
Don't use checksum on a public server you don't own, I was discussing this with Jason from planetmirror one time and apparently it can kill the server.
also good to know ;-)
K.