I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
My disks (same model drive for both sda and sdb) /boot - /dev/md1 (/dev/sda1 & /dev/sdb1) / - /dev/md6 (/dev/sda6 & /dev/sdb6) swap - /dev/md3 (/dev/sda3 & /dev/sdb3) /var - /dev/md5 (/dev/sda5 & /dev/sdb5) /usr - /dev/md2 (/dev/sda2 & /dev/sdb2) /home - /dev/md7 (/dev/sda7 & /dev/sdb7)
The system gets to:
Waiting for driver initialization. md: md6 stopped. mdadm: no devices found for /dev/md6 md: md3 stopped. mdadm: no devices found for /dev/md3 Trying to resume from /dev/md3 Creating root device. Mounting root filesystem. EXT3-fs: unable to read superblock mount: error mounting /dev/root on /sysroot as ext3: Invalid argument Setting up other filesystems. Setting up noew root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Mount failed for selinuxfs on /selinux: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Booting has failed.
I've had this problem in the past and I've checked the past suggestions and I am still unable to get the system to boot.
What I have done so far (while using the rescue CD)
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Any suggestions would really be appreciated.
Thanks,
Jeff
On Sat, Oct 04, 2008 at 10:04:46 -0400,
Any suggestions would really be appreciated.
Did you change anything just before this started happening? I had a very similar problem after trying to yum upgrade from FC6 to F9. Running mkinitrd under the older kernel did not generate the correct set of modules for use with the newer kernel and I needed to run mkinitrd under the newer kernel.
I think what I did was boot with a rescue disk, chroot /mnt/sysimage (I don't remember the exact name, so double check that) and then did a yum reinstall kernel.
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
My disks (same model drive for both sda and sdb) /boot - /dev/md1 (/dev/sda1 & /dev/sdb1) / - /dev/md6 (/dev/sda6 & /dev/sdb6) swap - /dev/md3 (/dev/sda3 & /dev/sdb3) /var - /dev/md5 (/dev/sda5 & /dev/sdb5) /usr - /dev/md2 (/dev/sda2 & /dev/sdb2) /home - /dev/md7 (/dev/sda7 & /dev/sdb7)
The system gets to:
Waiting for driver initialization. md: md6 stopped. mdadm: no devices found for /dev/md6 md: md3 stopped. mdadm: no devices found for /dev/md3 Trying to resume from /dev/md3 Creating root device. Mounting root filesystem. EXT3-fs: unable to read superblock mount: error mounting /dev/root on /sysroot as ext3: Invalid argument Setting up other filesystems. Setting up noew root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Mount failed for selinuxfs on /selinux: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Booting has failed.
I've had this problem in the past and I've checked the past suggestions and I am still unable to get the system to boot.
What I have done so far (while using the rescue CD)
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Any suggestions would really be appreciated.
Thanks,
Jeff
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
My disks (same model drive for both sda and sdb) /boot - /dev/md1 (/dev/sda1 & /dev/sdb1) / - /dev/md6 (/dev/sda6 & /dev/sdb6) swap - /dev/md3 (/dev/sda3 & /dev/sdb3) /var - /dev/md5 (/dev/sda5 & /dev/sdb5) /usr - /dev/md2 (/dev/sda2 & /dev/sdb2) /home - /dev/md7 (/dev/sda7 & /dev/sdb7)
The system gets to:
Waiting for driver initialization. md: md6 stopped. mdadm: no devices found for /dev/md6 md: md3 stopped. mdadm: no devices found for /dev/md3 Trying to resume from /dev/md3 Creating root device. Mounting root filesystem. EXT3-fs: unable to read superblock mount: error mounting /dev/root on /sysroot as ext3: Invalid argument Setting up other filesystems. Setting up noew root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Mount failed for selinuxfs on /selinux: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Booting has failed.
I've had this problem in the past and I've checked the past suggestions and I am still unable to get the system to boot.
What I have done so far (while using the rescue CD)
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Any suggestions would really be appreciated.
Thanks,
Jeff
I extracted the initrd file and looked at the file "init". in this file I can see the following (relevant) information..
echo Creating block device nodes. mkblkdevs echo "loading raid1 module" modprobe -q raid1
some other modules look like USB related
echo "Loading ext3 module" modprobe -q raid456 echo "Loading scsi_mod module" modprobe -q scsi_mod echo "Loading sd_mod module" modprobe -q sd_mod
the libata and ahci modules are next
echo Waiting for driver initialization. <-last thing I see before it fails! stabilized --hash --interval 250 /proc/scsi/scsi modprobe scsi_wait_scan rmmod scsi_wait_scan mkblkdevs mdadm -As --auto=yes --run /dev/md3 mdadm -As --auto=yes --run /dev/md6 resume /dev/md3
it appears as the mdadm is not starting properly or assembling the array properly. Can anybody confirm if the init file appears correct? I tried modifying the mdadm line to remove the /dev/md3 (and 6) and it tried to assemble all the volumes but said that /dev/mdX (listed all my array volums) didn't exist with "mdadm: no devices found for /dev/md..."
Thanks, Jeff
jeff@bubble.org wrote:
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
[...]
it appears as the mdadm is not starting properly or assembling the array properly. Can anybody confirm if the init file appears correct? I tried modifying the mdadm line to remove the /dev/md3 (and 6) and it tried to assemble all the volumes but said that /dev/mdX (listed all my array volums) didn't exist with "mdadm: no devices found for /dev/md..."
This is probably too obvious to mention, but are the underlying partition types set to FD (raid autodetect). I don't see why they'd work from the rescue boot if they aren't though unless you are assembling them by hand.
Les Mikesell wrote:
jeff@bubble.org wrote:
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
[...]
it appears as the mdadm is not starting properly or assembling the array properly. Can anybody confirm if the init file appears correct? I tried modifying the mdadm line to remove the /dev/md3 (and 6) and it tried to assemble all the volumes but said that /dev/mdX (listed all my array volums) didn't exist with "mdadm: no devices found for /dev/md..."
This is probably too obvious to mention, but are the underlying partition types set to FD (raid autodetect). I don't see why they'd work from the rescue boot if they aren't though unless you are assembling them by hand.
I wish that was the problem but alas, I checked, all the partitions are set to 0xfd, and the rescue disk is automatically assembling them.
Jeff
jeff@bubble.org wrote:
I extracted the initrd file and looked at the file "init". in this file I can see the following (relevant) information..
echo Creating block device nodes. mkblkdevs echo "loading raid1 module" modprobe -q raid1
Don't you need a "modprobe md" before this? Did you build this configuration with the installer or have you copied a working system to a different machine?
Les Mikesell wrote:
jeff@bubble.org wrote:
I extracted the initrd file and looked at the file "init". in this file I can see the following (relevant) information..
echo Creating block device nodes. mkblkdevs echo "loading raid1 module" modprobe -q raid1
Don't you need a "modprobe md" before this? Did you build this configuration with the installer or have you copied a working system to a different machine?
Not sure if I need the modprobe md first, I thought (incorrectly?) that when you load a module it loads all the required modules with it.
This was a system converted to a RAID 1 setup a while back, after the raid was configured I had rebuild the initrd with the following command: mkinitrd --preload=raid1 initrd-2.6 ......
The exact method I used can be found in this message I sent to the list about a year ago: http://www.redhat.com/archives/rhl-list/2007-June/msg06369.html
Jeff
Jeffrey Ross wrote:
Les Mikesell wrote:
jeff@bubble.org wrote:
I extracted the initrd file and looked at the file "init". in this file I can see the following (relevant) information..
echo Creating block device nodes. mkblkdevs echo "loading raid1 module" modprobe -q raid1
Don't you need a "modprobe md" before this? Did you build this configuration with the installer or have you copied a working system to a different machine?
Not sure if I need the modprobe md first, I thought (incorrectly?) that when you load a module it loads all the required modules with it.
This was a system converted to a RAID 1 setup a while back, after the raid was configured I had rebuild the initrd with the following command: mkinitrd --preload=raid1 initrd-2.6 ......
The exact method I used can be found in this message I sent to the list about a year ago: http://www.redhat.com/archives/rhl-list/2007-June/msg06369.html
I don't know enough about the technique to be more specific, but only the modules known to be needed will be included in the initrd. Maybe you need to tell it about md explicitly. I've occasionally found it easier to install a minimal system with a separate /boot partition on new/changed hardware, then copy over everything except /boot (tweaking /etc/fstab and /etc/modprobe.conf as needed) than to try to figure out the names of all the needed modules and the other stuff anaconda knows about.
jeff@bubble.org wrote:
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
What I have done so far (while using the rescue CD)
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Any suggestions would really be appreciated.
Thanks,
Jeff
I extracted the initrd file and looked at the file "init". in this file I can see the following (relevant) information..
it appears as the mdadm is not starting properly or assembling the array properly. Can anybody confirm if the init file appears correct? I tried modifying the mdadm line to remove the /dev/md3 (and 6) and it tried to assemble all the volumes but said that /dev/mdX (listed all my array volums) didn't exist with "mdadm: no devices found for /dev/md..."
Thanks, Jeff
Just thought I'd post a follow-up, the problem has been solved and I learned something new in the process.
I extracted the files from the initrd file but didn't realize that a copy of the mdadm.conf file is stored there as well. Sometime between the last two kernel updates I had modified the /etc/mdadm.conf file with incorrect settings for the arrays, I had then installed a new kernel update which picked up my bad /etc/mdadm.conf file and inserted it into the initrd. Once I learned that a copy of the conf file is stored in the initrd it was a simple process to manually adjust the initrc file with the correct file.
Problem solved!
Only remaining question, if I should adjust the /etc/mdadm.conf file again how can I get the initrc file to update without having to manually uncompress and extract the cpio archive and then re-archive and re-compress it by hand?
Thanks, Jeff
On Sun, Oct 05, 2008 at 07:08:14 -0400, Jeffrey Ross jeff@bubble.org wrote:
Only remaining question, if I should adjust the /etc/mdadm.conf file again how can I get the initrc file to update without having to manually uncompress and extract the cpio archive and then re-archive and re-compress it by hand?
I believe that reinstalling the kernel (yum reinstall kernel) will do this. You should also be able to run mkinitrd with the appropiate options which will do a lot less work than a reinstall.
Jeffrey Ross wrote:
Just thought I'd post a follow-up, the problem has been solved and I learned something new in the process.
I extracted the files from the initrd file but didn't realize that a copy of the mdadm.conf file is stored there as well. Sometime between the last two kernel updates I had modified the /etc/mdadm.conf file with incorrect settings for the arrays, I had then installed a new kernel update which picked up my bad /etc/mdadm.conf file and inserted it into the initrd. Once I learned that a copy of the conf file is stored in the initrd it was a simple process to manually adjust the initrc file with the correct file. Problem solved!
Only remaining question, if I should adjust the /etc/mdadm.conf file again how can I get the initrc file to update without having to manually uncompress and extract the cpio archive and then re-archive and re-compress it by hand?
I thought /etc/mdadm.conf wasn't really necessary anymore - and you said the rescue mode boot worked when it obviously couldn't read it. What happens if you just remove it?
On Sunday 05 October 2008 12:08:14 Jeffrey Ross wrote:
Only remaining question, if I should adjust the /etc/mdadm.conf file again how can I get the initrc file to update without having to manually uncompress and extract the cpio archive and then re-archive and re-compress it by hand?
for version in $( rpm -q --qf '[%{version}-%{release}\n]' kernel mkinitrd -f /boot/initrd-$version $version done
puilt from jeff@bubble.org wrote:
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
I have a few questions below...
My disks (same model drive for both sda and sdb) /boot - /dev/md1 (/dev/sda1 & /dev/sdb1) / - /dev/md6 (/dev/sda6 & /dev/sdb6) swap - /dev/md3 (/dev/sda3 & /dev/sdb3) /var - /dev/md5 (/dev/sda5 & /dev/sdb5) /usr - /dev/md2 (/dev/sda2 & /dev/sdb2) /home - /dev/md7 (/dev/sda7 & /dev/sdb7)
The system gets to:
Waiting for driver initialization. md: md6 stopped. mdadm: no devices found for /dev/md6 md: md3 stopped. mdadm: no devices found for /dev/md3 Trying to resume from /dev/md3 Creating root device. Mounting root filesystem. EXT3-fs: unable to read superblock mount: error mounting /dev/root on /sysroot as ext3: Invalid argument Setting up other filesystems. Setting up noew root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Mount failed for selinuxfs on /selinux: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Booting has failed.
I've had this problem in the past and I've checked the past suggestions and I am still unable to get the system to boot.
What I have done so far (while using the rescue CD)
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Since I've never had to do that on any of 10-12 systems, I'm really suspicious that this is not a step forward.
Any suggestions would really be appreciated.
If you are naming any partitions in the mdadm.conf, that's probably your problem. You want to be using the PARTITIONS method, giving only the UUID as a hint, and letting mdadm find the partitions and put them together. You also may want to check that the correct mdadm.conf file is in your init file (.img) to get correct function, particularly if you have chhanged arrays.
Final suggestion: take this to the linux-raid list, it's highly unlikely that this is a Fedora problem.
Bill Davidsen wrote:
puilt from jeff@bubble.org wrote:
I've again run into a problem with RAID-1 on my system. I can mount all the raid volumes using the linux rescue boot option from the install CD.
I have a few questions below...
verified that the UUID's listed in the /etc/mdadm.conf file are correct (mdadm -D /dev/mdX matches what's in the conf file). Reinstalled grub with "grub-install /dev/sda" and "grub-install /dev/sdb" Reviewed bugzilla bugs 447818 & 444237 and tried modifying the /etc/rc.sysinit file to create the arrays at boot time.
Since I've never had to do that on any of 10-12 systems, I'm really suspicious that this is not a step forward.
Any suggestions would really be appreciated.
If you are naming any partitions in the mdadm.conf, that's probably your problem. You want to be using the PARTITIONS method, giving only the UUID as a hint, and letting mdadm find the partitions and put them together. You also may want to check that the correct mdadm.conf file is in your init file (.img) to get correct function, particularly if you have chhanged arrays.
I'm not completely following you regarding the /etc/mdadm.conf file, currently it has entries such as:
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=e947ca93:d82a9cdf:822a12f9:1933def1
obviously a unique line per raid device (and I verified the UUID numbers are correct). However the initrd...img "init" file makes no reference to this file, how can mdadm get access to this file if the partition really hasn't been mounted yet?
Final suggestion: take this to the linux-raid list, it's highly unlikely that this is a Fedora problem.
I'll post my questions there as well, its probably something simple.
Thanks, Jeff