"AvdV" == Arjan van de Ven arjan@infradead.org writes:
AvdV> "dmraid" is the component that does the raid, and I think it has AvdV> some form of monitoring, but I don't know all that much about AvdV> it, I hope that at least it's a start for you to look for more AvdV> info...
dmraid will alert you if the RAID set becomes incomplete or experiences an error.
smartd will alert you if the disks themselves experience errors (even ones that are remapped by the drive, in my experience).
I don't know if there's any sort of functionality like the drive scrubbing that 3ware cards have. (They periodically read, reconstruct if necessary and rewrite all of the sectors on all of the drives, which prevents sectors from going bad silently.) You can run a cron job that asks the drives in your RAID set to do online self fests; smartd may even support doing that automatically.
- J<
On Wed, 2007-07-04 at 12:36 -0500, Jason L Tibbitts III wrote:
I don't know if there's any sort of functionality like the drive scrubbing that 3ware cards have. (They periodically read, reconstruct if necessary and rewrite all of the sectors on all of the drives, which prevents sectors from going bad silently.) You can run a cron job that asks the drives in your RAID set to do online self fests; smartd may even support doing that automatically.
As far as I can tell, you can get SMART to check drives, but it offers nothing to fix them up.
At 12:50 PM +0930 7/5/07, Tim wrote:
On Wed, 2007-07-04 at 12:36 -0500, Jason L Tibbitts III wrote:
I don't know if there's any sort of functionality like the drive scrubbing that 3ware cards have. (They periodically read, reconstruct if necessary and rewrite all of the sectors on all of the drives, which prevents sectors from going bad silently.) You can run a cron job that asks the drives in your RAID set to do online self fests; smartd may even support doing that automatically.
As far as I can tell, you can get SMART to check drives, but it offers nothing to fix them up.
SMART's automatic offline test will scan the drive's surface every 4 hours, correcting soft errors and remapping any bad sectors that could still be read. However, the main goal of SMART is to give about a day's warning of impending drive failure, so if SMART has anything to say it's usually rather late to do much repair, but rather time to make one last backup. (That's the goal; whether SMART succeeds is a different matter.) See man smartctl to enable --offlineauto. Note that, unlike RAID, if a sector on a drive can't be read, it's data is lost.
Tim:
As far as I can tell, you can get SMART to check drives, but it offers nothing to fix them up.
Tony Nelson:
SMART's automatic offline test will scan the drive's surface every 4 hours, correcting soft errors and remapping any bad sectors that could still be read.
Well, the manual off-line test didn't do anything helpful about correcting errors when I tried it. It discovered some, but that was all.
However, the main goal of SMART is to give about a day's warning of impending drive failure, so if SMART has anything to say it's usually rather late to do much repair, but rather time to make one last backup.
I tend to agree. However, I've found SMART complain about a drive that nothing else had a problem with. I've had a dodgy drive that SMART didn't warn about. I've had a dodgy drive that SMART did warn about, yet also said it passed the health checks. :-\ I don't have much faith in it anymore.
I'd been giving a few duff drives a while back. Things like that are always useful to experiment with. I did find that using dd if=/dev/zero of=/dev/hdb to write to all sectors on the drive caused the drive to sort itself out. Afterwards it passed the SMART check with flying colours.
The drives have, since, also been reliable in two test boxes with FC4, 6 & 7 over several months. Before the zeroing, they were continuously getting errors like the one mentioned below, and (as expected) mangling file contents.
hdb: drive_cmd: error=0x04 { DriveStat ...: 17 Time(s) hdb: drive_cmd: status=0x51 { DriveReady SeekComplete Error } ...: 17 Time(s)
The first zeroing out resulted in slightly less sectors *working* than were tried. The second zeroing used the same number as *worked* in the first time. i.e. Bad blocks had been mapped out, the drive was slightly smaller (now), and that portion of the drive was okay for use.
Before someone points out the obvious, these are test/play boxes. I'd be loathe to trust doing this for drives with data I wanted to keep.
At 2:14 PM +0930 7/5/07, Tim wrote:
Tim:
As far as I can tell, you can get SMART to check drives, but it offers nothing to fix them up.
Tony Nelson:
SMART's automatic offline test will scan the drive's surface every 4 hours, correcting soft errors and remapping any bad sectors that could still be read.
Well, the manual off-line test didn't do anything helpful about correcting errors when I tried it. It discovered some, but that was all.
If the error is correctable (the data could be read) then the disk will rewrite the data and remap the sector if it is bad. If the data could not be read, then the sector is put in a list of offline-uncorrectable errors and the sector will be reallocated the next time it is read.
When the off-line long test (not the short one) is run, it has been more than 4 hours since the last test. The advantage of the automatic offline test is that it sees each sector frequently, and may be able to rescue the data before it is lost. What you did is not comparable. It's rather more like the difference between taking your car in for scheduled routine maintenance and waiting for it to break and then getting it fixed. Much more may be wrong in the latter case, and it may not be fixable.
...
I'd been giving a few duff drives a while back. Things like that are always useful to experiment with. I did find that using dd if=/dev/zero of=/dev/hdb to write to all sectors on the drive caused the drive to sort itself out. Afterwards it passed the SMART check with flying colours.
This is not the same as a low-level format. What you did used some of the spare sectors, leaving fewer spare sectors for remapping in normal operation, and requiring a long seek for each spared sector. Doing a low-level format with the manufacturer's utility skips bad sectors in-line, with no long seeks, and makes a full-size spare sector list.
The drives have, since, also been reliable in two test boxes with FC4, 6 & 7 over several months. ...
...
Having sectors go bad is normal for modern hard drives, both because of their high density and their large number of sectors. If sectors go bad at a low rate, it is simply a waste to replace the drive with one that will hopefully perform the same way and not just fail early.
Tim:
Well, the manual off-line test didn't do anything helpful about correcting errors when I tried it. It discovered some, but that was all.
Tony Nelson:
If the error is correctable (the data could be read) then the disk will rewrite the data and remap the sector if it is bad. If the data could not be read, then the sector is put in a list of offline-uncorrectable errors and the sector will be reallocated the next time it is read.
And if it can never read that sector, you see that error each and every time. Of course I have no idea which file is on the dodgy point, it's not like you see "error trying to read named.conf" *then* the hda error message, it's not obviously correlatable to anything.
When the off-line long test (not the short one) is run, it has been more than 4 hours since the last test.
Yes, it was the long test, and it was done a few times over a few days, as I did a few experiments with things.
The advantage of the automatic offline test is that it sees each sector frequently, and may be able to rescue the data before it is lost. What you did is not comparable.
There's no indication in the manual, or other files, that the auto tests does anything other than test. I didn't see anything, anywhere, that'd indicate that things would get fixed.
Anyway, with this error (being unable to read something), the only fix would have been trying to replace the (unknown) data with the correct information from an external source.
I'd been giving a few duff drives a while back. Things like that are always useful to experiment with. I did find that using dd if=/dev/zero of=/dev/hdb to write to all sectors on the drive caused the drive to sort itself out. Afterwards it passed the SMART check with flying colours.
This is not the same as a low-level format. What you did used some of the spare sectors, leaving fewer spare sectors for remapping in normal operation, and requiring a long seek for each spared sector.
Oh I know that. It was just a way of exercising the drive with unimportant data, so it can sort itself out. It (the drive) found an error, it moved that part of the drive out of further use.
Even an ordinary high-level format should have done the same trick, though I'd imagine you'd have to do it twice (I'd forsee it finding an error and complaining, the first time), but Linux doesn't really format the drive during the install routine.
After doing that zeroing trick with the last drive, I formatted the drives myself, using the checking option, to see if any problems were discovered, before the install routine. Nothing came up.
e.g. mkfs.ext3 -c -v -L lazarus/home /dev/sdb3
A drive check used to be an install option with older Red Hat releases.
The drives have, since, also been reliable in two test boxes with FC4, 6 & 7 over several months. ...
Having sectors go bad is normal for modern hard drives, both because of their high density and their large number of sectors. If sectors go bad at a low rate, it is simply a waste to replace the drive with one that will hopefully perform the same way and not just fail early.
That's my way of thinking, too. They have some in-built level of error handling, for that reason. So long as the faults are insignificant, you're okay. I'm certainly not keen on coughing up $150 for a new drive, *just* because two sectors stopped working, for instance.
Tim wrote:
Having sectors go bad is normal for modern hard drives, both because of their high density and their large number of sectors. If sectors go bad at a low rate, it is simply a waste to replace the drive with one that will hopefully perform the same way and not just fail early.
That's my way of thinking, too. They have some in-built level of error handling, for that reason. So long as the faults are insignificant, you're okay. I'm certainly not keen on coughing up $150 for a new drive, *just* because two sectors stopped working, for instance.
Modern drives have spare sectors that are mapped into use internally if bad ones are detected. However this is normally transparent. If SMART says you've had a few errors that were recovered I wouldn't worry about it. On the other hand if you are seeing errors at the OS level it probably means you are out of usable spares and the drive is about to go. Most manufacturers have an on-line warranty check - if it is still covered, get a replacement. If it isn't, you can probably get twice the space for the same price now.
"T" == Tim ignored_mailbox@yahoo.com.au writes:
T> As far as I can tell, you can get SMART to check drives, but it T> offers nothing to fix them up.
If a drive in your array starts throwing errors, fixing it up consists of removing it from the array and sending it back to the manufacturer for replacement.
Unless, of course, you don't actually care about your data. Drive errors are hardware errors.
- J<