On Thu, 1 Jan 2009, Kostas Georgiou wrote:
On Thu, Jan 01, 2009 at 01:17:38AM -0600, Mike McGrath wrote:
> On Thu, 1 Jan 2009, James Antill wrote:
> > On Wed, 2008-12-31 at 14:42 -0600, Mike McGrath wrote:
> > > Lets pool some knowledge together because at this point, I'm missing
> > > something.
> > >
> > > I've been doing all measurements with sar as bonnie, etc, causes
> > > timeout.
> > >
> > > Problem: We're seeing slower then normal disk IO. At least I think
> > > are. This is a PERC5/E and MD1000 array.
> > >
> > > When I try to do a normal copy "cp -adv /mnt/koji/packages
/tmp/" I get
> > > around 4-6MBytes/s
> > This _might_ not be "IO" in a normal sense, -a to cp means:
> > file data + file inode + ACLs + selinux + xattrs [+ file capabilities]
> > ...esp. given that you aren't getting large IOWait times, you might want
> > to strace -T the cp and do some perl/whatever on the result to see what
> > is eating up the time.
> Even with non cp type things (like a bacula backup) it just doesn't seem
> as fast as I would expect it to be. I've never actually done trending at
> this level / scale on a filesystem / drive before. So I really don't have
> a good baseline except that it just seems slow to me.
> Other then the much faster direct block access and the large file reads, I
> don't have much else to go on that makes me think its slow.
Do writes show the same pattern? If you use selinux/ACLs/xattrs the default
inode size of 128 can cause slowdowns (#205161 for example).
One reason I'm trying to ramp this up now is because the koji share is
still under 50% utilized. If it turns out to be something in the
filesystem, its not too late for us to shrink the main filesystem, create
the new, copy, and grow the new.
Can you run blktrace+seekwatcher (both in EPEL) to get an idea on
what is going on? An iostat -x -k /dev/sde 1 output will also be
I'll take a look at those two applications as well, here's the iostat:
Linux 2.6.18-92.1.18.el5xen (xen2.fedora.phx.redhat.com
avg-cpu: %user %nice %system %iowait %steal %idle
0.55 0.01 1.35 0.10 6.28 91.71
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await
sde 1389.22 95.13 161.74 270.46 6693.75 1670.16 38.70 1.09 2.51