On Mon, Mar 30, 2015 at 12:58 PM, Dave Johansen <davejohansen(a)gmail.com> wrote:
I noticed on RHEL 6 that when a large amount of disk I/O is happening
that
CPU bound tasks "slow down". I have been able to reproduce it in Fedora 21
as well and here are the instructions of how I can reproduce it with a
simple test:
1) Build the disk_test.cc (the "CPU bound task") and run it.
2) Create a large file to copy ( fallocate -l 10G junk ).
3) Copy that file with a one minute delay between copies ( while true; do cp
junk junk2; sleep 60; done )
If you direct the output of disk_test.cc to a file, then you can plot the
results in gnuplot with the following commands to see the change in the mean
time between "finishing the work cycle" when the file is being copied:
set xdata time
set timefmt "%s"
plot "out.txt" using 1:3 with lines
You can also notice that the load average is also going up, so it seems like
something in the kernel/scheduler is getting some sort of exclusive lock in
the disk I/O process and that's causing the CPU bound task to not be able to
execute when it should. Any ideas?
Thanks,
Dave
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct:
http://fedoraproject.org/code-of-conduct
This has been an issue in Linux for a long time. It was old news in
2008 when I dug into it. Basically, the response you will get when you
say "It hurts when I do that" is "Don't do that!" ;-)
As a well-known performance engineer once said, "The only good I/O is
a dead I/O". Cache as if your life depended on it, use SSDs when you
can afford them, etc. If you *must* build on top of rotating disks,
there are tools to tune the I/O schedulers - blktrace and systemtap
are the best ones at the low level.
By the way, the Linux "load average" counts both CPU-bound tasks ("R"
state) and I/O-bound tasks ("D" state). That's why the load average is
going up. But the cure is to not design I/O-bound applications. ;-)
--
OSJourno: Robust Power Tools for Digital Journalists
http://www.znmeb.mobi/stories/osjourno-robust-power-tools-for-digital-jou...
Remember, if you're traveling to Bactria, Hump Day is Tuesday and Thursday.