How to identify what processes are generating IO Wait load.
An easy way to identify what process is generating your IO Wait load is to enable block I/O debugging. This is done by setting /proc/sys/vm/block_dump to a non zero value like:
echo 1 > /proc/sys/vm/block_dump
This will cause messages like the following to start appearing in dmesg:
bash(6856): dirtied inode 19446664 (ld-2.5.so) on md1
Using the following one-liner will produce a summary output of the dmesg entries:
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
354 md
324 export
288 kjournald
53 irqbalance
45 pdflush
14 portmap
14 bash
10 egrep
10 crond
8 ncftpput
Once you are finished you should disable block I/O debugging by setting /proc/sys/vm/block_dump to a zero value like:
echo 0 > /proc/sys/vm/block_dump
Link to this post!
July 9th, 2009 at 11:12
Another option is to use the "pidstat" tool part of the sysstat package on ubuntu.
Jun run "pidstat -d 5" and it will print the list of processes doing IO over the last 5 seconds.
July 9th, 2009 at 13:48
Good suggestion.
At least for RH/CentOS pidstat is not part of the main line repo yet. pidstat was introduced in sysstat v8.0 and CentOS (5.3) is still using v7.0.2.
July 9th, 2009 at 14:02
Kernel 2.6.20 and above maybe?
July 11th, 2009 at 08:58
No both distros (and i think most/all others) are on a 2.6 kernel now. Its just the difference in the systat package version. Hopefully RH/CentOS will incorporate the newer version in a future release.