ScriptBits, Little Bits Of Code You Cannot Live Without

An easy way to identify what process is generating your IO Wait load is to enable block I/O debugging. This is done by setting /proc/sys/vm/block_dump to a non zero value like:

echo 1 > /proc/sys/vm/block_dump

This will cause messages like the following to start appearing in dmesg:

bash(6856): dirtied inode 19446664 (ld-2.5.so) on md1

Using the following one-liner will produce a summary output of the dmesg entries:

dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
    354 md
    324 export
    288 kjournald
     53 irqbalance
     45 pdflush
     14 portmap
     14 bash
     10 egrep
     10 crond
      8 ncftpput

Once you are finished you should disable block I/O debugging by setting /proc/sys/vm/block_dump to a zero value like:

echo 0 > /proc/sys/vm/block_dump