Why is there a huge difference in disk I/O performance with different ioengines using FIO?
Environment
- Red Hat Enterprise Linux 7
Issue
FIOperformance benchmarking test results differ from one ioengine to another.- Why does
libaiorandwrite(w/s) perform many times better thansync?
Resolution
-
It is an
expected behaviourto get better performance withlibaioioengine than that ofsync. -
For more details regarding
fiocommand, refermanpage offio.
Root Cause
- Using
"-ioengine=sync"would bypass the cache and directly access the disk, which is whyhigh disk utilization (%util)is observed whereas "-ioengine=libaio" makes uses of cache. Hence, a quite better performance is observed withlibaio.
Diagnostic Steps
-
Run the following commands specifying
"-ioengine=sync -fdatasync=1"in one and"-ioengine=libaio"in the other:# fio -direct=1 -filename=/fio-sync.txt -iodepth=64 -rw=randwrite -ioengine=sync -fdatasync=1 -bs=4k -size=50G -numjobs=32 -runtime=1000 -group_reporting -name=write # fio -direct=1 -filename=/fio-libaio.txt -iodepth=64 -rw=randwrite -ioengine=libaio -bs=4k -size=50G -numjobs=32 -runtime=1000 -group_reporting -name=write -
iostat outputs for
syncandlibaiorespectively.avg-cpu: %user %nice %system %iowait %steal %idle 0.48 0.00 0.50 10.00 0.00 89.01 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdc 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdf 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sde 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdd 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdi 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdg 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdh 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdj 0.00 0.00 0.00 1217.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdk 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdl 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.99 1.63 0.00 1.63 0.82 100.00 sdn 0.00 0.00 0.00 1218.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdm 0.00 0.00 0.00 1219.00 0.00 2.38 4.00 1.98 1.63 0.00 1.63 0.82 100.00 sdq 0.00 0.00 0.00 1203.00 0.00 2.35 4.00 0.99 0.82 0.00 0.82 0.82 98.80 sdp 0.00 0.00 0.00 1202.00 0.00 2.35 4.00 0.99 0.82 0.00 0.82 0.82 98.60 sds 0.00 0.00 0.00 1204.00 0.00 2.35 4.00 0.99 0.82 0.00 0.82 0.82 98.90 avg-cpu: %user %nice %system %iowait %steal %idle 2.15 0.00 6.54 0.00 0.00 91.31 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 6026.00 0.00 23.54 8.00 0.28 0.05 0.00 0.05 0.03 18.90 sdc 0.00 0.00 0.00 6099.00 0.00 23.82 8.00 0.27 0.05 0.00 0.05 0.03 17.70 sdf 0.00 0.00 0.00 25699.00 0.00 100.39 8.00 40.86 1.57 0.00 1.57 0.03 66.80 sde 0.00 0.00 0.00 6016.00 0.00 23.50 8.00 0.24 0.04 0.00 0.04 0.03 17.80 sdd 0.00 1.00 0.00 17889.00 0.00 69.88 8.00 24.10 1.34 0.00 1.34 0.03 47.70 sdi 0.00 0.00 0.00 6016.00 0.00 23.50 8.00 0.26 0.04 0.00 0.04 0.03 17.30 sdg 0.00 0.00 0.00 6060.00 0.00 23.67 8.00 0.25 0.04 0.00 0.04 0.03 17.50 sdh 0.00 0.00 0.00 6079.00 0.00 23.75 8.00 0.27 0.04 0.00 0.04 0.03 18.80 sdj 0.00 0.00 0.00 6042.00 0.00 23.60 8.00 0.28 0.05 0.00 0.05 0.03 17.60 sdk 0.00 0.00 0.00 6030.00 0.00 23.55 8.00 0.25 0.04 0.00 0.04 0.03 17.10 sdl 0.00 0.00 0.00 6053.00 0.00 23.64 8.00 0.25 0.04 0.00 0.04 0.03 20.60 sdn 0.00 0.00 0.00 6057.00 0.00 23.66 8.00 0.27 0.05 0.00 0.05 0.03 20.40 sdm 0.00 0.00 0.00 6020.00 0.00 23.52 8.00 0.29 0.05 0.00 0.05 0.03 18.80 sdq 0.00 0.00 0.00 3044.00 0.00 11.89 8.00 0.16 0.05 0.00 0.05 0.03 8.50 sdp 0.00 0.00 0.00 3032.00 0.00 11.84 8.00 0.13 0.04 0.00 0.04 0.03 10.00 sds 0.00 0.00 0.00 3008.00 0.00 11.75 8.00 0.13 0.04 0.00 0.04 0.02 7.40
SBR
Product(s)
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.