fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --sync=1 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.28 Starting 1 process fiotest: Laying out IO file (1 file / 8192MiB) Jobs: 1 (f=1): [m(1)][100.0%][r=321MiB/s,w=106MiB/s][r=82.3k,w=27.0k IOPS][eta 00m:00s] fiotest: (groupid=0, jobs=1): err= 0: pid=9908: Thu Apr 27 09:01:43 2023 read: IOPS=81.4k, BW=318MiB/s (333MB/s)(6141MiB/19314msec) bw ( KiB/s): min=207568, max=373304, per=99.92%, avg=325348.63, stdev=43565.60, samples=38 iops : min=51890, max=93326, avg=81337.16, stdev=10891.57, samples=38 write: IOPS=27.2k, BW=106MiB/s (111MB/s)(2051MiB/19314msec); 0 zone resets bw ( KiB/s): min=69456, max=123008, per=99.92%, avg=108641.47, stdev=14384.58, samples=38 iops : min=17364, max=30752, avg=27160.37, stdev=3596.14, samples=38 cpu : usr=26.00%, sys=51.47%, ctx=362376, majf=0, minf=363 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=318MiB/s (333MB/s), 318MiB/s-318MiB/s (333MB/s-333MB/s), io=6141MiB (6440MB), run=19314-19314msec WRITE: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=2051MiB (2150MB), run=19314-19314msec Disk stats (read/write): nvme0n1: ios=1558625/554266, merge=0/6467, ticks=562573/50516, in_queue=613089, util=99.73% fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75 --sync=1 fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.28 Starting 1 process Jobs: 1 (f=1): [m(1)][100.0%][r=330MiB/s,w=111MiB/s][r=84.6k,w=28.4k IOPS][eta 00m:00s] fiotest: (groupid=0, jobs=1): err= 0: pid=11381: Thu Apr 27 09:02:02 2023 read: IOPS=83.9k, BW=328MiB/s (344MB/s)(6141MiB/18727msec) bw ( KiB/s): min=262888, max=372984, per=99.95%, avg=335646.27, stdev=30659.98, samples=37 iops : min=65722, max=93246, avg=83911.57, stdev=7664.99, samples=37 write: IOPS=28.0k, BW=110MiB/s (115MB/s)(2051MiB/18727msec); 0 zone resets bw ( KiB/s): min=88520, max=125488, per=99.94%, avg=112071.14, stdev=10131.25, samples=37 iops : min=22130, max=31372, avg=28017.78, stdev=2532.81, samples=37 cpu : usr=24.34%, sys=52.49%, ctx=386802, majf=0, minf=200 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=6141MiB (6440MB), run=18727-18727msec WRITE: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=2051MiB (2150MB), run=18727-18727msec Disk stats (read/write): nvme0n1: ios=1564706/555377, merge=0/6310, ticks=564794/49801, in_queue=614595, util=99.73% dd if=/dev/zero of=test bs=1M status=progress oflag=direct count=50k 52917436416 bytes (53 GB, 49 GiB) copied, 51 s, 1.0 GB/s 51200+0 records in 51200+0 records out 53687091200 bytes (54 GB, 50 GiB) copied, 51.6896 s, 1.0 GB/s dd if=test bs=1M status=progress > /dev/null 53461647360 bytes (53 GB, 50 GiB) copied, 98 s, 546 MB/s 51200+0 records in 51200+0 records out 53687091200 bytes (54 GB, 50 GiB) copied, 98.3106 s, 546 MB/s