Error "FATAL: Failed to fsync file! file: 0 errno = 22 ()" while running Sysbench test with 4500 IOPS and 2 threads.

Asked by Nitin Manjalkar on 2017-01-10

The test case is like this:

1) Using sysbench to test the disk throughput.
2) Using write back and disk cache with raid 5.
3) disks : Intel S3610 X 6

root@bb-blr-prod-db-01:/cryptfs# sysbench --num-threads=2 --test=fileio --file-total-size=350G --file-test-mode=rndrw --init-rng=on -
-max-time=3600 --max-requests=0 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2
Initializing random number generator from timer.

Extra file open flags: 0
128 files, 2.7344Gb each
350Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
FATAL: Failed to fsync file! file: 0 errno = 22 ()
FATAL: Incorrect file discovered in request
Done.

Operations performed: 2475038 Read, 1650025 Write, 5306656 Other = 9431719 Total
Read 37.766Gb Written 25.177Gb Total transferred 62.943Gb (176.76Mb/sec)
11312.79 Requests/sec executed

Test execution summary:
   total time: 364.6370s
   total number of events: 4125063
   total time taken by event execution: 475.9527
   per-request statistics:
        min: 0.00ms
        avg: 0.12ms
        max: 12.39ms
        approx. 95 percentile: 0.53ms

Threads fairness:
   events (avg/stddev): 2062531.5000/3374.50
   execution time (avg/stddev): 237.9764/0.04

IOSTAT result:

avg-cpu: %user %nice %system %iowait %steal %idle
          0.05 0.00 0.98 2.25 0.00 96.72

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 2775.00 4573.00 54708.00 72828.00 34.71 1.90 0.26 0.50 0.11 0.12 90.00
dm-0 0.00 0.00 2777.00 4569.00 54740.00 72764.00 34.71 2.18 0.30 0.55 0.15 0.13 94.00

avg

My Raid volume config :

Advanced Properties

Status
Name sdb
Device Description Virtual Disk 1 on Integrated RAID Controller 1
State Online
Layout RAID-5
Size 7449.38 GB
Span Depth 1
Block Size 512 bytes
Bus Protocol SATA
Media Type SSD
Operational State Not Applicable
Read Policy Read Ahead

Write Policy Write Back
Stripe Size 64K
Disk Cache Policy Enabled
Enhanced Cache Not Applicable
Progress Not Applicable
Bad Blocks Found No
Secured No
Remaining Redundancy 1
T10 PI Status Disabled

As you saw, the IO is getting choked @ 4500 IOPS in a long run test. I am also not able to run this test with 2 threads.

Pls advise.

Question information

Language:
English Edit question
Status:
Answered
For:
sysbench Edit question
Assignee:
No assignee Edit question
Last query:
2017-01-10
Last reply:
2017-01-12
Mark Callaghan (mdcallag) said : #1

I have a vague memory of fixing a bug for that and I don't remember whether
the fix was only in my branch or pushed upstream. But that bug was in the
old version of sysbench, before integration with Lua was added. Modern
sysbench is here -> https://github.com/akopytov/sysbench

I think my fix was
http://bazaar.launchpad.net/~mdcallag/sysbench/0.4-dev/revision/76

On Tue, Jan 10, 2017 at 5:57 AM, Nitin Manjalkar <
<email address hidden>> wrote:

> New question #434586 on sysbench:
> https://answers.launchpad.net/sysbench/+question/434586
>
> The test case is like this:
>
> 1) Using sysbench to test the disk throughput.
> 2) Using write back and disk cache with raid 5.
> 3) disks : Intel S3610 X 6
>
>
>
> root@bb-blr-prod-db-01:/cryptfs# sysbench --num-threads=2 --test=fileio
> --file-total-size=350G --file-test-mode=rndrw --init-rng=on -
> -max-time=3600 --max-requests=0 run
> sysbench 0.4.12: multi-threaded system evaluation benchmark
>
> Running the test with following options:
> Number of threads: 2
> Initializing random number generator from timer.
>
>
> Extra file open flags: 0
> 128 files, 2.7344Gb each
> 350Gb total file size
> Block size 16Kb
> Number of random requests for random IO: 0
> Read/Write ratio for combined random IO test: 1.50
> Periodic FSYNC enabled, calling fsync() each 100 requests.
> Calling fsync() at the end of test, Enabled.
> Using synchronous I/O mode
> Doing random r/w test
> Threads started!
> FATAL: Failed to fsync file! file: 0 errno = 22 ()
> FATAL: Incorrect file discovered in request
> Done.
>
> Operations performed: 2475038 Read, 1650025 Write, 5306656 Other =
> 9431719 Total
> Read 37.766Gb Written 25.177Gb Total transferred 62.943Gb (176.76Mb/sec)
> 11312.79 Requests/sec executed
>
> Test execution summary:
> total time: 364.6370s
> total number of events: 4125063
> total time taken by event execution: 475.9527
> per-request statistics:
> min: 0.00ms
> avg: 0.12ms
> max: 12.39ms
> approx. 95 percentile: 0.53ms
>
> Threads fairness:
> events (avg/stddev): 2062531.5000/3374.50
> execution time (avg/stddev): 237.9764/0.04
>
> IOSTAT result:
>
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.05 0.00 0.98 2.25 0.00 96.72
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
> avgqu-sz await r_await w_await svctm %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> sdb 0.00 0.00 2775.00 4573.00 54708.00 72828.00
> 34.71 1.90 0.26 0.50 0.11 0.12 90.00
> dm-0 0.00 0.00 2777.00 4569.00 54740.00 72764.00
> 34.71 2.18 0.30 0.55 0.15 0.13 94.00
>
> avg
>
>
>
> My Raid volume config :
>
>
> Advanced Properties
>
> Status
> Name sdb
> Device Description Virtual Disk 1 on Integrated RAID Controller 1
> State Online
> Layout RAID-5
> Size 7449.38 GB
> Span Depth 1
> Block Size 512 bytes
> Bus Protocol SATA
> Media Type SSD
> Operational State Not Applicable
> Read Policy Read Ahead
>
> Write Policy Write Back
> Stripe Size 64K
> Disk Cache Policy Enabled
> Enhanced Cache Not Applicable
> Progress Not Applicable
> Bad Blocks Found No
> Secured No
> Remaining Redundancy 1
> T10 PI Status Disabled
>
>
>
> As you saw, the IO is getting choked @ 4500 IOPS in a long run test. I am
> also not able to run this test with 2 threads.
>
> Pls advise.
>
>
> --
> You received this question notification because your team sysbench-
> developers is an answer contact for sysbench.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~sysbench-developers
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~sysbench-developers
> More help : https://help.launchpad.net/ListHelp
>

--
Mark Callaghan
<email address hidden>

Alexey Kopytov (akopytov) said : #2

Yes, sysbench 0.4 is too old. This should be fixed in the current code.

Can you help with this problem?

Provide an answer of your own, or ask Nitin Manjalkar for more information if necessary.

To post a message you must log in.