IO performance decreases after change to kernel 3.x

Asked by Vladimir

Hi everbody,

I got to setups and one has about half as good IO performance than the other.

a) Dell R710 + PERC H700 + RAID1 + HDD SAS 15k
Ubuntu 10.04 + kernel 2.6.32-44-server (from the official repository)

b) Dell R720 + PERC H710 mini + RAID1 + HDD SAS 15k
Ubuntu 10.04 + kernel 3.0.0-32-generic (from the official repository)

After changing to R720 and kernel 3.0.0-32-server I was told writing to disk feels laggy.

So I did two tests (multiple times to avoid rogue results). Both systems ran no load generating services.

a) Dell R710

#> dd if=/dev/zero of=/root/dd_10000 bs=512 count=10000 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5.1 MB) copied, 0.673255 s, 7.6 MB/s

b) Dell R720
#> dd if=/dev/zero of=/root/dd_10000 bs=512 count=10000 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 2,06752 s, 2,5 MB/s

I recognized that using a grml image with kernel 2.6.38 on Dell R720 gave same good results like kernel 2.6.32-44-server on Dell R710.

I further tested this setups:

- Dell R720 + Ubuntu 14.04 + kernel linux-image-3.13.0-53-generic (from the official repository)
- Dell R720 + Ubuntu 14.04 + kernel linux-image-3.16.0-38-generic (from the official repository)
- Dell R720 + Ubuntu 14.04 + kernel 4.0.4 (compiled with make oldconfig)

I was not able to reach the results from the 2.6.x kernel with any of it. All results were between 2.5 - 3.2 MB/s.

Also changing IO scheduler didn't bring better results.

My assumption is that the issue is kernel or driver (megaraid_sas) related.

---

Has anybody recognized similar issues?
Or do you have hints how to narrow further down this issue?

---

Thanks for your assistance.
Kind regards

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said :
#1

What is the output of:

cat /etc/issue; uname -a

Thanks

Revision history for this message
Manfred Hampl (m-hampl) said :
#2

Please look at https://wiki.ubuntu.com/Releases and you will see that Ubuntu 10.04 has gone out of support a month ago.
It is strongly recommended that you upgrade your systems to a supported release.

Revision history for this message
Vladimir (lp-x) said :
#3

a) Dell R710:

$> cat /etc/issue; uname -a
Ubuntu 10.04.4 LTS \n \l

Linux localhost 2.6.32-44-server #98-Ubuntu SMP Mon Sep 24 17:41:33 UTC 2012 x86_64 GNU/Linux

b) Dell R720: (I'm aware Ubuntu 10.04 is eol so I stayed with Ubuntu LTS 14.04 but the performance is still poor under Ubuntu 14.04)

$> cat /etc/issue; uname -a
Ubuntu 14.04.2 LTS \n \l

Linux localhost-v5 3.13.0-53-generic #88-Ubuntu SMP Wed May 13 18:10:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

---

I can provide further information if needed.

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#4

Lucid is EOL and not supported in any way.

I suggest you upgrade to a newer (supported release). Trusty is the latest LTS and is supported til April 2019.

https://wiki.ubuntu.com/Releases

Revision history for this message
Vladimir (lp-x) said :
#5

The new server (Dell R720) meanwhile runs with Ubuntu 14.04 Trusty but also shows the write performance issues in contrast to the tested 2.6.x kernels.

Even the recent upstream kernel 4.0.4 did perform poor (like the recent Trusty kernel too).

Revision history for this message
Vladimir (lp-x) said :
#6

For test purpose I wrote directly to the device of the Dell R720 (Trusty) with a much better result.

#> uname -a
Linux localhost 3.13.0-53-generic #89-Ubuntu SMP Wed May 20 10:34:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

#> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

#> dd if=/dev/zero of=/dev/vg0/lv_wo_fs count=10000 seek=20971520 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 0,80695 s, 6,3 MB/s

Does this imply that I'm faced with an filesystem issue? If this would be the case I would try to tweak some file system settings (data, barrier...).

---

Here some further tests with different file systems for comparison:

#> blkid | grep lv
/dev/mapper/vg0-lv_xfs: UUID="ee63b8ec-227e-41d5-9115-23a149186dc2" TYPE="xfs"
/dev/mapper/vg0-lv_ext3: UUID="518e8f9d-81a9-4641-b9c8-9d07d2fc3212" TYPE="ext3"
/dev/mapper/vg0-lv_ext4: UUID="a800664a-8ba6-4bce-a133-a59089b8867c" TYPE="ext4"

#> dd if=/dev/zero of=/mnt/lv_ext3/dd count=10000 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 2,15098 s, 2,4 MB/s

#> dd if=/dev/zero of=/mnt/lv_ext4/dd count=10000 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 2,11697 s, 2,4 MB/s

#> dd if=/dev/zero of=/mnt/lv_xfs/dd count=10000 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 1,55431 s, 3,3 MB/s

Revision history for this message
Vladimir (lp-x) said :
#7

After choosing the System Profile "Performance" [1] in Dell R720 BIOS I got some better values.

Unfortunately Dell seems to ship the BIOS with the System Profile "Performance-per-watt (DAPC)" which uses some power saving features.

Here the new results:

#> dd if=/dev/zero of=/dev/vg0/lv_wo_fs count=10000 seek=20971520 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 0,507091 s, 10,1 MB/s

#> dd if=/dev/zero of=/mnt/lv_ext4/dd_bla count=10000 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 1,28449 s, 4,0 MB/s

#> dd if=/dev/zero of=/mnt/lv_xfs/dd count=10000 bs=512 oflag=dsync
10000+0 records in
10000+0 records out
5120000 bytes (5,1 MB) copied, 0,885294 s, 5,8 MB/s

The values are still not as good as with 2.6.x kernel.

If I look at the values when writing to the volume without file system I don't understand where the huge loss of write speed (50-60%) comes when using ext4 (10,1 vs. 4,0).

[1] http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-24-87-40/12g_5F00_bios_5F00_tuning_5F00_for_5F00_performance_5F00_power.pdf

Revision history for this message
Launchpad Janitor (janitor) said :
#8

This question was expired because it remained in the 'Open' state without activity for the last 15 days.