Degraded performance when executing benchmark tests in 10.04 vs 12.04 and 14.04

So we were using Ubuntu 10.04 boxes to run several of our services via AWS and ran into some issues when upgrading to 12.04 and 14.04.

Basically we have seen across the board that instances running 10.04 run much better and at lower loads than ones running 12.04 or 14.04. 12.04 and 14.04 (Trusty) perform about the same, but we see a pretty drastic difference between them and 10.04 (Lucid).

We have seen this happen with various types of servers including MongoDB instances which we tried to upgrade, then had to downgrade again because they were falling over. In order to solve the problem, we focussed on our web servers which run Nginx and PHP-FPM.

When these servers receive the same amount of traffic, Lucid instances have about half the load of Trusty instances, and it is not just a measurement problem because as we tune the traffic up, the Trusty instances will hit their max load and the response time of the web servers starts to get worse, while the Lucid instances are still working fine.

One thing we have noticed is that the number of context switches and interrupts seem to be about double on Trusty instances compared to Lucid instances, but we haven't been able to track down the source of them.

We also tried a few different kernel versions in Trusty and both PV and HVM with no real change.

With regards to resources, we are spinning up both the instances with the same amount of resources (approx. 4 gb memory, 2 Cores and 32GB SSD drive ie C3.Large) so that is not the issue.

We thought that this may be an issue with our application, so we devised a reproducible method to determine that.

For benchmarking and tests, we used http://www.phoronix-test-suite.com/.

Our plan was to spin up four different instances with the same amount of resources (mentioned above) allocated to each instance.

- Instance 1: A Vanilla Ubuntu 10.04
- Instance 2: A Vanilla Ubuntu 14.04
- Instance 3: A Ubuntu 10.04 running our application
- Instance 4: A Ubuntu 14.04 running our application

We then used Phoronix-Test-Suite and ran the following tests:

- https://openbenchmarking.org/test/pts/nginx
“This test profile measures how many requests per second a given system can sustain when carrying out 500,000 requests with 100 requests being carried out concurrently.”

-https://openbenchmarking.org/test/pts/apache
“This test profile measures how many requests per second a given system can sustain when carrying out 1,000,000 requests with 100 requests being carried out concurrently.”

Here were the results that we obtained:

https://drive.google.com/open?id=1xNy4x31smP1LXEdbhygDDyr6CL7GfzZbqkicK2b6e0w

(Note: The Load Avg reported is the highest value observed in the duration of the test)

Taking a quick look at the load averages between the vanilla instances and the instances running our application, we see that the max load average is about the same. From this, we can determine that it is not our application that is causing the load average spike observed in 14.04.

Another point to mention is that the Nginx Test is relatively lighter than the Apache Test and from the test data, we can see that the results show that during lighter traffic loads, Ubuntu 14.04 outperforms Ubuntu 10.04, as expected. In the Apache Test however, we see that the Ubuntu 10.04 greatly outperforms Ubuntu 14.04 in responding to the traffic load, which is clearly not the expected behavior.

So due to the difference in performance, we are still using Ubuntu 10.04 in certain cases simply because switching to 14.04 instances would require much more instances to serve the same amount of traffic.

So far we’ve been unable to identify the source of the performance discrepancy, if anyone has any insights or info we would really appreciate it.