Keep powersave CPU frequency scaling governor for CPUs that support intel_pstate

As Theodore Ts'o has pointed out[1]:

"""
... with modern Intel processors, the ondemand CPU governor is actually counterproductive because waking up to decide whether the CPU is idle keeps it from entering the deepest sleep states, and so (somewhat counterintuitively) the performance governor will actually result in the best battery life.
"""

[1]https://plus.google.com/+TheodoreTso/posts/2vEekAsG2QT

| - 26.02% 0.08% swapper [kernel.kallsyms] [k] cpu_startup_entry
| - 25.94% cpu_startup_entry
| - 23.65% call_cpuidle
| - 23.63% cpuidle_enter
| - 17.31% cpuidle_enter_state
| 17.04% intel_idle
| 0.04% poll_idle
| + 0.03% ktime_get
| 0.02% leave_mm
| + 0.02% ret_from_intr
| + 0.00% apic_timer_interrupt
| 0.00% sched_idle_set_state
| 0.00% read_tsc
| + 4.41% ret_from_intr
| + 1.79% apic_timer_interrupt
| 0.04% native_irq_return_iret
| 0.02% ktime_get
| 0.01% irq_entries_start
| + 0.01% reschedule_interrupt
| 0.01% common_interrupt
| 0.01% intel_idle
| 0.00% sched_idle_set_state
| 0.00% do_IRQ
| 0.00% restore_regs_and_iret
| 0.00% native_iret
| 0.00% restore_c_regs_and_iret
| + 0.00% call_function_single_interrupt
| + 0.00% call_function_interrupt
| + 0.00% ret_from_intr
| 0.00% cpuidle_enter_state

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1579278

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Looking at the kernel CPUFreq Governors documentation[1], It looks like powersave is actually a pretty poor default value for many applications, but especially servers.

I'm not sure about low-power applications like laptops, and how power usage is affected by frequency on other architectures.

Perhaps defaulting to the performance governor unless one of the more intelligent governors is available?

[1]https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt

The default is already CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y. However, it can be changed from user space. I suggest looking at /etc/init.d/ondemand

Ah thanks. So maybe /etc/init.d/ondemand should have something to override or disable it (say DISABLE=1 in /etc/default/ondemand)?

Looking at it currently, it seems to prefer governors in this order - interactive, ondemand, powersave. Even an option in /etc/default/ondemand to specify the preferred would be nice (e.g. PREFERRED_GOVERNOR=performance).

It looks like /etc/init.d/ondemand doesn't have any support for the performance governor at all. On systems that only have "performance" and "powersave" the only thing /etc/init.d/ondemand will do is set the governor to powersave.

I'll file the appropriate bug against the initscripts package.

In that same Google+ post, Arjan van de Ven wrote:

"""
Now, about ondemand and cpufreq.
The ondemand algorithm was designed roughly 10 years ago, for CPUs from that era. If you look at what ondemand really ends up doing, is managing the frequency during idle periods, and 10 years ago, that mattered for power.

Today (well, last 5 years), the frequency in idle is zero, and even the voltage is now zero (NHM and later).... so what frequency the OS picks during the idle period is completely irrelevant. This, and other things, make ondemand not a good algorithm for current Intel processors.

...

The new code in the 3.9 kernel, under, CONFIG_X86_INTEL_PSTATE, is a fresh approach to all of this.
"""

Bug #1188647 enables Intel PSTATE by default.

So it seems we should make the "ondemand" init script a no-op if /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver contains "intel_pstate", since in this case "ondemand" is worse than "performance"?

In yakkety I'll add an ondemand.service to systemd, as /etc/init.d/ondemand fell out of the standard installation (as "initscripts" is now -- thankfully -- gone). I'll keep the xenial task in case someone is interested in SRUing this.

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=2558ca88

This bug was fixed in the package systemd - 230-3git1

---------------
systemd (230-3git1) yakkety; urgency=medium

  Upload current Debian packaging git to fix tests.

  [ Martin Pitt ]
  * tmp.mount: Add nosuid and nodev mount options. This restores compatibility
    with the original SysV int RAMTMP defaults. (Closes: #826377)
  * debian/tests/upstream: Some tests fail on platforms without QEMU at the
    moment due to upstream PR#3587; blacklist these for now if QEMU is not
    available.
  * debian/rules: Don't run the "anything links against /usr" check for
    upstream tests, as those run on Ubuntu 16.04 LTS which does not yet have
    libidn moved to /lib.
  * debian/tests/upstream: Clean up old journals before running a test, to
    avoid printing a wrong one on failure.
  * debian/tests/upstream: Do not run the QEMU tests on i386. Nested QEMU on
    i386 causes testbed hangs on Ubuntu's cloud infrastructure, which is the
    only place where these actually run.

  [ Laurent Bigonville ]
  * Build with IDN support. (Closes: #814528)

 -- Martin Pitt <email address hidden> Thu, 23 Jun 2016 10:51:14 +0200

What has been done here is incorrect.

The old "ondemand" script was modified so that if the intel_pstate driver was being used, and therefore "ondemand" did not exist, it would fall through to setting the "powersave" governor (refer to bug #1314643). Note that "powersave" with the intel_pstate CPU frequency scaling driver is NOT the same as "powersave" with the acpi-cpufreq CPU frequency scaling driver.

The preferred governor with the intel_pstate driver is powersave. I'm saying that this:

# more modern processors that support intel_pstate behave worse with
# ondemand/powersave and should use performance
if [ `cat /sys/devices/system/cpu/cpu$FIRSTCPU/cpufreq/scaling_driver` = intel_pstate ]; then
    echo 'CPU supports intel_pstate, keeping "performance"'
    exit 0
fi

is incorrect.

> The preferred governor with the intel_pstate driver is powersave.

Do you have some references/proof for that? This is contrary to what kernel developers say, see comment 1.

>> The preferred governor with the intel_pstate driver is powersave.

> Do you have some references/proof for that? This is contrary to what
> kernel developers say, see comment 1.

In my opinion, that reference is obsolete.

While I don't work for Intel, the intel_pstate CPU frequency driver is pretty much the only thing that I work on, and for a few years now, and through 3 maintainers. Why would Intel expend so much effort on powersave mode, if it were better to merely set performance mode and forget about it? The objective with powersave mode is best energy/ performance tradeoff. There are still issues, yes. For example whenever clock modulation becomes involved. Also, there can be a tendency to incorrectly drive up the CPU frequency. There is work in progress that should address these issues (see: http://askubuntu.com/questions/812530/cpu-frequency-scaling-not-working-as-intended-on-vanilla-ubuntu-16-04/812721#812721 and the reference links therein.)

I'll try to come back, maybe this weekend, with some energy comparison numbers.

Using kernel 4.8-rc5 I did some energy tests, using a SpecPower simulator that I made one time. I reused intel_pstate powersave data from a previous test. Reference:
http://marc.info/?l=linux-kernel&m=147326197513427&w=2

Big numbers are Joules (package Joules from turbostat)
Smaller numbers are watts, 1500 Seconds test run time.

X ~= 18.2% on a real SpecPower.

Load: idle 0.5X X 2X 3X 4X 5X 100%
powersave 5708 11037 16075 29147 45913 61165 76650 81695
  3.81 7.36 10.72 19.43 30.61 40.78 51.10 54.46
performance 5749 14536 22841 38475 51757 64425 77000 81733
  3.83 9.69 15.23 25.65 34.50 42.95 51.33 54.49
  0.7% 31.7% 42.1% 32.0% 12.7% 5.3% 0.5% 0.0%

Note: A real SpecPower test-bed tends to get less dramatic results than my simulator.

The table formatting got messed up, trying again:

Load: idle 0.5X X 2X 3X 4X 5X 100%
powersave 5708 11037 16075 29147 45913 61165 76650 81695
                3.81 7.36 10.72 19.43 30.61 40.78 51.10 54.46
performance 5749 14536 22841 38475 51757 64425 77000 81733
                3.83 9.69 15.23 25.65 34.50 42.95 51.33 54.49
                0.7% 31.7% 42.1% 32.0% 12.7% 5.3% 0.5% 0.0%

Thanks Doug! Ack, I'll change it to use "powersave" again then.

pitti, can we please have a config option to disable "ondemand"?

As originally reported, one some workloads, it seems setting to powersave/ondemand causes high load with CPUs checking to see if they need to enter powersave state.

While the original report included perf report for the acpi-cpufreq driver, it was also seen with intel_pstate (and was why we decided to try switch to acpi-cpufreq).

With https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=94a70093 it uses powersave on intel_pstate again, like in xenial.

> pitti, can we please have a config option to disable "ondemand"?

Indeed, right now it is statically enabled, so it can only be disabled with "systemctl mask ondemand". I'll make it dynamically enabled so that it can be disabled with the usual systemctl enable/disable.

> I'll make it dynamically enabled

https://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?id=19e67c70

@Haw Loeung: It would be good to understand in more detail your findings. As far as I know (and I am not an expert in this area), great care has been taken to avoid things like wakeups just to decide to go idle.

I did a test on my computer (Ubuntu 16.04 server, no GUI), but to avoid extra overheads, I used the /proc/timer_stats method. I also did the test over 300 seconds, so as to dilute influence from the command itself. Kernel: 4.8.0-040800rc5-generic

intel_pstate, "idle" system:
powersave: 14722 events, 49.073 events / second
performance: 14964 events, 49.879 events / second

intel_pstate: extra "idle" system:
powersave: 6381 total events, 21.269 events / second
performance: 6788 events, 22.626 events / second

Which kernel are you using? There was a flurry of high wakeups activity in June, but I thought it all got resolved. Note that there was a slight increase in wakeups with the recent change to a utilization based algorithm, but that was good thing because it solved a very long standing issue whereby the CPU frequency might not increase under high load for periodic workflows that just so happened to be idle on jiffy boundaries.

You mention "on some workloads". Could you be more specific? i.e. so that I can attempt to recreate your scenario on my computer.

My script:
$ cat ./set_cpu_timer_stats_2
#! /bin/bash
# Take a 300 second time sample
# A long time is needed to amortize/dilute the effect of the command itself.
# Reset the counters
echo "0" > /proc/timer_stats
echo "1" > /proc/timer_stats
# Now let the sample time elapse
sleep 300
# Now observe the timer counters
cat /proc/timer_stats
# And turn off timer stats.
echo "0" > /proc/timer_stats

What does "extra idle" mean?:
$ cat set_cpu_turn_off_services
#! /bin/bash
# Turn off some services to try to get "idle" to be more "idle"
sudo systemctl stop mysql.service
sudo systemctl stop apache2.service
sudo systemctl stop nmbd.service
sudo systemctl stop smbd.service
sudo systemctl stop cron.service
sudo systemctl stop winbind.service
sudo systemctl stop apt-daily.timer

Recent wakeups thread: http://marc.info/?t=146617757000003&r=1&w=2

@Haw Loeung: In addition to what I wrote earlier:

> With the new Ubuntu archive servers, we saw constantly high load
> and after some tinkering, we found that it was mostly CPUs
> being woken up to see if they should enter idle states.
> Changing the CPU frequency scaling governor to "performance" saw a considerable drop.

What do you mean by "high load"?
And when you say "saw a considerable drop", does that mean in wakeups per second or load?

Note that you should observe a significant difference in load average on a server between powersave and performance mode, and that actually indicates things are working as they should be. For the SpecPower simulator test I posted above, I'll add some more data for the 0.5X and X lines:

0.5X, where Performance used 31.7% more package power:
Powersave: Busy%: 12.58% (load average = 1.01) Bzy MHz: 1651
Performance: Busy%: 5.04% (load average = 0.40) Bzy MHz: 3686

X, where Performance used 42.1% more package power:
Powersave: Busy%: 23.66% (load average = 1.89) Bzy MHz: 1798
Performance: Busy%: 10.56% (load average = 0.84) Bzy MHz: 3681

Isn't energy consumption what really matters, as long as performance doesn't suffer too much?
What I would like to see for your servers is the results from:

sudo turbostat -J -S --debug sleep 300

For the intel_pstate CPU frequency scaling driver and the powersave and performance scaling governors with your work flow.

This bug was fixed in the package systemd - 231-6git1

---------------
systemd (231-6git1) yakkety; urgency=medium

  Upload current Debian packaging git.

  [ Michael Biebl ]
  * fsckd: Do not exit on idle timeout if there are still clients connected
    systemd-fsckd's event loop terminates if nothing happens for 30 seconds.
    Exiting prematurely while fsck is still running but simply too slow to
    send us progress updates would close the socket and fsck would receive
    SIGPIPE when it writes to the socket. If this happens, the fsck process
    is aborted and the file system check is not completed. (Closes: #788050)
    (LP: #1547844)

  [ Martin Pitt ]
  * 73-usb-net-by-mac.rules: Split kernel command line import line.
    Reportedly this makes the rule actually work on some platforms. Thanks Alp
    Toker! (LP: #1593379)
  * debian/tests/boot-smoke: Only run 5 iterations
  * systemd.postinst: Drop obsolete setcap call for systemd-detect-virt.
    Drop corresponding libcap2-bin dependency.
  * debian/tests/systemd-fsckd: Robustify check for "unit was running"
    (LP: #1624406)
  * debian/extra/set-cpufreq: Use powersave with intel_pstate.
    This is what we did on xenial, and apparently powersave is still actually
    better than performance. Thanks to Doug Smythies for the measurements!
    (LP: #1579278)
  * Ubuntu: Move ondemand.service from static to runtime enablement.
    This makes it easier to keep performance, by disabling ondemand.service.
    Side issue in LP: #1579278

 -- Martin Pitt <email address hidden> Mon, 19 Sep 2016 22:37:51 +0200

On my Dell XPS 13 (i7-6500U) the only governors available are performance and powersave. Ubuntu 16.04 seems to set this always to 'powersave' for me. This causes the CPU to clock down to 300-400Mhz when idle. Unfortunately it's not clocking up again when under load sometimes. Setting the governor to performance 'fixes' it.

I don't know if this is *this* issue or another one, so happy to open a new issue if you prefer.

@fish : Yes your issue does not belong here.

To figure out where your issue actually belongs, some details are missing:

If your stuck at low frequency issue only occurs after a suspend and on battery power, then your issue is covered by: https://bugzilla.kernel.org/show_bug.cgi?id=90041 (and it is not resolved quite yet, even though that is the status).

If your issue always occurs and you are not using the proper Dell AC adapter, then that is why.

Otherwise, I don't know. Try the acpi_cpufreq CPU frequency scaling driver instead. Also have a look at: https://bugzilla.kernel.org/show_bug.cgi?id=189861

@dsmythies, sorry, missed your question. We're actually seeing performance issues/load on a couple of our Ubuntu Archive servers (archive.ubuntu.com). They're high traffic servers with 10GbE NICs with just apache2 serving .deb packages from disk. We've recently upgraded a few to 4.11.0-14-generic from linux-image-generic-hwe-16.04-edge (for TCP BBR).

NOTE: That these are systems using the pcc-cpufreq and acpi-cpufreq drivers.

The last outstanding open task for sysvinit/xenial is also invalid, to my understanding. Both /etc/init.d/ondemand in <= xenial and /lib/systemd/set-cpufreq in >= bionic implement the intended default policy of interactive/ondemand/powersave, not performance.

> Setting the governor to performance 'fixes' it.
> I don't know if this is *this* issue or another one, so happy to open a new issue if you prefer.

we will be changing the built-in 'ondemand' systemd service (set-cpufreq script, technically) to allow end-user configuration instead of using only its hardcoded values. So if you want to use the 'performance' governor you will be able to configure it to do so. See bug 1806012