wpa-roam broken by fix for ifupdown #1337873

Status changed to 'Confirmed' because the bug affects multiple users.

The attachment "patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Hello Paul,
The locking mechanism was implemented to avoid race conditions in ifupdown (such as the one reported as bug #1337873). Unsetting the variable may lead to unexpected consequences. I belive a proper solution would be to modify wpa-roam implementation to play along with ifupdown.

Could you please share your configuration? I tried reproducing it but was unable to do so with the simplest wpa-roam config I checked (ifup brings the interface up as expected with ifupdown 0.8.10ubuntu1 on Xenial.

Updating my patch ... The environment variable also needs to be unset before calling ifdown.

My configuration:

/etc/network/interfaces
auto wlan0
iface wlan0 inet manual
    wpa-driver wext
    wpa-roam /etc/wpa_supplicant/wpa.conf
    wpa-roam-default-iface dhcp
iface dhcp inet dhcp
    dns-nameservers 8.8.8.8 8.8.4.4
iface dhcp_dns

/etc/wpa_supplicant/wpa.conf
network={
    ssid="x"
    priority=10
    key_mgmt=WPA-PSK
    psk="..."
    id_str="dhcp_dns"
}
network={
    ssid="y"
    priority=9
    key_mgmt=WPA-PSK
    psk="..."
}

After connecting to either the x or y networks, wpa_supplicant calls `wpa_action connect` which calls `ifup wlan0=dhcp_dns` or `ifup wlan0=dhcp`, which fails with "ifup: recursion detected for parent interface wlan0 in post-up phase". Therefore, dhclient is never run, so wlan0 never gets an IP address.

Note that the wpa_supplicant daemon is started by the /etc/network/if-up.d/wpasupplicant script, and it inherits the environment from `ifup wlan0` (including the environment variable used to detect recursion). `wpa_action connect` then inherits the environment from wpa_supplicant, and `ifup wlan0=dhcp` inherits it from wpa_action, hence the error from ifup.

I understand that this variable is meant to avoid certain race conditions, but I don't believe the type of race condition reported in bug #1337873 is applicable to this situation. Because wpa_supplicant is started by ifup itself (and not a boot script), there is no way for the `ifup wlan0` and `ifup wlan0=dhcp` to be called out of order, so there is no chance of a race condition. However, maybe I'm missing something here. Could you explain how the wpa-roam implementation could play along with ifupdown given that it is started by ifup itself and needs to call ifup to reconfigure the interface?

Oops ... missed "inet dhcp" on the "iface dhcp_dns" line above. That line should be:
iface dhcp_dns inet dhcp

Paul, I think that the real cause here is that (by looking into debian/ifupdown/functions.sh) wpasupplicant depends on the previous single-file ifstate.
Currently, for the sake of the locking mechanism, it has been split into an individual ifstate.$IFACE per interface, while wpasupplicant still checks for the state /run/network/ifstate.

This leads to loosing track of aliases and misusing of the locking.

I will work on fixing this for Ubuntu and Debian and will appreciate your help in testing it as soon as a hotfix is ready.

Thanks!

See bug #1545363 - my patch for that bug happens to remove the use of the ifstate file. However, fixing that does not solve the "ifup: recursion detected ..." issue. The environment variable still needs to be removed to fix the recursion issue.

After thinking about it some more, I think it may make more sense to unset the variable in /etc/wpa_supplicant/ifupdown.sh rather than /etc/wpa_supplicant/functions.sh (so the variable is removed from the wpa_supplicant daemon's environment, rather than being removed each time wpa_action calls ifup or ifdown). This new attached patch file does this.

Let me know when you have something for me to test. Thanks!

Paul, I have finally managed to reproduce this on Trusty. The ifstate looks to be a false trail.

However, please note that with 0.8.10ubuntu1 on Xenial this problem does not occur.

I am investigating which change needs to be backported. I will keep you updated.

Hello Paul,
I have identified the cause of the problem and prepared a fix.
My tests show that it solves the issue for Trusty and Wily (while Xenial does not require a fix since it's ifupdown has been synced with Debian to a version containing a fix already).

Could you please revert your wpasupplicant to the original state (without your patches) and upgrade your ifupdown from the one available in my ppa?

sudo add-apt-repository ppa:dgadomski/lp1545302
sudo apt-get update
sudo apt-get install ifupdown

In my understanding this should solve your problem. Please leave your feedback and so we will be able to proceed with an official release.

Thank you.

Adding SRU proposal for Trusty.

Adding SRU proposal for Wily.

Dariusz - your PPA fixes the issue for me on Trusty. Thanks.

Cheers... Mike

Hello Mike,
Dariusz ppa fix does not work for me on trusty, @Mike can you post your configuration for?:
/etc/network/interfaces
/etc/wpa_supplicant/wpa.conf?

I am using this configuration:

/etc/wpa_supplicant/wpa.conf:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
 ssid="duckietown"
 scan_ssid=1
 psk="something"
 priority=10
}

/etc/network/interfaces:
# interfaces(5) file used by ifup(8) and ifdown(8)
# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp

# Wifi settings
allow-hotplug wlan0
iface wlan0 inet manual
wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
iface default inet dhcp

Any advice?

/etc/network/interfaces:
--------------------------------------
auto lo
allow-hotplug wlan1

iface lo inet loopback

iface wlan1 inet manual
    wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf

iface default inet dhcp

/etc/wpa-supplicant/wpa-supplicant.conf
-----------------------------------------------------------
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="Sooty"
    key_mgmt=WPA-PSK
    psk="secret"
    priority=5
}

Basically the same as yours. It's worth confirming the symptoms: for me this setup was working fine until a recent apt-get upgrade. After that, wireless appeared to come up but no address was assigned over DHCP. I could get one by manually running "dhclient wlan1". It was apparent looking at the logs that dhclient was not being run after the wireless connection came up, and the PPA from Dariusz has dhclient running again.

Hi folks.
Dariusz fix worked for me.
I reverted my config to failure mode -- while i was connected to the wifi, I did not get an IP address.

Changed the wifi to use wpa-conf (so I had a network to do the upgrade).
Upgraded to the ppa
Changed the wifi back to using wpa-roam
ifdown / ifup -- all good
rebooted -- all good.

here is my network/interfaces:
========================
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

iface eth0 inet dhcp

auto wlan0
allow-hotplug wlan0
iface wlan0 inet manual
  wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
#iface wlan0 inet dhcp
# wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

iface default inet dhcp
===================

Here is my wpa_supplicant.conf:
===================
update_config=1
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

network={
        ssid="MySSID"
        psk="MyPSK"
        id_str="home"
}
===================

@Robert: Just checking, but your wpa.conf file is actually named wpa_supplicant.conf -- Yes??

@Mike, @DavidS: thank you for feedback. I'm glad it's working for you. The official fix will be on it's way.

@Robert: please confirm that your wpa-roam settings below "iface wlan0 inet manual" is in fact pointing to the right file as DavidS noticed. If it is please check:
sudo wpa_cli status

If the wpa_state is anything other than COMPLETED - your WiFi connection was not established. If it is ok - please let me know, I'll need to check if this may be caused by anything else.

Thanks!

@Dariusz: Works for me on Wily. Nice find, this is a much better fix than my proposed patch. Thanks!

@Paul: glad it works for you. Thank you for reporting the issue and the feedback.

@Dariusz: Your patch fixes the issue on Trusty. Thanks!
Is there any estimate when a patched package would land into trusty-proposed?

Guys, could you please forward the patch to Guus using Debian BTS?

Thanks!

Disregard the last comment, from reading the comments I see the bug isn't relevant to Debian.

That's right Andrew, it's backported from Debian so no forwarding needed.

Thanks Dariusz! I sponsored the trusty and wily fixes, sitting in the SRU review queues now.

@Dariusz: my wpa-roam settings point to the right place. my wifi does establish and wpa_state=COMPLETED

$sudo wpa_cli status
Selected interface 'wlan0'
bssid=02:18:4a:b0:f5:11
ssid=StataCenter
id=0
id_str=drl
mode=station
pairwise_cipher=NONE
group_cipher=NONE
key_mgmt=NONE
wpa_state=COMPLETED
address=00:0f:60:07:b6:c9
uuid=8b3d14ff-8548-52b8-a27d-fc2770017ea0

$ifconfig:
eth0 Link encap:Ethernet HWaddr b8:27:eb:85:1b:e8
          inet addr:10.0.1.9 Bcast:10.0.1.255 Mask:255.255.255.0
          inet6 addr: 2001:db8:1:0:ba27:ebff:fe85:1be8/64 Scope:Global
          inet6 addr: fe80::ba27:ebff:fe85:1be8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1537 errors:0 dropped:0 overruns:0 frame:0
          TX packets:680 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:197521 (197.5 KB) TX bytes:157220 (157.2 KB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2368 (2.3 KB) TX bytes:2368 (2.3 KB)

wlan0 Link encap:Ethernet HWaddr 00:0f:60:07:b6:c9
          inet6 addr: fe80::20f:60ff:fe07:b6c9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:11345 errors:0 dropped:0 overruns:0 frame:0
          TX packets:22 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2036197 (2.0 MB) TX bytes:4019 (4.0 KB)

Any ideas what is causing this?

reassigning to the right package

Hello Paul, or anyone else affected,

Accepted ifupdown into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.7.54ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Paul, or anyone else affected,

Accepted ifupdown into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ifupdown/0.7.47.2ubuntu4.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ifupdown 0.7.54ubuntu1.3 fixes this bug for me in Wily.

@dbilunov: Would you mind testing the trusty-proposed package?

@robert-katzschmann:

Make sure wpa_cli is running:
# ps aux | grep wpa
/sbin/wpa_supplicant -s -B -P /run/wpa_supplicant.wlan0.pid -i wlan0 -W -D wext -c /etc/wpa_supplicant/wpa_supplicant.conf
/sbin/wpa_cli -B -P /run/wpa_action.wlan0.pid -i wlan0 -p /var/run/wpa_supplicant -a /sbin/wpa_action

If it is, then check /var/log/syslog - it should contain something like:
wpa_supplicant[11998]: wlan0: CTRL-EVENT-CONNECTED - Connection to 18:1b:eb:b7:4a:27 completed [id=0 id_str=default]
wpa_action: WPA_IFACE=wlan0 WPA_ACTION=CONNECTED
wpa_action: WPA_ID=0 WPA_ID_STR=default WPA_CTRL_DIR=/var/run/wpa_supplicant
wpa_action: ifup wlan0=default
dhclient: Internet Systems Consortium DHCP Client 4.3.1

I'm guessing one or more of the above syslog messages will be missing ... knowing which messages are missing will help us determine the cause of your problem.

I did test into trusty-proposed - work fine

This bug was fixed in the package ifupdown - 0.7.47.2ubuntu4.4

---------------
ifupdown (0.7.47.2ubuntu4.4) trusty; urgency=medium

  * Fix false-positive recursion detection. (LP: #1545302)

 -- Dariusz Gadomski <email address hidden> Wed, 17 Feb 2016 17:45:58 +0100

The verification of the Stable Release Update for ifupdown has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package ifupdown - 0.7.54ubuntu1.3

---------------
ifupdown (0.7.54ubuntu1.3) wily; urgency=medium

  * Fix false-positive recursion detection. (LP: #1545302)

 -- Dariusz Gadomski <email address hidden> Wed, 17 Feb 2016 17:48:41 +0100