[rt61] Lockups running Feisty on x86-64

In desperation, I started trimming down my module list as much as possible. I have a wifi card (1814:0301, "RaLink RT2561/RT61 802.11g PCI") driven by rt61 that I am not using at all (it shipped with my system and I never bothered to take it out, but I am using wired networking with e1000 exclusively).

Blacklisting the "rt61" module seems to have made my system much more stable. So perhaps rt61 has some locking bugs?

If someone could tell me how to build a linux image and restricted modules (I need that pesky fglrx module to see anything under X) package from source, I could turn on CONFIG_PROVE_LOCKING etc. and try to get some more info about the specific problem. Unfortunately I've gotten lost trying to follow the Ubuntu kernel packaging...

Hi Roland,

If you run "apt-get source linux-source-2.6.20" it will download and unpack the latest copy of the source. "apt-get build-dep linux-source-2.6.20" will fetch all the corresponding build-dependencies of the source package. Run "apt-get install build-essential dpkg-dev fakeroot" to make sure you have the latest Debian package building stuff.

To build the source tree, you can run "dpkg-buildpackage -rfakeroot" which will build the whole shebang. If you only want to rebuild the linux-image you can run "fakeroot debian/rules binary-debs flavours=FLAVOUR" to only build the binary packages for a specific flavour.

You can use similar commands to fetch the "linux-restricted-modules-2.6.20" source package. You'll need to reinstall your linux-headers package from building the linux-source package to build l-r-m against the new headers.

Cheers,
  Kyle

There's a few other rt61-ish bugs, I'll see if any of those can be of any help.

Thanks Kyle, but it seems that rebuilding the kernel is a little harder, or perhaps the linux-source package is broken. I had tried basically your suggestion before and failed, and I just tried again.

I installed linux-source-2.6.20 and unpacked /usr/src/linux-source-2.6.20.tar.bz2, and then in the resulting directory I ran "dpkg-buildpackage -rfakeroot" as you suggested. That did some stuff but eventually died with:

====== making target clean [new prereqs: stamp-clean]======
 dpkg-source -b linux-source-2.6.20
dpkg-parsechangelog: error: cannot open ./linux-source-2.6.20/debian/changelog to find format: No such file or directory
dpkg-source: error: syntax error in parsed version of changelog at line 0: empty file

the command "fakeroot debian/rules binary-debs flavours=generic" dies with

====== making target .config [new prereqs: Makefile]======

test -f .config || test ! -f .config.save || \
                            cp -pf .config.save .config
test -f .config || test ! -f ./debian/Config/config || \
                            cp -pf ./debian/Config/config .config
test -f .config || test ! -f ./debian/config || \
                            cp -pf ./debian/config .config
test -f .config || (echo "*** Need a config file .config" && false)
====== making .config because of Makefile ======

test -f .config || test ! -f .config.save || \
                            cp -pf .config.save .config
test -f .config || test ! -f .config || \
                            cp -pf .config .config
test -f .config || test ! -f ./debian/config || \
                            cp -pf ./debian/config .config
test -f .config || (echo "*** Need a config file .config" && false)
make: *** No rule to make target `binary-debs'. Stop.

So it ain't that easy :)

I can confirm lockups with rt61 driver loaded. I have a ralink card in a computer:
04:08.0 Network controller: RaLink RT2561/RT61 802.11g PCI

When the rt61 driver is loaded the computer locks up sometimes during boot, sometimes after a few minutes of usage, and even sometimes after a couple of hours.

When using a graphical environment, I experience the same symptoms as the poster. The keyboard hangs first, mouse still works a couple of seconds, soon everything is locked.

I once got the following message in a virtual console just before the hang:
soft lockup on cpu#0

I have a dual core Athlon X2 processor (SMP-related problem??)

Without the rt61 module loaded, the system is completely stable. I have now deleted the ko file to ensure it doesn't get loaded by accident until the problem is resolved.

I also can confirm this problem. A Ralink pci card

00:09.0 Network controller: RaLink RT2561/RT61 802.11g PCI

is driven by the rt61 driver. I have a 32bit single core system, so I don't think that the problem is SMP related. Same problems here: The system freezes completly from one second to another, sometimes after a few seconds, sometimes after some hours. Like Frode Solheim I also got the message soft lockup on cpu#0 as the system freezed without X loaded.

OK, I managed to get a traceback of the soft lockup on my system, but unfortunately I screwed up and it didn't get captured like I thought it was. So I don't have the traceback, and I didn't read it that closely before I rebooted, since I thought I had it saved to study later.

However, I did look at it enough to get the idea that some "MlmeXXX" function was calling del_timer_sync() from a timer callback (which is of course an instant deadlock).

And indeed, the first del_timer_sync() call I check shows the following problem. MlmeAssocReqAction() (which is called from the STATE_MACHINE obfuscation) does

       del_timer_sync(&pAd->MlmeAux.AssocTimer);

and AssocTimer runs AssocTimeout(), which calls MlmeHandler(), which runs the STATE_MACHINE crap. So the AssocTimer callback can deadlock easily.

Based on this, rt61 seems utterly broken by design and prone to deadlocking if anything times out. Aside from this deadlock problem, the source is bloated and unreadable, so I think the best thing to do would be to throw it away once and for all. The rt61pci driver seems a lot saner to me, and indeed a complete rewrite like that seems to be the only way to salvage this driver. I don't know what the problems with rt61pci are, but I don't see how rt61pci could be worse than rt61.

If I get a chance I may try (out of morbid curiousity) to get the exact traceback that's deadlocking my system, but I don't have any hope that the issue will be fixable without a driver rewrite.

Regarding my last post, I managed to resolve my problem by downloading: rt2x00 nightly CVS tarball, from
http://rt2x00.serialmonkey.com/wiki/index.php?title=Downloads
a couple of weeks ago. I still use the same kernel version.
If the drivers shipped with ubuntu are the same as the ones from rt2x00.serialmonkey.com, the problem may (for all I know) be fixed in CVS. It worked for me.

This is still a problem as of 4/9/2007 (kernel 2.6.20-14). I just removed the rt61 card from my system for now, but if it's at all possible at this late date, please do -not- ship feisty with this bug. It's very hard to diagnose ('Everything just stops, except mouse movement!'), and rt61 cards aren't all that uncommon.

May I ask what exactly 'needs info' about this bug?

In my logs I found the following, but the machine ran for a while after this and then locked up hours later. When it actually locks, there are no messages saved in the logs, and I don't have a serial console available right now. Removing the rt61 card from my system has restored stability.

Apr 9 01:26:26 jennifer kernel: [ 357.311049]
Apr 9 01:26:26 jennifer kernel: [ 357.311050] Call Trace:
Apr 9 01:26:26 jennifer kernel: [ 357.311052] <IRQ> [softlockup_tick+250/288] softlockup_tick+0xfa/0x120
Apr 9 01:26:26 jennifer kernel: [ 357.311107] [_end+129904608/2130489456] :rt61:BeaconTimeout+0x0/0x30
Apr 9 01:26:26 jennifer kernel: [ 357.311112] [update_process_times+87/144] update_process_times+0x57/0x90
Apr 9 01:26:26 jennifer kernel: [ 357.311120] [smp_local_timer_interrupt+52/96] smp_local_timer_interrupt+0x34/0x60
Apr 9 01:26:26 jennifer kernel: [ 357.311126] [smp_apic_timer_interrupt+89/128] smp_apic_timer_interrupt+0x59/0x80
Apr 9 01:26:26 jennifer kernel: [ 357.311133] [apic_timer_interrupt+102/112] apic_timer_interrupt+0x66/0x70
Apr 9 01:26:26 jennifer kernel: [ 357.311145] [_end+129901472/2130489456] :rt61:MlmeJoinReqAction+0x0/0xe0
Apr 9 01:26:26 jennifer kernel: [ 357.311156] [_spin_unlock_irqrestore+8/16] _spin_unlock_irqrestore+0x8/0x10
Apr 9 01:26:26 jennifer kernel: [ 357.311164] [try_to_del_timer_sync+83/96] try_to_del_timer_sync+0x53/0x60
Apr 9 01:26:26 jennifer kernel: [ 357.311172] [del_timer_sync+12/32] del_timer_sync+0xc/0x20
Apr 9 01:26:26 jennifer kernel: [ 357.311183] [_end+129901509/2130489456] :rt61:MlmeJoinReqAction+0x25/0xe0
Apr 9 01:26:26 jennifer kernel: [ 357.311195] [_end+129877818/2130489456] :rt61:MlmeHandler+0xda/0x160
Apr 9 01:26:26 jennifer kernel: [ 357.311205] [_end+129904608/2130489456] :rt61:BeaconTimeout+0x0/0x30
Apr 9 01:26:26 jennifer kernel: [ 357.311211] [run_timer_softirq+322/448] run_timer_softirq+0x142/0x1c0
Apr 9 01:26:26 jennifer kernel: [ 357.311221] [__do_softirq+95/208] __do_softirq+0x5f/0xd0
Apr 9 01:26:26 jennifer kernel: [ 357.311229] [call_softirq+28/40] call_softirq+0x1c/0x28
Apr 9 01:26:26 jennifer kernel: [ 357.311235] [do_softirq+44/144] do_softirq+0x2c/0x90
Apr 9 01:26:26 jennifer kernel: [ 357.311240] [smp_apic_timer_interrupt+94/128] smp_apic_timer_interrupt+0x5e/0x80
Apr 9 01:26:26 jennifer kernel: [ 357.311246] [apic_timer_interrupt+102/112] apic_timer_interrupt+0x66/0x70
Apr 9 01:26:26 jennifer kernel: [ 357.311249] <EOI>

I hope this helps, and I hope this will be addressed before feisty ships.

Looking at the traceback that John Clemens just posted -- it looks pretty obviously like another variation of the same class of deadlock that I described before:

 - BeaconTimeout() ran from a timer, and calls into MlmeHandler()
 - MlmeHandler() runs the table-based state machine
 - The state machine calls MlmeJoinReqAction()
 - Right at the top of the function, MlmeJoinReqAction() does
        del_timer_sync(&pAd->MlmeAux.BeaconTimer);
 - But we're running from the BecaonTimer callback already

and voila, we have another "wait for timer to finish from within timer callback" deadlock.

Looking at the rt61 driver source, I really feel it was a mistake to merge this into the Ubuntu kernel in the first place. I realize that there are probably some people who are using the rt61 driver without lock ups, but at this point I don't think feisty should ship with rt61 enabled by default.

I think the reason this bug is having a more severe impact now is that all kernels are built with CONFIG_SMP now, so del_timer_sync() is never converted to del_timer(). So another possible fix would be to enable the code in rtmp.h that does:

#undef del_timer_sync
#define del_timer_sync(x) del_timer(x)

that would probably convert this easily-triggered deadlock into much rarer strange crashes on SMP systems. (Although I know that almost every modern system is dual-core at least)

I resolved my problem with the rt61 driver by switching to the 386 kernel and the system is rock-solid again. As the 386 Kernel isn't build with CONFIG_SMP this is just a work-around for people using old cpu's wich don't need the SMP feature. Of course this is not a solution to the problem as the SMP feature is important for the future, but for me it's the only possibillity that I've found, which let's me work with this adapter.

The status of this bug is still "needs info", but I think that we've gathered enough information on this issue. I know it's very late, but please throw this driver out of feisty and include the one from the rt2x00 project, if it realy works well. Also if that one isn't stable enough as the developers say, it should be better than this one, at least for the generic version of the kernel fof feisty. Cars using rt61 aren't that exotic at all and I think it would be very important to finish this porblem before the release.

I'm experiencing the same problem with the following configuration:

00:00.0 Host bridge: Silicon Integrated Systems [SiS] SiS645DX Host & Memory & AGP Controller
00:01.0 PCI bridge: Silicon Integrated Systems [SiS] Virtual PCI-to-PCI bridge (AGP)
00:02.0 ISA bridge: Silicon Integrated Systems [SiS] SiS961 [MuTIOL Media IO] (rev 10)
00:02.1 SMBus: Silicon Integrated Systems [SiS] SiS961/2 SMBus Controller
00:02.2 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 07)
00:02.3 USB Controller: Silicon Integrated Systems [SiS] USB 1.0 Controller (rev 07)
00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (rev d0)
00:03.0 Ethernet controller: Silicon Integrated Systems [SiS] SiS900 PCI Fast Ethernet (rev 90)
00:05.0 Multimedia audio controller: C-Media Electronics Inc CM8738 (rev 10)
00:09.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
00:09.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 61)
00:09.2 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 63)
00:09.3 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
00:09.4 RAID bus controller: VIA Technologies, Inc. VT6421 IDE RAID Controller (rev 50)
01:00.0 VGA compatible controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (rev 01)
01:00.1 Display controller: ATI Technologies Inc RV280 [Radeon 9200 PRO] (Secondary) (rev 01)

(note the lack of the wireless card now)

Basically, it's a vanilla P4 system. If I put the Linksys WMP54G 4.1 (Rt61 in) within 30 seconds to 60 minutes after bootup, I *will* get a soft lockup on cpu #0. Before, I had this system configured with Dapper and the proprietary driver, now I'm on Fiesty and no wireless. Ah, progress. I wish this horrible driver was never included with the kernel, so at least I would have been aware that it was the network card's fault after I compiled the kernel module and inserted it. Instead, it's taken me hours to figure out this situation.

try appending acpi=force irqpoll to your kernel parameters

possibly related to my issue, if eth0 is up and ra0 is brought up the system get soft lockup, try removing either of these entries from /etc/network/interfaces and see if the crashing persists.

Rather disappointing that Feisty went live with this bugged driver from SerialMonkey

I had gone at length in Edgy to get the rt61 running with WEP, by removing the rt61pci module and downloading and compiling the rt61 driver from ralinktech.com, finally with success.

Now after the upgrade to Feisty everything was broken again: the distribution ships with a rt61 (same name to add to the confusion), which is not configured to my WEP needs, and connects to my neighbors unprotected router. After endless ifconfigs, ifups/downs, /etc/netwrok/interfaces, iwconfigs etc, the system got completely instable with the afore mentioned deadlock.

Reverting back to ralinktech.com learns thereafter that now I also cannot compile the available driver anymore, due to kernel changes for net_device. Cute.

Many hours later I managed to got it to compile without errors and all is working. That's all thanks to having a multiboot system, where I still was able to get to some files and network ...

I was a big fan of Ubuntu, but my enthusiasm has diminished. I wonder how our less experienced users (which are the target group for Ubuntu - wasn't it?) are doing with this

Hi!

I don't know how much this could be useful, but I have unistalled Feisty and installed linux mint "cassandra" and I had no more freeze!

Since the two OS 99% identical (linux mint has codecs-flash plug in and Java preinstalled and different menus and themes; nothing really essential) it may be interesting to figure out what is this 1%: my guess is that the problem is there.

Same freeze here, and I confirm that it only happens on linux-generic (2.6.20). The bug does not occur on linux-386.
The symptoms are the same as detailed above, i.e. complete lockup of the system except for the mouse and sometimes keyboard repeating indefinitely.
My system is the following:

00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02)
00:01.0 PCI bridge: Intel Corporation 82P965/G965 PCI Express Root Port (rev 02)
00:19.0 Ethernet controller: Intel Corporation 82566DC Gigabit Network Connection (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #5 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02)
00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2)
00:1f.0 ISA bridge: Intel Corporation 82801HH (ICH8DH) LPC Interface Controller (rev 02)
00:1f.2 RAID bus controller: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) SATA RAID Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation GeForce 7300 LE (rev a1)
04:03.0 Network controller: RaLink RT2561/RT61 802.11g PCI

Hope it helps narrowing the issue a little further. Let me know if I can do anything else.

Same here except I am running Gutsy, not Feisty.

The computer locks up sometimes during boot, sometimes after a few minutes of usage, and even sometimes after a couple of hours.

When using a graphical environment. The keyboard hangs first, mouse still works a couple of seconds, soon everything is locked.

Have not tried using the linux-386 kernel.

The current settings are:
Linux 2.6.22-11-generic #1 SMP Mon Sep 17 03:45:58 GMT 2007 i686 GNU/Linux

My system is a Dell Vostro 200 (2GHz Intel Duo Core)

00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller (rev 02)
00:01.0 PCI bridge: Intel Corporation 82G33/G31/P35/P31 Express PCI Express Root Port (rev 02)
00:02.0 VGA compatible controller: Intel Corporation 82G33/G31 Express Integrated Graphics Controller (rev 02)
00:1a.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 02)
00:1a.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 02)
00:1a.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 02)
00:1a.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 02)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 02)
00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IR (ICH9R) LPC Interface Controller (rev 02)
00:1f.2 IDE interface: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 4 port SATA IDE Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA IDE Controller (rev 02)
02:01.0 Network controller: RaLink RT2561/RT61 802.11g PCI

See also https://bugs.launchpad.net/ubuntu/+bug/34902

The same issue.

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this bug to the new "linux" package. However, development has already began for the upcoming Intrepid Ibex 8.10 release. It would be helpful if you could test the upcoming release and verify if this is still an issue - http://www.ubuntu.com/testing . If the issue still exists, please update this report by changing the Status of the "linux" task from "Incomplete" to "New". We appreciate your patience and understanding as we make this transition. Thanks!

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.

I have a level one wpc-301 v4 that is affected by rt61pci lockup issue in hardy using 2.6.24-19-generic and version 2.0.10 of the rt61pci module. I slapped on linux-backports-modules which seemed to resolve the issue with the hard lockups, but activity light on the card stayed lit regardless of activity.

I have now installed linux-image-2.6.27-4-generic_2.6.27-4.6_i386, and so far all seems good, as it has not locked up yet and problem with activity light is gone.

linux-image-2.6.27-4-generic and rt61pci on my machine has serious problems with resume from suspend. events/0 pegs the cpu at 100% and keyboard becomes unresponsive. It happens on about 80% of time, the other 20% of time it will resume fine - but removing the network card and reinserting it will cause the issue after the case of a initial successful resume.

This problem does not happen with 2.6.24-19-generic and version 2.1.4 of the rt61pci in linux-backports-modules.

Went Intrepid today -- The events/0 problems is gone. It was perhaps a the relationship between the version nm-manager in 8.04 and rt61pci driver in 2.6.27-4.

For me this issue is gone with Intrepid and 2.6.27-4.

many thanks

Based on the previous comments I'm marking this Fix Released against Intrepid. Thanks.