netboot: aic94xx sequencer firmware is missing from initrd.gz

Same issue with Jaunty install using both the alternate install, and the desktop install. Firmware fails to load. Using alternate install, there are no issues on install, but after install, boot fails with aic94xx firmware error below. On desktop install, it doesn't load on boot of the cd. Here are the errors:

ubuntu@ubuntu:~$ dmesg | grep aic94xx
[ 6.164408] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded
[ 6.164679] aic94xx 0000:05:02.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
[ 6.549828] aic94xx: found Adaptec AIC-9410W SAS/SATA Host Adapter, device 0000:05:02.0
[ 6.549841] scsi6 : aic94xx
[ 6.609511] aic94xx 0000:05:02.0: firmware: requesting aic94xx-seq.fw
[ 66.608563] aic94xx: Failed to load sequencer firmware file aic94xx-seq.fw, error -2
[ 66.608568] aic94xx: couldn't init seqs for 0000:05:02.0
[ 66.608570] aic94xx: couldn't init the chip
[ 66.608747] aic94xx 0000:05:02.0: PCI INT A disabled
[ 66.608756] aic94xx: probe of 0000:05:02.0 failed with error -2

Confirmed. Same problems:

kmallory@pandora:~$ dmesg | grep aic
[ 3.104198] aic94xx: Adaptec aic94xx SAS/SATA driver version 1.0.3 loaded
[ 5.388293] aic94xx 0000:03:02.0: power state changed by ACPI to D0
[ 5.388313] aic94xx 0000:03:02.0: PCI INT A -> GSI 32 (level, low) -> IRQ 32
[ 5.388373] aic94xx: found Adaptec AIC-9410W SAS/SATA Host Adapter, device 0000:03:02.0
[ 5.388378] scsi3 : aic94xx
[ 5.411492] aic94xx 0000:03:02.0: firmware: requesting aic94xx-seq.fw
[ 65.408063] aic94xx: Failed to load sequencer firmware file aic94xx-seq.fw, error -2
[ 65.408067] aic94xx: couldn't init seqs for 0000:03:02.0
[ 65.408070] aic94xx: couldn't init the chip
[ 65.408226] aic94xx 0000:03:02.0: PCI INT A disabled
[ 65.408236] aic94xx: probe of 0000:03:02.0 failed with error -2

I tried symlinking /lib/firmware/aic94xx-seq.fw into the /lib/firmware/<kernel-version>/aic94xx-seq.fw, and also download the latest firmware from Adaptec. Still no luck. Ubuntu/Debian needs this fixed. I've got 4TB offline right now because of it.

I can also confirm this on an IBM x3550 server with the Adaptec aic94xx RAID. I'm getting exactly the same error messages as shown above, and I've also tried to symlink the aic94xx-seq.fw file as in the previous comment with the same result. In my case this is not yet a production system, but it will definitely set me back a few hours wiping the disk, reinstalling with hardy or intrepid and then restoring the data.

Following up on this, I found the following two bugs on RHEL. The first reports the same issues, and the solution of rmmod and modprobe aic94xx works for me. Ie, if I 'sudo rmmod aic94xx' and then 'sudo modprobe aic94xx', I can then mount my drives without issue, but I have to do this after each reboot.

The second bug points to a simple code change in hotplug.c, which seems to address another issue. According to the first bug, the patch provided in the second link fixes this problem in RHEL4/5. hotplug.c is responsible for loading the firmware through initramfs.

https://bugzilla.redhat.com/show_bug.cgi?id=240105

https://bugzilla.redhat.com/show_bug.cgi?id=378651

I will investigate the code, but my environment is a non-build environment, so I cannot compile and apply the patch myself. :-( In my particular case, I don't need the drives on the aic94xx interface until after the machine boots, so I'm inclined to build a init.d script and put it at level 99, or something.

From http://forums.fedoraforum.org/showthread.php?t=114819

I found a way to get at the Ram disk image and put in the aic94xx firmware.

Ive tried putting it in /firmware and /lib/firmware

I cannot get it to boot correctly still. Still says firmware is missing.

Any Ideas / Fixes?

After much tinkering, I think I have found what is missing. if you update-initramfs -k all -c
it #warns #W the firmware /lib/firmware/<kernelversion>/aic94xx may be missing, sure enough it was in /lib/firmware
I Copied the firmware file to /lib/firmware/kernelversion/ still says its missing... Did some more reading... found this may be related to udev rules.

copied:
 /lib/udev/firmware.sh
/lib/udev/rules.d/50-firmware.rules

into the initramfs img

procedure for adding / removing from initramfs:
mkdir /boot/tmp
cd /boot/tmp
cat <initrd> | gzip -c | cpio -id

/* Add and remove files / edit ... etc... */

cd /boot/tmp
find . | cpio --create --format='newc' | gzip > ../newinitramfs.img
cd /boot
/* backup old initramfs file */
/* rename your initramfs file to the origional in this case it was initrd.img-2.6.28-13-server */

I still think this is a bug. as this creates some udev output that I am currently unsure how to quiet.
I found this to be a great learning experience but would not expect people trying to get something to work to have to deal with.

Hope this helps someone.

Building on Delbert's work, the following commands are what I had to do to get this working. Basically I use an initramfs hook to make sure the firmware and supporting files are included on the initrd image every time it's built (like when you do a kernel upgrade). Then you just use update-initramfs to re-build the image.
According to this bug: https://bugs.launchpad.net/ubuntu/+source/udev/+bug/328550 it looks like this won't be getting fixed any time soon?
Also, it looks like you don't actually need the firmware to be copied to /lib/firmware/<kernelversion>/aic94xx even though update-initramfs whines about it.

I hope this gets a more official fix, or at least some notification during the install process that you can't use drives requiring this driver as your root partiton

cat << EOF > /etc/initramfs-tools/hooks/firmware_aic94xx
#!/bin/sh

PREREQ="udev"

prereqs()
{
echo "$PREREQ"
}

case $1 in
prereqs)
prereqs
exit 0
;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /lib/udev/firmware.sh /lib/udev

cp /lib/udev/rules.d/50-firmware.rules $DESTDIR/lib/udev/rules.d/

mkdir -p $DESTDIR/lib/firmware
cp /lib/firmware/aic94xx-seq.fw $DESTDIR/lib/firmware
EOF

chmod +x /etc/initramfs-tools/hooks/firmware_aic94xx

update-initramfs -k all -c

Same problem still exists on Karmic amd64 server install. Our Supermicro servers have only Adaptec storage controller for SAS disks. Since this is server distribution, I think this should work out of the box!

Installation CD obviously has firmware issue fixed, because I can do installation without any problems. Booting from hd fails.

im trying to install ubuntu 9.04/9.10 server edition to an old x306m that has adaptec 9405w… and two sata 160gb HDD… since i don’t need RAID for these two… what i need is to just install an Ubuntu OS to the first HDD and use the 2nd HDD as data storage… taking note that i have both plugged in to adaptec 9405w…

i disabled the raid in bios, and when ubuntu prompted to Activate Serial ATA Raid devices:

when i said yes: i always end up in a ‘No root file system is defined’
when i said no: i used a guided partitioning and used the whole 160gb with LVM.. ubuntu detects a scsi3 (0,0,0) (sda) and installs the system…

but upon rebooting after installation:

Boot from (hd0,4) ext2
Starting up …
Loading, please wait…
no block devices found
no block devices found
no block devices found
no block devices found

Gave up waiting for root device… etc…

thanks for the help in advance…

FYI: bug 328550 has been fixed in the package initramfs-tools - 0.92bubuntu54

Hi azer,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/releases/lucid . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 315763

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

I'm running 9.10 at this time, and every single official kernel release I still have to extract the initrd, add aic94xx-seq.fw to {initrdroot}/lib/firmware NOT {initrdroot}/lib/firmware/{kernel-version}/ and repack the initrd, or it will fail to find root on booting of the system.

I can't test the alpha at this time to verify it still fails (as I enjoy a relatively stable system), but it's failed in certainly the last 2-3 ubuntu releases, and I highly doubt that it's resolved unless someone can verify it.

When Lucid is GA, I'll test it and update this bug with some verification.

I can confirm that PXE booting (I suspect from a CD as well) of Lucid SERVER (desktop work fine) does not have the firmware available, so the modprobe fails.

I can bypass this by:
* Wget'ing the firmware in an alternate firmware ( cd /lib/firmware ; wget http://kernel.org/pub/linux/kernel/people/jejb/aic94xx-seq.fw)
* Removing the module ( modprobe -r aic94xx )
* Switching back to the main screen and selecting the module to load again.

Indeed, aic94xx firmware is missing from initrd.gz in 10.04.1. Perhaps it was generated by initramfs-tools before they were fixed (in 0.92bubuntu54)?

For PXE boot, the workaround is to create another initrd with a single file inside (the firmware), and then pass its name like this:

"initrd=ubuntu-installer/amd64/initrd.gz,ubuntu-installer/amd64/aic94xx-seq.gz"

Strange, it's also missing from 10.10's initrd.gz.

I'm not sure which package generates initrd.gz, but debian-installer is my best guess.

Sergey, you're conflating two things here. You're talking about the installer initrd, which does not need to include aic94xx-seq.fw because it's in the scsi-firmware udeb which is loaded at run-time. However, the folks reporting this bug are talking about the initramfs that's used on the installed system, which is generated using an entirely separate system (initramfs-tools).

On my Natty system, /lib/firmware/aic94xx-seq.fw is in the initramfs. The odd thing is that the code that puts it there has been there for a long time, back to Ubuntu 8.10, so I don't quite see why this wouldn't have worked in older releases; it's possible that this is system-specific, and that maybe I'm just lucky that the file is present here. This needs some more analysis.

Actually, I have both problems (and more):

- firmware is not loaded by the netbooted installer in Karmic and in Lucid. I had to force it with "initrd=ubuntu-installer/amd64/initrd.gz,ubuntu-installer/amd64/aic94xx-seq.gz" hack. I didn't know about scsi-firmware udeb; looks like it's not working the way it's intended to.

- installer says "One or more drives containing Serial ATA RAID configurations have been found. Do you wish to activate these RAID devices?" and then fails to detect any drives (even though "fdisk -l" does list them). "d-i disk-detect/activate_dmraid boolean false" fixed that for me (https://answers.launchpad.net/ubuntu/+source/util-linux/+question/134126)

- after installation, firmware is not present in initrd (at least in Karmic); late_command like this fixes it:

for m in `echo /lib/firmware/*-generic /lib/firmware/*-server`; do
        cp -p /lib/firmware/aic94xx-seq.fw $m
done
update-initramfs -k all -t -u