/boot full unable to clean successfully

For diagnostic purposes please provide the output of the commands

uname -a
lsb_release -crid
ls -l /boot
dpkg -l | grep ' linux-i'
df -h

(In case that you see the need to hide certain information from the public, you are free to obfuscate it - as long as the data that are required for diagnostics stay readable.)

Thanks you for responding,I apologize about the delay in response, I had to type this all out as I could not find a way to copy from my terminal on the VM to another source:

uname -a

Linux NASBUBUNTU 4.4.0-164-generic #192-Ubuntu SMP Fri Sep 13 12:02:50 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -crid

Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

ls -l /boot

total 960496
-rw-r--r-- 1 root root 1252309 Oct 24 2018 abi-4.4.0-139-generic
-rw-r--r-- 1 root root 1252309 Nov 14 2018 abi-4.4.0-140-generic
-rw-r--r-- 1 root root 1252429 Dec 5 2018 abi-4.4.0-141-generic
-rw-r--r-- 1 root root 1252376 Jan 16 2019 abi-4.4.0-142-generic
-rw-r--r-- 1 root root 190611 Oct 24 2018 config-4.4.0-139-generic
-rw-r--r-- 1 root root 190611 Nov 14 2018 config-4.4.0-140-generic
-rw-r--r-- 1 root root 190611 Dec 5 2018 config-4.4.0-141-generic
-rw-r--r-- 1 root root 190580 Jan 16 2019 config-4.4.0-142-generic
-rw-r--r-- 1 root root 190560 Feb 7 2019 config-4.4.0-143-generic
-rw-r--r-- 1 root root 190580 Mar 26 2019 config-4.4.0-145-generic
-rw-r--r-- 1 root root 190862 Apr 3 2019 config-4.4.0-146-generic
-rw-r--r-- 1 root root 190862 May 7 2019 config-4.4.0-148-generic
-rw-r--r-- 1 root root 190871 May 29 15:11 config-4.4.0-150-generic
-rw-r--r-- 1 root root 190871 June 11 04:47 config-4.4.0-151-generic
-rw-r--r-- 1 root root 190871 June 25 01:43 config-4.4.0-154-generic
-rw-r--r-- 1 root root 190871 July 23 06:05 config-4.4.0-157-generic
-rw-r--r-- 1 root root 190943 Aug 1 12:47 config-4.4.0-159-generic
-rw-r--r-- 1 root root 190973 Aug 27 04:24 config-4.4.0-161-generic
-rw-r--r-- 1 root root 190973 Sep 13 08:29 config-4.4.0-164-generic
-rw-r--r-- 1 root root 190973 Sep 17 15:18 config-4.4.0-165-generic
drwxr-xr-x 5 root root 4096 Sep 18 01:49 grub
-rw-r--r-- 1 root root 18050833 Dec 12 2018 initrd.img-4.4.0-104-generic.old-dkms
-rw-r--r-- 1 root root 18100094 Dec 12 2018 initrd.img-4.4.0-119-generic.old-dkms
-rw-r--r-- 1 root root 18136042 Dec 12 2018 initrd.img-4.4.0-131-generic.old-dkms
-rw-r--r-- 1 root root 18134435 Dec 12 2018 initrd.img-4.4.0-133-generic.old-dkms
-rw-r--r-- 1 root root 18136404 Dec 12 2018 initrd.img-4.4.0-134-generic.old-dkms
-rw-r--r-- 1 root root 18134681 Dec 12 2018 initrd.img-4.4.0-135-generic.old-dkms
-rw-r--r-- 1 root root 18134687 Dec 12 2018 initrd.img-4.4.0-137-generic.old-dkms
-rw-r--r-- 1 root root 18141564 Dec 12 2018 initrd.img-4.4.0-138-generic.old-dkms
-rw-r--r-- 1 root root 40754033 Jul 19 01:52 initrd.img-4.4.0-139-generic
-rw-r--r-- 1 root root 40747007 Jul 19 01:51 initrd.img-4.4.0-140-generic
-rw-r--r-- 1 root root 40747009 Jul 19 01:51 initrd.img-4.4.0-141-generic
-rw-r--r-- 1 root root 40764929 Jul 19 01:50 initrd.img-4.4.0-142-generic
-rw-r--r-- 1 root root 40764147 Jul 19 01:50 initrd.img-4.4.0-143-generic
-rw-r--r-- 1 root root 40771776 Jul 19 01:49 initrd.img-4.4.0-145-generic
-rw-r--r-- 1 root root 40771314 Jul 19 01:49 initrd.img-4.4.0-146-generic
-rw-r--r-- 1 root root 40763262 Jul 19 01:48 initrd.img-4.4.0-148-generic
-rw-r--r-- 1 root root 40764571 Jul 19 01:48 initrd.img-4.4.0-150-generic
-rw-r--r-- 1 root root 40767807 Jul 19 01:48 initrd.img-4.4.0-151-generic
-rw-r--r-- 1 root root 40768715 Jul 19 01:47 initrd.img-4.4.0-154-generic
-rw-r--r-- 1 root root 40774202 Jul 25 01:49 initrd.img-4.4.0-157-generic
-rw-r--r-- 1 root root 40786072 Aug 14 01:50 initrd.img-4.4.0-159-generic
-rw-r--r-- 1 root root 40783652 Sep 3 01:51 initrd.img-4.4.0-161-generic
-rw-r--r-- 1 root root 40785985 Sep 18 01:49 initrd.img-4.4.0-164-generic
-rw-r--r-- 1 root root 10527127 Jul 19 01:52 initrd.img-4.4.0-77-generic
-rw-r--r-- 1 root root 37838848 Apr 9 2018 initrd.img-4.4.0-77-generic.old-dkms
drwx------ 2 root root 16384 May 1 2017 lost+found
-rw-r--r-- 1 root root 255 Oct 24 2018 retpoline-4.4.0-139-generic
-rw-r--r-- 1 root root 255 Nov 14 2018 retpoline-4.4.0-140-generic
-rw-r--r-- 1 root root 255 Dec 5 2018 retpoline-4.4.0-141-generic
-rw-r--r-- 1 root root 255 Jan 16 2019 retpoline-4.4.0-142-generic
-rw------- 1 root root 3904765 Oct 24 2018 System.map-4.4.0-139-generic
-rw------- 1 root root 3905265 Nov 14 2018 System.map-4.4.0-140-generic
-rw------- 1 root root 3905747 Dec 5 2018 System.map-4.4.0-141-generic
-rw------- 1 root root 3904797 Jan 16 2019 System.map-4.4.0-142-generic
-rw------- 1 root root 3905647 Feb 7 2019 System.map-4.4.0-143-generic
-rw------- 1 root root 3906115 Mar 26 2019 System.map-4.4.0-145-generic
-rw------- 1 root root 3906510 Apr 3 2019 System.map-4.4.0-146-generic
-rw------- 1 root root 3907764 May 7 2019 System.map-4.4.0-148-generic
-rw------- 1 root root 3908727 May 29 15:11 System.map-4.4.0-150-generic
-rw------- 1 root root 3908794 Jun 11 04:47 System.map-4.4.0-151-generic
-rw------- 1 root root 3908872 Jun 25 01:43 System.map-4.4.0-154-generic
-rw------- 1 root root 3909878 Jul 23 06:05 System.map-4.4.0-157-generic
-rw------- 1 root root 3910823 Aug 1 12:47 System.map-4.4.0-159-generic
-rw------- 1 root root 3911629 Aug 27 04:24 System.map-4.4.0-161-generic
-rw------- 1 root root 3911629 Sep 13 04:24 System.map-4.4.0-164-generic
-rw------- 1 root root 3911756 Sep 17 15:18 System.map-4.4.0-165-generic
-rw------- 1 root root 7168736 Oct 24 2018 vmlinuz-4.4.0-139-generic
-rw------- 1 root root 7169888 Nov 14 2018 vmlinuz-4.4.0-140-generic
-rw------- 1 root root 7171392 Dec 5 2018 vmlinuz-4.4.0-141-generic
-rw------- 1 root root 7184032 Jan 16 2019 vmlinuz-4.4.0-142-generic
-rw------- 1 root root 7190840 Feb 12 2019 vmlinuz-4.4.0-143-generic
-rw------- 1 root root 7188984 Mar 27 2019 vmlinuz-4.4.0-145-generic
-rw------- 1 root root 7197208 Apr 3 2019 vmlinuz-4.4.0-146-generic
-rw------- 1 root root 7196216 May 14 13:09 vmlinuz-4.4.0-148-generic
-rw------- 1 root root 7198616 May 30 03:28 vmlinuz-4.4.0-150-generic
-rw------- 1 root root 7198776 Jun 17 12:19 vmlinuz-4.4.0-151-generic
-rw------- 1 root root 7199000 Jun 25 11:58 vmlinuz-4.4.0-154-generic
-rw------- 1 root root 7202616 Jul 23 11:08 vmlinuz-4.4.0-157-generic
-rw------- 1 root root 7203416 Aug 6 12:34 vmlinuz-4.4.0-159-generic
-rw------- 1 root root 7206200 Aug 28 08:03 vmlinuz-4.4.0-161-generic
-rw------- 1 root root 7206264 Sep 17 14:28 vmlinuz-4.4.0-164-generic

dpkg -l | grep 'linux-i'

rc linux-image-4.4.0-101-generic 4.4.0-101.124
rc linux-image-4.4.0-104-generic 4.4.0-104.127
rc linux-image-4.4.0-119-generic 4.4.0-119.143
rc linux-image-4.4.0-131-generic 4.4.0-131.157
rc linux-image-4.4.0-133-generic 4.4.0-133.159
rc linux-image-4.4.0-134-generic 4.4.0-134.160
rc linux-image-4.4.0-135-generic 4.4.0-135.161
rc linux-image-4.4.0-137-generic 4.4.0-137.163
rc linux-image-4.4.0-138-generic 4.4.0-138.164
ii linux-image-4.4.0-139-generic 4.4.0-139.165
ii linux-image-4.4.0-140-generic 4.4.0-140.166
ii linux-image-4.4.0-141-generic 4.4.0-141.167
ii linux-image-4.4.0-142-generic 4.4.0-142.168
ii linux-image-4.4.0-143-generic 4.4.0-143.169
ii linux-image-4.4.0-145-generic 4.4.0-145.171
ii linux-image-4.4.0-146-generic 4.4.0-146.172
ii linux-image-4.4.0-148-generic 4.4.0-148.174
ii linux-image-4.4.0-150-generic 4.4.0-150.176
ii linux-image-4.4.0-151-generic 4.4.0-151.178
ii linux-image-4.4.0-154-generic 4.4.0-154.181
ii linux-image-4.4.0-157-generic 4.4.0-157.185
ii linux-image-4.4.0-159-generic 4.4.0-159.187
ii linux-image-4.4.0-161-generic 4.4.0-161.189
ii linux-image-4.4.0-164-generic 4.4.0-164.192
rc linux-image-4.4.0-62-generic 4.4.0-62.83
rc linux-image-extra-4.4.0-101-generic 4.4.0-101.124
rc linux-image-extra-4.4.0-104-generic 4.4.0-104.127
rc linux-image-extra-4.4.0-119-generic 4.4.0-119.143
rc linux-image-extra-4.4.0-131-generic 4.4.0-131.157
rc linux-image-extra-4.4.0-133-generic 4.4.0-133.159
rc linux-image-extra-4.4.0-134-generic 4.4.0-134.160
rc linux-image-extra-4.4.0-135-generic 4.4.0-135.161
rc linux-image-extra-4.4.0-137-generic 4.4.0-137.163
rc linux-image-extra-4.4.0-138-generic 4.4.0-138.164
ii linux-image-extra-4.4.0-139-generic 4.4.0-139.165
ii linux-image-extra-4.4.0-140-generic 4.4.0-140.166
ii linux-image-extra-4.4.0-141-generic 4.4.0-141.167
ii linux-image-extra-4.4.0-142-generic 4.4.0-142.168
rc linux-image-extra-4.4.0-62-generic 4.4.0-62.83
rc linux-image-extra-4.4.0-77-generic 4.4.0-77.98
iU linux-image-generic 4.4.0-166.174

df -h

Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 779M 8.7M 770M 2% /run
/dev/sda1 182G 11G 161G 7% /
tmpfs 3.9G 132K 3.9G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda6 967M 947M 0 100% /boot
tmpfs 779M 0 779 0% /run/user/1000

I added a comment, was I supposed to press "I'm providing more information"

General remark to avoid retyping:
What you could have done (or might do in future) is redirecting all output to a file,
e.g.
uname -a | tee mytempfile.txt
lsb_release -crid | tee -a mytempfile.txt #note the -a for append for subsequent commands
etc.
and then uploading that file to pastebin.ubuntu.com with the command
pastebinit mytempfile.txt
This last command will tell you an URL which you can type into this question document, and we can then see your output.

For your space problem:
I see some obsolete files that can be removed:

sudo rm /boot/*.old-dkms
sudo rm /boot/initrd.img-4.4.0-77-generic
sudo rm /boot/*-4.4.0-139-generic

This should give enough free space that the package management will work again.

Then try

sudo dpkg --purge linux-image-4.4.0-139-generic linux-image-extra-4.4.0-139-generic
sudo dpkg --configure -a

and report the results.

sudo dpkg -P linux-image-4.4.0-101-generic
sudo dpkg -P linux-image-4.4.0-104-generic
sudo dpkg -P linux-image-4.4.0-119-generic
sudo dpkg -P linux-image-4.4.0-131-generic
sudo dpkg -P linux-image-4.4.0-133-generic
sudo dpkg -P linux-image-4.4.0-134-generic
sudo dpkg -P linux-image-4.4.0-135-generic
sudo dpkg -P linux-image-4.4.0-137-generic
sudo dpkg -P linux-image-4.4.0-138-generic
sudo dpkg -P linux-image-4.4.0-62-generic
sudo dpkg -P linux-image-extra-4.4.0-101-generic
sudo dpkg -P linux-image-extra-4.4.0-104-generic
sudo dpkg -P linux-image-extra-4.4.0-119-generic
sudo dpkg -P linux-image-extra-4.4.0-131-generic
sudo dpkg -P linux-image-extra-4.4.0-133-generic
sudo dpkg -P linux-image-extra-4.4.0-134-generic
sudo dpkg -P linux-image-extra-4.4.0-135-generic
sudo dpkg -P linux-image-extra-4.4.0-137-generic
sudo dpkg -P linux-image-extra-4.4.0-138-generic

Will also help

Manfred,

Thank you fro your patience and help, I thought I could run the output command at the end, that didnt work so I printed the screen and uploaded here:

https://imgur.com/a/eF2Qgsn

Im not certain but is there still an issue? I do have 171mb free on the boot drive so thats good.

actionparsnip, thank you for your suggestions.

What is the output of:

df -h

Thanks

Hey, here you go

http://paste.ubuntu.com/p/hNDFdkK77j/

Thank you

Why make a pastebin? The forum is text based and there is no limit to posts so why not just post it in line?

You have 20% free on /boot

If you remove old kernels then it will free more space

Hey,

I have a HyperV VM of Ubuntu, I have no way of copy text from terminal to clipboard that I know of. So do I run the commands you posted earlier? I am a novice at linux, please be as detailed as possible.

Also what do the messages of my last image above mean.....

dpkg: error processing package linux-image-generic (--configure)

It says that multiple times with different versions on each, should I be concerned about this?

Thanks

Your uploaded image shows some missing dependencies, such that the command "sudo dpkg --configure" failed. There is already some free space on /boot, but I suggest to free some more first, end then try installing the missing packages.

Run the following commands

sudo dpkg --purge linux-image-4.4.0-140-generic linux-image-extra-4.4.0-140-generic
sudo dpkg --purge linux-image-4.4.0-141-generic linux-image-extra-4.4.0-141-generic
sudo apt install -f

and report what the last five lines of the last command are.

Ok when I run

sudo dpkg --purge linux-image-4.4.0-140-generic linux-image-extra-4.4.0-140-generic

I got this:

dpkg: dependency problems prevent removal of linux-image-4.4.0-140-generic:
  linux-image-extra-4.4.0-140-generic depends on linux-image-4.4.0-140-generic.

dpkg: error processing package linux-image-4.4.0-140-generic (--purge):
  dependcy problems - not removing
dpkg: warning: ignoring request to remove linux-image-extra-4.4.0-140-generic which isn't installed
Errors were encountered while processing:
 linux-image-4.4.0-140-generic

I have not run any other of your posted commands since this failed.

Thanks

Try the other way round

sudo dpkg --purge linux-image-extra-4.4.0-140-generic linux-image-4.4.0-140-generic

Manfred,

Seriously thank you for your patience and help. I was able to run those commands the other way, when I run:

sudo apt install -f

These are the last five line, I have not proceeded:

0 upgraded, 3 newly installed, 0 to remove and 136 not upgraded.
4 not fully installed or removed
Need to get 0 B/25.8 MB of archives.
After this operation, 74.2 MB of additional disk space will be used.
Do you want to continue [Y/n]

I am assuming I should continue?

Thanks

Yes, press enter to continue. And at the end tell us whether there is success, or iof there still are error messages.

Ok here it is

http://paste.ubuntu.com/p/v7HbJjMbWq/

Thanks

When looking at the Webmin URL it appears I should reboot, should I do that now or wait?

Ok, no error shown, and all broken dependencies seem to be resolved. I assume that all normal package management commands will work again.

My suggestion is that you try rebooting whenever convenient, and checking the kernel version (with the command "uname -a") afterwards. It should then be 4.4.0-166.

What you also should do (if rebooting into the new kernel is working well):

1. purge the obsolete versions of the kernel packages, e.g. with the command
sudo apt --purge autoremove

2. install all available package updates, e.g. with the command
sudo apt full-upgrade

Ok done,

It appears only 171MB are free in the /boot, how do I free up more of that space so this doesnt happen again?

Is there a better way of maintaining the /boot that I should learn about?

Thank you for you help

Have you already executed the command "sudo apt --purge autoremove"?

If yes, then this can be caused by a known problem.

What is the contents of the file
/etc/apt/apt.conf.d/01autoremove-kernels
(upload it to pastebin)
and what is now the output of
uname -a
and what is the output of
dpkg -l | grep ' linux-'
?

Ok,

I had already run the command

sudo apt --purge autoremove

When that ran I thought it said it would free up 603MB, but that never re-appeared on the /boot

When I run

cd /etc/apt/apt.conf.d/01autoremove-kernels

I get:

-bash: cd: /etc/apt/apt.conf.d/01autoremove-kernels: Not a directory

uname -a

4.4.0-166-generic

Output of

dpkg -l | grep ' linux-' | tee mytempfile.txt

http://paste.ubuntu.com/p/VV8M24WcRZ/

Thank you

to see the contents of a file you have to use the command cat

cat /etc/apt/apt.conf.d/01autoremove-kernels

(or to upload: "pastbinit /etc/apt/apt.conf.d/01autoremove-kernels")

Please also show the output of the command

apt-mark showmanual linux-*

Here it is

http://paste.ubuntu.com/p/c6kJXctC9g/

and the next command you request output for

http://paste.ubuntu.com/p/YRW4cjNY5R/

Thanks

Ok, as expected. several kernel packages are wrongly marked as "manually installed" (a known problem).

To reset the apt-mark flags on the packages and remove all obsolete kernel packages issue the following commands:

sudo apt-mark auto linux-headers-4*
sudo apt-mark auto linux-image-4*
sudo apt-mark auto linux-image-extra-4.4.0-142-generic
sudo apt-mark auto linux-modules-*
sudo apt --purge autoremove

The commands ending with '*' will complain "... cannot be marked as it is not installed", but that is ok.
The last command will produce lots of output. If you look closely you should see that the list of grub options becomes shorter and shorter, until there area only two or three pairs left at the end.

Finally as verification execute the command
df -h
Result should be that the space usage of boot has gone down to about 20 or 30 %.
And the command
ls -l /boot
should show only packages with three version numbers containing -164-, -165- and -166-.

Manfred,

I apologize for the delay, I will be getting back to this today and will let you know my results.

Thank you

Manfred,

It appears I have 69% free on boot, nice! Here is the output of "ls -l /boot"

http://paste.ubuntu.com/p/j4SwBSRHqv/

I did notice there were old-dkms of 139-161, do I need to do anything with those?

Thanks

The /boot/*.old-dkms files are obsolete and should also be deleted (already recommended in comment #4).

In case that they re-appear (e.g. after an upgrade that installs a new version of the kernel), then there seems to be some obsolete Information in the dkms configuration.

What is the output of

dkms status

Manfred,

Here is it:

http://paste.ubuntu.com/p/cw4WDf2ysP/

Thanks

Looks good. What is the output of:

df -h

Thanks

Agree with actionparsnip.

Looks good.

Possible further actions:
- verify whether all available package updates have been installed
(commands sudo apt update; apt list --upgradeable)
- remove the obsolete config files for packages that have already been removed
(those listed by the command dpkg -l | grep "^rc")

Here is the output of "df -h"

http://paste.ubuntu.com/p/fZFCpvgw7h/

Thanks

70% used. Looks like Manfred saved the day

He did!

Manfred,

Serious kudos to you my friend, thank you for your patience and continued support/help through this process. If I could buy you a beer I would. This help was greatly appreciated!!

-daniel

619 MB used on /boot is still more than expected,

Are those initrd.img-*-generic.old-dkms files still there? If I add their sizes (based on the listing in comment #26), they use about 460 MB. Removing them would end up with about 160 MB used (=16% usage), which is in line with my expectation.

There are several discussions and bug Reports about these old-dkms files, and all agree that they can (and should) be deleted - at least those für kernel versions that already have been uninstalled.
https://github.com/dell/dkms/issues/2
Bug #1515513

Manfred,

I am out of office today but will be back tomorrow and will update you on the the status of the old-dkms files.

Thanks

-daniel

Manfred,

Sorry for the long delay, I had a week long conference to attend last week. So I am back in office and it appears I am back down to 21% free on /boot. Can we try to clean up the old dkms files or do you have a better suggestion?

Thanks

-daniel

There is no need for those *.old-dkms files. They are not serving any purpsose (except for shor moment during the update run) and are just occupying space.

For diagnostic purposes, what is currently the output of the commands

df -h
ls -l /boot

Hi Manfred, Thank you for your quick response.

df -h

http://paste.ubuntu.com/p/X9WSFnjhXq/

ls -l /boot

http://paste.ubuntu.com/p/3Z9ffvH8M6/

-daniel

Execute

sudo rm -v /boot/initrd.img-*-generic.old-dkms

and then repeat

df -h

What is now the usage percentage of /dev/sda6 ?

Hi Mandfred,

When I run:

sudo rm -v /boot/initrd.img-*-generic.old-dkms

I get:

rm: cannot remove '/boot/initrd.img-*-generic.old-dkms' : No such file or directory

Output of:

df -h

http://paste.ubuntu.com/p/dTH9N4cbh8/

Thanks

Strange.
http://paste.ubuntu.com/p/3Z9ffvH8M6/ lists the files, but your command leads to "http://paste.ubuntu.com/p/3Z9ffvH8M6/"?

What is the output of

ls -l /boot/initrd.img-*-generic.old-dkms

Manfred,

When I try that command I get:

ls: cannot access '/boot/initrd.img-*-generic.old-dkms' : No such file or directory

Thanks

-daniel

and what is now the output of

ls -l /boot/

?

Manfred,

Here:

http://paste.ubuntu.com/p/9mjxMzQcGn/

Thanks

-daniel

Strange. There must be some kind of mistyping.

What is the output of

ls -l /boot/init*dkms

Manfred,

Here you go:

http://paste.ubuntu.com/p/24SPhgf3Br/

Thanks

ok, then execute the command:

sudo rm -v /boot/init*dkms

(no need to copy/paste the output, but check that only those *old-dkms files have been deleted)

What free percentage do you then have on /deb/sda6 ?

Manfred,

Ok, looking better. It removed 139-165 and I now have 742mb of the 933mb on /dev/sda6 only 160mb used.

Kudos man!!

-daniel

Ok, target achieved.

For the future:
If there again are some old-dkms files in the /boot directory after the next kernel version upgrade, then this should be investigated.
After the installation of a new version of the kernel packages, you should (after a certain testing period) run the command "sudo apt --purge autoremove" to uninstall the oldest version of the kernel packages.

I think that's it.
If you have further questions, don't hesitate to ask.

Manfred,

Again, thank you for your continued support and help through this issue, I even managed to learn a few things so thank you.

-daniel

The very last thing:
please mark this question document as solved.