DNS reporting bad update servers for ubuntu.security.com so package install fails.

Asked by Alastair Growcott

For the last few weeks I have been getting "404 Not found" when Docker is installing packages when building an image.

On several completely different machines, nslookup on security.ubuntu.com gives:

Server: 37.235.1.174
Address: 37.235.1.174#53

Non-authoritative answer:
Name: security.ubuntu.com
Address: 91.189.88.161
Name: security.ubuntu.com
Address: 91.189.88.162
Name: security.ubuntu.com
Address: 91.189.88.149
Name: security.ubuntu.com
Address: 91.189.91.26
Name: security.ubuntu.com
Address: 91.189.88.152
Name: security.ubuntu.com
Address: 91.189.91.23

I have flushed the cache on my DNS server, which feeds of Google's, and also tried directly accessing other DNS server's as above. All give exactly the same results.

If I use a web browser to browse to http://91.189.91.26/ubuntu then I get a sensible page. In fact using any of the 91.189.91.x addresses works.

If I use a web browser to browse to http://91.189.88.152/ubuntu then I get a 404 Not Found result. In fact I get the same result with any of the 91.189.88.x addresses.

I have not tried updating any of our Ubuntu servers. I am a little bit scared to.

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Solved by:
actionparsnip
Solved:
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said :
#1

What is the full output of:

sudo apt-get update

Thanks

Revision history for this message
Alastair Growcott (agrowcott) said :
#2

On the host machine, the output is:

Hit:1 http://gb.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:4 http://download.virtualbox.org/virtualbox/debian xenial InRelease
Ign:5 http://download.webmin.com/download/repository sarge InRelease
Get:6 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:7 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [705 kB]
Hit:8 https://download.docker.com/linux/ubuntu xenial InRelease
Hit:9 http://download.webmin.com/download/repository sarge Release
Get:11 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [657 kB]
Get:12 http://gb.archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [7,600 B]
Get:13 http://gb.archive.ubuntu.com/ubuntu xenial-updates/restricted i386 Packages [7,548 B]
Get:14 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [577 kB]
Get:15 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages [536 kB]
Get:16 http://gb.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [16.2 kB]
Get:17 http://gb.archive.ubuntu.com/ubuntu xenial-updates/multiverse i386 Packages [15.3 kB]
Fetched 2,827 kB in 10s (280 kB/s)
Reading package lists... Done

From within an old docker image, the output is:

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [57.4 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [547 kB]
Get:7 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [237 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [911 kB]
Get:9 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [249 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3492 B]
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [741 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.5 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [5162 B]
Get:16 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7146 B]
Fetched 3108 kB in 10s (286 kB/s)
Reading package lists... Done

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#3

Can't see an issue. The repository is being hit and the package lists updated....

Revision history for this message
Alastair Growcott (agrowcott) said :
#4

It isn't "apt-get update" that reports the error, it is apt-get install, and probably "apt-get upgrade" as well. Here is the output when trying to build the image:

Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  binutils bzip2 cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc gcc-5
  libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl
  libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0 libcilkrts5 libdpkg-perl
  libfakeroot libfile-fcntllock-perl libgcc-5-dev libgomp1 libisl15 libitm1
  liblsan0 libmpc3 libmpfr4 libmpx0 libquadmath0 libstdc++-5-dev libtsan0
  libubsan0 linux-libc-dev make manpages manpages-dev xz-utils
Suggested packages:
  binutils-doc bzip2-doc cpp-doc gcc-5-locales debian-keyring g++-multilib
  g++-5-multilib gcc-5-doc libstdc++6-5-dbg gcc-multilib autoconf automake
  libtool flex bison gdb gcc-doc gcc-5-multilib libgcc1-dbg libgomp1-dbg
  libitm1-dbg libatomic1-dbg libasan2-dbg liblsan0-dbg libtsan0-dbg
  libubsan0-dbg libcilkrts5-dbg libmpx0-dbg libquadmath0-dbg glibc-doc
  libstdc++-5-doc make-doc man-browser
The following NEW packages will be installed:
  binutils build-essential bzip2 cpp cpp-5 dpkg-dev fakeroot g++ g++-5 gcc
  gcc-5 libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libasan2 libatomic1 libc-dev-bin libc6-dev libcc1-0
  libcilkrts5 libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-5-dev
  libgomp1 libisl15 libitm1 liblsan0 libmpc3 libmpfr4 libmpx0 libquadmath0
  libstdc++-5-dev libtsan0 libubsan0 linux-libc-dev make manpages manpages-dev
  xz-utils
0 upgraded, 40 newly installed, 0 to remove and 6 not upgraded.
Need to get 40.0 MB of archives.
After this operation, 146 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpfr4 amd64 3.1.4-1 [191 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libmpc3 amd64 1.0.3-1 [39.7 kB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/main amd64 bzip2 amd64 1.0.6-8 [32.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 manpages all 4.04-2 [1087 kB]
Err:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 binutils amd64 2.26.1-1ubuntu1~16.04.5
  404 Not Found [IP: 91.189.88.162 80]
Err:6 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libc-dev-bin amd64 2.23-0ubuntu9
  404 Not Found [IP: 91.189.88.162 80]
Ign:7 http://security.ubuntu.com/ubuntu xenial-security/main amd64 linux-libc-dev amd64 4.4.0-101.124
Ign:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 libc6-dev amd64 2.23-0ubuntu9
Get:9 http://archive.ubuntu.com/ubuntu xenial/main amd64 libisl15 amd64 0.16.1-1 [524 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 cpp-5 amd64 5.4.0-6ubuntu1~16.04.5 [7786 kB]

You can see that the request on "http://archive.ubuntu.com/ubuntu" appears to succeed 4 times before failing on the fifth time. I haven't seen this issue on this address before, it usually only happens on "http://security.ubuntu.com/ubuntu". I also see that the same IP address is being reported for both archive.ubuntu.com and security.ubuntu.com.

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#5

It's not DNS because the name resolves OK.

404 means the file isn't found when the system tried to download it. Sounds like the package lists and the actual files are out of sync.

Have you tried using a different package server?

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#6

Could also try:

cd $HOME
wget https://dl.dropbox.com/u/8850924/fixpackage
chmod +x ./fixpackage
sudo ./fixpackage

This wipes all package knowledge from the system and downloads it fresh. May help.

Revision history for this message
Alastair Growcott (agrowcott) said :
#7

DNS reports 6 IP addresses for security.ubuntu.com:

91.189.88.161
91.189.88.162
91.189.88.149
91.189.91.26
91.189.88.152
91.189.91.23

Browsing http://91.189.88.161/ubuntu does not work.
Browsing http://91.189.88.162/ubuntu does not work.
Browsing http://91.189.88.149/ubuntu does not work.
Browsing http://91.189.91.26/ubuntu works.
Browsing http://91.189.88.152/ubuntu does not work.
Browsing http://91.189.91.23/ubuntu works.

Here we are only checking the /ubuntu path, not any additional path that might be added for a package. And all these IP addresses resolved from the same internet address, so it isn't a problem with the package lists. Even if the package lists are 100% correct we will get this problem, and if the package lists are incorrect, the failure will occur at the stage where the process checks for /ubuntu, before it tries to get an actual package.

If an FQDN resolves to multiple IP addresses, all the machines at those IP addresses should behave identically the same - as they all have the same FQDN. Otherwise it would be like typing in http://www.google.com and ending up looking at the search page for DuckDuckGo.

As an extra investigative step I did nslookup for archive.security.com:

Addresses:
          2001:67c:1560:8001::14
          2001:67c:1360:8001::17
          2001:67c:1560:8001::11
          2001:67c:1360:8001::21
          91.189.88.152
          91.189.88.149
          91.189.88.161
          91.189.88.162

Note that here we get the same 4 spurious addresses appearing. Are these machines archive.ubuntu.com or security.ubuntu.com? either way they do no work. As can be seen from the apt-get output earlier, this is also causing problems.

The solution to this problem is for someone from Canonical who has access to those servers to go and work out which are serving package content and which aren't and either make the ones that aren't serve the content out or to update the DNS address to only point to those that are. If there was a way to directly report problems I would have used it, but sadly this is the only port of call.

Revision history for this message
Best actionparsnip (andrew-woodhead666) said :
#8

I suggest you report a bug

Revision history for this message
actionparsnip (andrew-woodhead666) said :
#9

If you get a system that can update and compare IPs then you can update /etc/hosts so it resolves correctly. Does it then work OK? Add this result to your bug report too

Revision history for this message
Alastair Growcott (agrowcott) said :
#10

Thanks actionparsnip, that solved my question.