network unreachable for ppa.launchpad.net

Launchpad maintainers have asked those reporting similar issues on IRC to re-report them to https://answers.launchpad.net/launchpad/+addquestion (which they monitor). I am converting this bug report (which is clearly not a software bug but a network issue affecting the Launchpad.net instance of the Launchpad software) into such a question - so you can just continue reporting here.

Hi Milos,

I won't be able to handle this issue for you, but I'm loosely following discussions on IRC (libera.chat's #launchpad primarily) where others are reporting network connectivity issues/outages while accessing ppa.launchpad.net from Amazon AWS's us-east-1 region (which also seems to host some Bitbucket CI systems).

Those reporting similar issues have been asked to post the output of

  mtr --report -T -P 80 ppa.launchpad.net

running on an affected system. While you already provided a (normal) traceroute, the "mtr" output would provide some additional information which could support them in diagnosing this issue - if it is still ongoing.

Thanks!

Tom

+ mtr --report -T -P 80 ppa.launchpad.net
Start: 2021-08-16T02:35:57+0000
HOST: 26001bd5-4d62-4dd6-a4f5-cdd Loss% Snt Last Avg Best Wrst StDev
  1.|-- ip-10-203-199-145.ec2.int 0.0% 10 0.3 0.3 0.3 0.5 0.1
  2.|-- ip-10-203-141-118.ec2.int 0.0% 10 0.5 0.5 0.3 0.7 0.1
  3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  6.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  7.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  8.|-- 100.65.14.81 40.0% 10 1.2 1.6 1.1 3.7 1.0
  9.|-- 52.93.29.59 10.0% 10 12.7 3.1 1.3 12.7 3.6
 10.|-- 52.93.29.39 10.0% 10 1.4 3.5 1.4 17.2 5.1
 11.|-- ix-ae-14-0.tcore3.aeq-ash 50.0% 10 1.8 2.4 1.5 5.0 1.4
 12.|-- ix-ae-14-0.tcore3.aeq-ash 60.0% 10 2.1 2.1 1.7 2.5 0.3
 13.|-- if-be-55-2.ecore1.aeq-ash 10.0% 10 2.0 2.7 2.0 4.9 0.9
 14.|-- ae-1-3110.edge3.London1.L 0.0% 10 2.6 39.4 1.6 77.4 39.1
 15.|-- brd1-level3.ifl2.lateroom 0.0% 10 2.3 53.3 1.6 76.3 35.4
 16.|-- brd1-level3.ifl2.lateroom 0.0% 10 3110. 479.2 73.7 3110. 977.5
 17.|-- ganondorf.canonical.com 10.0% 10 3105. 403.1 2.0 3105. 1013.6

commented on the irc channel too.

+ mtr --report -T -P 80 ppa.launchpad.net
Start: 2021-08-17T01:45:24+0000
HOST: 19e936e1-5184-4250-bc82-3cc Loss% Snt Last Avg Best Wrst StDev
  1.|-- ip-10-203-203-46.ec2.inte 0.0% 10 0.3 0.4 0.3 0.4 0.1
  2.|-- ip-10-203-142-66.ec2.inte 0.0% 10 0.4 0.7 0.4 1.7 0.4
  3.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  4.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  5.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0
  6.|-- 240.0.60.59 40.0% 10 1.4 1.5 1.4 1.9 0.2
  7.|-- 240.0.28.30 40.0% 10 1.5 1.5 1.4 1.7 0.1
  8.|-- 240.0.28.21 20.0% 10 1.4 2.4 1.4 7.8 2.2
  9.|-- 52.93.28.241 10.0% 10 1.7 2.4 1.5 6.4 1.6
 10.|-- 100.100.36.50 30.0% 10 1.8 2.7 1.6 7.7 2.2
 11.|-- ix-ae-14-0.tcore3.aeq-ash 40.0% 10 2.0 1.9 1.6 2.2 0.3
 12.|-- 241.0.4.128 0.0% 10 1.6 1.9 1.3 3.9 0.8
 13.|-- 63.243.136.13 0.0% 10 1.6 2.8 1.5 7.4 1.7
 14.|-- 242.0.146.33 0.0% 10 2.9 20.6 1.7 76.5 30.4
 15.|-- 52.93.28.203 0.0% 10 2.1 16.8 1.4 78.7 31.8
 16.|-- bond0.erlking.canonical.c 0.0% 10 1.6 66.9 1.6 75.2 22.9
 17.|-- ganondorf.canonical.com 0.0% 10 75.7 67.6 1.8 75.9 23.1

I've not really got any word back on the IRC channel. Is there some other communication channel? or a list of questions/issues from other users?

milos, have you tried lowering the Maximum Transmission Unit (MTU)? It could be a path MTU issue. If the originating interface XXXX currently has an MTU of 1500

# ip link show dev XXXX

To change it

# ip link set mtu 1420 dev XXXX

1420 is chosen as the 'best' compromise in the face of certain types of tunnel and potentially carrying IPv6 traffic. You can of course adjust that as you think fit. The idea is to ensure the packets aren't being dropped by a router on the link because they either:

1) Are too big for the next hop, are IPv4 and have the Do Not Fragment (DF) bit set

2) Are too big for the next hop, are IPv6 (which doesn't support fragmentation)

If this proves to be the case you can apply a netfilter/iptables rule on the egress interface of your network gateway (not necessarily the same host as the origin of the packets) to clamp the TCP Maximum Segment Size (MSS) to the path MTU for all packets and all routes.

On an egress host (gateway) that is not the origin of the packets (or is the origin and is routing (maybe for VMs or containers)) . Use the CMD 'iptables" for IPv4 and "ip6tables" for IPv6:

for CMD in iptables ip6tables; do

  $CMD -t mangle --insert FORWARD -i XXXX -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "reduce MTU automatically" -j TCPMSS --clamp-mss-to-pmtu
  $CMD -t mangle --insert FORWARD -o XXXX -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "reduce MTU automatically" -j TCPMSS --clamp-mss-to-pmtu

done

On an origin host that is also the egress and packets originate locally:

for CMD in iptables ip6tables; do

  $CMD -t mangle --insert INPUT -i XXXX -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "reduce MTU automatically" -j TCPMSS --clamp-mss-to-pmtu
$CMD -t mangle --insert OUTPUT -o XXXX -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "reduce MTU automatically" -j TCPMSS --clamp-mss-to-pmtu

done

Hi, we are investigating what seems to be the same issue ( 216.182.230.247 is in aws us-east-1)

in https://rt.ubuntu.com/Ticket/Display.html?id=36786