ssh access to vm from public interface hangs using vlan, nova-network and folsom

Asked by arturo lorenzo

I have two nodes: controller (all folsom services but nova-network and nova-compute) and compute1 (nova-network and nova-compute). I can ssh to the VM OK from compute1, I can ping public IP from outside OK but I cannot ssh from outside. ssh hangs when running from outside:
Replaced real IP with x.x.x.x
===========================================================
ssh -vvv -i user_onek.pem x.x.x.x -l ubuntu
OpenSSH_5.1p1, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to x.x.x.x [x.x.x.x] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Not a RSA1 key file user_onek.pem.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file user_onek.pem type -1
Hangs ...
===========================================================
Using same key and running from compute1 I can ssh OK:
ssh -vvv -i user_onek.pem x.x.x.x -l ubuntu
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to x.x.x.x [x.x.x.x] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "user_onek.pem" as a RSA1 public key
debug1: identity file user_onek.pem type -1
debug1: identity file user_onek.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "x.x.x.x" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: <email address hidden>,<email address hidden>,<email address hidden>,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
more ....
debug2: shell request accepted on channel 0
Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-35-virtual x86_64)
OK
and the VM can access the internet OK
===========================================================
I have two NICs eth0 for management network and eth1 for public access.
Here is my /etc/nova/nova.conf from compute1
[DEFAULT]
# LOGS/STATE
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/run/lock/nova
verbose=True

# RABBITMQ
rabbit_host=100.10.10.51

# SCHEDULER
scheduler_driver=nova.scheduler.multi.MultiScheduler
compute_scheduler_driver=nova.scheduler.filter_scheduler.FilterScheduler

# APIS
s3_host=100.10.10.51
ec2_host=100.10.10.51
ec2_dmz_host=100.10.10.51
ec2_url=http://100.10.10.51:8773/services/Cloud
rabbit_host=100.10.10.51
cc_host=100.10.10.51
dmz_cidr=169.254.169.254/32
metadata_host=100.10.10.52
metadata_listen=0.0.0.0
keystone_ec2_url=http://100.10.10.51:5000/v2.0/ec2tokens
nova_url=http://100.10.10.51:8774/v1.1/
enabled_apis=ec2,osapi_compute,metadata

# DATABASE
sql_connection=mysql://novaUser:novaPass@100.10.10.51/nova

# Auth
use_deprecated_auth=false
auth_strategy=keystone

# Imaging service GLANCE
glance_api_servers=100.10.10.51:9292
image_service=nova.image.glance.GlanceImageService

# Vnc configuration
novnc_enabled=true
novncproxy_base_url=http://192.168.100.52:6080/vnc_auto.html
novncproxy_port=6080
vncserver_proxyclient_address=100.10.10.52
vncserver_listen=100.10.10.52

# Network settings
network_manager=nova.network.manager.VlanManager
vlan_interface=eth0
#vlan_start=100
fixed_range=172.16.169.0/24
network_size=256
public_interface=eth1
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
connection_type=libvirt
multi_host=True
dhcpbridge=/usr/bin/nova-dhcpbridge
force_dhcp_release=True
dhcpbridge_flagfile=/etc/nova/nova.conf
auto_assign_floating_ip=True
#routing_source_ip=192.168.100.52

# Compute #
compute_driver=libvirt.LibvirtDriver
libvirt_type=kvm
libvirt_use_virtio_for_bridges=True
start_guests_on_host_boot=True
resume_guests_state_on_host_boot=True
api_paste_config=/etc/nova/api-paste.ini
allow_admin_api=True
use_deprecated_auth=False
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf

# Cinder #
volume_api_class=nova.volume.cinder.API
osapi_volume_listen_port=5900

# /etc/nova/nova.conf Done
===========================================
Here is my ifconfig -a from compute1:
br169 Link encap:Ethernet HWaddr 00:1e:67:29:03:23
          inet addr:10.10.169.4 Bcast:10.10.169.255 Mask:255.255.255.0
          inet6 addr: fe80::7004:a7ff:fedb:620a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:13317 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15326 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7392382 (7.3 MB) TX bytes:5922013 (5.9 MB)

eth0 Link encap:Ethernet HWaddr 00:1e:67:29:03:23
          inet6 addr: fe80::21e:67ff:fe29:323/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:26041 errors:0 dropped:101 overruns:0 frame:0
          TX packets:14101 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12170467 (12.1 MB) TX bytes:5809215 (5.8 MB)
          Memory:d0920000-d0940000

eth1 Link encap:Ethernet HWaddr 00:1e:67:29:03:24
          inet addr:192.168.100.52 Bcast:192.168.100.255 Mask:255.255.255.0
          inet6 addr: fe80::21e:67ff:fe29:324/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:28660 errors:0 dropped:0 overruns:0 frame:0
          TX packets:47301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5393133 (5.3 MB) TX bytes:13295718 (13.2 MB)
          Memory:d0900000-d0920000

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:724 errors:0 dropped:0 overruns:0 frame:0
          TX packets:724 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:61103 (61.1 KB) TX bytes:61103 (61.1 KB)

vlan169 Link encap:Ethernet HWaddr 00:1e:67:29:03:23
          inet6 addr: fe80::21e:67ff:fe29:323/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:11667 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13659 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7268580 (7.2 MB) TX bytes:5780283 (5.7 MB)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:25:8d:34
          inet6 addr: fe80::fc16:3eff:fe25:8d34/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1694 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:150686 (150.6 KB) TX bytes:148552 (148.5 KB)
================================================
and here is my ip addr show from compute1 (public ip replaced with x.x.x.x)
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 169.254.169.254/32 scope link lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1e:67:29:03:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21e:67ff:fe29:323/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1e:67:29:03:24 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.52/24 brd 192.168.100.255 scope global eth1
    inet x.x.x.x/32 scope global eth1
    inet6 fe80::21e:67ff:fe29:324/64 scope link
       valid_lft forever preferred_lft forever
5: br169: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:1e:67:29:03:23 brd ff:ff:ff:ff:ff:ff
    inet 10.10.169.4/24 brd 10.10.169.255 scope global br169
    inet 100.10.10.52/24 brd 100.10.10.255 scope global br169
    inet6 fe80::7004:a7ff:fedb:620a/64 scope link
       valid_lft forever preferred_lft forever
6: vlan169@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br169 state UP
    link/ether 00:1e:67:29:03:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::21e:67ff:fe29:323/64 scope link
       valid_lft forever preferred_lft forever
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br169 state UNKNOWN qlen 500
    link/ether fe:16:3e:25:8d:34 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe25:8d34/64 scope link
       valid_lft forever preferred_lft forever
=========================================
and here is most of the iptables output:
Chain nova-network-OUTPUT (1 references)
 pkts bytes target prot opt in out source destination
    0 0 DNAT udp -- * * 0.0.0.0/0 192.168.100.52 udp dpt:1000 to:10.10.169.2:1194
    2 120 DNAT all -- * * 0.0.0.0/0 13.141.237.129 to:10.10.169.3

Chain nova-network-POSTROUTING (1 references)
 pkts bytes target prot opt in out source destination
    0 0 ACCEPT all -- * * 172.16.169.0/24 100.10.10.52
    0 0 ACCEPT all -- * * 172.16.169.0/24 169.254.169.254
    0 0 ACCEPT all -- * * 172.16.169.0/24 172.16.169.0/24 ! ctstate DNAT

Chain nova-network-PREROUTING (1 references)
 pkts bytes target prot opt in out source destination
   24 1440 DNAT tcp -- * * 0.0.0.0/0 169.254.169.254 tcp dpt:80 to:100.10.10.52:8775
    0 0 DNAT udp -- * * 0.0.0.0/0 192.168.100.52 udp dpt:1000 to:10.10.169.2:1194
 1022 45288 DNAT all -- * * 0.0.0.0/0 x.x.x.x to:10.10.169.3

Chain nova-network-float-snat (1 references)
 pkts bytes target prot opt in out source destination
    2 136 SNAT all -- * eth1 10.10.169.3 0.0.0.0/0 to:x.x.x.x

Chain nova-network-snat (1 references)
 pkts bytes target prot opt in out source destination
 1527 79816 nova-network-float-snat all -- * * 0.0.0.0/0 0.0.0.0/0
    0 0 SNAT all -- * eth1 172.16.169.0/24 0.0.0.0/0 to:192.168.100.52

Chain nova-postrouting-bottom (1 references)
 pkts bytes target prot opt in out source destination
 1518 79160 nova-compute-snat all -- * * 0.0.0.0/0 0.0.0.0/0
 1527 79816 nova-network-snat all -- * * 0.0.0.0/0 0.0.0.0/0
 1850 101K nova-api-metadat-snat all -- * * 0.0.0.0/0 0.0.0.0/0
==================================================
everytime I run the ssh command I can see the packet # increase on the nova-network-PREROUTING chain but I don't see the nova-network-float-snat doing the same.
any hint?
Thanks!

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
arturo lorenzo
Solved:
Last query:
Last reply:
Revision history for this message
arturo lorenzo (arturo-lorenzo) said :
#1

Hi again, my problem is solved. The switches far away in the internal network did not have VLAN enabled. I had to re-architect the network and now everything is working OK.