OpenStack Compute (nova)

nova-network not forwarding properly from VM->public

Asked by Chris Meisinger

Hey there,

I currently have a 3 node Openstack configuration using Vlan networking. One host (os-controller) runs Nova-network/Nova-scheduler and Nova-API. The other two hosts run compute and volume.

This setup was working as expected until we had to reboot the controller on Monday. After that happened, the following odd traffic patterns were noted.

Floating IPs bound to instances worked as expected. Fixed IPs would reply from the IP of the controller. i.e., if I opened up an SSH session, I could see the connection from my workstation make it to the VM and I could see the ssh banner response from the VM -> OS controllers vlan interface. However when the OS-Controller replied, my workstation would see the response as coming from the controller and not the IP of the instance (via tcpdump -A -s 65000 and looking for the SSH response banner in the payload).

So the situation now is that when I try to SSH to the IP of a VM, traffic goes to 10.1.100.11 (for example), but responses come back from 10.1.0.76.

I've put what I believe to be the relevant info in the paste at http://paste.openstack.org/show/2263/ .

Thank you for your help. :)

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
Chris Meisinger
Solved:
Last query:
Last reply:
Revision history for this message
Chris Meisinger (cmeisinger) said : 		#1

It should be noted that I removed the dmz_cidr option from the nova.conf as this was working previously without it. With that enabled, I could ssh to the fixed_ip of an instance, however after about 45 seconds, the connection would hang.

The IP I've been using to test in this scenario is 10.1.100.11.

Revision history for this message
Chris Meisinger (cmeisinger) said : 		#2

iptables-save output.

# Generated by iptables-save v1.4.4 on Wed Aug 24 13:03:30 2011
*nat
:PREROUTING ACCEPT [45899:4908527]
:POSTROUTING ACCEPT [523:32419]
:OUTPUT ACCEPT [396:27954]
:nova-network-OUTPUT - [0:0]
:nova-network-POSTROUTING - [0:0]
:nova-network-PREROUTING - [0:0]
:nova-network-floating-snat - [0:0]
:nova-network-snat - [0:0]
:nova-postrouting-bottom - [0:0]
-A PREROUTING -j nova-network-PREROUTING
-A POSTROUTING -j nova-network-POSTROUTING
-A POSTROUTING -j nova-postrouting-bottom
-A OUTPUT -j nova-network-OUTPUT
-A nova-network-OUTPUT -d 10.1.0.89/32 -j DNAT --to-destination 10.1.100.8
-A nova-network-OUTPUT -d 10.1.0.84/32 -j DNAT --to-destination 10.1.100.9
-A nova-network-OUTPUT -d 10.1.0.85/32 -j DNAT --to-destination 10.1.100.10
-A nova-network-POSTROUTING -s 10.1.100.0/22 -d 10.128.0.0/24 -j ACCEPT
-A nova-network-POSTROUTING -s 10.1.100.0/22 -d 10.1.100.0/22 -j ACCEPT
-A nova-network-PREROUTING -d 10.1.0.76/32 -p udp -m udp --dport 1000 -j DNAT --to-destination 10.1.100.2:1194
-A nova-network-PREROUTING -d 10.1.0.76/32 -p udp -m udp --dport 1000 -j DNAT --to-destination 10.1.101.2:1194
-A nova-network-PREROUTING -d 10.1.0.89/32 -j DNAT --to-destination 10.1.100.8
-A nova-network-PREROUTING -d 10.1.0.84/32 -j DNAT --to-destination 10.1.100.9
-A nova-network-PREROUTING -d 10.1.0.85/32 -j DNAT --to-destination 10.1.100.10
-A nova-network-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.1.0.76:8773
-A nova-network-floating-snat -s 10.1.100.8/32 -j SNAT --to-source 10.1.0.89
-A nova-network-floating-snat -s 10.1.100.9/32 -j SNAT --to-source 10.1.0.84
-A nova-network-floating-snat -s 10.1.100.10/32 -j SNAT --to-source 10.1.0.85
-A nova-network-snat -j nova-network-floating-snat
-A nova-network-snat -s 10.1.100.0/22 -j SNAT --to-source 10.1.0.76
-A nova-postrouting-bottom -j nova-network-snat
COMMIT
# Completed on Wed Aug 24 13:03:30 2011
# Generated by iptables-save v1.4.4 on Wed Aug 24 13:03:30 2011
*filter
:INPUT ACCEPT [1983975:182109122]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1432744:140808755]
:nova-filter-top - [0:0]
:nova-network-FORWARD - [0:0]
:nova-network-INPUT - [0:0]
:nova-network-OUTPUT - [0:0]
:nova-network-local - [0:0]
-A INPUT -j nova-network-INPUT
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-network-FORWARD
-A OUTPUT -j nova-filter-top
-A OUTPUT -j nova-network-OUTPUT
-A nova-filter-top -j nova-network-local
-A nova-network-FORWARD -i br_vlan100 -j ACCEPT
-A nova-network-FORWARD -o br_vlan100 -j ACCEPT
-A nova-network-FORWARD -d 10.1.100.2/32 -p udp -m udp --dport 1194 -j ACCEPT
-A nova-network-FORWARD -i br_vlan101 -j ACCEPT
-A nova-network-FORWARD -o br_vlan101 -j ACCEPT
-A nova-network-FORWARD -d 10.1.101.2/32 -p udp -m udp --dport 1194 -j ACCEPT
COMMIT

Revision history for this message
Chris Meisinger (cmeisinger) said : 		#3

Hey there,

This was solved and due to a quirk in our network rather than a quirk with nova-network. We trunk our vlans from the border router down to the switches, and in doing so essentially provided multiple gateways for traffic to/from an instance. Removing the vlans from the BR so they exist only on the switchgear connected to the OpenStack instance have fixed this problem.