OpenStack Compute (nova)

VlanManager networking mode, first private bridge interface, and iptables NAT rules

Asked by xsited on 2013-01-16

I have a Folsom All in One Node and a Compute Node installation using VlanManager networking mode. I need some assistance in finding the source of a (S)NAT rule that is being added that seems erroneous for the my current network topology and causes outbound forwarding issues.

How can iptables corrections be made so they don't have to be made manually after each VM is started?

A brief background on how I arrived here and some more details are provided below.

When I first used nova-manage network to create a range of networks things did not quite work out as planned right away and when I attempted to launch my first VM I ran into the following issue.

https://bugs.launchpad.net/nova/+bug/1076309

I am sure alot of these troubles might have been created by what I did along the way and could have been avoided. But once I deleted all the networks and created new networks VM started correctly again. I wrote a couple of handy script I have included below to create and delete the networks. When the NIC first came up it was assigned a 10.0.0.0/8 adderss, which appears to be the default tenant address and iptables rules accordingly. A restart of the nova processes failed to make the correction, but a reboot did get the private network bridge configured with the correct IP address and subnet mask.

These are rules I need to do execute manually to get the outbound SNAT routes to work properly.

iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT
iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.101.0.0/16 -m conntrack ! --ctstate DNAT -j ACCEPT
iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.1.1.216/32 -j ACCEPT
iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.1.1.216/32 -j ACCEPT
iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

I have flipped through a little bit where this magic might be occurring in nova/network/linux_net.py -- how can these correction be made so I don't have to perform the manually?

Here are some other observation and question I collected along the way. I have not found any specific bugs already reported, but i am happy to file them if warranted.

1. network-manage create --dns1 and --dns2 did not seem to add anything to the table.

2. if the network table entries where added they did not seem to be used any way by dnsmasq as dhcp lease attributes

3. if the dhcp_start was populated dnsmasq was properly i.e --dhcp-range=set:'Network101',10.101.0.11,static,120s but the first address assigned to VMs have been 103 to 109 so far. My original intent was clear about eight IP addresses for eight possible compute nodes.

4. which left me with the question of how nova-network issues IP addresses for the private network bridge/ vm gateway on each compute node when the first VM is spun up on a compute node?

Thanks for reading,

-t

----------------8<----Cut------------------------
#!/bin/sh

nova-manage network create --label=Network$1 --multi_host=T --fixed_range_v4=10.$1.0.0/16 --num_networks=1 --bridge=br$1 --bridge_interface=eth3 --network_size=65536 --vlan=$1 --dns1=8.8.8.8 --dns2=8.8.4.4
nova-manage network list
echo "update networks set dns1='8.8.8.8' where label='Network$1' \G" | mysql -unova -pnova nova
echo "update networks set dns2='8.8.4.4' where label='Network$1' \G" | mysql -unova -pnova nova
echo "update networks set dhcp_start='10.$1.0.11' where label='Network$1' \G" | mysql -unova -pnova nova
echo "update networks set vpn_public_port='1$1' where label='Network$1' \G" | mysql -unova -pnova nova
echo "select * from networks where label='Network$1' \G" | mysql -unova -pnova nova

nova-manage network list
nova-manage db sync

----------------8<----Cut------------------------
#!/bin/sh

if [ "$1" == "" ]; then
echo "Invalid VLAN ..."
exit
fi

echo "select * from networks where label='Network$1' \G" | mysql -unova -pnova nova
PROJECT=`echo "select project_id from networks where label='Network$1'" | mysql -unova -pnova nova -BN`
if [ "$PROJECT" != "NULL" ]; then
echo "nova-manage project scrub $PROJECT"
nova-manage project scrub $PROJECT
fi

UUID=`echo "select uuid from networks where label='Network$1'" | mysql -unova -pnova nova -BN`
if [ "$UUID" != "NULL" ]; then
   echo "nova-manage network delete --uuid=$UUID"
   nova-manage network delete --uuid=$UUID
   echo "Network deleted."
fi

nova-manage db sync
nova-manage network list

----------------8<----Cut------------------------

# nova-manage network list
id IPv4 IPv6 start address DNS1 DNS2 VlanID project uuid
2013-01-15 17:18:22 DEBUG nova.utils [req-325123e5-1f08-46c2-976f-292f1b430e0d None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=55956) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494
21 10.101.0.0/16 None 10.101.0.11 8.8.8.8 8.8.4.4 101 b9e0dcc82b9e4837bbefba9bdb9c4f4e f6f7517a-eb0a-4476-b156-007960b42ef8
22 10.102.0.0/16 None 10.102.0.11 8.8.8.8 8.8.4.4 102 None 93b2c6e5-d7fd-4db5-af3d-60fcb7314693
23 10.103.0.0/16 None 10.103.0.11 8.8.8.8 8.8.4.4 103 None 283a9752-4729-41c1-a0ba-12211b6f25e2
24 10.104.0.0/16 None 10.104.0.11 8.8.8.8 8.8.4.4 104 None 7a2e9cb1-a399-4d00-81bb-b43791dd518f
25 10.105.0.0/16 None 10.105.0.11 8.8.8.8 8.8.4.4 105 None f6202053-fed4-4b2d-aa59-89ef1eb015cb
26 10.106.0.0/16 None 10.106.0.11 8.8.8.8 8.8.4.4 106 None c285335d-a8f3-44ec-a94a-043752a35a4d
27 10.107.0.0/16 None 10.107.0.11 8.8.8.8 8.8.4.4 107 None e2f8966b-0d68-4189-b920-7f751d9cec0e

# public 10.4.0.0/21
# ifconfig br-ex
br-ex Link encap:Ethernet HWaddr 00:1e:67:4f:bb:26
          inet addr:10.4.0.216 Bcast:10.4.7.255 Mask:255.255.248.0
          inet6 addr: fe80::21e:67ff:fe4f:bb26/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:16070861 errors:0 dropped:30 overruns:0 frame:0
          TX packets:6551736 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:35832866517 (35.8 GB) TX bytes:33707775912 (33.7 GB)

# private 10.101.0.0/16
# ifconfig br101
br101 Link encap:Ethernet HWaddr fa:16:3e:33:2a:2f
          inet addr:10.101.0.4 Bcast:10.101.255.255 Mask:255.255.0.0
          inet6 addr: fe80::446b:6bff:fe06:3e9c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1352789 errors:0 dropped:1 overruns:0 frame:0
          TX packets:1459590 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:117111465 (117.1 MB) TX bytes:2793206776 (2.7 GB)

# vlan host interface
# ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:1b:21:5c:81:03
          inet6 addr: fe80::21b:21ff:fe5c:8103/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3312 errors:0 dropped:78 overruns:0 frame:0
          TX packets:767 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:424558 (424.5 KB) TX bytes:106218 (106.2 KB)

# ifconfig vlan101
vlan101 Link encap:Ethernet HWaddr fa:16:3e:33:2a:2f
          inet6 addr: fe80::f816:3eff:fe33:2a2f/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:743 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:102826 (102.8 KB)

# Openstack management interface
eth1 Link encap:Ethernet HWaddr 00:1e:67:4f:bb:27
          inet addr:10.1.1.216 Bcast:10.1.1.255 Mask:255.255.255.0
          inet6 addr: fe80::21e:67ff:fe4f:bb27/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:381196 errors:0 dropped:101 overruns:0 frame:0
          TX packets:12380 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:108545104 (108.5 MB) TX bytes:1782787 (1.7 MB)
          Memory:d0900000-d0920000

# brctl show br101

bridge name bridge id STP enabled interfaces
br101 8000.fa163e332a2f no vlan101
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
                                                        vnet4
                                                        vnet5

# nova.conf

# NETWORK
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
network_manager=nova.network.manager.VlanManager
public_interface=br_ex
vlan_interface=eth3
routing_source_ip=10.4.0.216
my_ip=10.1.1.216
# Change my_ip to match each host
# network_manager=nova.network.manager.FlatManager
# network_manager=nova.network.manager.FlatDHCPManager
# flat_network_bridge=br100
# flat_interface=br_ex

# what iptables looks like after launching a VM
# iptables -t nat --line-numbers -n -L

Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
4 MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
5 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
6 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
7 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-api-OUTPUT (1 references)
num target prot opt source destination

Chain nova-api-POSTROUTING (1 references)
num target prot opt source destination

Chain nova-api-PREROUTING (1 references)
num target prot opt source destination

Chain nova-api-float-snat (1 references)
num target prot opt source destination

Chain nova-api-snat (1 references)
num target prot opt source destination
1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-compute-OUTPUT (1 references)
num target prot opt source destination

Chain nova-compute-POSTROUTING (1 references)
num target prot opt source destination

Chain nova-compute-PREROUTING (1 references)
num target prot opt source destination

Chain nova-compute-float-snat (1 references)
num target prot opt source destination

Chain nova-compute-snat (1 references)
num target prot opt source destination
1 nova-compute-float-snat all -- 0.0.0.0/0 0.0.0.0/0

Chain nova-network-OUTPUT (1 references)
num target prot opt source destination
1 DNAT udp -- 0.0.0.0/0 10.1.1.216 udp dpt:1101 to:10.101.0.2:1194

Chain nova-network-POSTROUTING (1 references)
num target prot opt source destination
1 ACCEPT all -- 10.0.0.0/8 10.1.1.216
2 ACCEPT all -- 10.0.0.0/8 10.0.0.0/8 ! ctstate DNAT

Chain nova-network-PREROUTING (1 references)
num target prot opt source destination
1 DNAT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 to:10.1.1.216:8775
2 DNAT udp -- 0.0.0.0/0 10.1.1.216 udp dpt:1101 to:10.101.0.2:1194

Chain nova-network-float-snat (1 references)
num target prot opt source destination

Chain nova-network-snat (1 references)
num target prot opt source destination
1 nova-network-float-snat all -- 0.0.0.0/0 0.0.0.0/0
2 SNAT all -- 10.0.0.0/8 0.0.0.0/0 to:10.4.0.216

Chain nova-postrouting-bottom (1 references)
num target prot opt source destination
1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-snat all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0

# manual correction made to achieve desired functionality

Question information

Language:: English Edit question

Status:: Solved

For:: OpenStack Compute (nova) Edit question

Assignee:: No assignee Edit question

Solved by:: xsited

Solved:: 2013-01-27

Last query:: 2013-01-27

Last reply:: 2013-01-18

Link existing bug

Revision history for this message

Vish Ishaya (vishvananda) said on 2013-01-16:

On Jan 16, 2013, at 3:05 PM, xsited <email address hidden> wrote:

> https://bugs.launchpad.net/nova/+bug/1076309
>
> I am sure alot of these troubles might have been created by what I did along the way and could have been avoided. But once I deleted all the networks and created new networks VM started correctly again. I wrote a couple of handy script I have included below to create and delete the networks. When the NIC first came up it was assigned a 10.0.0.0/8 adderss, which appears to be the default tenant address and iptables rules accordingly. A restart of the nova processes failed to make the correction, but a reboot did get the private network bridge configured with the correct IP address and subnet mask.
>
> These are rules I need to do execute manually to get the outbound SNAT routes to work properly.
>
> iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.0.0.0/8 -m conntrack ! --ctstate DNAT -j ACCEPT
> iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.101.0.0/16 -m conntrack ! --ctstate DNAT -j ACCEPT
> iptables -t nat -D nova-network-POSTROUTING -s 10.0.0.0/8 -d 10.1.1.216/32 -j ACCEPT
> iptables -t nat -A nova-network-POSTROUTING -s 10.101.0.0/16 -d 10.1.1.216/32 -j ACCEPT
> iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216
> iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216
> iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

Simply set:

fixed_range=10.101.0.0/16

in your nova.conf and restart nova-network.

The last two rules appear to duplicate each other so I wouldn't be suprised if you don't need the last one. I'm assuming br_ex contains the default route for the host, if not then you might need to set:

public_interface=xxx

where xxx is the interface that the contains the default route (or a bridge that contains that interface).

Vish

Revision history for this message

xsited (xsited) said on 2013-01-17:

Vish,

Thanks for your reply. Yes, the public_interface is a bridge with an enslaved interface that provides the default route.

I added the fixed_range variable and restested both after a all nova services restart and a full reboot.

All appears well. I have include the iptables as they appear without any manual modifications. As I went to each of the VMs I could ping the VMs default route (10.101.0.4/16) but not any external routes and I was using 8.8.8.8 as a target. It was not until I manually added this command

iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216

could all the VMs go anywhere they wanted to. Yup, I understand that it appears to be similar to rule 2 of the chain nova-network-snat that already exists.

wrt fixed_range, how will the system and the iptables ruleset be affected by firing up the next tenants network, say 10.102.0.0/16. While the solution you suggested largely cleans up the issue I was experiencing with this network, what additional configuration items, if any, will I need to consider as I am adding VLAN/networks for tenants?

Thanks,

-t

# iptables -t nat --line-numbers -n -L
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0

Chain INPUT (policy ACCEPT)
num target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
2 MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
3 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
4 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
5 nova-network-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
6 nova-api-POSTROUTING all -- 0.0.0.0/0 0.0.0.0/0
7 nova-postrouting-bottom all -- 0.0.0.0/0 0.0.0.0/0