VlanManager networking mode, first private bridge interface, and iptables NAT rules
I have a Folsom All in One Node and a Compute Node installation using VlanManager networking mode. I need some assistance in finding the source of a (S)NAT rule that is being added that seems erroneous for the my current network topology and causes outbound forwarding issues.
How can iptables corrections be made so they don't have to be made manually after each VM is started?
--
A brief background on how I arrived here and some more details are provided below.
When I first used nova-manage network to create a range of networks things did not quite work out as planned right away and when I attempted to launch my first VM I ran into the following issue.
https:/
I am sure alot of these troubles might have been created by what I did along the way and could have been avoided. But once I deleted all the networks and created new networks VM started correctly again. I wrote a couple of handy script I have included below to create and delete the networks. When the NIC first came up it was assigned a 10.0.0.0/8 adderss, which appears to be the default tenant address and iptables rules accordingly. A restart of the nova processes failed to make the correction, but a reboot did get the private network bridge configured with the correct IP address and subnet mask.
These are rules I need to do execute manually to get the outbound SNAT routes to work properly.
iptables -t nat -D nova-network-
iptables -t nat -A nova-network-
iptables -t nat -D nova-network-
iptables -t nat -A nova-network-
iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216
I have flipped through a little bit where this magic might be occurring in nova/network/
Here are some other observation and question I collected along the way. I have not found any specific bugs already reported, but i am happy to file them if warranted.
1. network-manage create --dns1 and --dns2 did not seem to add anything to the table.
2. if the network table entries where added they did not seem to be used any way by dnsmasq as dhcp lease attributes
3. if the dhcp_start was populated dnsmasq was properly i.e --dhcp-
4. which left me with the question of how nova-network issues IP addresses for the private network bridge/ vm gateway on each compute node when the first VM is spun up on a compute node?
Thanks for reading,
-t
-------
#!/bin/sh
nova-manage network create --label=Network$1 --multi_host=T --fixed_
nova-manage network list
echo "update networks set dns1='8.8.8.8' where label='Network$1' \G" | mysql -unova -pnova nova
echo "update networks set dns2='8.8.4.4' where label='Network$1' \G" | mysql -unova -pnova nova
echo "update networks set dhcp_start=
echo "update networks set vpn_public_
echo "select * from networks where label='Network$1' \G" | mysql -unova -pnova nova
nova-manage network list
nova-manage db sync
-------
#!/bin/sh
if [ "$1" == "" ]; then
echo "Invalid VLAN ..."
exit
fi
echo "select * from networks where label='Network$1' \G" | mysql -unova -pnova nova
PROJECT=`echo "select project_id from networks where label='Network$1'" | mysql -unova -pnova nova -BN`
if [ "$PROJECT" != "NULL" ]; then
echo "nova-manage project scrub $PROJECT"
nova-manage project scrub $PROJECT
fi
UUID=`echo "select uuid from networks where label='Network$1'" | mysql -unova -pnova nova -BN`
if [ "$UUID" != "NULL" ]; then
echo "nova-manage network delete --uuid=$UUID"
nova-manage network delete --uuid=$UUID
echo "Network deleted."
fi
nova-manage db sync
nova-manage network list
-------
# nova-manage network list
id IPv4 IPv6 start address DNS1 DNS2 VlanID project uuid
2013-01-15 17:18:22 DEBUG nova.utils [req-325123e5-
21 10.101.0.0/16 None 10.101.0.11 8.8.8.8 8.8.4.4 101 b9e0dcc82b9e483
22 10.102.0.0/16 None 10.102.0.11 8.8.8.8 8.8.4.4 102 None 93b2c6e5-
23 10.103.0.0/16 None 10.103.0.11 8.8.8.8 8.8.4.4 103 None 283a9752-
24 10.104.0.0/16 None 10.104.0.11 8.8.8.8 8.8.4.4 104 None 7a2e9cb1-
25 10.105.0.0/16 None 10.105.0.11 8.8.8.8 8.8.4.4 105 None f6202053-
26 10.106.0.0/16 None 10.106.0.11 8.8.8.8 8.8.4.4 106 None c285335d-
27 10.107.0.0/16 None 10.107.0.11 8.8.8.8 8.8.4.4 107 None e2f8966b-
# public 10.4.0.0/21
# ifconfig br-ex
br-ex Link encap:Ethernet HWaddr 00:1e:67:4f:bb:26
inet addr:10.4.0.216 Bcast:10.4.7.255 Mask:255.255.248.0
inet6 addr: fe80::21e:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:16070861 errors:0 dropped:30 overruns:0 frame:0
TX packets:6551736 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:35832866517 (35.8 GB) TX bytes:33707775912 (33.7 GB)
# private 10.101.0.0/16
# ifconfig br101
br101 Link encap:Ethernet HWaddr fa:16:3e:33:2a:2f
inet addr:10.101.0.4 Bcast:10.
inet6 addr: fe80::446b:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1352789 errors:0 dropped:1 overruns:0 frame:0
TX packets:1459590 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:117111465 (117.1 MB) TX bytes:2793206776 (2.7 GB)
# vlan host interface
# ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:1b:21:5c:81:03
inet6 addr: fe80::21b:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3312 errors:0 dropped:78 overruns:0 frame:0
TX packets:767 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:424558 (424.5 KB) TX bytes:106218 (106.2 KB)
# ifconfig vlan101
vlan101 Link encap:Ethernet HWaddr fa:16:3e:33:2a:2f
inet6 addr: fe80::f816:
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:743 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:0 (0.0 B) TX bytes:102826 (102.8 KB)
# Openstack management interface
eth1 Link encap:Ethernet HWaddr 00:1e:67:4f:bb:27
inet addr:10.1.1.216 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::21e:
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:381196 errors:0 dropped:101 overruns:0 frame:0
TX packets:12380 errors:0 dropped:0 overruns:0 carrier:0
RX bytes:108545104 (108.5 MB) TX bytes:1782787 (1.7 MB)
# brctl show br101
bridge name bridge id STP enabled interfaces
br101 8000.fa163e332a2f no vlan101
# nova.conf
# NETWORK
firewall_
network_
public_
vlan_interface=eth3
routing_
my_ip=10.1.1.216
# Change my_ip to match each host
# network_
# network_
# flat_network_
# flat_interface=
# what iptables looks like after launching a VM
# iptables -t nat --line-numbers -n -L
Chain PREROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-
2 nova-network-
3 nova-api-PREROUTING all -- 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
1 nova-compute-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0
Chain POSTROUTING (policy ACCEPT)
num target prot opt source destination
1 nova-compute-
2 nova-network-
3 nova-api-
4 MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
5 MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
6 MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
7 nova-postroutin
Chain nova-api-OUTPUT (1 references)
num target prot opt source destination
Chain nova-api-
num target prot opt source destination
Chain nova-api-PREROUTING (1 references)
num target prot opt source destination
Chain nova-api-float-snat (1 references)
num target prot opt source destination
Chain nova-api-snat (1 references)
num target prot opt source destination
1 nova-api-float-snat all -- 0.0.0.0/0 0.0.0.0/0
Chain nova-compute-OUTPUT (1 references)
num target prot opt source destination
Chain nova-compute-
num target prot opt source destination
Chain nova-compute-
num target prot opt source destination
Chain nova-compute-
num target prot opt source destination
Chain nova-compute-snat (1 references)
num target prot opt source destination
1 nova-compute-
Chain nova-network-OUTPUT (1 references)
num target prot opt source destination
1 DNAT udp -- 0.0.0.0/0 10.1.1.216 udp dpt:1101 to:10.101.0.2:1194
Chain nova-network-
num target prot opt source destination
1 ACCEPT all -- 10.0.0.0/8 10.1.1.216
2 ACCEPT all -- 10.0.0.0/8 10.0.0.0/8 ! ctstate DNAT
Chain nova-network-
num target prot opt source destination
1 DNAT tcp -- 0.0.0.0/0 169.254.169.254 tcp dpt:80 to:10.1.1.216:8775
2 DNAT udp -- 0.0.0.0/0 10.1.1.216 udp dpt:1101 to:10.101.0.2:1194
Chain nova-network-
num target prot opt source destination
Chain nova-network-snat (1 references)
num target prot opt source destination
1 nova-network-
2 SNAT all -- 10.0.0.0/8 0.0.0.0/0 to:10.4.0.216
Chain nova-postroutin
num target prot opt source destination
1 nova-compute-snat all -- 0.0.0.0/0 0.0.0.0/0
2 nova-network-snat all -- 0.0.0.0/0 0.0.0.0/0
3 nova-api-snat all -- 0.0.0.0/0 0.0.0.0/0
# manual correction made to achieve desired functionality
iptables -t nat -D nova-network-
iptables -t nat -A nova-network-
iptables -t nat -D nova-network-
iptables -t nat -A nova-network-
iptables -t nat -D nova-network-snat -s 10.0.0.0/8 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -o br_ex -j SNAT --to-source 10.4.0.216
iptables -t nat -A nova-network-snat -s 10.101.0.0/16 -j SNAT --to-source 10.4.0.216
Question information
- Language:
- English Edit question
- Status:
- Solved
- Assignee:
- No assignee Edit question
- Solved by:
- xsited
- Solved:
- Last query:
- Last reply: