Failed to ping from linux bridge's gw ip

Asked by yong sheng gong

I failed to set up a molti node env with linux bridge plugin. So I want to have a simulation first.

First I did a test between two hosts:
host a:
ip link add link eth0 name eth0.1000 type vlan id 1000
ip link set eth0.1000 up
ip addr add 4.4.4.1/24 dev eth0.1000
host b:
ip link add link eth0 name eth0.1000 type vlan id 1000
ip link set eth0.1000 up
ip addr add 4.4.4.2/24 dev eth0.1000

This way I can ping each other from both hosts

Then I add in linux bridge on one host:
host b:
ip link add link eth0 name eth0.1000 type vlan id 1000
ip link set eth0.1000 up
brctl addbr br200
brctl setfd br200 0
brctl stp br200 off
ip link set br200 up
brctl addif br200 eth0.1000

ip tuntap add gw200 mode tap
ip link set gw200 up
brctl addif br200 gw200
ip addr add 4.4.4.2/24 dev gw200

but I cannot ping from each other.

Who can help with this?
Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
yong sheng gong
Solved:
Last query:
Last reply:
Revision history for this message
Sumit Naiksatam (snaiksat) said :
#1

I believe what happens in the later case is that the tap device by itself does not have a L3 stack to respond to the ping unless that tap device has a VM associated with it.

In the former case, the host's network stack itself will respond to the ping.

Hence in the current implementation, I assign the gateway IP address to the bridge (however we still need the gw interface so as to anchor the MAC address to the bridge).

Revision history for this message
yong sheng gong (gongysh) said :
#2

Can you tell me more about (however we still need the gw interface so as to anchor the MAC address to the bridge)?

and
how Should I modify the scripts so that they can ping each other without any VMs?

Thanks

Revision history for this message
Sumit Naiksatam (snaiksat) said :
#3

In the experiment that you are trying, you can assign an IP address to the bridge.

When using Quantum/Linux-bridge plugin, you should not have to change anything. The only problem I have seen recently is that the VMs that are created on the default network (the one created when the installation is done using devstack) are not able to reach the metadata server, hence they do not acquire an IP address. I am trying to investigate this.

I have some more documentation on the gw interace here (see the section on handling the gateway interface):
http://wiki.openstack.org/Quantum-Linux-Bridge-Plugin

Revision history for this message
yong sheng gong (gongysh) said :
#4

Hi Summit,
Now I can ping each other with following scripts:
hosta:
ip link add link eth0 name eth0.1000 type vlan id 1000
ip link set eth0.1000 up

ip tuntap add gw200 mode tap
ip link set gw200 address 00:21:CC:6D:44:43
ip link set gw200 up

brctl addbr br200
brctl setfd br200 0
brctl stp br200 off
ip link set br200 address 00:21:CC:6D:44:43
ip link set br200 up

brctl addif br200 eth0.1000

brctl addif br200 gw200
ip addr add 3.3.3.1/24 dev br200
ip addr add 3.3.3.1/24 dev gw200

hostb:
ip link add link eth0 name eth0.1000 type vlan id 1000
ip link set eth0.1000 up

ip tuntap add gw200 mode tap
ip link set gw200 address 00:21:CC:6D:44:45
ip link set gw200 up

brctl addbr br200
brctl setfd br200 0
brctl stp br200 off
ip link set br200 address 00:21:CC:6D:44:45
ip link set br200 up

brctl addif br200 eth0.1000

brctl addif br200 gw200
ip addr add 3.3.3.2/24 dev br200
ip addr add 3.3.3.2/24 dev gw200

I have a new problem for the VM on each host. I created one vm with following steps on hosta:
ip tuntap add tap4cce2519-02 mode tap
ip link set tap4cce2519-02 address fa:16:3e:48:31:dc
 ip link set tap4cce2519-02 up
 brctl addif br200 tap4cce2519-02

virsh dumpxml instance-00000001:
...
    <interface type="ethernet">
      <mac address="fa:16:3e:48:31:dc"/>
      <script path=""/>
      <target dev="tap4cce2519-02"/>
    </interface>
...
and in the vm, I config the eth0:
sudo ifconfig eth0 3.3.3.4 broadcast 3.3.3.255 netmask 255.255.255.0

after that I can ping hosta in VM:
ping 3.3.3.1
but I cannot ping hostb:
ping 3.3.3.2

By the same token, I created a vm on hostb with ip 3.3.3.3,
I can ping hostb, but cannot ping hosta.

That means the ping cannot go across the host.

any ideas?
Thanks

Revision history for this message
Hua Zhang (zhhuabj) said :
#5

you can ping hosta in VM from hosta, but cann't ping hostb, so I think you try following command on hosta ( only a little of theory analysis, I am not sure it can work ):
sysctl -w net.ipv4.ip_forward=1
route add -host 3.3.3.4 dev tap4cce2519-02

then on VM from hosta:
route add default gw 3.3.3.1

Revision history for this message
yong sheng gong (gongysh) said :
#6

Hi Hua,
I have tried, no chance to work.
Thanks

Revision history for this message
Sumit Naiksatam (snaiksat) said :
#7

Few things to check for -

Can you ping the gateway address 3.3.3.1 (in this case) from the VM on the hostb?

Can you ping the gateway address 3.3.3.1 from just the hostb?

Are there in iptables rules in place blocking the VM traffic?

If there are no further clues, you probably might want to do a tcpdump on the relevant interfaces to see how far the packets are reaching across the hosts.

Revision history for this message
yong sheng gong (gongysh) said :
#8

I can ping just hosta 3.3.3.1 which is a br200 ip address from VMs on hosta, but cannot ping hostb's 3.3.3.2, which is hostb's br200 ip address.

I will use tcpdump later.
Thanks

Revision history for this message
Hua Zhang (zhhuabj) said :
#9

Hi Sumit,

         I also done this experiment in multi physical machines, but faied.

         but I found one article, it said should use TUN to communicate in two TAP virtual NIC from two physical machines.

         http://kovyrin.net/2006/04/05/connecting-two-remote-local-networks-with-transparent-bridging/

         So I was confused in following issues:

         1) can linux bridge with physical NICs be used in multi physical machines ?

         2) can linux bridge with TAP virtual NICs be used in multi physical machines ? need to use TUN ?

         3) why we create a TAP NIC named gw200 ? what's it's role ? can we directly use br200 ?

         4) another other not related problem with this question, assume compute node only has one physical NIC ( eth0 ), and we don't use vlan ( means no eth0.1000 at this time ), meanwhile I also want have two network ( manament nework 172.17.1.0/24, and service network 10.0.0.0/8) in this single NIC, can do do it ?
            my question is , the network of bridge is 10.0.0.0/8, if I add this single NIC eth0 into bridge, many docs said should remove ip from eth0 at this time. can eth0 sitll keep it's original ip ( 17.17.1.0/24) ? if so, what's their route path ?

         look forward to your input, many thanks.

Revision history for this message
Hua Zhang (zhhuabj) said :
#10

Hi yong sheng,

I have successfully run this testing when I boot the vm using "libvirt bridge mode" rather than "libvirt tap mode" or "qemu tap mode".

libvirt bridge mode:
<interface type='bridge'>
      <mac address='fa:16:3e:48:31:dc'/>
      <source bridge='br0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

libvirt tap mode:
<interface type="ethernet">
      <mac address="fa:16:3e:48:31:dc"/>
      <script path=""/>
      <target dev="tap4cce2519-02"/>
 </interface>

qemu tap mode:
qemu-system-x86_64 -boot c -hda /bak/kvmimages/linux-0.2.img -net nic,macaddr=fa:16:3e:48:31:dc -net tap,ifname=tap0,script=no -m 128 -vnc :1 -monitor stdio

you use "libvirt tap mode", I use "qemu tap mode", we have the same problem.
so I guess the reason is as I said above " should use TUN to communicate in two TAP virtual NIC from two physical machines.", perhalps libvirt bridge has help us do those things automatically. the next I will do experient to prove this point.

Revision history for this message
yong sheng gong (gongysh) said :
#11

Hi Sumit,
MyCurrent env is:
First I create a VM on hosta with network definition:
    <interface type='bridge'>
      <mac address='52:54:00:e4:2e:c1'/>
      <source bridge='br200'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
I can ping hostb in the VM;
while I clone the vm and redefine the derived VM's network:
    <interface type='ethernet'>
      <mac address='52:54:00:ad:21:49'/>
      <script path=''/>
      <target dev='tapb79b2704-2c'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
I failed to ping the hostb.

 I have used the tshark show that hostb has responded the ARP request, but the derived VM cannot accept it.
These are my bridge settting on hosta:
[root@robinlinux ~]# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.0021cc6d4440 no eth0
br200 8000.0021cc6d4443 no eth0.1000
       gw200
       tapb79b2704-2c
       vnet0
[root@robinlinux ~]# ifconfig
br100 Link encap:Ethernet HWaddr 00:21:CC:6D:44:40
          inet addr:10.10.10.1 Bcast:0.0.0.0 Mask:255.255.255.255
          inet6 addr: fe80::221:ccff:fe6d:4440/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:291 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:37655 (36.7 KiB) TX bytes:6115 (5.9 KiB)

br200 Link encap:Ethernet HWaddr 00:21:CC:6D:44:43
          inet addr:3.3.3.1 Bcast:0.0.0.0 Mask:255.255.255.0
          inet6 addr: fe80::221:ccff:fe6d:4443/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:4128 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2897 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:304856 (297.7 KiB) TX bytes:225043 (219.7 KiB)

eth0 Link encap:Ethernet HWaddr 00:21:CC:6D:44:40
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:3163 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2924 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:287833 (281.0 KiB) TX bytes:219959 (214.8 KiB)
          Interrupt:20 Memory:f3900000-f3920000

eth0.1000 Link encap:Ethernet HWaddr 00:21:CC:6D:44:40
          inet6 addr: fe80::221:ccff:fe6d:4440/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2802 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2807 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:186849 (182.4 KiB) TX bytes:211562 (206.6 KiB)

gw200 Link encap:Ethernet HWaddr 00:21:CC:6D:44:43
          inet6 addr: fe80::221:ccff:fe6d:4443/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:1702 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:28912 errors:0 dropped:0 overruns:0 frame:0
          TX packets:28912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:85366878 (81.4 MiB) TX bytes:85366878 (81.4 MiB)

tapb79b2704-2c Link encap:Ethernet HWaddr 52:54:00:AD:21:49
          inet6 addr: fe80::5054:ff:fead:2149/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2108 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1553 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:212561 (207.5 KiB) TX bytes:143614 (140.2 KiB)

tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:9.77.178.7 P-t-P:9.77.178.7 Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1362 Metric:1
          RX packets:79 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:52248 (51.0 KiB) TX bytes:15127 (14.7 KiB)

vnet0 Link encap:Ethernet HWaddr FE:54:00:E4:2E:C1
          inet6 addr: fe80::fc54:ff:fee4:2ec1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:20 errors:0 dropped:0 overruns:0 frame:0
          TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1704 (1.6 KiB) TX bytes:4720 (4.6 KiB)
[root@robinlinux ~]# iptables-save
# Generated by iptables-save v1.4.7 on Thu Jun 14 11:30:31 2012
*filter
:INPUT ACCEPT [83106:63445198]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [83632:13299389]
COMMIT
# Completed on Thu Jun 14 11:30:31 2012

Thanks

Revision history for this message
yong sheng gong (gongysh) said :
#12

After I added an ARP entry into VM's arp table by:
arp -s 3.3.3.2 mac_address,
Then it seems the Vm cannot accept the ICMP response.

Thanks

Revision history for this message
Sumit Naiksatam (snaiksat) said :
#13

Thanks Yong. Want to make sure I understand your response #11, you mention that the response is not coming back to the VM. Are you seeing the packets from the VM on hosta reaching the hostb?

I also noticed that the gw200 interface has dropped packets, I am wondering if this offers any clue.

Revision history for this message
yong sheng gong (gongysh) said :
#14

one more thing:
for <interface type='bridge'>
      <mac address='52:54:00:e4:2e:c1'/>
      <source bridge='br200'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
the kvm process is using:
-netdev tap,fd=24,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:e4:2e:c1,bus=pci.0,addr=0x3

I am wondering who created and opened vnet0 for KVM process.

for <interface type='ethernet'>
      <mac address='52:54:00:ad:21:49'/>
      <script path=''/>
      <target dev='tapb79b2704-2c'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
it is using:
-netdev tap,ifname=tap10,script=,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=f6:19:e0:6c:84:c3,bus=pci.0,addr=0x3
Note, KVM is using ifname=tap10 to run.

Revision history for this message
yong sheng gong (gongysh) said :
#15

tap dev should have different mac than the VM