tap devices getting tagged to 4095

4095 is a special "dead" vlan for which all traffic is dropped. the
OVS quantum agent puts a linux device on this vlan if it does not find
a quantum port associated with the device, or if the port that is
found should not be forwarding traffic.

Its possible there's a problem with the communication channel between
the OVS agent on node 2 and the main quantum-server process on node 1.
 Providing the logs for the agent on node 2 would probably help.

dan

On Thu, Aug 23, 2012 at 8:11 PM, Vijay
<email address hidden> wrote:
> New question #206661 on quantum:
> https://answers.launchpad.net/quantum/+question/206661
>
> I have installed 2 node environment.
> node 1- controller - nova-compute/nova-api/nova-volume/nova-network/quanutm-server/OpenVswitch in vlan mode/quantum-openvswitch-agent.
> node 2 - compute node - nova-compute/nova-api/openvswitch/quantum-openvswitch-agent
>
> VMs launched on controller work fine. I am able to ping/ssh to them.
>
> However, VMs launched on compute node cannot be pinged or sshed.
>
> When i do, sudo ovs-vsctl show, I see that VMs (tap devices) on the compute node are all getting tag: 4095
>
> Also, when look at the console log of the VMs, I see that
>
> No lease, forking to background
> starting DHCP forEthernet interface eth0 [ OK ]
> cloud-setup: checking http:\/\/169.254.169.254\/2009-04-04\/meta-data\/instance-id
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 1\/30: up 10.46. request failed
>
> Any clue is appreciated.
> Thanks,
> vj
>
>
> --
> You received this question notification because you are an answer
> contact for quantum.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks Dan for the reply!

Here is the log from ovs quantum agent on the second compute node:

DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 list-ports br-int
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface eth2 external_ids
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface eth2 ofport
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface tap89afd232-ce external_ids
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface tap89afd232-ce ofport
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface tapeb806203-34 external_ids
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 get Interface tapeb806203-34 ofport
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 set Port tap89afd232-ce tag=4095
DEBUG:root:## running command: sudo ovs-ofctl add-flow br-int priority=2,in_port=10,actions=drop
DEBUG:root:## running command: sudo ovs-vsctl --timeout=2 set Port tapeb806203-34 tag=4095
DEBUG:root:## running command: sudo ovs-ofctl add-flow br-int priority=2,in_port=-1,actions=drop
ovs-ofctl: -1: value too large for 2-byte field in_port

Here is the nova-compute log:

2012-08-23 20:15:57 DEBUG nova.compute.manager [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] [instance: 5e7de5ad-5335-4b17-a779-92237ae8878e] Instance network_info: |[VIF({'network': Network({'bridge': u'', 'subnets': [Subnet({'ips': [FixedIP({'meta': {}, 'version': 4, 'type': u'fixed', 'floating_ips': [], 'address': u'192.168.4.20'})], 'version': 4, 'meta': {u'dhcp_server': u'192.168.4.1'}, 'dns': [IP({'meta': {}, 'version': 4, 'type': u'dns', 'address': u'8.8.4.4'})], 'routes': [], 'cidr': u'192.168.4.0/24', 'gateway': IP({'meta': {}, 'version': 4, 'type': u'gateway', 'address': u'192.168.4.1'})}), Subnet({'ips': [], 'version': None, 'meta': {u'dhcp_server': None}, 'dns': [], 'routes': [], 'cidr': None, 'gateway': IP({'meta': {}, 'version': None, 'type': u'gateway', 'address': None})})], 'meta': {u'tenant_id': u'53cd8a9405b24921a7a1d0223ff54e0d'}, 'id':
 u'5dc620b6-0277-4dd0-87bb-8f36b493deff', 'label': u'net-1'}), 'meta': {}, 'id': u'89afd232-ce33-49ef-b95a-28bf2b5c835f', 'address': u'fa:16:3e:76:22:72'})]| from (pid=308) _allocate_network /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/compute/manager.py:578
2012-08-23 20:15:57 DEBUG nova.virt.libvirt.connection [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] [instance: 5e7de5ad-5335-4b17-a779-92237ae8878e] Starting toXML method from (pid=308) to_xml /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/libvirt/connection.py:1544
2012-08-23 20:15:57 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Running cmd (subprocess): ip link show dev tap89afd232-ce from (pid=308) execute /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:219
2012-08-23 20:15:57 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Result was 1 from (pid=308) execute /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:235
2012-08-23 20:15:57 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Running cmd (subprocess): sudo ip tuntap add tap89afd232-ce mode tap from (pid=308) execute /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:219
2012-08-23 20:15:57 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Running cmd (subprocess): sudo ip link set tap89afd232-ce up from (pid=308) execute /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:219
2012-08-23 20:15:57 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Running cmd (subprocess): sudo ovs-vsctl -- --may-exist add-port br-int tap89afd232-ce -- set Interface tap89afd232-ce external-ids:iface-id=89afd232-ce33-49ef-b95a-28bf2b5c835f -- set Interface tap89afd232-ce external-ids:iface-status=active -- set Interface tap89afd232-ce external-ids:attached-mac=fa:16:3e:76:22:72 -- set Interface tap89afd232-ce external-ids:vm-uuid=5e7de5ad-5335-4b17-a779-92237ae8878e from (pid=308) execute /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:219
2012-08-23 20:15:57 DEBUG nova.virt.libvirt.connection [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] block_device_list [] from (pid=308) _volume_in_mapping /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/libvirt/connection.py:1419
2012-08-23 20:15:57 DEBUG nova.virt.libvirt.connection [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] block_device_list [] from (pid=308) _volume_in_mapping /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/libvirt/connection.py:1419
2012-08-23 20:15:57 DEBUG nova.virt.libvirt.connection [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] [instance: 5e7de5ad-5335-4b17-a779-92237ae8878e] Finished toXML method from (pid=308) to_xml /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/libvirt/connection.py:1548
2012-08-23 20:15:57 INFO nova.virt.libvirt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] [instance: 5e7de5ad-5335-4b17-a779-92237ae8878e] Called setup_basic_filtering in nwfilter
2012-08-23 20:15:57 INFO nova.virt.libvirt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] [instance: 5e7de5ad-5335-4b17-a779-92237ae8878e] Ensuring static filters
2012-08-23 20:15:58 DEBUG nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Adding security group rule: <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x5852b50> from (pid=308) instance_rules /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/firewall.py:291
2012-08-23 20:15:58 INFO nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Using cidr u'0.0.0.0/0'
2012-08-23 20:15:58 INFO nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Using fw_rules: ['-m state --state INVALID -j DROP', '-m state --state ESTABLISHED,RELATED -j ACCEPT', '-j $provider', u'-s 192.168.4.1 -p udp --sport 67 --dport 68 -j ACCEPT', u'-s 192.168.4.0/24 -j ACCEPT', u'-j ACCEPT -p tcp --dport 22 -s 0.0.0.0/0']
2012-08-23 20:15:58 DEBUG nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Adding security group rule: <nova.db.sqlalchemy.models.SecurityGroupIngressRule object at 0x5852c50> from (pid=308) instance_rules /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/firewall.py:291
2012-08-23 20:15:58 INFO nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Using cidr u'0.0.0.0/0'
2012-08-23 20:15:58 INFO nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Using fw_rules: ['-m state --state INVALID -j DROP', '-m state --state ESTABLISHED,RELATED -j ACCEPT', '-j $provider', u'-s 192.168.4.1 -p udp --sport 67 --dport 68 -j ACCEPT', u'-s 192.168.4.0/24 -j ACCEPT', u'-j ACCEPT -p tcp --dport 22 -s 0.0.0.0/0', u'-j ACCEPT -p icmp -s 0.0.0.0/0']
2012-08-23 20:15:58 DEBUG nova.virt.firewall [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Filters added to instance 5e7de5ad-5335-4b17-a779-92237ae8878e from (pid=308) prepare_instance_filter /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/virt/firewall.py:137
2012-08-23 20:15:58 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Attempting to grab semaphore "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=308) inner /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:927
2012-08-23 20:15:58 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Got semaphore "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=308) inner /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:931
2012-08-23 20:15:58 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Attempting to grab file lock "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=308) inner /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:935
2012-08-23 20:15:58 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Got file lock "iptables" for method "_do_refresh_provider_fw_rules"... from (pid=308) inner /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:942
2012-08-23 20:15:58 DEBUG nova.utils [req-93bac5d9-d054-481a-8f46-824bac779fe5 2ad891f6904946d8b08dd38f50ce0801 53cd8a9405b24921a7a1d0223ff54e0d] Attempting to grab semaphore "iptables" for method "apply"... from (pid=308) inner /usr/local/lib/python2.7/dist-packages/nova-2012.1.1-py2.7.egg/nova/utils.py:927

Thanks,
-vj

________________________________
From: dan wendlandt <email address hidden>
To: <email address hidden>
Sent: Thursday, August 23, 2012 9:45 PM
Subject: Re: [Question #206661]: tap devices getting tagged to 4095

Your question #206661 on quantum changed:
https://answers.launchpad.net/quantum/+question/206661

    Status: Open => Answered

dan wendlandt proposed the following answer:
4095 is a special "dead" vlan for which all traffic is dropped.  the
OVS quantum agent puts a linux device on this vlan if it does not find
a quantum port associated with the device, or if the port that is
found should not be forwarding traffic.

Its possible there's a problem with the communication channel between
the OVS agent on node 2 and the main quantum-server process on node 1.
Providing the logs for the agent on node 2 would probably help.

dan

On Thu, Aug 23, 2012 at 8:11 PM, Vijay
<email address hidden> wrote:
> New question #206661 on quantum:
> https://answers.launchpad.net/quantum/+question/206661
>
> I have installed 2 node environment.
> node 1- controller - nova-compute/nova-api/nova-volume/nova-network/quanutm-server/OpenVswitch in vlan mode/quantum-openvswitch-agent.
> node 2 - compute node - nova-compute/nova-api/openvswitch/quantum-openvswitch-agent
>
> VMs launched on controller work fine. I am able to ping/ssh to them.
>
> However, VMs launched on compute node cannot be pinged or sshed.
>
> When i do, sudo ovs-vsctl show, I see that VMs (tap devices) on the compute node are all getting tag: 4095
>
> Also, when look at the console log of the VMs, I see that
>
> No lease, forking to background
> starting DHCP forEthernet interface eth0 [ OK ]
> cloud-setup: checking http:\/\/169.254.169.254\/2009-04-04\/meta-data\/instance-id
> wget: can't connect to remote host (169.254.169.254): Network is unreachable
> cloud-setup: failed 1\/30: up 10.46. request failed
>
> Any clue is appreciated.
> Thanks,
> vj
>
>
> --
> You received this question notification because you are an answer
> contact for quantum.

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: http://www.nicira.com/
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

--
If this answers your question, please go to the following page to let us
know that it is solved:
https://answers.launchpad.net/quantum/+question/206661/+confirm?answer_id=0

If you still need help, you can reply to this email or go to the
following page to enter your feedback:
https://answers.launchpad.net/quantum/+question/206661

You received this question notification because you asked the question.

Hi Vijay, are you running quantum from source? If so I believe if you do a git pull this should fixed this issue for you. Alternatively if you do; sudo ovs-vsctl list-ports and then remove the interfaces from br-int (ovs-vsctl del-port br-int tapxxxx) that don't show up when you do ifconfig -a this should hopefully resolve the issue for you.

P.S: you can also do ovs-vsctl del-br br-int; ovs-vsctl add-br br-int; to fix this but it will break the network for the vms that are currently attached to br-int.

Can u post which version of quantum u are using, and how u start the agent?
Thanks