ovs router cannot access vm, but vms can access out

Asked by smarta94

Recently after a power failure, all my compute nodes and controller needed to be restarted. Prior to the failure, everything was running great. After starting back up, every configuration looks the same, the instances can ping out to the network, however, the instances are no longer accessible by public or private (via namespace execution) ip for ssh -- only through the console, which makes their purpose in the network fail, as they are to control various vital services I am running.
The odd thing is, the namespace router can ping itself, the instances show up in an arp -an on the namespace, the router can ping the external network, and the vms can ping the routers private ip for the subnet they are on, as well as other instances on that private network (so they are obtaining their dhcp addresses from the network node). The problem is, the router cannot ping the instances in the namespace, and nothing can ping their public floating ip address (but the meta data still shows up on the instance with the correct ip).
I have tried restarting the controller/network node, followed by restarting the computes and vice versa. The same thing results. I have also tried to restart the services on compute nodes in different orders (openvswitch followed by neutron services and nova services in nearly all permutations I can think of). GRE tunnels are there, otherwise my instances wouldn't be able to ping out and ovs-vsctl shows them normally.

What would cause this issue? Is there a database somewhere stopping flows or something else? The iptables in the namespace look fine to me:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N neutron-filter-top
-N neutron-l3-agent-FORWARD
-N neutron-l3-agent-INPUT
-N neutron-l3-agent-OUTPUT
-N neutron-l3-agent-local
-N neutron-vpn-agen-FORWARD
-N neutron-vpn-agen-INPUT
-N neutron-vpn-agen-OUTPUT
-N neutron-vpn-agen-local
-A INPUT -j neutron-l3-agent-INPUT
-A INPUT -j neutron-vpn-agen-INPUT
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-l3-agent-FORWARD
-A FORWARD -j neutron-vpn-agen-FORWARD
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-l3-agent-OUTPUT
-A OUTPUT -j neutron-vpn-agen-OUTPUT
-A neutron-filter-top -j neutron-l3-agent-local
-A neutron-filter-top -j neutron-vpn-agen-local
-A neutron-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT
-A neutron-vpn-agen-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT

Any help is appreciated, I need to get things back up.

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
actionparsnip (andrew-woodhead666) said :
#1

Have you tried rebooting your router?

You may also want to reboot the systems gracefully as the sudden power outage may have affected the networking as it was ungracefully closed down.

Revision history for this message
smarta94 (smarta94) said :
#2

How would one reboot the SDN router? I have restarted the ovs and neutron services on the controller and compute nodes assuming that would be the equivalent but when that didnt' work I posted the question here. Also, yes I had previously attempted a clean reboot of the machines (as stated in my question previously). The rouer refered to in the question is the SDN one via dhcp as neutron is the networking component I use (there is no problem with the physcial switches I use over the rest of the network, the issue is somwhere in the SDN).

Revision history for this message
smarta94 (smarta94) said :
#3

Are there any other suggestions. I have another node I installed fresh, everything worked fine on it and pingable and sshable in both directions with namespace and without using floating ips. Then, just to confirm the problem, I rebooted. Now that node is also in a state where any vm's hosted on it can access the outside as well, but nothing can reach it except through the console, rendering openstack USELESS.

Revision history for this message
smarta94 (smarta94) said :
#4

Running openstack juno, neutron networking (2014.2.1-1) neutronclient 2.3.9-1, and openvswitch 2.1.2-2 if that helps at all

Revision history for this message
Launchpad Janitor (janitor) said :
#5

This question was expired because it remained in the 'Open' state without activity for the last 15 days.