To isolate or not to isolate

Looking at the newly added (or soon to be added) code for OVS, in particular the Tunneling switch stuff, the above may not be very strange. For having inter-hypervisor connection, should we wait for the new code? Thanks.

Here are several questions (seven to be precise) regarding my Quantum setup. Any help is much appreciated.
I have a two node system running openstack; I install the first node using devstack with all the services specified there by default and Quantum with openvswitch. (Essentially everything but Swift.) On the second node, I only start n-cpu and q-agt using devstack with a few minor changes to that script.
I can create VMs on both machines using Dashboard. Everything looks good. On both machines I connect eth0 to my br-int switch. (I have eth1 for outside world but don't need that for my current experiments.) I have the 10.0.0.0 network for my VMs as specified in devstack.
Then I go about creating a second network. I do that from the cli interface.

[SB] Which CLI? I am assuming Nova CLI using nova-manage commands as that would be the proper way to do it.
Question 0: I know Dan explained once why it is better not to have a way to create a network from dashboard but I am still confused as why this is the case.

[SB] The Dashboard allows for networks to be created in Quantum Service but today there is no way to associate a Quantum network ( that can be created with Dashboard) with Nova IPAM or Melange IPAM or an entry into Nova networks table. Creating a network using 'nova-manage' command creates a Quantum network, associates it with a Nova network table entry, and Nova or Melange IPAM.

In the future, all this can be done via OpenStack Dashboard but today we would encourage you to use 'nova-manage' command.

Having no other option I try the CLI but I cannot find the tenant id using the following instruction: keystone-manage tenant list. Nothing is printed out when this is executed.

[SB] This seems like a keystone issue, have you created tenants in Nova and Keystone, and made the proper associations? If you would like, you can run Nova without keystone using legacy Nova auth and specify Nova's project_id as the tenant_id to 'nova-manage' CLI command for Quantum.

Question 1: Why is that even though I have the keystone service running.

[SB] This seems like a keystone issue, even if you have keystone running, you have to manually create keystone id's credentials and make associations with nova projects.

Question 2: Are there other ways of finding the tenant id? I do not seem to find it in the nova tables in the database.

[SB] Tenant is a keystone term, its equivalence in terms of Nova us 'Project' you can associate a network with a Nova project, if you are having keystone issues, you can use legacy Nova Authentication.

So I try the following command for creating a new network:
nova-manage network create --label=secondNet --fixed_range_v4=8.8.8.0/24
I get a second network and create a bunch of VMs some using only 10.0.0.0 network and some using only the 8.8.8.0 network. (I have changed the GUI so I can pick the network at Laucnh.)

[SB] Using the above command, what you have done is created a 'shared' global network instead of a private network for a particular project/tenant. For experimentation purposes, this would work fine but all VMs irrespective of which project they belong to are placed on this global "shared" network that you have created.

Here is what I have:
Node1:
=========================================
running all services
has two gw-* interface (one for each network)
and two tap-* interface for two VMs
The IP address of VM 1: 10.0.0.2
The IP address of VM 2: 8.8.8.2
mb:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 9.2.xxx.65 0.0.0.0 UG 100 0 0 eth1
8.8.8.0 * 255.255.255.0 U 0 0 0 gw-c55299df-9c
9.2.xxx.64 * 255.255.255.192 U 0 0 0 eth1
10.0.0.0 * 255.255.255.0 U 0 0 0 gw-c21c442b-f3
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
Node 2:
=========================================
running n-cpu and q-agt
has two tap-* interface for two VMs
The IP address of VM 1: 10.0.0.3
The IP address of VM 2: 8.8.8.3
mb:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 9.2.xxx.65 0.0.0.0 UG 100 0 0 eth1
localnet * 255.255.255.192 U 0 0 0 eth1
192.168.1.0 * 255.255.255.0 U 0 0 0 eth0
192.168.122.0 * 255.255.255.0 U 0 0 0 virbr0
Question 3: Shouldn't I have the gw- interfaces on the second node as well? Am I missing something on the second node?

[SB] No, the gw-* interfaces are on the node where you are running nova-network. The OpenStack controller that is running
nova-network is housing the gateway for those 2 networks you have created ( one through dev stack and another through nova-manage command) You get one gateway per network, like it should be.

from 8.8.8.2 I can ping 8.8.8.3, and 10.0.0.3 but not 10.0.0.2
from 8.8.8.3 I can ping 8.8.8.2, and 10.0.0.3 but not 10.0.0.2
All the nodes are up and happily running.

Question 4: So I am wondering what exactly the gw- interface does. from the code I think it is supposed to forward all the messages betwwen different layer 3 networks (in addition to serving as dhcp server). Is this correct. If yes, shouldn't I have something automatically set up on my second machine? Why the gw-* gets created only on one machine, that is where I create the network.

[SB] The gw-* interface is used for NATing to pubic networks, as well as providing dnsmasq dhcp service to VMs. As mentioned above, the gw-* is bound to a network, Quantum networks are virtualized networks and span multiple hosts.

Question 5: What I really want is creating two networks and have the VMs on each not seeing the ones on the other network but potentiaaly able to access say the internet through eth1. Is that doable? What am I missing?

[SB] A key thing to understand is quantum abstracts or effectively "virtualized" you network and its resources, it doesn't segment your resources. For e.g. it doesn't segment this nic belong to this "virtual network". The way to achieve that would be using L2 QoS where you can bond both NICs, and then set a certain QoS on the port associated with a VM. Then on the "virtual network" that represents the public network, you will have a nova-network node serving as the gateway/NAT to the public internet.

To do this, You would need to be able to create private network per project using the nova-manage command with a associated project id, this will ensure that the created Quantum network is an isolated L2 segment and therefore another VM on another isolated L2 segment, can't ping VMs in this L2 segment. The second "virtual" network you will create is a shared public network that has a nova-network node or some hardware gateway that is performing NAT to the internet. Only the nova-network node needs to be physically connected to the internet or have a route to the internet.

In summary, your use cases can implemented using Quantum and is very typical but not implemented in the way you are currently envisioning.

Question 6: What's going on in my setup. As you can see it is not the case that all VMs see each other. Nor it is the case where they are isolated on different networks.

[SB] I think something funky is going with the VM 10.0.0.2, I would delete this VM and spin up another instance and also check connectivity from the 10.0.0.2 VM to others.

Thanks,
-Mohammad

Response to comment #1:

The new code just adds a new mechanism to achieve data path connectivity and isolation, and you dont have to wait for that the be able to achieve your use case.

Current mechanism for isolating L2 segments is VLANs, the new code adds an ability to use tunneling in some scenarios.

Thanks for the response. To verify the wierd behavior wrt ping VMs, I followed your suggestion.
I created a network using the project-id of my user that I can get from nova "instances" table.
I created two VMs on 10.0.0.0 which is my shared public network and two VMs on my 6.6.6.0 private network. I can verify that on each node of my two-node system, I have one VM from each network. I have repeated this with different networks and several iterations. To my surprise I always see the same pattern as reported earlier:

from 6.6.6.2( on node 1) I can ping 6.6.6.3 (on node 2), and 10.0.0.3 (on node 2) but not 10.0.0.2 (on node 1)
from 6.6.6.3 I can ping 6.6.6.2, and 10.0.0.3 but not 10.0.0.2

Hi Mohammad,

I feel like this is something with that particular VM( 10.0.0.2) that is troublesome. Can you do one of two things:

1) Delete 10.0.0.2 VM and create another VM and see if things are now working.

or

2) Provide who can 10.0.0.2 VM can ping and tcpdump/tcpdump analysis of where the ping packet is going and where it stops moving forward. i.e. bridge, OpenVSwitch, vNIC, physical NIC, on which host etc.

Thanks for the quick response. In my email I mentioned 10.0.0.2 but I tried other VMs. I removed 10.0.0.2 VM and created new ones (10.0.0.6 and 10.0.0.7), I even stopped nova and Quantum alltogether and restarted from scratch. All cases lead to the same pattern.
Regarding tcpdump, where do you want me to collect the data from? From the main nodes? I am not sure how to collect the details you mentioned above, thanks again for your help.

Hi Mohammad,

Is your issue resolved or have you been able to identify at which point do we loose the packet during your ping?

Thanks,
Somik

No the problem is still There. The thing is that there is not much one can gather from tcpdump as I run it. That's why I asked how you want to run it? I can also collect tracerout on each VM. Is that of any value?

Closing this and reopening another question with clearer description of the problem.