neutron

Quantum - cant reach metadata service using custom gateway

Asked by Mercadolibre CloudBuilders

Hi all,

In E3, flat_injected was working ok, to inject the ip info on the virtual instance, since E4, flat injected wasnt working anymore and we managed to get it working by modifying the network dict on quantum.manager.py .
So right now the ip is getting injected ok, as well as our custom gateway, why custom? because its not the gw-interface that ovs creates in network creation time, its our L3 switch direct IP , we are in a multinode sceneario, so we dont want that the gw interface on the controller were nova-network resides routes all the traffic, we want all the traffic routed trough our switch .
So, if we configure the ovs gw interface as the VMs default gateway everything works ok, but if we configure our custom gateway, the metadata fetch is failing (when the VM finish booting we can reach all the networks, but the metadata ip of course), cause its not matching any iptables rules or route to get to the metadata service on the controller.

So the question is :

- Is there any way to configure a rule via iptables or openflow to reach the metadata service from multiple compute nodes without using the ovs gw interface but our custom L3 Switch ip as the gateway ?

Regards

Question information

Language:
English Edit question
Status:
Solved
For:
neutron Edit question
Assignee:
No assignee Edit question
Solved by:
Mercadolibre CloudBuilders
Solved:
Last query:
Last reply:
Revision history for this message
dan wendlandt (danwent) said : 		#1

Generally speaking, no.

You could try to do the DNAT rewrite as part of your physical gateway.

You may also be able to give VMs an explicit gateway that is the network node gateway for the IP range that matches the metadata server. That way, they would use your external gateway for most traffic, but metadata traffic would be forwarded to the network node, and properly DNATed.

I wonder if Trey's recent changes changed the injection flag. We used to just hardcode it to 'true' in E-3. We already had a bug to use the --flat_injected flag in Quantum manager: https://bugs.launchpad.net/quantum/+bug/900864 Any chance your team would be able to make that change and push it upstream?

Revision history for this message
Alejandro Comisario (alejandro-comisario) said : 		#2

We can deffinitelly push the change to upstream, just tell us what it takes to do it !?
Of course we can improve it, and make Quantum Manager read --flat_injected flag.

We'll keep trying how to read the metaData service and get back to you !
We just tought that we had to manipulate OpenFlow to redirect trafic from the VM's to handle traffic going out of the tap interfaces.

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said : 		#3

Like Alejandro said.
What do we have to do to commit the --flat_injected consideration into quantum.manager ?

Revision history for this message
Alejandro Comisario (alejandro-comisario) said : 		#4

Dan.

I think we fixed the metadata issue in a very clever way !

We added into the interface.template on the compute node file the line :

post-up /sbin/iptables -t nat -A OUTPUT -d 169.254.169.254 -p tcp --dport 80 -j DNAT --to ${network-node}:8775

So, every intance actualy get the iptables rule to know how to get the metadata using an external gateway!

For the --flat_injected fix, we are still waiting for the structions on how to do it !

Revision history for this message
dan wendlandt (danwent) said : 		#5

if your change basically:

diff --git a/nova/network/quantum/manager.py b/nova/network/quantum/manager.py
index 8f2aa0a..50bd1f1 100644
--- a/nova/network/quantum/manager.py
+++ b/nova/network/quantum/manager.py
@@ -480,6 +480,7 @@ class QuantumManager(manager.FloatingIP, manager.FlatManager):
                            'uuid': network['uuid'],
                            'bridge': '', # Quantum ignores this field
                            'label': network['label'],
+ 'injected': FLAGS.flat_injected,
                            'project_id': net_tenant_id}
                 networks[vif['uuid']] = network

If so, you can just propose this into Nova, or if that is a lot of work, I can propose it on your behalf.

If you want to push this fix, assign the bug to yourself (https://bugs.launchpad.net/nova/+bug/900864), and then follow the process described at http://wiki.openstack.org/GerritWorkflow

Revision history for this message
dan wendlandt (danwent) said : 		#6

btw, that's a clever fix for the metadata issue. I hadn't realized that all of your VMs where linux VMs. definitely the simplest way to solve the problem in that case!

Revision history for this message
Mercadolibre CloudBuilders (cloudbuilders-n) said : 		#7

Thanks Dan !
And yes, all our vms are Linux, so yes, we can have the luxury of applying the interfaces template fix !

Ok, we will assign the bug to ourselves, and push it mainstream !
Best regards