Quantum with Openvswitch and stack.sh no gateway interface

hi eoghan

i'm not sure why vm's in your setup are not getting an ip but based on my experience so far, one of the "tap" ports in your setup should be gateway interface. that tap port gets created when you have created a subnet for that network and it get's the gateway ip assigned to it. so if you do ifconfig, that tap port should have an ip assigned. if that's not the case then the best place to look for errors is in /opt/stack/logs/screen/screen-q-*.log

thanks
rohit

Rohit

I looked at all the quantum logs and could not find any errors and none of the tap interfaces get IP addresses assigned. I ran the folsom2 version of Quantum on this same server and it worked fine and created tap and gateway devices (and the gateway was assigned an IP). This is what the tap interface looks like after creating a network and subnet.

tap9618cae3-7e Link encap:Ethernet HWaddr 36:13:13:d4:71:b5
          inet6 addr: fe80::3413:13ff:fed4:71b5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:153 errors:0 dropped:0 overruns:0 frame:0
          TX packets:60 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:11664 (11.6 KB) TX bytes:7982 (7.9 KB)

Is there anything else I can look at?

Thanks
Eoghan

Do you have q-dhcp in ENABLED_SERVICES? I don't see a port for the dhcp agent attached to your bridge.

I have q-dhcp enabled and it is running.

Looked back at the logs and think maybe this is what causes the problems (I am running Ubuntu 12.04).

2012-08-23 13:40:30 DEBUG [quantum.agent.linux.dhcp] Unable to access /opt/stack/data/dhcp/910bebe6-13a3-438c-a021-a217269d43b4/pid
2012-08-23 13:40:30 DEBUG [quantum.agent.linux.utils] Running command: sudo cat /proc/None/cmdline
2012-08-23 13:40:30 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'cat', '/proc/None/cmdline']
Exit code: 1
Stdout: ''
Stderr: 'cat: /proc/None/cmdline: No such file or directory\n'
.............................
2012-08-23 13:40:30 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec 910bebe6-13a3-438c-a021-a217269d43b4 ip -o link show tape9e2be92-ca
2012-08-23 13:40:30 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'exec', '910bebe6-13a3-438c-a021-a217269d43b4', 'ip', '-o', 'link', 'show', 'tape9e2be92-ca']
Exit code: 1
Stdout: ''
Stderr: 'Cannot open network namespace: No such file or directory\n'

Hi Eoghan,

I think that error you see there is fine. I believe the agent does a show to see if the interface exists and if it doesn't it goes ahead and creates the interface. I just ran stack.sh with the following:

ENABLED_SERVICES="g-api,g-reg,key,n-api,n-cpu,n-sch,n-vnc,mysql,rabbit,openstackx,q-svc,quantum,q-agt,n-cpu,q-dhcp"
Q_PLUGIN=openvswitch

 and everything worked fine. Can you try rm -fr /opt/stack; and then do a git pull in your devstack dir to make sure that you have the most up to date devstack code?

Hi Aaron

I did a git pull and tried with this in my stackrc:
ENABLED_SERVICES="g-api,g-reg,key,n-api,n-cpu,n-sch,n-vnc,mysql,rabbit,openstackx,q-svc,quantum,q-agt,n-cpu,q-dhcp"
Q_PLUGIN=openvswitch

I created a network and subnet:
quantum net-create private
quantum subnet-create b7076534-e1b5-42fc-ac4b-14196fb9f21d 10.0.1.0/24

Then booted up the VM with this:
nova boot --image facb609d-5399-4023-808a-bf10fb39e47c --flavor 1 --nic net-id=b7076534-e1b5-42fc-ac4b-14196fb9f21d test

From the console log it looks like the VM got an IP but I still cannot ping it:

ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 10.0.1.3 255.255.255.0 fa:16:3e:d3:f2:f1
ci-info: route-0: 0.0.0.0 10.0.1.1 0.0.0.0 eth0 UG
ci-info: route-1: 10.0.1.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 24 Aug 2012 00:22:41 +0000. up 6.52 seconds
2012-08-24 00:22:44,681 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [timed out]

And I still do not see the any gateways. Ifconfig still only shows the tap devices:

tap17199918-5d Link encap:Ethernet HWaddr de:c9:18:64:8a:aa
          inet6 addr: fe80::dcc9:18ff:fe64:8aaa/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:77 errors:0 dropped:0 overruns:0 frame:0
          TX packets:35 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:5346 (5.3 KB) TX bytes:3626 (3.6 KB)

tap455c4a70-f3 Link encap:Ethernet HWaddr a6:f9:4c:e4:f2:cc
          inet6 addr: fe80::a4f9:4cff:fee4:f2cc/64 Scope:Link
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:1836 (1.8 KB) TX bytes:468 (468.0 B)

And ovs-vsctl still shows this:

8d3731cf-1ac2-4df8-86e5-47ed0a5106ac
    Bridge br-int
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port "tapc0474b52-85"
            tag: 2
            Interface "tapc0474b52-85"
                type: internal
        Port br-int
            Interface br-int
                type: internal
        Port "tap196eecae-49"
            tag: 2
            Interface "tap196eecae-49"
        Port "tap647b6989-d5"
            tag: 1
            Interface "tap647b6989-d5"
                type: internal
    Bridge br-tun
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "1.4.0+build0"

I'm having the exact same problem Eoghan, with the same setup.

There's some confusion here:

1) unlike nova-network, quantum DHCP does not use the gateway IP address of the subnet. Thus, the lack of a tap device with the gateway IP does not mean that the DHCP server is not running. In fact, by default, quantum runs DHCP in its own isolated network namespace, so you would not see the IP address of the DHCP server from the main test host.

2) The actual gateway is implemented using the quantum-l3-agent, which just landed after F-3. Devstack support for this is still under review (should merge within a few days). The quantum-l3-agent uses namespaces as well though, so by default, you wouldn't see the .1 IP address using ifconfig even using L3 is in use.

Thanks Aaron and Dan,

I thought that may be the case (gateway and IP's being inaccessible from the hyperverisor) but was not sure. And I can see the VM getting an IP in the nova console-log output so quantum dhcp is working.

Thanks Aaron Rosen, that solved my question.

> I thought that may be the case (gateway and IP's being inaccessible from
> the hyperverisor) but was not sure. And I can see the VM getting an IP
> in the nova console-log output so quantum dhcp is working.

Its actually not that the IP is inaccessible from the hypervisor, just
that it requires some tricky knowledge about how network namespaces
work. We're working on some tools that will hopefully help make
troubleshooting these scenarios easier.

dan

Egohan:

Are you basing your belief that the VM got an IP address based on the 169.254.169.254 you are seeing in the logs you posted?

If so, you may be mistaken -- if configured for fixed range addresses, my understanding is you should be seeing some evidence that dhcp is returning an IP address to the VM in the range in the console logs, e.g., 10.4.128.0 is the range I have assigned.

ci-info: lo : 1 127.0.0.1 255.0.0.0
ci-info: eth0 : 1 10.0.1.3 255.255.255.0 fa:16:3e:d3:f2:f1
ci-info: route-0: 0.0.0.0 10.0.1.1 0.0.0.0 eth0 UG
ci-info: route-1: 10.0.1.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Fri, 24 Aug 2012 00:22:41 +0000. up 6.52 seconds
2012-08-24 00:22:44,681 - DataSourceEc2.py[WARNING]: 'http://169.254.169.254' failed: url error [timed out]

Turns out I am having almost the same issues as you were describing. I followed your steps exactly to create a private net and spin up a VM. In the console logs of the VM, I see:

Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending discover...
Sending discover...
No lease, failing
WARN: /etc/rc3.d/S40-network failed
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 1/30: up 16.39. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 2/30: up 17.46. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 3/30: up 18.51. request failed
wget: can't connect to remote host (169.254.169.254): Network is unreachable
cloud-setup: failed 4/30: up 19.57. request failed

This is distinctly different than what I was getting with devstack/OVS quantum V1 pre-F3 (I would get a DHCP-returned address in the fixed range I specified in my devstack localrc), and I'm tearing my hair out trying to figure out why.

For completeness sake, here is what I did based on your examples:

stack@openstack4:~/devstack$ quantum net-create private
Created a new network:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| admin_state_up | True |
| id | 3983ff37-b018-451e-a668-1f062d1924ad |
| name | private |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | a0e72cd263f041ca9ad977bb6889d2ad |
+----------------+--------------------------------------+
stack@openstack4:~/devstack$ quantum subnet-create 3983ff37-b018-451e-a668-1f062d1924ad 10.0.1.0/24
Created a new subnet:
+------------------+--------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------+
| allocation_pools | {"start": "10.0.1.2", "end": "10.0.1.254"} |
| cidr | 10.0.1.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.1.1 |
| host_routes | |
| id | 345280cf-a4fd-4538-a9d1-421ed73f9229 |
| ip_version | 4 |
| name | |
| network_id | 3983ff37-b018-451e-a668-1f062d1924ad |
| tenant_id | a0e72cd263f041ca9ad977bb6889d2ad |
+------------------+--------------------------------------------+
stack@openstack4:~/devstack$ nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| 6b465660-b99a-4e37-b1a2-2f1f2fbfcc8a | cirros-0.3.0-x86_64-uec | ACTIVE | |
| 04319fb4-ee1d-4104-b8b1-27c9400167dd | cirros-0.3.0-x86_64-uec-kernel | ACTIVE | |
| 8bcca951-388f-4fc1-a286-73159409183d | cirros-0.3.0-x86_64-uec-ramdisk | ACTIVE | |
+--------------------------------------+---------------------------------+--------+--------+
stack@openstack4:~/devstack$ nova boot --image 6b465660-b99a-4e37-b1a2-2f1f2fbfcc8a --flavor 1 --nic net-id=3983ff37-b018-451e-a668-1f062d1924ad test
+------------------------+--------------------------------------+
| Property | Value |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | ptq3iEhSt3xz |
| config_drive | |
| created | 2012-08-29T20:06:52Z |
| flavor | m1.tiny |
| hostId | |
| id | 68a13a7a-5bdb-403e-9e45-12dc11682dfe |
| image | cirros-0.3.0-x86_64-uec |
| key_name | None |
| metadata | {} |
| name | test |
| progress | 0 |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| tenant_id | a0e72cd263f041ca9ad977bb6889d2ad |
| updated | 2012-08-29T20:06:52Z |
| user_id | e12eeaab09364ca7ae967c53855a2c94 |
+------------------------+--------------------------------------+

I then looked in the log for the VM (/opt/stack/nova/instances/instance-000000001/console.log to see if I got a network.

Is the dhcp agent running?

slogan@openstack4:~/devstack/devstack$ ps -eaf | grep quantum
stack 4194 4133 0 12:57 pts/3 00:00:02 python /opt/stack/quantum/bin/quantum-server --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
root 4264 4196 0 12:57 pts/6 00:00:00 sudo python /opt/stack/quantum/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
root 4267 4264 0 12:57 pts/6 00:00:09 python /opt/stack/quantum/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini

Short answer, no. Maybe a devstack issue? It is what is supposed to launch it. I see evidence that it is being launched in the stack logs:

...

+ screen -S stack -p q-agt -X stuff 'sudo python /opt/stack/quantum/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini^M'
+ screen_it q-dhcp 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini'
++ echo -ne '\015'
+ NL=$'\r'
+ is_service_enabled q-dhcp
+ services=q-dhcp
+ for service in '${services}'
+ [[ ,g-api,g-reg,key,n-api,n-cpu,n-sch,n-vnc,mysql,rabbit,openstackx,q-svc,quantum,q-agt,n-cpu,q-dhcp, =~ ,q-dhcp, ]]
+ return 0
+ screen_rc q-dhcp 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini'
+ SCREENRC=/opt/stack/devstack/stack-screenrc
+ [[ ! -e /opt/stack/devstack/stack-screenrc ]]
+ grep q-dhcp /opt/stack/devstack/stack-screenrc
++ echo -ne '\015'
+ NL=$'\r'
+ echo 'screen -t q-dhcp bash'
+ echo 'stuff "sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini^M"'
+ screen -S stack -X screen -t q-dhcp
+ sleep 1.5
+ [[ -n '' ]]
+ screen -S stack -p q-dhcp -X stuff 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini^M'
+ NOVA_CONF_DIR=/etc/nova
+ [[ ! -d /etc/nova ]]
+ sudo mkdir -p /etc/nova
++ whoami
...

 I tried running the command manually and same results in ps, perhaps it runs but for a short time. The fact it is being passed two arguments with same flag (--config-file) looks worrisome depending on how argc, argv is being parsed by the process (is that a good idea?). Nonetheless, both files exist:

/etc/quantum/dhcp_agent.ini:

[DEFAULT]
admin_password = password
admin_user = quantum
admin_tenant_name = service
auth_url = http://192.168.4.1:35357/v2.0
db_connection = mysql://root:password@localhost/ovs_quantum?charset=utf8
use_namespaces = True
debug = True
verbose = True
# Show debugging output in log (sets DEBUG log level output)
# debug = true

# Where to store dnsmasq state files. This directory must be writable by the
# user executing the agent. The value below is compatible with a default
# devstack installation.
state_path = /opt/stack/data

# The DHCP requires that an inteface driver be set. Choose the one that best
# matches you plugin.

# OVS
interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
# LinuxBridge
#interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver
# Ryu
#interface_driver = quantum.agent.linux.interface.RyuInterfaceDriver

# The agent can use other DHCP drivers. Dnsmasq is the simplest and requires
# no additional setup of the DHCP server.
dhcp_driver = quantum.agent.linux.dhcp.Dnsmasq

# Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y and
# iproute2 package that supports namespaces).
# use_namespaces = True

/etc/quantum/quantum.conf:

[DEFAULT]
rabbit_password = password
rabbit_host = localhost
auth_strategy = keystone
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = True

# Address to bind the API server
bind_host = 0.0.0.0

# Port the bind the API server to
bind_port = 9696

# Path to the extensions. Note that this can be a colon-separated list of
# paths. For example:
# api_extensions_path = extensions:/path/to/more/extensions:/even/more/extensions
# The __path__ of quantum.extensions is appended to this, so if your
# extensions are in there you don't need to specify them here
# api_extensions_path =

# Quantum plugin provider module
core_plugin = quantum.plugins.openvswitch.ovs_quantum_plugin.OVSQuantumPluginV2

# Paste configuration file
api_paste_config = api-paste.ini

# The strategy to be used for auth.
# Supported values are 'keystone'(default), 'noauth'.
# auth_strategy = keystone

# Base MAC address. The first 3 octets will remain unchanged. If the
# 4h octet is not 00, it will also used. The others will be
# randomly generated.
# 3 octet
# base_mac = fa:16:3e:00:00:00
# 4 octet
# base_mac = fa:16:3e:4f:00:00

# Maximum amount of retries to generate a unique MAC address
# mac_generation_retries = 16

# DHCP Lease duration (in seconds)
# dhcp_lease_duration = 120

# Enable or disable bulk create/update/delete operations
# allow_bulk = True
# RPC configuration options. Defined in rpc __init__
# The messaging module to use, defaults to kombu.
# rpc_backend = quantum.openstack.common.rpc.impl_kombu
# Size of RPC thread pool
# rpc_thread_pool_size = 64,
# Size of RPC connection pool
# rpc_conn_pool_size = 30
# Seconds to wait for a response from call or multicall
# rpc_response_timeout = 60
# Seconds to wait before a cast expires (TTL). Only supported by impl_zmq.
# rpc_cast_timeout = 30
# Modules of exceptions that are permitted to be recreated
# upon receiving exception data from an rpc call.
# allowed_rpc_exception_modules = quantum.openstack.common.exception, nova.exception
# AMQP exchange to connect to if using RabbitMQ or QPID
control_exchange = quantum

# If passed, use a fake RabbitMQ provider
# fake_rabbit = False

# Configuration options if sending notifications via kombu rpc (these are
# the defaults)
# SSL version to use (valid only if SSL enabled)
# kombu_ssl_version =
# SSL key file (valid only if SSL enabled)
# kombu_ssl_keyfile =
# SSL cert file (valid only if SSL enabled)
# kombu_ssl_certfile =
# SSL certification authority file (valid only if SSL enabled)'
# kombu_ssl_ca_certs =

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Aaron Rosen
Sent: Wednesday, August 29, 2012 1:31 PM
To: Syd (Sydney) Logan
Subject: Re: [Question #206604]: Quantum with Openvswitch and stack.sh no gateway interface

Question #206604 on quantum changed:
https://answers.launchpad.net/quantum/+question/206604

Aaron Rosen posted a new comment:
Is the dhcp agent running?

--
You received this question notification because you are a direct
subscriber of the question.

I tried running with just the --config-file=/etc/quantum/dhcp_agent.ini and spun up a second VM. Still no network assigned by DHCP, but at least quantum-dhcp-agent doesn't exit immediately anymore. Going to take a look at the sources for the agent, maybe I need that second argument but under a different name...

You said you were able to use devstack etc. no problem in an earlier post on this thread. If you do ps -eaf | grep quantum-dhcp, what is your output?

Hrmmm, ovs-quantum-agent takes two --config-file options, and it survives it seems. Still seems weird, but maybe not the issue.

Syd

My VM's did get IP's assigned by the quantum agent and by disabling namespaces I was able to ping them (or by switching to the correct network namespace). I tried spinning up VM's with the default network and subnet that devstack creates as well and that worked fine.

And this is what dhcp shows as running the 10.0.0.0/24 subnet:

 3180 1 0 17:24 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap76a31515-4c --except-interface=lo --domain=openstacklocal --pid-file=/opt/stack/data/dhcp/87aca91f-e5e0-4149-86e1-a9d62ab31a35/pid - dhcp-hostsfile=/opt/stack/data/dhcp/87aca91f-e5e0-4149-86e1-a9d62ab31a35/host --dhcp-optsfile=/opt/stack/data/dhcp/87aca91f-e5e0-4149-86e1-a9d62ab31a35/opts --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s

Eoghan

So, I killed the dhcp agent execution where I specified only the one --config-file arg, tried again with both args. This time, it persisted (didn't exit), and I got a bunch of dnsmasq processes as well.

Created a new net, created a subnet for 10.4.128.0/20, and then spun up a VM. And the console logs reflected that in fact it was due to dhcp agent not running:

Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending select for 10.4.128.3...
Lease of 10.4.128.3 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 10.4.128.2
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 1/30: up 7.50. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 2/30: up 11.59. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 3/30: up 14.59. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 4/30: up 18.65. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 5/30: up 22.71. request failed
wget: can't connect to remote host (169.254.169.254): No route to host
cloud-setup: failed 6/30: up 26.78. request failed

This attempt to hit up a web server at 169.254.169.254 is very weird, something I don't recall seeing previously when working with essex quantum v1/OVS.

Even weirder, now that I rebooted the host and re-ran devstack, quantum-dhcp-agent runs. And my VMs get IP addresses in the specified subnet. WTF? Did running the agent once with only one argument have some kind of side effect?

Eoghan:

Glad to hear it. My dnsmasq process is slightly different, it has one additional arg (--dhcp-script)

root 5524 5523 0 14:49 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap0af8ef80-41 --except-interface=lo --domain=openstacklocal --pid-file=/opt/stack/data/dhcp/3416e686-fc5e-4efc-b204-9f92f88e9d9c/pid --dhcp-hostsfile=/opt/stack/data/dhcp/3416e686-fc5e-4efc-b204-9f92f88e9d9c/host --dhcp-optsfile=/opt/stack/data/dhcp/3416e686-fc5e-4efc-b204-9f92f88e9d9c/opts --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.4.128.0,static,120s

Can you detail more about what you did with namespaces? Not familiar with that.

Aaron:

I'm going to scp my localrc to a safe place, reimage my blade with a fresh copy of Ubuntu 12.04, and see if I can reproduce this issue with quantum-dhcp-agent not starting. Give me a couple of hours to report back... :-)

I think it would be awesome to the community to publish a wiki page with a mostly copy-pasteable localrc for using devstack with quantum v2 and OVS. If one looks hard enough, it can be pieced together from various posts and such, but that can be error prone and tedious (as you have seen from my recent questions, I have never been sure about keystone, the need to specify v2 flag, what exactly I should use for enabled services...). Let me know if I can help with this if you agree it is worthwhile (and I hope you do :-).

Hi Syd,

I agree documentation can definitely be improved. I did not need to make any changes in order to get this working. I suspect that your devstack had fallen out of date and some how during your process you updated it (that's why things started working :) )

Those errors you are seeing in the vm is because the vm is configured to run that cloud-setup command. This is exactly what I get to. :)

Things seem to be working from what you posted.

Aaron

Aaron:

I'm suspecting that I was on the fence between a setup that was old-devstack and new, which is why I am going thru this blade reimaging exercise.

Totally encourage you to post a localrc somewhere prominent. Would buying you a beer at the openstack summit help to encourage you more? :-)

I agree that your issue was because you were using an older version of devstack. That's explains why the dhcp agent was starting with one config file instead of two.

This page seems up to date to me: http://wiki.openstack.org/QuantumDevstack (and shows localrc) . I agree it can definitely be challenging getting everything working together when you are running on master :)

Aaron

Well, a clean install of ubuntu 12.04 and still running into the same issue -- O need to launch the quantum-dhcp-agent by hand. Then things work. Boo.

I think I'll file a bug on this, my setup is too generic and my localrc is correct based on our discussions.

Just to confirm did you have q-dhcp in ENABLED_SERVICES? If so I'm not sure why stack.sh isn't starting it for you. Is there an error?

screen_it q-dhcp "sudo python $AGENT_DHCP_BINARY --config-file $Q_CONF_FILE --config-file=$Q_DHCP_CONF_FILE"

See the referenced bug report. ENABLED_SERVICES is one I copied from an earlier post you made on this thread, and from the devstack logs (attached to the bug), it sure looks like it is trying to start it.

+ screen -S stack -p q-agt -X stuff 'sudo python /opt/stack/quantum/quantum/plugins/openvswitch/agent/ovs_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini^M'
+ screen_it q-dhcp 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini'
++ echo -ne '\015'
+ NL=$'\r'
+ is_service_enabled q-dhcp
+ services=q-dhcp
+ for service in '${services}'
+ [[ ,g-api,g-reg,key,n-api,n-cpu,n-sch,n-vnc,mysql,rabbit,openstackx,q-svc,quantum,q-agt,n-cpu,q-dhcp, =~ ,q-dhcp, ]]
+ return 0
+ screen_rc q-dhcp 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini'
+ SCREENRC=/opt/stack/devstack/stack-screenrc
+ [[ ! -e /opt/stack/devstack/stack-screenrc ]]
+ grep q-dhcp /opt/stack/devstack/stack-screenrc
+ screen -S stack -X screen -t q-dhcp
+ sleep 1.5
+ [[ -n '' ]]
+ screen -S stack -p q-dhcp -X stuff 'sudo python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini^M'
+ NOVA_CONF_DIR=/etc/nova
+ [[ ! -d /etc/nova ]]
++ whoami

Syd:

To your question on namespaces, quantum creates a separate namespace for each subnet created. You can view the namespaces with "ip netns list" and the namespace id's match those of the quantum subnets.

$ quantum subnet-list
+--------------------------------------------+-------------+-----------------+-------------+------------+-------------+--------------------------------------+------------+------+--------------------------------------+----------------------------------+
| allocation_pools | cidr | dns_nameservers | enable_dhcp | gateway_ip | host_routes | id | ip_version | name | network_id | tenant_id |
+--------------------------------------------+-------------+-----------------+-------------+------------+-------------+--------------------------------------+------------+------+--------------------------------------+----------------------------------+
| {"start": "10.0.0.2", "end": "10.0.0.254"} | 10.0.0.0/24 | [] | True | 10.0.0.1 | [] | 8582c300-c86c-42d8-9fe2-8e6680815486 | 4 | | cb731bd2-ea13-4537-90a7-65efe6312c00 | 65b7bd9ca5ff4a20874d589cbe9c97b0 |
+--------------------------------------------+-------------+-----------------+-------------+------------+-------------+--------------------------------------+------------+------+--------------------------------------+----------------------------------+
$ ip netns list
cb731bd2-ea13-4537-90a7-65efe6312c00

To change network namespaces and be able to ping the VM's run this and you are placed in the subnet's namespace:

$ ip netns exec cb731bd2-ea13-4537-90a7-65efe6312c00 bash

Eoghan

Eoghan,

Thanks! I just got a 4 node blade up with quantum V2, crossing two separate VLANs managed by an intervening switch, VMs are getting IP addresses. The above hints for dealing with namespaces were the remaining issue for me. So thanks for posting, it was a big help.