OpenStack Compute (nova)

network configuration

Asked by Hao Li

Hello there,

I keep getting the following error from nova-network.log no matter it is VLAN or FlatDHCP.

(nova): TRACE: Command: sudo vconfig add br100 100
(nova): TRACE: Exit code: 3
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'ERROR: trying to add VLAN #100 to IF -:br100:- error: No such device\n'

If I set up the bridge in /etc/network/interfaces, ubuntu 11.10 will go to a dead loop. See this bug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/856417/comments/8. I am also wondering how nova-network tries to create a vlan even I specify to use flas dhcp in nova.conf.

Below is my nova.conf and interface. I have spent several days on this issue and can't figure it out. Thanks for help!

============== interface ====================
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
# ALTHOUGH IT IS A DHCP, YOU CAN VIEW IT AS STATIC.
auto eth0
iface eth0 inet dhcp
# This is an autoconfigured IPv6 interface
iface eth0 inet6 auto

auto eth1
iface eth1 inet static
address 192.168.3.1
netmask 255.255.255.0
network 192.168.3.0
broadcast 192.168.3.255

============== nova.conf for flat dhcp ===================
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--use_deprecated_auth
--verbose
--libvirt_type=qemu
--ec2_host=10.0.0.2
--osapi_host=10.0.0.2
--s3_host=10.0.0.2
--rabbit_host=10.0.0.2
--nova_url=http://10.0.0.2:8774/v1.1/
--ec2_url=http://10.0.0.2:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=8
--glance_api_servers=10.0.0.2:9292
--glance_host=10.0.0.2
--routing_source_ip=10.0.0.2
--sql_connection=mysql://novadbadmin:novadbadmin@10.0.0.2/nova
--image_service=nova.image.glance.GlanceImageService
--network_manager=nova.network.manager.FlatDHCPManager
--flat_network_dhcp_start=192.168.0.2
--public_interface=eth0
--flat_interface=eth1
--flat_injected=False

============== nova.conf for vlan ===================
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--use_deprecated_auth
--verbose
--libvirt_type=qemu
--ec2_host=10.0.0.2
--osapi_host=10.0.0.2
--s3_host=10.0.0.2
--rabbit_host=10.0.0.2
--nova_url=http://10.0.0.2:8774/v1.1/
--ec2_url=http://10.0.0.2:8773/services/Cloud
--fixed_range=192.168.0.0/16
--network_size=8
--glance_api_servers=10.0.0.2:9292
--glance_host=10.0.0.2
--routing_source_ip=10.0.0.2
--sql_connection=mysql://novadbadmin:novadbadmin@10.0.0.2/nova
--image_service=nova.image.glance.GlanceImageService
--public_interface=eth0
--vlan_interface=eth1

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
james feister (openjaf) said : 		#1

Have the same issue

euca-describe-instances always returns with my instance 'pending test'

>> INSTANCE i-00000001 ami-00000002 192.168.4.3 192.168.4.3 pending test (proj, skynet) 0 m1.tiny 2011-12-15T20:44:26Z nova aki-00000001 ami-00000000

Looking throught the nova-network.log shows a trance ending with
(nova): TRACE: Exit code: 3
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'ERROR: trying to add VLAN #100 to IF -:br100:- error: No such device\n'

if I list the bridges it shows a virbr0 not br100. Changing the nova.conf to use virbr0 seems not to work as the nova service will restart still looking for the br100 even after a reboot. But would changing this in the database be the appropriate solution or is something not working properly with nova-networking

Revision history for this message
lgs (luca-soave) said : 		#2

I resolved same issue by setting the correct routing into /etc/nova/nova.conf.

In my case I've got the following :

root@node105:~# netstat -rn|grep UG
0.0.0.0 10.246.61.65 0.0.0.0 UG 0 0 0 eth0
root@node105:~#

root@node105:~# grep routing /etc/nova/nova.conf
--routing_source_ip=10.246.61.65
root@node105:~#

than it works.

bye
Luca G. Soave

Revision history for this message
james feister (openjaf) said : 		#3

Tried proposed solution, nova-network will fail with the following dump:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 336, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
    server.start()
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
    self.manager.init_host()
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1008, in init_host
    NetworkManager.init_host(self)
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 373, in init_host
    self._setup_network(ctxt, network)
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1088, in _setup_network
    dev = self.driver.plug(network_ref, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 831, in plug
    return interface_driver.plug(network, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 869, in plug
    mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 889, in ensure_vlan_bridge
    bridge_interface, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 688, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 903, in ensure_vlan
    vlan_num, run_as_root=True)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 745, in _execute
    return utils.execute(*cmd, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 191, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo vconfig add br100 100
Exit code: 3
Stdout: ''
Stderr: 'ERROR: trying to add VLAN #100 to IF -:br100:- error: No such device\n'

Removing the change and creating the br100 bridge with brctl will give the following error.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 336, in fire_timers
    timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
    server.start()
  File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
    self.manager.init_host()
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1008, in init_host
    NetworkManager.init_host(self)
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 373, in init_host
    self._setup_network(ctxt, network)
  File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1088, in _setup_network
    dev = self.driver.plug(network_ref, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 831, in plug
    return interface_driver.plug(network, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 869, in plug
    mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 889, in ensure_vlan_bridge
    bridge_interface, mac_address)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 688, in inner
    retval = f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 909, in ensure_vlan
    _execute('ip', 'link', 'set', interface, 'up', run_as_root=True)
  File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 745, in _execute
    return utils.execute(*cmd, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 191, in execute
    cmd=' '.join(cmd))
ProcessExecutionError: Unexpected error while running command.
Command: sudo ip link set vlan100 up
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: Network is down\n'

That being the case I checked the interface with ifconfig and br100 is down. Bringing up the br100 interface with 'ifconfig br100 up' completely locks up the box' The only way out is to perform a force shutdown by killing the power, which has already caused me problems. This is the second rebuild I have performed.

This is following the 'Openstack Compute starter guide' for the Diablo release, taking account of the user comments on errors for the glance ip addresses.

Any help is appreciated.

James

Revision history for this message
Brian Brown (brianwbrown) said : 		#4

I have the same problem. Nova-network will not start with 2 instances pending. I tried the suggestion in post #2. All that did was crash nova-scheduler & nova-volume in addition to nova-network.

euca-terminate-instances i-0000000X

will not remove the instance when nova-network is down. Is the any way to force the deletion of instances?

-------------------------
2012-02-02 20:00:25,076 CRITICAL nova [-] Unexpected error while running command.
Command: sudo vconfig add br100 100
Exit code: 3
Stdout: ''
Stderr: 'ERROR: trying to add VLAN #100 to IF -:br100:- error: No such device\n'
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-network", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1008, in init_host
(nova): TRACE: NetworkManager.init_host(self)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 373, in init_host
(nova): TRACE: self._setup_network(ctxt, network)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1088, in _setup_network
(nova): TRACE: dev = self.driver.plug(network_ref, mac_address)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 831, in plug
(nova): TRACE: return interface_driver.plug(network, mac_address)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 869, in plug
(nova): TRACE: mac_address)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 889, in ensure_vlan_bridge
(nova): TRACE: bridge_interface, mac_address)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 688, in inner
(nova): TRACE: retval = f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 903, in ensure_vlan
(nova): TRACE: vlan_num, run_as_root=True)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 745, in _execute
(nova): TRACE: return utils.execute(*cmd, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 191, in execute
(nova): TRACE: cmd=' '.join(cmd))
(nova): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova): TRACE: Command: sudo vconfig add br100 100
(nova): TRACE: Exit code: 3
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'ERROR: trying to add VLAN #100 to IF -:br100:- error: No such device\n'
(nova): TRACE:

Revision history for this message
Vish Ishaya (vishvananda) said : 		#5

The problem here is that you have br100 set as the bridge_interface in the networks table. This is incorrect. You should set --vlan_interface=eth0 and modify bridge_inteface in your networks table to eth0 as well

Can you help with this problem?

Provide an answer of your own, or ask Hao Li for more information if necessary.

To post a message you must log in.