OpenStack Compute (nova)

Error in network tests when running dhcpbridge with fake option

Asked by Salvatore Orlando on 2011-03-01

6 out of 10 tests in nova.tests.test_network are failing on my dev machine.
I tried with fresh code from trunk, so I guess there must be a problem with the setup on my machine.

NetworkTestCase
    test_allocate_deallocate_fixed_ip ERROR
    test_available_ips OK
    test_ips_are_reused ERROR
    test_private_ipv6 OK
    test_public_network_association ERROR
    test_side_effects ERROR
    test_subnet_edge ERROR
    test_too_many_addresses ERROR
    test_too_many_networks OK
    test_vpn_ip_and_port_looks_valid OK

Both with and without venv it looks like the problem lies with nova-dhcpbridge, as it is trying to connect with mysql (I use it as a db for my development). I think the db connection should be faked in tests, so I don't understand why it is trying to connect to mysql.

With venv I get the following:

Without venv I get the following:

Traceback (most recent call last):
  File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/tests/test_network.py", line 306, in test_too_many_addresses
    lease_ip(address)
  File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/tests/test_network.py", line 352, in lease_ip
    (out, err) = utils.execute(cmd, addl_env=env)
  File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/utils.py", line 151, in execute
    cmd=cmd)
ProcessExecutionError: Unexpected error while running command.
Command: /home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge add 02:16:3e:32:68:5d 10.0.0.3 fake
Exit code: 1
Stdout: ''
Stderr: '2011-03-01 10:50:33,382 DEBUG nova.dhcpbridge [-] TESTING!!!! from (pid=8692) main /home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge:108\n2011-03-01 10:50:33,396 DEBUG nova.dhcpbridge [-] Called add for mac 02:16:3e:32:68:5d with ip 10.0.0.3 and hostname fake on interface br100 from (pid=8692) main /home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge:117\n2011-03-01 10:50:33,396 DEBUG nova.dhcpbridge [-] leasing ip from (pid=8692) add_lease /home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge:58\n2011-03-01 10:50:33,396 DEBUG nova.network.manager [E0C3P9RLAQZ-5V98RUZB None None] Leasing IP 10.0.0.3 from (pid=8692) lease_fixed_ip /home/salvatore/workspace/Openstack-xenapi-netinject/nova/network/manager.py:223\n2011-03-01 10:50:33,516 DEBUG nova.utils [-] backend <module \'nova.db.sqlalchemy.api\' from \'/home/salvatore/workspace/Openstack-xenapi-netinject/nova/db/sqlalchemy/api.pyc\'> from (pid=8692) __get_backend /home/salvatore/workspace/Openstack-xenapi-netinject/nova/utils.py:376\n2011-03-01 10:50:36,580 CRITICAL nova [-] (OperationalError) (2003, "Can\'t connect to MySQL server on \'10.70.177.19\' (113)") None None\n(nova): TRACE: Traceback (most recent call last):\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge", line 123, in <module>\n(nova): TRACE: main()\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge", line 118, in main\n(nova): TRACE: globals()[action + \'_lease\'](mac, ip, hostname, interface)\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/bin/nova-dhcpbridge", line 62, in add_lease\n(nova): TRACE: ip_address)\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/network/manager.py", line 224, in lease_fixed_ip\n(nova): TRACE: fixed_ip_ref = self.db.fixed_ip_get_by_address(context, address)\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/db/api.py", line 298, in fixed_ip_get_by_address\n(nova): TRACE: return IMPL.fixed_ip_get_by_address(context, address)\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/db/sqlalchemy/api.py", line 108, in wrapper\n(nova): TRACE: return f(*args, **kwargs)\n(nova): TRACE: File "/home/salvatore/workspace/Openstack-xenapi-netinject/nova/db/sqlalchemy/api.py", line 602, in fixed_ip_get_by_address\n(nova): TRACE: options(joinedload(\'instance\')).\\\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py", line 1496, in first\n(nova): TRACE: ret = list(self[0:1])\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py", line 1405, in __getitem__\n(nova): TRACE: return list(res)\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py", line 1565, in __iter__\n(nova): TRACE: return self._execute_and_instances(context)\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py", line 1570, in _execute_and_instances\n(nova): TRACE: mapper=self._mapper_zero_or_none())\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py", line 734, in execute\n(nova): TRACE: return self._connection_for_bind(engine, close_with_result=True).execute(\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py", line 701, in _connection_for_bind\n(nova): TRACE: return engine.contextual_connect(**kwargs)\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/base.py", line 1742, in contextual_connect\n(nova): TRACE: self.pool.connect(),\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 158, in connect\n(nova): TRACE: return _ConnectionFairy(self).checkout()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 329, in __init__\n(nova): TRACE: rec = self._connection_record = pool.get()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 177, in get\n(nova): TRACE: return self.do_get()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 692, in do_get\n(nova): TRACE: con = self.create_connection()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 138, in create_connection\n(nova): TRACE: return _ConnectionRecord(self)\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 213, in __init__\n(nova): TRACE: self.connection = self.__connect()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/pool.py", line 279, in __connect\n(nova): TRACE: connection = self.__pool._creator()\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/strategies.py", line 76, in connect\n(nova): TRACE: return dialect.connect(*cargs, **cparams)\n(nova): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/engine/default.py", line 238, in connect\n(nova): TRACE: return self.dbapi.connect(*cargs, **cparams)\n(nova): TRACE: File "/usr/lib/pymodules/python2.6/MySQLdb/__init__.py", line 81, in Connect\n(nova): TRACE: return Connection(*args, **kwargs)\n(nova): TRACE: File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 170, in __init__\n(nova): TRACE: super(Connection, self).__init__(*args, **kwargs2)\n(nova): TRACE: OperationalError: (OperationalError) (2003, "Can\'t connect to MySQL server on \'10.70.177.19\' (113)") None None\n(nova): TRACE: \n'

The "TESTING!!!" string in nova-dhcpbridge output proves the TESTING environment variable is set, and therefore nova-dhcpbridge should be loading the fake flags.
Any help?

Question information

Language:: English Edit question

Status:: Solved

For:: OpenStack Compute (nova) Edit question

Assignee:: No assignee Edit question

Solved by:: Salvatore Orlando

Solved:: 2011-03-01

Last query:: 2011-03-01

Last reply:: 2011-03-01

Link existing bug

Revision history for this message

Salvatore Orlando (salvatore-orlando) said on 2011-03-01:

I fixed it by removing by the --mysql_connection flag in /etc/nova/nova-dhcpbridge.conf
However, another possible way of sorting this out (in my opinion) is to prevent nova-dhcpbridge from reading the flagfile when it is being ran for tests. Something like the following will do:

def main():
    """Parse environment and arguments and call the approproate action."""
    interface = os.environ.get('DNSMASQ_INTERFACE', 'br0')
    argv = FLAGS(sys.argv)
    if int(os.environ.get('TESTING', '0')):
        LOG.debug("TESTING!!!!")
        from nova.tests import fake_flags
    else:
        flagfile = os.environ.get('FLAGFILE', FLAGS.dhcpbridge_flagfile)
        utils.default_flagfile(flagfile)
    logging.setup()
    action = argv[1]
    if action in ['add', 'del', 'old']:
        mac = argv[2]
        ip = argv[3]
        hostname = argv[4]
        msg = _("Called %(action)s for mac %(mac)s with ip %(ip)s and"
                " hostname %(hostname)s on interface %(interface)s") % locals()
        LOG.debug(msg)
        globals()[action + '_lease'](mac, ip, hostname, interface)
    else:
        print init_leases(interface)

Revision history for this message

Vish Ishaya (vishvananda) said on 2011-03-01:

This makes sense to me. The dhcpbridge flagfile shouldn't need to be read during tests.

Vish

On Mar 1, 2011, at 3:15 AM, Salvatore Orlando wrote:

> Question #147352 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/147352
>
> Salvatore Orlando posted a new comment:
> I fixed it by removing by the --mysql_connection flag in /etc/nova/nova-dhcpbridge.conf
> However, another possible way of sorting this out (in my opinion) is to prevent nova-dhcpbridge from reading the flagfile when it is being ran for tests. Something like the following will do:
>
> def main():
> """Parse environment and arguments and call the approproate action."""
> interface = os.environ.get('DNSMASQ_INTERFACE', 'br0')
> argv = FLAGS(sys.argv)
> if int(os.environ.get('TESTING', '0')):
> LOG.debug("TESTING!!!!")
> from nova.tests import fake_flags
> else:
> flagfile = os.environ.get('FLAGFILE', FLAGS.dhcpbridge_flagfile)
> utils.default_flagfile(flagfile)
> logging.setup()
> action = argv[1]
> if action in ['add', 'del', 'old']:
> mac = argv[2]
> ip = argv[3]
> hostname = argv[4]
> msg = _("Called %(action)s for mac %(mac)s with ip %(ip)s and"
> " hostname %(hostname)s on interface %(interface)s") % locals()
> LOG.debug(msg)
> globals()[action + '_lease'](mac, ip, hostname, interface)
> else:
> print init_leases(interface)
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Revision history for this message

Salvatore Orlando (salvatore-orlando) said on 2011-03-01:

Thanks, Vish.

I'll file a bug and attach a patch for this.

Salvatore

To post a message you must log in.

Ask a question

Edit question

OpenStack Compute (nova)

Error in network tests when running dhcpbridge with fake option

Question information

Related bugs

Related FAQ:

Subscribers