neutron

neutron unit tests never complete

Asked by Bhuvan Arumugam

$ tox -e py26

When I run unit tests it hang forever and continue to run beyond 24hrs. The tests pass, but test suite never complete. The tests are quick until around 2000 tests. The test_basic tests seem to run forever. The testr logs for last 50 tests are here. It complain different error for each test, though it's passed:
http://pastebin.com/djcGnpP3

This is the case even if we use py27 (tox -e py27). This is the case even if we run ./run_test.sh script manually. In both cases testr with venv is used.

However if I run test_basic tests separately, it is successful and completed within 2mins.
testr list-tests | grep test_basic > test_basic
./run_tests.sh -P -V -- --load-list=basic-tests

Does anyone face this problem? Is it specific to running testr in parallel mode? Should I run testr in serial mode?

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Bhuvan Arumugam (bhuvan) said : 		#1

Everytime it hang, it spit the following error message. The tests PASS though.

Traceback (most recent call last):
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/plugins/linuxbridge/agent/linuxbridge_neutron_agent.py", line 759, in _report_state
    self.agent_state)
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/agent/rpc.py", line 74, in report_state
    return self.cast(context, msg, topic=self.topic)
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/openstack/common/rpc/proxy.py", line 171, in cast
    rpc.cast(context, self._get_topic(topic), msg)
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/openstack/common/rpc/__init__.py", line 158, in cast
    return _get_impl().cast(CONF, context, topic, msg)
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/openstack/common/rpc/impl_fake.py", line 166, in cast
    check_serialize(msg)
  File "/home/jenkins/workspace/csi-neutron-upstream/neutron/openstack/common/rpc/impl_fake.py", line 131, in check_serialize
    json.dumps(msg)
  File "/usr/lib64/python2.6/json/__init__.py", line 230, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.6/json/encoder.py", line 367, in encode
    chunks = list(self.iterencode(o))
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 309, in _iterencode
    for chunk in self._iterencode_dict(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 275, in _iterencode_dict
    for chunk in self._iterencode(value, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 317, in _iterencode
    for chunk in self._iterencode_default(o, markers):
  File "/usr/lib64/python2.6/json/encoder.py", line 323, in _iterencode_default
    newobj = self.default(o)
  File "/usr/lib64/python2.6/json/encoder.py", line 344, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <MagicMock name='LinuxBridgeManager().local_ip' id='666599248'> is not JSON serializable
ERROR:neutron.plugins.linuxbridge.agent.linuxbridge_neutron_agent:Failed reporting state!

Revision history for this message
ZhiQiang Fan (aji-zqfan) said : 		#2

there is a bug report https://bugs.launchpad.net/neutron/+bug/1232155

if you're urgent to pass ut, i think you can add mock.patch for neutron.openstack.common.loopingcall.FixedIntervalLoopingCall in neutron/tests/unit/linuxbridge/test_lb_neutron_agent.py: class TestLinuxBridgeAgent.setUp

the root cause is TestLinuxBridgeAgent.test_update_devices_failed() create a LinuxBridgeNeutronAgentRPC agent but with LinuxBridgeManager mocked, the agent will report state periodically (because it is handled via FixedIntervalLoopingCall) with configuration, which needs LinuxBridgeManager.local_ip attr, as that object is mocked, json cannot serialize it

you can catch up with bug/1232155 for the real patch

good luck

Can you help with this problem?

Provide an answer of your own, or ask Bhuvan Arumugam for more information if necessary.

To post a message you must log in.