MySQL Connection Timeout

Asked by kcarlson on 2011-10-23

When launching nova-compute service, connection to mysql times out. (The compute node is separate from the node running mysql)

In slow network situations, it seems that the default timeout in db_pool.py::BaseConnectionPool, line 18 (5 sec) is too low. If the default is changed to 10, the connection succeeds (in my situation).

With that said, i'm not sure this is configurable.

So, my question is, can the mysql connection timeout be configured in nova.conf? (I tried sql_idle_timeout with no affect)

Trace:
2011-10-23 21:34:37,784 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
2011-10-23 21:34:42,791 CRITICAL nova [-]
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in init_host
(nova): TRACE: instances = self.db.instance_get_all_by_host(context, self.host)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 536, in instance_get_all_by_host
(nova): TRACE: return IMPL.instance_get_all_by_host(context, host)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 101, in wrapper
(nova): TRACE: return f(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1392, in instance_get_all_by_host
(nova): TRACE: session = get_session()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 53, in get_session
(nova): TRACE: _ENGINE = get_engine()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 87, in get_engine
(nova): TRACE: creator = eventlet.db_pool.ConnectionPool(MySQLdb, **pool_args)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 50, in __init__
(nova): TRACE: order_as_stack=True)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/pools.py", line 108, in __init__
(nova): TRACE: self.free_items.append(self.create())
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 246, in create
(nova): TRACE: **self._kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 253, in connect
(nova): TRACE: conn = tpool.execute(db_module.connect, *args, **kw)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 119, in execute
(nova): TRACE: rv = e.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: ConnectTimeout

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
2011-10-23
Last reply:
2011-10-24
Jian Wen (wenjianhn) said : #1

Could you paste the output of the commands listed below?
$ cat /etc/resolv.conf
$ dig google.com | grep Query

My team leader has solved a problem like this by using a
 much faster DNS.

Kristoffer (kcarlson) said : #2

$ cat /etc/resolv.conf
nameserver 192.168.1.xxx
search mydomain.com

$ dig google.com | grep Query
;; Query time: 34 msec

Can you help with this problem?

Provide an answer of your own, or ask kcarlson for more information if necessary.

To post a message you must log in.