OpenStack Object Storage (Swift)

system hang with swift1.7.4 & syslog-ng

Asked by jinkyung hwang on 2012-11-26

Swift folks

Recently we upgrade system from 1.4.2 to 1.7.4 release package.

We experienced the Swift object server failure (server hang) with the system,
The faiilure message shows that it is related with syslog-ng.

The observed situation is
- (On systems with no service traffic) 'Number of allowed concurrent connections exceeded' messages are increasing and in a two days, the 'Too many open files' happens and finally the system hanged. (not able to ssh, nor ping) (below messages)
- (Upon googling, too many open files seems related to ulimit) we increase the ulimit # and syslog-ng tcp connection # enough, but system hang eventually.
- We also observed that the # of sockets of lsof that the ‘swift’ user use continuously increase, and it seems it reaches the eventual system hang. Is there any known error/bugs about socket leaks in swift 1.7.4 ?

---- below messages -----------------------
root@mnode02:~# knife ssh roles:snode "cat /var/log/syslog | grep 'Too many open files' | head -n1"
obj-r2-1.ktswift-dev.local Nov 4 10:15:01 obj-r2-1.ktswift-dev.local syslog-ng[2207]: Error accepting new connection; error='Too many open files (24)'
obj-r1-2.ktswift-dev.local Nov 4 10:19:31 obj-r1-2.ktswift-dev.local syslog-ng[2179]: Error accepting new connection; error='Too many open files (24)'
obj-r1-1.ktswift-dev.local Nov 4 10:10:01 obj-r1-1.ktswift-dev.local syslog-ng[2244]: Error accepting new connection; error='Too many open files (24)'
obj-r2-2.ktswift-dev.local Nov 4 10:20:01 obj-r2-2.ktswift-dev.local syslog-ng[2185]: Error accepting new connection; error='Too many open files (24)'
obj-r1-3.ktswift-dev.local Nov 4 10:15:01 obj-r1-3.ktswift-dev.local syslog-ng[2205]: Error accepting new connection; error='Too many open files (24)'

root@mnode02:~# knife ssh roles:snode "cat /var/log/syslog | grep 'Number of allowed concurrent connections exceeded' | head -n1"
obj-r1-2.ktswift-dev.local Nov 5 10:04:26 obj-r1-2.ktswift-dev.local syslog-ng[2197]: Number of allowed concurrent connections exceeded; num='256', max='256'
obj-r1-1.ktswift-dev.local Nov 5 10:04:01 obj-r1-1.ktswift-dev.local syslog-ng[2243]: Number of allowed concurrent connections exceeded; num='256', max='256'
obj-r2-1.ktswift-dev.local Nov 5 10:03:35 obj-r2-1.ktswift-dev.local syslog-ng[2179]: Number of allowed concurrent connections exceeded; num='256', max='256'
obj-r2-2.ktswift-dev.local Nov 5 10:04:00 obj-r2-2.ktswift-dev.local syslog-ng[2219]: Number of allowed concurrent connections exceeded; num='256', max='256'
obj-r1-3.ktswift-dev.local Nov 5 10:03:23 obj-r1-3.ktswift-dev.local syslog-ng[2189]: Number of allowed concurrent connections exceeded; num='256', max='256'

It would be appreciated if you have any similar experiences and solutions.

Thanks,

Jinkyung Hwang

Question information

Language:
English Edit question
Status:
Expired
For:
OpenStack Object Storage (swift) Edit question
Assignee:
No assignee Edit question
Last query:
2012-11-26
Last reply:
2012-12-11
Launchpad Janitor (janitor) said : #1

This question was expired because it remained in the 'Open' state without activity for the last 15 days.