"Too many open files" problem with mysql-proxy 0.8.1

I think that the mysql proxy has an option to also increase the number
of open files it can handle.
check --help or --proxy-help or --help-proxy (I can;t remember the
right command to get the help)

Regards,

  Diego

On Sun, Jul 15, 2012 at 8:15 AM, Todd S
<email address hidden> wrote:
> New question #203228 on MySQL Proxy:
> https://answers.launchpad.net/mysql-proxy/+question/203228
>
> Once a day or so I'll get a block of this in the mysql-proxy log:
>
> 2012-07-14 19:19:44: (critical) network-socket.c.311: socket(10.10.31.150:3306) failed: Too many open files (24)
> 2012-07-14 19:19:44: (message) proxy-plugin.c.1634: connecting to backend (10.10.31.150:3306) failed, marking it as down for ...
> 2012-07-14 19:19:44: (critical) proxy-plugin.c.1612: Cannot connect, all backends are down.
> 2012-07-14 19:19:44: (critical) (lua-error) [/usr/local/lib/mysql-proxy/lua/proxy/mysql-proxy-cache.lua]
> /usr/local/lib/mysql-proxy/lua/Memcached.lua:401: No servers available
> 2012-07-14 19:19:45: (critical) last message repeated 1 times
> 2012-07-14 19:19:45: (critical) proxy-plugin.c.1612: Cannot connect, all backends are down.
> 2012-07-14 20:40:46: (critical) last message repeated 1 times
> 2012-07-14 20:40:46: (critical) network-socket.c.311: socket(10.10.31.150:3306) failed: Too many open files (24)
> 2012-07-14 20:40:46: (message) proxy-plugin.c.1634: connecting to backend (10.10.31.150:3306) failed, marking it as down for ...
> 2012-07-14 20:40:46: (critical) proxy-plugin.c.1612: Cannot connect, all backends are down.
> 2012-07-14 20:58:26: (critical) last message repeated 2 times
> 2012-07-14 20:58:26: (critical) network-socket.c.311: socket(10.10.31.150:3306) failed: Too many open files (24)
> 2012-07-14 20:58:26: (message) proxy-plugin.c.1634: connecting to backend (10.10.31.150:3306) failed, marking it as down for ...
> 2012-07-14 20:58:26: (critical) proxy-plugin.c.1612: Cannot connect, all backends are down.
>
> Sometimes it resolves itself by shutting down and restarting, other times I have to go in and restart it manually. Of course, there are front-end (website) ramifications while this is going on.
>
> We're using mysql-proxy as an extended middleware cache - no connection pooling and no load balancing of servers. When we remove the proxy the front end errors go away but performance is horrible, so it's our preference to keep something in the middle and mysql-proxy is preferred to other methods reviewed (moving caching into mysql or rails).
>
> In mysql I have innodb_open_files set to 500 which seems more than adequate for our purposes. The server itself is running on an AWS instance with 15GB of RAM, Ubuntu 10.04.3 LTS (kernel version 2.6.32-309-ec2) and mysql 5.5.8.
>
> Any insights, tips etc are most welcome - I'm fresh out of ideas and this is making me crazy.
>
> Thanks!
>
> Todd
>
> --
> You received this question notification because you are a member of
> MySQL Proxy Discuss, which is an answer contact for MySQL Proxy.

--
Diego Medina
Lift/Scala Developer
<email address hidden>
http://www.fmpwizard.com

I found this option:

--max-open-files

which defaults to (ulimit -n) which in this server's case is 1024. That, however, is shared between 2 running proxy instances.

I suppose it would be possible to bump the OS's file limit and give it a try.

On 20.07.2012 15:06, Todd S wrote:
> Question #203228 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/203228
>
> Todd S posted a new comment:
> I found this option:
>
> --max-open-files
>
> which defaults to (ulimit -n) which in this server's case is 1024. That,
> however, is shared between 2 running proxy instances.

It is a per process limit which can be increased if you are starting the
proxy as root.

> I suppose it would be possible to bump the OS's file limit and give it a
> try.

Jan

They are running out of upstart as root, so I should be able to add

ulimit -n xxxx

to the upstart script, and

max-files-open=xxxx

to the mysql-proxy config file? Or does it have to be --max-files-open on the command line?

Thanks!
Todd

On 20.07.2012 16:21, Todd S wrote:
> Question #203228 on MySQL Proxy changed:
> https://answers.launchpad.net/mysql-proxy/+question/203228
>
> Todd S posted a new comment:
> They are running out of upstart as root, so I should be able to add
>
> ulimit -n xxxx
>
> to the upstart script, and
>
> max-files-open=xxxx
>
> to the mysql-proxy config file? Or does it have to be --max-files-open
> on the command line?

Both do the same thing. You only need to do one of them.

> Thanks!
> Todd

Jan

At the risk of jinxing the situation... :) there hasn't been a glitch since the 22nd when I raised the file limit on both proxy processes to 4096. If it continues to hold for a week I'll mark Jan's response as a definitive answer but at this point I'm fairly confident.

Thanks!
  Todd