"Too Many Open Files" - Solve It Easy!
Application errors can sometimes be very frustrating to solve, more so in a situation where you cannot pinpoint its source. Below, you'll discover a few simple tips and tricks for resolving one error that isn't very frequent yet a show stopper when it does strike.
"Too many open files"
Sound familiar? If you are an application developer, you may break into a sweat at the very mention of this error. Most of applications, at some point in their life cycles – be it in development or testing or production – have come across this error. Some might argue about its frequency; but as an application developer, you would want to avoid this error at any cost.
- sudden surge in app usage caused by authorized users
- a malicious hacker attack on the application trying to bring it down
- failure of some app servers in a distributed deployment leading to overload in the remaining app servers
3. Trying to connect to a non-existing host for an external service. For example, if the hostname url to connect to the external service is maintained in a database table and it has multiple entries for existing and non-existing hostnames, then the application will try to connect to all the entries in the database, thus ending up in connecting to a non-existing host, resulting in open files piling up.
The usual tendency to overcome this error is to increase the open files limit at the system level. But that only defers the problem. Below are the various items that an application developer should check do to resolve the error.
- Configure the external services maximum open files or connections to the limit suggested by the service.
- Look for any redundant entries in the configuration file or database where the external service connection url is configured.
- If load is genuine, add more app servers.
- Use third party agents like http://file-leak-detector.kohsuke.org/ to detect open resources and to find the trace of the source that caused the file leak.
We had once faced such an issue in one of our test beds by having wrong entries connecting to the memcache server and had used this file leak detector tool to detect and resolve the error. We now are planning to add this feature to APM Insight. We shall keep this space updated once the feature is in place.
So the next time you face "Too many open files" error, you know what to do. Just follow these simple tips and don't lose your sleep over it.