Re: slapd runs out of open files (ITS#2813)

[andrew.findlay@skills-1st.co.uk]

On Wed, Nov 05, 2003 at 10:16:53PM +0000, ragnar@skolelinux.no wrote:

> In an installation using the Debian Woody derivative Skolelinux we have once
> seen slapd report in the syslog messages similar to this:
> 
> tjener slapd[353]: warning: cannot open /etc/hosts.deny: Too many open files
> 
> Other files were also mentioned. We checked that this was not a file-max/file-nr
> issue. Until slapd was restarted no users could log in. Restarting slapd cured
> the problem. This situation has only occurred once. The system has about 800 -
> 1000 users, with typically around 500 active users being authenticated by the
> OpenLDAP server at a time when the problem occured. OpenLDAP is used mainly for
> authentication of Linux and Windows (Samba 2) users.

Remember that each active user will keep *at least* one connection
open to the LDAP server. The actual number depends on the client
systems: I tested a few in Jan 2002 and the results are in the
'Performance' section of my paper on Security with LDAP:

http://www.skills-1st.co.uk/papers/security-with-ldap-jan-2002/security-with-ldap.html

Thus, if you have 500 active users you are likely to have a base load
of 500 open connections plus a variable load that might peak almost
twice as high during busy periods. Each connection needs a file
descriptor so you could easily run into limits imposed by the
operating system. The 'ulimit -a' command should show you the default
limits - e.g. on my Linux 2.4.18 system the default limit on open files is
1024.

Next time you hit the problem it would be worth using 'lsof' to find
out what files the slapd process actually has open. This may well give
a clue to the problem (e.g. if file descriptors are being leaked, or
if some clients are opening vast numbers of connections).

Andrew
-- 
-----------------------------------------------------------------------
|                 From Andrew Findlay, Skills 1st Ltd                 |
| Consultant in large-scale systems, networks, and directory services |
|     http://www.skills-1st.co.uk/                +44 1628 782565     |
-----------------------------------------------------------------------