[Date Prev][Date Next] [Chronological] [Thread] [Top]

Too many filehandles


I'm looking into a problem we're getting on our production master LDAP

There are around 10 servers connecting rapidly and repeatedly with lots
of short LDAP read only queries. 

When operating at a sustained level of around 77 binds and 300 searches
per second (peeking at 140 binds and 500 searchs per sec) we're seeing
lots of logs like this:

Jun 14 09:28:53 ldap-pna slapd[1756]: warning: cannot
open /etc/hosts.allow: Too many open files
Jun 14 09:28:53 ldap-pna slapd[1756]: error: bad option name:

where is one of the servers making repeated connections.

repeated every few seconds thru /var/log/messages

Curiously, we don't appear to be hitting any system limits, and ulimit
for the ldap user is unlimited.

# lsof | grep slapd | wc
    419    3788   60393

far less than files-nr 

# cat /proc/sys/fs/file-nr 
1800    0       430000

you can also see file-max is set to 430000.

I've followed the advice on :
http://www.openldap.org/faq/data/cache/1126.html and rebuild 2.2.27 with
CPPFLAGS set to 8192 to no avail. I know the setting took because I
deliberately broke it and broke the build process. If there's some way I
can really tell it took, I'd be grateful for the info.

Operating system is Fedora Core 3 for x86_64 + updates, kernel
2.6.10-1.770_FC3smp with bespoke OpenLDAP rpm based from Fedora Core 4
rebuilt with 2.2.27 + DB 4.2.52 + patches. The system is a Sun dual
Opteron SunFire 20z.

Load is getting up to 5 or 6, but right now is less than 2. The server
looks and feels like it can take double this load and probably more so.

I'm not quite sure what else to try. We're working on rerouting load
temporarily to LDAP consumers. It's only a transitional thing as we're
migrating systems, but would like to get to the bottom of this as it has
scalability concerns. We'll probably look at installing overlays on the
servers too to limit ro queries.

Best regards,

Rob Fielding

www.dsvr.co.uk                                             Development
Designer Servers                                    Business Serve Plc

Attachment: signature.asc
Description: This is a digitally signed message part