Re: (ITS#5860) slapd memeory leak under openldap 2.4

[rlvcosta@yahoo.com]

Quanah,

I follow your recommendation and compiled the Berkeley Db 4.7.25NC and openldap 2.4.13.

After that load my LDIF file to have 1million entraces in my DB just to test the dncachesize control.

During my initialization I have :

module back_bdb.la: null module registered
line 29 (moduleload     back_monitor.la)
loaded module back_monitor.la
module back_monitor.la: null module registered
line 74 (database       bdb)
line 75 (suffix          "ou=CONTENT,o=alcatel,c=fr")
line 76 (rootdn          "cn=admin,ou=CONTENT,o=alcatel,c=fr")
line 80 (rootpw ***)
line 86 (directory       /var/openldap-data/bdb1)
line 89 (index userid,pnnumber,submxid,maillogin,abookaliasid eq,pres)
index uid 0x0006
index pnnumber 0x0006
index submxid 0x0006
index maillogin 0x0006
index abookaliasid 0x0006
line 90 (index objectClass eq)
index objectClass 0x0004
line 93 (cachesize      1000)
line 94 (idlcachesize   1000)
line 95 (dncachesize    1000)
line 105 (database        bdb)
line 106 (suffix          "ou=INDEXES,o=alcatel,c=fr")
line 107 (rootdn          "cn=admin,ou=INDEXES,o=alcatel,c=fr")
line 111 (rootpw ***)
line 114 (directory       /var/openldap-data/bdb2)
line 117 (index weblogin,maillogin,mail,pnnumber eq,pres)
index weblogin 0x0006
index maillogin 0x0006
index mail 0x0006
index pnnumber 0x0006
line 118 (index profileid,loginname eq,pres)
index profileid 0x0006
index loginname 0x0006
line 119 (index objectClass eq)
index objectClass 0x0004
line 122 (cachesize       1000)
line 123 (idlcachesize    1000)
line 124 (dncachesize   1000)
line 129 (database        monitor)
slapd starting

See that I have 2 DBs and in each DB area in slapd.conf, read without error by slapd as seen above, I have the cachesize, idlcachesize and dncachesize specified with small values just to monitor the memory usage by slapd.

Unfortunately even with this configuration I still having slapd consuming memory without release it. So if a new entrance is queried by some LDAP client, slapd will consume more memory. Since my DB dn's are bigger than 3GB then soon or later the slapd will crash.

See my DB information :

7.2G    dn2id.bdb
12G     id2entry.bdb
110M    maillogin.bdb
3.3M    objectClass.bdb
108M    pnnumber.bdb
1.3M    submxid.bdb
323M    uid.bdb

See my dn2id is much bigger than yours and maybe you just did not saw this issue happenig since your dn can be loaded in your memory and you in a 64 bits environment can use more than 3GB per process.

But in any case looks like there is a problem in slapd that do not respect the cache limitations imposed.

In this way we cannot use slapd for large databases. Please see my configuration above and let me know if a made a mistake(I hope so) since dncachesize is not documented at man pages.

Just as example, with the DB I loaded for tests with 1 million entrances, with the cache sizes definitions as in the slapd.conf described above and making a ldapsearch on all entrances from the CONTENT DB, in the end I have :

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3189 ldap      15   0  842m 724m  67m S   99  6.1   7:00.48 slapd

See the slapd process already consumed 842m and I just read 1 million entrances, even dncachesize is defined to be 1000.

I do not see anyway to control the memory usage by slapd.

Is this "disrespect" to slapd.conf directives a possible problem?

I hope there is a configuration problem with my slapd.conf where this could be work around.

Regards,

Rodrigo.

--- On Wed, 1/21/09, Quanah Gibson-Mount <quanah@zimbra.com> wrote:

> From: Quanah Gibson-Mount <quanah@zimbra.com>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com, openldap-its@openldap.org
> Date: Wednesday, January 21, 2009, 8:30 PM
> --On Wednesday, January 21, 2009 12:49 PM -0800 Rodrigo
> Costa <rlvcosta@yahoo.com> wrote:
> 
> > Quanah,
> > 
> > That's is exactly the point. My system is a 32
> bits and with 64 bits if
> > there are enough physical memory available even slapd
> cache grows without
> > never release it will be difficult(or very long) to
> cache enough
> > entrances before system consumes all memory.
> > 
> > With the 32 bits 3GB maximum, depend on OS too, this
> issue appears more
> > often. My impression is if you ldapsearch all DB in a
> 32 bits HW/OS slapd
> > will consume all possible memory unless to keep
> boundaries for its
> > cache(erase and overwrite).
> 
> As I noted, in 2.3, you cannot constrain the DN cachesize. 
> You can do this with OL 2.4.  You need to try a current
> RE2.4 release, and set the dncachesize.  There was no dn
> cache at all in OL 2.1.
> 
> --Quanah
> 
> --
> 
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and
> collaboration