[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#5860) slapd memeory leak under openldap 2.4



Howard,

I have the always the same DB_CONFIG file as below :

set_cachesize 0 52428800 1

set_flags DB_LOG_AUTOREMOVE

set_lg_regionmax 1048576
set_lg_max 10485760
set_lg_bsize 2097152

set_tmp_dir /var/tmp

I just removed the comment lines so it fits better on screen.

About versions I have :

[root@brtldp12 yum.repos.d]# /usr/libexec/slapd  -V
@(#) $OpenLDAP: slapd 2.4.13 (Jan 22 2009 02:28:12) $
        root@brtldp12:/root/openldap-2.4.13/servers/slapd

[root@brtldp12 yum.repos.d]# db_verify -V
Berkeley DB 4.7.25: (May 15, 2008)

About the man pages I have :

[root@brtldp12 yum.repos.d]# ll /usr/man/man8/slapd.8.gz
-rw-r--r-- 1 root root 4392 Jan 22 03:28 /usr/man/man8/slapd.8.gz

[root@brtldp12 yum.repos.d]# ll /usr/man/man5/slapd-bdb.5.gz
-rw-r--r-- 1 root root 4089 Jan 22 03:28 /usr/man/man5/slapd-bdb.5.gz

I was expecting to find information about dncachesize in slapd-bdb but there isn't nothing there.

Where at config.c I have :

                "( OLcfgDbAt:1.5 NAME 'olcDbDirtyRead' "
                "DESC 'Allow reads of uncommitted data' "
                "SYNTAX OMsBoolean SINGLE-VALUE )", NULL, NULL },
        { "dncachesize", "size", 2, 2, 0, ARG_UINT|ARG_OFFSET,
                (void *)offsetof(struct bdb_info, bi_cache.c_eimax),
                "( OLcfgDbAt:1.12 NAME 'olcDbDNcacheSize' "
                        "DESC 'DN cache size' "
                        "SYNTAX OMsInteger SINGLE-VALUE )", NULL, NULL }

So the dncachesize is supposed to be in the code.

In any case slapd never respects memory limitation and consumes it without boundaries. It also never releases any memory.

Then after sometime the process cannot allocate more memory and crashes.

Best Regards,

Rodrigo.

--- On Thu, 1/22/09, Howard Chu <hyc@symas.com> wrote:

> From: Howard Chu <hyc@symas.com>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com
> Cc: openldap-its@openldap.org
> Date: Thursday, January 22, 2009, 1:35 PM
> rlvcosta@yahoo.com wrote:
> 
> > See that I have 2 DBs and in each DB area in
> slapd.conf, read without error by slapd as seen above, I
> have the cachesize, idlcachesize and dncachesize specified
> with small values just to monitor the memory usage by slapd.
> >
> > Unfortunately even with this configuration I still
> having slapd consuming memory without release it. So if a
> new entrance is queried by some LDAP client, slapd will
> consume more memory. Since my DB dn's are bigger than
> 3GB then soon or later the slapd will crash.
> >
> > See my DB information :
> >
> > 7.2G    dn2id.bdb
> > 12G     id2entry.bdb
> > 110M    maillogin.bdb
> > 3.3M    objectClass.bdb
> > 108M    pnnumber.bdb
> > 1.3M    submxid.bdb
> > 323M    uid.bdb
> >
> > See my dn2id is much bigger than yours and maybe you
> just did not saw this
> issue happenig since your dn can be loaded in your memory
> and you in a 64 bits
> environment can use more than 3GB per process.
> >
> > But in any case looks like there is a problem in slapd
> that do not respect
> the cache limitations imposed.
> >
> > In this way we cannot use slapd for large databases.
> Please see my
> configuration above and let me know if a made a mistake(I
> hope so) since
> dncachesize is not documented at man pages.
> 
> You're mistaken, dncachesize is in the 2.4 manpages.
> You seem to not be 
> reading the documentation that came with the software
> release you're using.
> 
> > Just as example, with the DB I loaded for tests with 1
> million entrances, with the cache sizes definitions as in
> the slapd.conf described above and making a ldapsearch on
> all entrances from the CONTENT DB, in the end I have :
> >
> > PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
> TIME+  COMMAND
> > 3189 ldap      15   0  842m 724m  67m S   99  6.1  
> 7:00.48 slapd
> >
> > See the slapd process already consumed 842m and I just
> read 1 million entrances, even dncachesize is defined to be
> 1000.
> >
> > I do not see anyway to control the memory usage by
> slapd.
> >
> > Is this "disrespect" to slapd.conf
> directives a possible problem?
> 
> Looks like slapd is behaving normally so far, though
> it's hard to tell without 
> also seeing your DB_CONFIG settings. Test again with your
> full database and 
> report the results.
> 
> -- 
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com
>    Director, Highland Sun     http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP 
> http://www.openldap.org/project/