Re: (ITS#5860) slapd memeory leak under openldap 2.4

[rlvcosta@yahoo.com]

Pierangelo,

Executing some test I'm not sure anymore if this is really a memory leakage or if some cache issue.

I decided to install CentOS5.2 where I could have OpenLdap 2.3.27 compiled by the distribution and then execute more tests to check if some library in RHEL4 would be causing some problem.

I then use a similar slapd.conf as in my original e-mail just without explicitly define an size for cache and idlcache there. My DB_CONFIG file has a cache defined as 50MB what is respected.

Looking at some other ITS I found the ITS#3938 what I believe is very similar to my problem.

I still have a DB with 3882998 entrances and where the DB files at disk is around 15GB. My machine has 12GB of memory where for sure all the DB cannot be cached into memory.

In any case what is happening is that every time a transaction is done with OpenLdap, even a distribution formal compilation, the cache continues to grow until consumes all memory. If the DB is bigger than machine memory then soon or later a memory out of range will happen.

The cache is never release by OpenLdap and then if a sequential search is done and since DB is bigger than memory we can speed up the problem.

I also used Valgrind where I obtained :

==4963== LEAK SUMMARY:
==4963==    definitely lost: 0 bytes in 0 blocks.
==4963==      possibly lost: 0 bytes in 0 blocks.
==4963==    still reachable: 336,091 bytes in 12,547 blocks.
==4963==         suppressed: 0 bytes in 0 blocks.
==4963== Reachable blocks (those to which a pointer was found) are not shown.
==4963== To see them, rerun with: --show-reachable=yes
[root@brtldp12 bdbBackupContent]# ==4964== Conditional jump or move depends on uninitialised value(s)
...
Valgrind's memory management: out of memory:
   newSuperblock's request for 1048576 bytes failed.
   2952183808 bytes have already been allocated.
Valgrind cannot continue.  Sorry.

See it consume all the available memory and then could not allocate more memory.

I also have an old version of OpenLdap in other machine with the same HW where the same DB has even more entrances. I'm trying to upgrade the OpenLdap to improve other itens but this appears to not be possible. At this old version I do not have this infinite cache problem and the only memory consumed is the Berkeley DB cache defined as 50MB.

I could ldapsearch all the DB but once cache is allocated it never release it. At top I can see the memory continues associated with slapd process even the ldapsearch ends. Then if we start other ldapsearch then memory continues to be allocated.

For information the old version informations is :

openldap 2.1.30
and
BerkeleyDB 4.2.52

With this version above I have a DB bigger than the available memory and I do not have this cache problem. I also do not have entrances like idelcache or anything else at slapd.conf file.

At this way at new versions if for some reason you have a file DB bigger than your available memory with time you will have problems with OpenLdap.

I will try again to force the idlcache and openldap cache but the last time tried this did not work.

In this way I would identify better this problem and ask if you see this is as a bug or if some configuration would "force" openldap to cache(clean it) only using certain maximum size?

Regards,

Rodrigo.

--- On Tue, 12/30/08, Pierangelo Masarati <ando@sys-net.it> wrote:

> From: Pierangelo Masarati <ando@sys-net.it>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com
> Cc: openldap-its@openldap.org
> Date: Tuesday, December 30, 2008, 2:06 PM
> rlvcosta@yahoo.com wrote:
> > --0-1358063323-1230648878=:64021
> > Content-Type: text/plain; charset=iso-8859-1
> > Content-Transfer-Encoding: quoted-printable
> > 
> > Pierangelo,
> > 
> > I could run the valgrind but at this time I do not
> have the DB with the aro=
> > und 4 million subscribers. In this condition, with a
> small number of users,=
> >  I already could see some "definitive"
> memory leaks and I believe they are =
> > associated with OpenLdap.
> > 
> > Just for a history I already compiled the openldap
> 2.4.11 souce code withou=
> > t modules and TLS since many lists make reference to
> some possible issues i=
> > n RHEL with TLS. Even so I could not have a load
> without the possible memor=
> > y leak.
> > 
> > So I do not believe the problems are in the RHEL
> libtool or glibc.
> > 
> > In any case the result for the valgrind is (short
> number of entrances at th=
> > is time):
> 
> You probably need to use an unstripped binary in order to
> find out where mallocs/frees occur.  Most of the occurrences
> you notice are either one-time leaks.  Moreover, 2.4.13 is
> out, and it might have fixed some.  I suggest you further
> investigate using the latest code, using an unstripped
> binary.
> 
> p.
> 
> 
> Ing. Pierangelo Masarati
> OpenLDAP Core Team
> 
> SysNet s.r.l.
> via Dossi, 8 - 27100 Pavia - ITALIA
> http://www.sys-net.it
> -----------------------------------
> Office:  +39 02 23998309
> Mobile:  +39 333 4963172
> Fax:     +39 0382 476497
> Email:   ando@sys-net.it
> -----------------------------------