Re: (ITS#5860) slapd memeory leak under openldap 2.4

[rlvcosta@yahoo.com]

Quanah,

In this specific test I reported in my previous e-mail it was used glibc.

Now I tested with tcmalloc and the memory again was much more consumed and never released.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10652 ldap      18   0 2006m 1.8g  68m S  100 15.8   8:36.75 slapd

Once it get this much memory it then stabilize but it is much more than before that was 385m.

The scenario was :

1) Start a full ldapsearch for one specific dn(the DB has 4 dn's);
2) After sometime start a new full ldapsearch to behave as queries to DB when replication for LDIF backup is being done;
3) Export one of the queries to a file so speed verification can be done with grep and wc.

After sometime the memory starts to be consumed in large chunks until it stopped in around 2G.

Since one of the queries is being redirect to a file I can tail to have a visual rate of the entrances being returned by slapd. As the issue before after sometime the rate appears to get stuck and come in small 16 entrances chunks but only after some minutes.

This behavior above appears to happen after the second ldapsearch fills its own dn cache even monitor always shows only one information as I could see.

[root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l
Mon Jan 26 21:43:31 BRST 2009
482432
[root@brtldp11 ~]# date; grep -e '^pnnum*' /backup/temp.ldif |wc -l
Mon Jan 26 21:44:56 BRST 2009
482448

This behavior is similar as before with the initial issue with a single process.

I just notice when slapd starts there are these threads :

[root@brtldp12 ~]# ps -mu ldap
  PID TTY          TIME CMD
10638 pts/2    00:00:00 slapd
    - -        00:00:00 -
    - -        00:00:00 -

Then after the 2 queries are started I have :

[root@brtldp12 ~]# ps -mu ldap
  PID TTY          TIME CMD
10195 pts/2    00:44:22 slapd
    - -        00:00:00 -
    - -        00:00:00 -
    - -        00:26:09 -
    - -        00:18:13 -
    - -        00:00:00 -

So 3 new threads. Looks like for some reason with multiples queries the issue with response getting stuck repeats.

Even one ldapsearch is stopped the number of threads continues the same, memory continues allocated and the other search get with this slow rate.

These outputs are using now tcmalloc but similar as with glibc.

Regards,

Rodrigo.

--- On Mon, 1/26/09, Quanah Gibson-Mount <quanah@zimbra.com> wrote:

> From: Quanah Gibson-Mount <quanah@zimbra.com>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com, openldap-its@openldap.org
> Date: Monday, January 26, 2009, 9:25 PM
> --On Monday, January 26, 2009 3:18 PM -0800 Rodrigo Costa 
> <rlvcosta@yahoo.com> wrote:
> 
> > Quanah and Howard,
> >
> > Sorry to come back.
> >
> > I tried to make 2 ldapsearch to the same DB
> simultaneously, doing some
> > stress test like dump the LDIF and queries coming.
> >
> > The slapd after sometime start to get more memory. By
> what I could
> > observe looks like new thread was started consuming
> again the same memory
> > as before so the slad now is consuming around the
> double of memory as
> > before.
> 
> Are you also using an alternate malloc() instead of glibc? 
> I know you were 
> investigating this, just want to confirm you are using
> tcmalloc or hoard 
> and not glibc.
> 
> --Quanah
> 
> 
> --
> 
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and
> collaboration