Re: (ITS#5860) slapd memeory leak under openldap 2.4

[rlvcosta@yahoo.com]

Howard,

I download the new HEAD version and made some testing. Now the olmBDBEntryCache is following the cache configuration.

I also made some small change in slapd.conf including cachefree parameter. Please see below in the end how slapd is configured related to bdb cache :

line 123 (cachesize       1000)
line 124 (cachefree     1000)
line 125 (idlcachesize    1000)
line 126 (dncachesize     1000000)

I increase the dncachesize since with a small value the search takes considerable more time.

First I tested with a value greater than the value that was being consumed before without the memory boundary. In this way I was expecting some order of magnitude as before, what was achieved (line 126 (dncachesize     5000000)):

BEFORE CACHED:
1000000

real    5m23.084s
user    0m28.026s
sys     0m6.686s

Then Cache has :

olmBDBEntryCache: 1001
olmBDBDNCache: 4000264
olmBDBIDLCache: 1

AFTER CACHED:

1000000

real    2m31.623s
user    0m28.145s
sys     0m8.637s

I would like to reinforce this test above was using a dncachesize knew that can store all DN's from my DB. This follow the same behavior as before where all DB can be stored into memory without slapd respecting the configuration boundaries(like it).

In my system I will have dn's that compose the entrance. Since my filter is for only one of the dn I was expecting this cache to be only related with the filter and then in 1,000,000.

So I'm expecting something greater than 6 minutes since for the first time system caches the information, or in other way used disk to read DB, the time was around 6 minutes.

With a cache smaller than 4 million entrances, I was expecting just a little more overhead since it would required slapd to flush cache and start to load the entrances more than once. In any case I would not expect something greater than 6 minutes since to flush memory and load from pieces should not cause too much overhead.

Only following queries would not take fully advantage, like half time at further tries, since the cache cannot keep all information.

But some strange things, by my perspective, happened. The dncache grew normally until reach the 1000000 value with cachesize following the boundary 1000 :

readOnly: FALSE
olmBDBEntryCache: 1001
olmBDBDNCache: 957271
olmBDBIDLCache: 1

But after the dncache limit is reached, by monitor, the cache became :

olmBDBEntryCache: 1
olmBDBDNCache: 1000262 (this is a very deterministic number)
olmBDBIDLCache: 1

Getting stuck here forever.

Also the performance degrades considerable. From this point on I let all night and the DB was not fully passed. This means that after the DN cache enter in restrictions, even a lower performance would be expected since disk access should be more often, it becomes unpractical.

I needed to stop the slapd so I could see how many entrances were parsed. This return a number around 250,000, meaning around 1/4 as the DN size. This appears to be that after cache boundary is reached for some reason the DB parse doesn't evolve anymore.

I lost the correct time information from my last night query running but I can send another query I did this morning with some similar information. I can say it took more than 8 hours without moving very further than 250,000 entrance(of 1,000,000).

ldap_result: Can't contact LDAP server (-1)
251512

real    121m51.697s
user    0m7.327s
sys     0m1.734s

This is very deterministic and normally get stuck around this value without much improvement. I do not believe query will end. This was the reason I needed to kill the slapd process to have an order of magnitude the search had reached since this search is never ending.

Something still looking with problems since performance is too much degraded even the disk would be more in use. Looks like search gets stuck after cache boundaries are reached.

Also looks like the entrance cache becomes 1(olmBDBEntryCache) after dn limit is reached.

Best Regards,

Rodrigo.

--- On Sun, 1/25/09, Howard Chu <hyc@symas.com> wrote:

> From: Howard Chu <hyc@symas.com>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com
> Cc: openldap-its@OpenLDAP.org
> Date: Sunday, January 25, 2009, 7:46 PM
> Rodrigo Costa wrote:
> > Howard,
> >
> > I tested the HEAD load last night. The results I
> believe are partial.
> 
> Thanks for the feedback, please try the new patch in HEAD.
> >
> > The memory stay stable and the use was much lower. I
> had in my slapd.conf the following cache boundaries :
> >
> > line 122 (cachesize       1000)
> > line 123 (idlcachesize    1000)
> > line 124 (dncachesize     2000)
> >
> > Which make with the new load to consume much less
> memory and stay stable.
> >
> > PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
> TIME+  COMMAND
> > 20715 ldap      18   0  184m  73m  68m S    0  0.6
> 174:05.81 slapd
> >
> > But by the other hand I use the monitor information to
> check the cache usage. I would notice that now dncache is
> keeping around the boundary I put of 20000 but the cache is
> now fixed as 1, not 1000. See information below :
> >
> > readOnly: FALSE
> > olmBDBEntryCache: 1
> > olmBDBDNCache: 2185
> > olmBDBIDLCache: 1
> > olmDbDirectory: /var/openldap-data/bdb2/
> > entryDN: cn=Database 2,cn=Databases,cn=Monitor
> >
> > This kept around this all the time I was making a
> search in DB.
> >
> > Before this load with this possible DN cache boundary
> solution I made the ldapsearch in around 7 minutes, for the
> first time not cached yet. And around 4 minutes after the
> entrances were cached.
> >
> > Now with this new load I made the search and it took :
> >
> > 1000000
> >
> > real    174m6.486s
> > user    0m43.746s
> > sys     0m16.923s
> >
> >
> > Almost 3 hours. It's ok that I didn't tune the
> cache sizes but I was at least expecting the
> olmBDBEntryCache would now follow the 1000 boundary and not
> single 1.
> >
> > I believe this is what is not making this huge
> performance difference and not the olmBDBDNCache.
> >
> > Please confirm this is some tuning missing or if
> something else like the olmBDBEntryCache would be improved.
> >
> > In terms of memory usage now it is respecting but the
> performance, probably based in the olmBDBEntryCache only
> using 1 position, is much worst.
> >
> > Please also let me know if you think this performance
> degradation and the olmBDBEntryCache marking 1 is a tuning
> issue or something still need to be modified in the code.
> >
> > Thanks for the fast solution for this issue since this
> would be affecting all systems and types of DB.
> >
> > Rodrigo.
> 
> 
> -- 
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com
>    Director, Highland Sun     http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP 
> http://www.openldap.org/project/