Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown

[quanah@zimbra.com]

--On Thursday, September 30, 2010 4:59 PM +0000 quanah@zimbra.com wrote:

> --On Thursday, September 30, 2010 11:14 AM +0000 masarati@aero.polimi.it
> wrote:
>
>> There's no mention of overlays; I was wondering whether or not it could
>> be related to some adverse interaction, e.g. with sssvlv, which has some
>> paged-results specific code.  Can you also post on overlays either global
>> or specific to "olcDatabase={2}hdb"?
>
> There are no overlays being used. ;)  As I said, a very vanilla config.

Howard and I continue to examine this problem, but as yet, have found no 
solution nor insight into why the problem exists.  It has nothing to do 
with paged results, and you can see the behavior relatively easily with a 
smaller DB than I was working with.  Right now, I'm using a 1 million entry 
DB.  If I execute two concurrent searches of the database with a 25,000 
entry cache limit, that frees 1,000 entries on purge, at the end of those 
searches, we end up with:

counted 1000087 entryInfos and 114476 entries, 0 notcached

What this shows is that after the search completed, there are 114,476 
entries in the entry cache, even though it only reports between 24,000 and 
25,000.  Repeated additional searches will continue to increase the number 
of cached entries until all memory is exhausted.


The following code is used to count up the cached entries, and it is called 
from inside gdb via: call bdb_lru_count(&bdb->bi_cache)


--- openldap-2.4.23/servers/slapd/back-bdb/cache.c.orig 2010-10-05 
16:20:27.085259568 -0700
+++ openldap-2.4.23/servers/slapd/back-bdb/cache.c      2010-10-05 
16:21:03.655255057 -0700
@@ -1530,6 +1530,35 @@
 }

 #ifdef LDAP_DEBUG
+static void
+bdb_lru_count( Cache *cache )
+{
+       EntryInfo       *e;
+       int ei = 0, ent = 0, nc = 0;
+
+       for ( e = cache->c_lrutail; ; ) {
+               ei++;
+               if ( e->bei_e ) {
+                       ent++;
+                       if ( e->bei_state & CACHE_ENTRY_NOT_CACHED )
+                               nc++;
+               }
+               e = e->bei_lrunext;
+               if ( e == cache->c_lrutail )
+                       break;
+       }
+       fprintf( stderr, "counted %d entryInfos and %d entries, %d 
notcached\n",
+               ei, ent, nc );
+       ei = 0;
+       for ( e = cache->c_lrutail; ; ) {
+               ei++;
+               e = e->bei_lruprev;
+               if ( e == cache->c_lrutail )
+                       break;
+       }
+       fprintf( stderr, "counted %d entryInfos (on lruprev)\n", ei );
+}
+
 #ifdef SLAPD_UNUSED
 static void
 bdb_lru_print( Cache *cache )


If anyone else has some thoughts on why the entry cache is growing 
unbounded, much appreciated.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration