Re: (ITS#8852) slapd memory use grows continuously with non-delta syncrepl and modifying groups

[hyc@symas.com]

ryan@nardis.ca wrote:
> bisect identifies c365ac359e9c9b483b934c2a1f0bc552645c32fa as the commit
> that introduced this behaviour.
> 
> 003dfbda574f37bbf1a2240f530ff9fa35ab0801 on RE24 (2.4.20)
> 
> commit c365ac359e9c9b483b934c2a1f0bc552645c32fa
> Author: Howard Chu <hyc@openldap.org>
> Date:   Sun Nov 22 04:42:00 2009 +0000
> 
>      ITS#6368 use dup'd entries in response queue

I've run your reproducer and see no memory leak. The response queue will 
indeed grow without bound if the consumer runs slower than the provider, and 
doesn't read responses fast enough. But in the case of this test script, 
eventually the client finishes and the consumer catches up.

The provider's process size may not decrease, but that just means the malloc 
implementation isn't returning freed memory to the kernel - it's not a leak.

This can be verified using mleak, and using SIGPROF to snapshot the memory 
usage of the provider. The simplest way to force the memory use to grow is to 
first suspend the consumer with SIGSTOP. Let the modify client run to 
completion. mleak / SIGPROF will show a large amount of memory in use. Resume 
the consumer with SIGCONT, let it run to completion, and then check with 
SIGPROF on the provider again - all of the response queue memory is freed.

So, conclusively, there is no actual leak. But there's a problem with 
sustained client modifications when the consumer is too slow. Our options here 
are to configure a size limit on the response queue, and hang the client when 
the limit is hit, or to return LDAP_BUSY to the client. Neither of these are 
very attractive options.

Doing batched commits will speed up the consumer, but that feature is only in 2.5.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/