Re: (ITS#7210) back-mdb endless database growth

[quanah@zimbra.com]

--On Tuesday, March 20, 2012 2:16 PM +0000 hyc@symas.com wrote:

> hyc@symas.com wrote:
>> quanah@OpenLDAP.org wrote:
>>> Full_Name: Quanah Gibson-Mount
>>> Version: 2.4.30
>>> OS: Linux 2.6
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (75.108.184.39)
>>>
>>>
>>> If you create a script to add&   delete the same 50,000 users, and then
>>> re-run that script multiple times, the MDB database on disk continues
>>> to grow, despite the fact that we are only adding and deleting the same
>>> data repeatedly.  There should be zero growth to the database.
>>>
>>> Instead, the data.mdb file went from 108MB in size to over 1.2GB after
>>> 5-6 runs. Time to add/delete all the users went from 3 minutes 32
>>>    seconds on the first run to over 90 minutes on the final run (5th or
>>> 6th, I lost count).
>>
>> At present, MDB is working as intended. In your test a number of overflow
>> pages are being used. Currently libmdb always allocates overflow pages
>> from new space, because it's not smart enough to search the free list for
>> contiguous pages. Can treat this ITS as an enhancement request for that
>> purpose, but this is not a new issue nor is it unexpected in the current
>> version of MDB.
>>
> Hm, I take this back. The overflow pages are not enough to account for
> the  difference. Still looking.

If this were truly the expected behavior, it would make using back-mdb with 
delta-syncrepl useless, as the accesslog would simply grow without bound. 
;) Even back-bdb/hdb don't do that.  While the DB will never be smaller 
than the maximum size ever reached without reloading, it won't 
unnecessarily grow past that either. :P

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration