Re: index files grow with repeated add and delete (ITS#186)

[Juan Carlos Gomez <gomez@cthulhu.engr.sgi.com>]

I believe I have fixed this problem in the current devel version of openldap.
Please Test and email feedback I am very interested in it.


                                                            Enjoy!, Juan


mharris@intermec.com wrote:

> Full_Name: Mike Harris
> Version: current stable version
> OS: HP-UX
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (204.57.247.20)
>
> I'm running slapd 3.3 on hpux using ndbm as the backend database.  I built
> it from source downloaded from www.umich.edu.  I have a simple test script
> that repeatedly adds and deletes a single entry from LDAP.  I have observed
> that the following backend database files grow over time when running the
> test script:
>
>         dn.ndbm.pag
>         objectclass.ndbm.pag
>
> The growth of the other files seems to  be limited only by the space available
> on the disk.
>
> In the umich source, I see calls to dn2id_delete() and id2entry_delete() in
> ldbm_back_delete() (back-ldbm/delete.c) that presumably update the
> dn2id.ndbm.pag and id2entry.ndbm.pag files when an entry is deleted.
>
> In openldap source, I see that a call to id2children_remove() has been added
> to ldbm_back_delete().  I presume that this call updates the
> id2children.ndbm.pag file when an entry is deleted.
>
> In neither source do I find where the index files dn.ndbm.pag and
> objectclass.ndbm.pag are updated when an entry is deleted.  Are the index
> files updated elsewhere in openldap?   Or is this a bug in both LDAP
> implementations?  Is there a work-around other than using ldbmcat and ldif2ldbm
> to dump and recreate the LDAP database?
>
> To create the database:
>
> ldif2ldbm -f slapd.conf -i domain.add
>
> To invoke the test script:
>
>         $ adddelete.ksh person.add person.del
>
> adddelete.ksh:
>
> while [ true ]
> do
>     ldapadd -D "dc=MyDomain" -w secret -f $1
>     ldapdelete -D "dc=MyDomain" -w secret -f $2
> done
>
> person.add:
>
> dn: cn=Nat, dc=MyDomain
> objectclass: person
> sn: Cole
> cn: Nat
>
> person.del:
>
> cn=Nat, dc=MyDomain
>
> domain.add:
>
> dn: dc=MyDomain
> dc: MyDomain
> objectclass: domain
>
> slapd.conf:
>
> include         slapd.at.conf
> include         slapd.oc.conf
> schemacheck     on
>
> #######################################################################
> # ldbm database definitions
> #######################################################################
>
> database        ldbm
> suffix          "dc=MyDomain"
> directory       .
> rootdn          "dc=MyDomain"
> rootpw          secret
> lastmod off
>
> Sizes before running test script:
>
> -rw-------   1 mharris    intermec      1024 May 28 10:57 dn.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:57 dn2id.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:57 id2children.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:57 id2entry.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:57 objectclass.ndbm.pag
>
> Sizes after running test script for about 2 minutes:
>
> -rw-------   1 mharris    intermec     86016 May 28 10:59 dn.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:59 dn2id.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:59 id2children.ndbm.pag
> -rw-------   1 mharris    intermec      1024 May 28 10:59 id2entry.ndbm.pag
> -rw-------   1 mharris    intermec      2048 May 28 10:59 objectclass.ndbm.pag