[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: back-bdb IDL limitations




During the performance evaluation and testing of the LDAP component matching in OpenLDAP, we developed an ldif split program written in C. It was a lot faster than doing it by scripting.
For the second point, isn't it possible to figure out what the dn of the last committed entry is in a consistent database by retrieving the last ID and the corresponding entry in the id2entry database ?

By the "incremental slapadd", I tried not to rule out a situation where the database is populated incrementally for example day by day by using slapadd. If there's a system failure during the current slapadd, it is possible with transactions to avoid the repopulation of the entire directory which was not populated in the current slapadd run. It might not be the common case scenario but it shows that it is desirbale to maintain the transactional operation of slapadd and I guess that's why the new quick slaptools mode is activated by a command line option.

- Jong-Hyuk

------------------------
Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979



Howard Chu <hyc@symas.com>

03/04/2005 01:13 PM

       
        To:        Jonghyuk Choi/Watson/IBM@IBMUS
        cc:        David Boreham <david_list@boreham.org>, openldap-devel@OpenLDAP.org
        Subject:        Re: back-bdb IDL limitations



We've tested ranges from 2 million to 20 million entries, and it seems
rather inconvenient to do incremental adds here. First we need to add a
flag to slapadd to tell it to skip the first N entries and start adding
from some offset into the input LDIF, because manually splitting such a
large input LDIF file is a pain. Second we need to add feedback from
slapadd to tell you exactly where it was in the input when it failed
(entry number may be enough, byte offset would probably be better) but
of course, if the add is interrupted because the entire machine crashed,
you may never see that feedback. I guess one question to answer is what
kind of failures are you protecting against, and what does it cost
either way. For the most part, it is simpler to just wipe out the
database and start over.

Jonghyuk Choi wrote:

>
> Is it so because of the recovery overhead can be huge ? Even so,
> transactions seem to be indispensable to situations like incremental
> slapadd.
> - Jong-Hyuk
>
> ------------------------
> Jong Hyuk Choi
> IBM Thomas J. Watson Research Center - Enterprise Linux Group
> P. O. Box 218, Yorktown Heights, NY 10598
> email: jongchoi@us.ibm.com
> (phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979
>
>
>
>
>                  *"David Boreham" <david_list@boreham.org>*
> Sent by: owner-openldap-devel@OpenLDAP.org
>
> 03/04/2005 12:06 PM
>
>                        
>         To:        <openldap-devel@OpenLDAP.org>
>         cc:        
>         Subject:        Re: back-bdb IDL limitations
>
>
>
>
>  
> 2) there should be non-zero need for transaction protected operations
> of slap tools to cope with a system failure during a lengthy directory
> population.
> I think you will find that it's almost always faster to simply disable
> transactions and
> re-start the slapadd in the event of a failure.
>  
>  



--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support


Attachment: smime.p7s
Description: S/MIME Cryptographic Signature