[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Faster slapadd in presence of multiple indices?

Actually I did examine this issue too when I was striving to enhance the performance of slapadd a few months ago. What I considered at that time is to combine the concurrent slapadding with the index clustering approach (ITS#3611). The index clustering consists of two phases: in-memory index creation and batched write of indices to DB. If the two phases can be arranged to overlap, it should be possible to keep CPU busy performing in-memory index creation during the time CPU would have been kept idle waiting for I/O completion. In this way, it would be possible to improve the performance of slapadd even in a single CPU system (in addition to multiple CPU system) if I/O bandwidth is abound (which is true esp. when index clustering approach is used). This has been in my todo lists before I switched to examining Berkeley DB. I guess this has a good potential to further improve the performance of slapadd beyond what has been achieved by the BDB scalability patch.
- Jong-Hyuk

Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979

Howard Chu <hyc@symas.com>
Sent by: owner-openldap-devel@OpenLDAP.org

09/07/2005 04:24 AM

        To:        ando@sys-net.it
        cc:        openldap-devel@OpenLDAP.org
        Subject:        Re: Faster slapadd in presence of multiple indices?

Pierangelo Masarati wrote:
> I haven't investigated this issue yet but would exploiting multiple CPUs
> allow faster slapadds (with -q, i.e. with less consistency checks) if, for
> instance, the entry and the indices are generated concurrently?  Much like
> the ancient ldif2index.  This comes from the consideration that on the
> machine we're testing giant slapadds, we have 75% to 87.5% of the CPU
> idle...
> Does anybody see any big stopper to this approach?
I actually added multi-threading code to slapadd in a previous
experiment. It's probably still in CVS, but I removed the code because
it yielded no measurable improvement. I think the problem is that this
will be I/O bound regardless of what CPU resources are available.

 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature