[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Faster slapadd in presence of multiple indices?

Well, now that I'm developing on a dual-core system (AMD64 X2) I guess I'm a bit more interested in re-examining the question, since I now see my system is 50% idle while slapadd is running...

My previous experiment just used two separate threads, one for reading/parsing the LDIF input and one for writing the entry+index data. I suppose, in the context of -q with transactions disabled, we could consider using a separate thread per indexed attribute. This implies activating the thread-pool code in tool mode, but the current code assumes there is no thread-pool in tool mode.

Since it looks like a large slapadd is mostly CPU bound (assuming a large enough BDB cache) this ought to be a win. And once the cache gets full and starts flushing pages, it should also be a win as we get past I/O waits. I guess I should test with a smaller cache (thus more frequent I/O) and see how it goes.

Jonghyuk Choi wrote:

Actually I did examine this issue too when I was striving to enhance the performance of slapadd a few months ago. What I considered at that time is to combine the concurrent slapadding with the index clustering approach (ITS#3611). The index clustering consists of two phases: in-memory index creation and batched write of indices to DB. If the two phases can be arranged to overlap, it should be possible to keep CPU busy performing in-memory index creation during the time CPU would have been kept idle waiting for I/O completion. In this way, it would be possible to improve the performance of slapadd even in a single CPU system (in addition to multiple CPU system) if I/O bandwidth is abound (which is true esp. when index clustering approach is used). This has been in my todo lists before I switched to examining Berkeley DB. I guess this has a good potential to further improve the performance of slapadd beyond what has been achieved by the BDB scalability patch.
- Jong-Hyuk


Pierangelo Masarati wrote:
> I haven't investigated this issue yet but would exploiting multiple CPUs
> allow faster slapadds (with -q, i.e. with less consistency checks) if, for
> instance, the entry and the indices are generated concurrently? Much like
> the ancient ldif2index. This comes from the consideration that on the
> machine we're testing giant slapadds, we have 75% to 87.5% of the CPU
> idle...
> Does anybody see any big stopper to this approach?
> I actually added multi-threading code to slapadd in a previous
experiment. It's probably still in CVS, but I removed the code because
it yielded no measurable improvement. I think the problem is that this
will be I/O bound regardless of what CPU resources are available.

-- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/