[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: thread pools, performance



David Boreham wrote:
Gavin Henry wrote:
Thanks for all this Howard. It certainly makes it clear where OpenLDAP
lies in the LDAP world (looking down from the top and around at
everyone else ;-) ).

Hmm...not so sure about the 'everyone' part because
I added this feature (parallel import with correctly implemented
partent/child entry interlocking) to the product that is now SunDS and
FedoraDS
in 1999.

With all due respect David, you've done some nice work on those products but OpenLDAP is still significantly faster than either SunDS or FedoraDS, both in importing data and in the authentication rates.


IIRC we had this conversation about parallel import before on this list. I had tried parallel entry importing before and gave up on it because it showed no benefit on my tests. Then based on your hints I tested a variety of other approaches. What I eventually settled on for OL 2.3 - serial entry import with parallel indexing - has been the fastest approach in my tests.

One obvious reason is that no entry locking is required because only a single thread ever operates on any given DB. With parallel writes of entries to the same DB, you still need full locking to prevent data corruption. Another reason is that BerkeleyDB's B-trees are optimized for sequential writes; parallelizing the entry writes defeats that optimization because it partially randomizes the I/Os. (I.e., it requires seeks instead of just sequential access.)

It's too bad that SunDS still disallows publishing benchmark results. The current license terms just tells the world they have something to hide.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/