speed up parallel add/delete

[Derek Zhou <derek@shannon-data.com>]

all

I have a patch that can speed up parallel ldap add/delete a lot; in my testing about 2.5X. The idea is described here: the mdb backend does not really support parallelism in rw transactions, so add/delete are serialized by a mutex inside the mdb backend as small transactions. In my patch, all add/delete are put in a queue and executed in a dedicated worker thread. It may seem pointless at first, but by doing them all in the same thread, now I can merge concurrent ops from different clients into larger transactions, and reduce the number of expensive txn_commit calls (fsyncs). 

my test results are ( your mileage may vary) 

10 thread parallel add:

1, unpatched slapd + default config, ~ 2800 op/s (performance stable over time)
2, unpatched slapd + dbnosync ~9000 op/s (lots of fluctuation due to background dirty page flush)
3, unpatched slapd + dpnosync + checkpoint every minute ~8000 op/s, with reduced fluctuation
4, patched slapd ~7300 op/s (fluctuation nearly gone)

2 is not a recommended config, because crashing or power lost will cause massive data lost. 3 is better, but still up to 1 minutes of data can be lost at crash. With this patch, I can achieve >90% of the performance of 3, with data durability as good as 1; or 2.5X performance of 1 with same data durability guaranty. 

The patch still need some tidy up; is this email list the place to send patches?
 
-- 
Derek Zhou
Shannon Systems
http://www.shannon-sys.com