Emmanuel Lécharny wrote:
Le 03/02/15 05:11, Howard Chu a écrit :Another option here is simply to perform batching. Now that we have the TXN api exposed in the backend interface, we could just batch up e.g. 500 entries per txn. much like slapadd -q already does. Ultimately we ought to be able to get syncrepl refresh to occur at nearly the same speed as slapadd -q.Batching is ok, except that you never know how many entries you'll going to have, thus you will have to actually write the data after a period of time, even if you don't have the 500 entries.
This isn't a problem - we know exactly when refresh completes, so we can finish the batch regardless of how many entries are left over.
Testing this out with the experimental ITS#8040 patch - with lazy commit the 2.8M entries (2.5GB data) takes ~10 minutes for the refresh to pull them across. With batching 500 entries/txn+lazy commit it takes ~7 minutes, a decent improvement. It's still 2x slower than slapadd -q though, which loads the data in 3-1/2 minutes.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/