[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: syncrepl consumer is slow

Emmanuel Lécharny wrote:
Le 03/02/15 05:11, Howard Chu a écrit :
Another option here is simply to perform batching. Now that we have
the TXN api exposed in the backend interface, we could just batch up
e.g. 500 entries per txn. much like slapadd -q already does.
Ultimately we ought to be able to get syncrepl refresh to occur at
nearly the same speed as slapadd -q.

Batching is ok, except that you never know how many entries you'll going
to have, thus you will have to actually write the data after a period of
time, even if you don't have the 500 entries.

This isn't a problem - we know exactly when refresh completes, so we can finish the batch regardless of how many entries are left over.

Testing this out with the experimental ITS#8040 patch - with lazy commit the 2.8M entries (2.5GB data) takes ~10 minutes for the refresh to pull them across. With batching 500 entries/txn+lazy commit it takes ~7 minutes, a decent improvement. It's still 2x slower than slapadd -q though, which loads the data in 3-1/2 minutes.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/