Re: syncrepl consumer is slow

[Emmanuel Lécharny <elecharny@gmail.com>]

Le 03/02/15 09:41, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Le 03/02/15 05:11, Howard Chu a écrit :
>>> Another option here is simply to perform batching. Now that we have
>>> the TXN api exposed in the backend interface, we could just batch up
>>> e.g. 500 entries per txn. much like slapadd -q already does.
>>> Ultimately we ought to be able to get syncrepl refresh to occur at
>>> nearly the same speed as slapadd -q.
>>
>> Batching is ok, except that you never know how many entries you'll going
>> to have, thus you will have to actually write the data after a period of
>> time, even if you don't have the 500 entries.
>
> This isn't a problem - we know exactly when refresh completes, so we
> can finish the batch regardless of how many entries are left over.

True for Refresh. I was thinking more specifically of updates when we
are connected.

The idea of pushing the expected number of updates within the cookie is
for information purposes : having this number traced in the
logs/monitored could help in some cases where the refresh phase takes
long : the users will not stop the server thinking it has stalled.
>
> Testing this out with the experimental ITS#8040 patch - with lazy
> commit the 2.8M entries (2.5GB data) takes ~10 minutes for the refresh
> to pull them across. With batching 500 entries/txn+lazy commit it
> takes ~7 minutes, a decent improvement. It's still 2x slower than
> slapadd -q though, which loads the data in 3-1/2 minutes.

Not bad at all. What makes it 2x slower, btw?