[Date Prev][Date Next]
Re: OL 2.3.18 syncrepl vs slurpd
Howard Chu wrote:
Francis Swasey wrote:
I'm attempting to convert from using slurpd to using syncrepl.
However, my testing is developing a definite belief that syncrepl is
hopelessly unable to keep up.
I have a test situation where I have loaded a 48,819 entry ldif
using slapadd -q -w on the master and slapadd -q on the replica. I
then proceed to perform 12,654 modrdns, 56 modifies, and 961
delete/add actions in rapid succession.
Did you verify that the syncrepl consumer was actually idle before you
started your tests? syncrepl requires a contextCSN attribute to be
present on both the provider and on the consumer. The "-w" option to
slapadd causes the contextCSN attribute to be written, so that means
your provider's database was immediately usable. But then you need to
copy that value over to the consumer. If the LDIF file that you
slapadd'd on the consumer came from slapcat'ing the provider, then
you're all set, because it contains all the operational attributes,
including the contextCSN attribute. But if you slapadd'd a plain input
LDIF file on the consumer, then it had no contextCSN attribute, and so
it would have to suck the entire database down from the provider before
it considered itself sync'd up.
I believe that I verified the syncrepl consumer was idle. I set the
loglevel on the consumer to 16640 (256 stats + 16384 LDAPSync) and the
syslog was quiet for several minutes as well as no timestamps changing
in the database directory before I started the test.
The ldif that I loaded on both the syncrepl provider and syncrepl
consumer was generated by a slapcat on the syncrepl provider after the
original plain ldif file was loaded with the -w flag to generate the
contextCSN attribute and I have verified that the contextCSN is in the
ldif that was loaded -- however, since I used the -w flag on the
slapadd, would that have regenerated a (possibly) different contextCSN
value on the syncrepl provider's database?
With that prerequisite aside, it's well understood that syncrepl is
slower than slurpd for a number of reasons. Since syncrepl sends whole
Ah, it wasn't well understood by me that it was designed to be slower.
entries rather than just modifications, it uses a lot more network
bandwidth than slurpd. It also causes a lot more database update
activity on the consumers. We can take steps to make some of the
database activity more efficient, but the network load is still an
issue. That's why Symas developed the delta-syncrepl mode of operation,
which uses the accesslog data format to propagate modifications instead
of whole entries. Of course, delta-syncrepl has its own performance cost
since it serializes write operations. (The serialization is two-phase,
so you can have two writes in progress at a time.) There's an up-side
and a down-side to this; the downside is serialization limiting the
number of simultaneous write operations, the upside is that you
generally get zero database deadlocks this way so every modification
completes much faster.
I guess I'd better investigate the delta-syncrepl mode.