[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: OL 2.3.18 syncrepl vs slurpd

Francis Swasey wrote:
Folks,
I'm attempting to convert from using slurpd to using syncrepl. However, my testing is developing a definite belief that syncrepl is hopelessly unable to keep up.

I have a test situation where I have loaded a 48,819 entry ldif using slapadd -q -w on the master and slapadd -q on the replica. I then proceed to perform 12,654 modrdns, 56 modifies, and 961 delete/add actions in rapid succession.

Did you verify that the syncrepl consumer was actually idle before you started your tests? syncrepl requires a contextCSN attribute to be present on both the provider and on the consumer. The "-w" option to slapadd causes the contextCSN attribute to be written, so that means your provider's database was immediately usable. But then you need to copy that value over to the consumer. If the LDIF file that you slapadd'd on the consumer came from slapcat'ing the provider, then you're all set, because it contains all the operational attributes, including the contextCSN attribute. But if you slapadd'd a plain input LDIF file on the consumer, then it had no contextCSN attribute, and so it would have to suck the entire database down from the provider before it considered itself sync'd up.

With that prerequisite aside, it's well understood that syncrepl is slower than slurpd for a number of reasons. Since syncrepl sends whole entries rather than just modifications, it uses a lot more network bandwidth than slurpd. It also causes a lot more database update activity on the consumers. We can take steps to make some of the database activity more efficient, but the network load is still an issue. That's why Symas developed the delta-syncrepl mode of operation, which uses the accesslog data format to propagate modifications instead of whole entries. Of course, delta-syncrepl has its own performance cost since it serializes write operations. (The serialization is two-phase, so you can have two writes in progress at a time.) There's an up-side and a down-side to this; the downside is serialization limiting the number of simultaneous write operations, the upside is that you generally get zero database deadlocks this way so every modification completes much faster.

Using slurpd, the script runs in 28 minutes on the master and within one (1) minute the replica is up to date. Using syncrepl, the script runs in 25 minutes on the master, but the replica will take about 45 minutes to get in sync (one test took over 90 minutes).

I haven't found any syncrepl tuning documents, so I'm kind of shooting blind after reading the Admin Guide and the syncrepl man pages.

  I'm using the following in the master slapd.conf:

overlay syncprov
syncprov-checkpoint 1000 10
syncprov-sessionlog 1000
syncprov-nopresent FALSE
syncprov-reloadhint FALSE

  I'm using the following in the replica slapd.conf:

syncrepl rid=100
        provider=ldaps://carcajou.uvm.edu
        type=refreshAndPersist
        retry=5,+
        searchbase="dc=uvm,dc=edu"
        filter="(objectclass=*)"
        scope=sub
        updatedn="cn=Replicator,dc=uvm,dc=edu"
        binddn="cn=SyncUser,dc=uvm,dc=edu"
        bindmethod=simple
        credentials=IAmNotThatStupidAsToGiveYouTheRealPassword

Doing ldapsearch calls at sixty second intervals, it appears that syncrepl is handling approximately 200 updates per minute.

Anyone have any insight to what I've done wrong or where I should start looking?

Thanks,

--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/