[Date Prev][Date Next] [Chronological] [Thread] [Top]

Syncrepl: full sync vs. delta

I recently had to process some 40,000+ modifications through my set of directory servers. Initially, I tested my changes through my dev boxes (2.3/HEAD using syncrepl). This run took approximately 2 hours, but the slaves were modified essentially as the master was. This length of time concerned me somewhat going forward, but on my production boxes (slurpd master->slave) it took only 45 minutes.

I discussed this time disparity with Howard, and he made some modifications to syncrepl (full sync mode) that allowed the master to take the changes in around 16 minutes. However, the slaves still took several hours to catch up on these modifications, which meant they were out of sync for long periods of time. Howard and I then discussed putting together the syncrepl delta method, using the accesslog backend (as previously discussed on -devel). This worked (after some bugs in accesslog were fixed), where it took 37 minutes to push the updates through the master. 2 of the 3 replica's finished within a few seconds of the master, and the 3rd slave finished within 5 minutes of the master.

However, the problem with the way we currently have to set up delta-syncrepl (via accesslog) is that there is no way for a slave to become fully refreshed if its contextCSN is out of date. It looks like this would take an extension to the syncrepl protocol for this to be done properly. Objections? comments?


Quanah Gibson-Mount
Principal Software Developer
ITSS/Shared Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

"These censorship operations against schools and libraries are stronger
than ever in the present religio-political climate. They often focus on
fantasy and sf books, which foster that deadly enemy to bigotry and blind
faith, the imagination." -- Ursula K. Le Guin