Re: syncrepl catchup (ITS#2713)

[quanah@stanford.edu]

--On Thursday, September 11, 2003 10:35 AM -0400 Jonghyuk Choi 
<jongchoi@us.ibm.com> wrote:

>
>> Hello,
>
>> The syncrepl process implemented in 2.2, makes from what I can tell, a
>> bad assumption.
>
>> It assumes the DB on the replica is empty, meaning that it wants to send
>> everything it has in its DB to the replica.  Since we pre-load the
>> replicas with a dump of the master's DB, there is no need for this to be
>> done.  What would be better, is for the master to generate a tag of some
>> sort, marking a starting point (probably 0?) that gets incremented as it
>> receives updates, from the time its DB was created.  That way, if I dump
>> the DB after it has gotten some updates
>
> Syncrepl uses cookies for that pupose. The contextCSN at the sync
> provider  is maintained such that it represents the current status of the
> provider replica  (this is far from trivial in the transaction
> environment).
> The syncreplCookie at the sync consumer is the largest cookie value it
> received  from the provider. When the consumer connects to the provider,
> it always  include its syncreplCookie in the request so that it can
> receive entries changed  after the corresponding state.
>
>> (so its tag is now 500, lets say), and I load that into a new replica,
>> and bring it up, the master will only catch it up from change #500
>> onward.  I have a feeling this has to be significantly faster than
>> sending our entire 450MB DB across the network to the replica, as well.
>
> If you populate the consumer with a dump of the consumer's DB
> (not of the provider's), incremental synchronization will take place.
> A syncreplCookie can be included in the consumer's dump by using slapcat
> -k.  Slapadding of this dump will create the consumerSyncSubentry
> which contains the syncreplCookie.
>
> When you start from the provider's dump, the contextCSN can be included
> in the dump by retrieving the providerSyncSubentry (use slapcat -m).
> When you want to move this dump directly to the sync consumer, the
> current workaround  would be to change the providerSyncSubentry to the
> corresponding consumerSyncSubentry  having the same cookie value.
>
> Currently, it is possible to add the providerSyncSubentry automatically
> when using slapadd (-w option).  A similar functionality can be supported
> for the consumerSyncSubentry  under the assumption that each such dump
> covers only one syncrepl search specification.  Then, it would be
> possible to populate the directory from any (provider or consumer) DB
> which includes entryCSN attribute of each entry.
>
> ------------------------
> Jong Hyuk Choi

Jong,

What would be very useful is a flag to dump a version of the database 
acceptable for a consumer.  Since our master is what receives all of our 
updates prior to propagating them, what we want is to be able to dump that 
DB and import it on a consumer.  I obviously can run a sed against what 
you've done as a workaround.  I'm coming from a standpoint of where we have 
had a catastrophic failure, and need to restore all of our servers.  To 
this end, we do a nightly dump of our master server, so that DB can be used 
to recover all of our suppliers & consumers.

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITSS/TSS/Computing Systems
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html