Re: (ITS#3939) min/max function extension to LDAP protocol

[Kurt@OpenLDAP.org]

At 11:22 PM 8/19/2005, Quanah Gibson-Mount wrote:


>--On Friday, August 19, 2005 11:11 PM -0700 "Kurt D. Zeilenga" <Kurt@OpenLDAP.org> wrote:
>
>>Well, what I noted was that there was an existing protocol
>>mechanism to request return the entry with the lowest/highest
>>CSN.
>>
>>I pretty much ignored most of rest of your post as it didn't
>>make much sense to me at the time (and still doesn't).
>
>Okay, maybe this will explain it more:
>
>When the consumer connects to the provider, it tries to determine if it needs to synch any data or not.

It seems here you that you intend 'it' to refer to the consumer.
That's incorrect.  When using LDAP sync, it is the provider
which determines what data needs to be sent to sync the consumer.
The provider does so by providing the cookie.

So most of what follows makes little sense to me.

>To do that, it uses the value of its cookie, and compares that value to the value of the cookie on the master.

The cookie is suppose to be opaque.  The consumer has no way to
compare it to anything on the provider.  The consumer should
not attempt to parse the CSN.  Doing so is simply counter to
the design of LDAP sync.

>If its cookie value is not equivalent to what is on the master, it then looks for *all* values <= to its value.  This works fine in a very small database (say 10k entries).  It takes around 20-30 minutes in my 400k database.

Well, an ordering index on the provider might speed that up.
But this is an provider side implementation detail.   No need
for a "mix/max function extension to LDAP" to do that.

>It would take even longer on a very large database (say 50 million entries).  The more consumers you have, the worse it gets, as well. By giving the consumer a way to immediately get the smallest cookie value that the provider has,

No need for the consumer to do anything special with the cookie.

>the consumer can immediately know whether or not its cookie is valid, there by skipping the <= search. If its cookie isn't valid, it does a full resynch of data.  If its cookie is valid, then it only looks for entries that have been modified since the value of its cookie.  That was the point of the recent CSN checking change done between 2.3.5 and 2.3.6, which took care of some cases to allow an immediate equality check (before it defaulted to always doing <=).

I'll have to examine these changes more closely.  I hope it was just
a provider implementation optimization...

Consumers MUST NOT parse the cookie.  Doing so will cause problems
when we run into other servers implementing LDAP sync.

>But there are still times when the master and replica can differ in value, and for syncrepl to be worthwhile, there needs to be a way to get an immediate yes/no answer as to whether or not it needs to do a full resync.
>
>
>--Quanah
>
>--
>Quanah Gibson-Mount
>Principal Software Developer
>ITSS/Shared Services
>Stanford University
>GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
>
>"These censorship operations against schools and libraries are stronger
>than ever in the present religio-political climate. They often focus on
>fantasy and sf books, which foster that deadly enemy to bigotry and blind
>faith, the imagination." -- Ursula K. Le Guin