RE: ITS#2512 wrong order of entries in replog file

[hyc@highlandsun.com]

> -----Original Message-----
> From: owner-openldap-bugs@OpenLDAP.org
> [mailto:owner-openldap-bugs@OpenLDAP.org]On Behalf Of Kurt@OpenLDAP.org

> At 02:22 AM 12/24/2003, hyc@highlandsun.com wrote:
> >There is a partial fix for this now in CVS HEAD. The replog
> entries are now
> >timestamped according to when the operation began, instead
> of when the
> >operation was logged. The entries are still written in an
> unpredictable
> >order, but it is now feasible for the replog to be sorted
> into correct order.

> You assume the order in which they are stamped is the "correct"
> (commit) order.  This is not true.  It is still possible that
> an add stamped after another needs to be processed before that
> other.

Perhaps the current patch sets the timestamp at the wrong point, but this
possibility *can* be completely eliminated. We know that a single client
issuing requests on a single session will have those requests decoded in
order, and they will be queued to the thread pool in order, and they will
begin execution in the same order.

> That is, now we have a race to commit instead of a race to log.
> This, however, may be a better race to have.

If in your example, the second Add depends on the first because the second
entry is a child of the first, then there is no race. If the first Add hasn't
committed by the time the second Add has started, the second Add will fail
because the parent entry doesn't exist. It's not possible for these two
requests to both succeed out of order.

Also, most clients submitting requests that depend on each other will run
synchronously - i.e., they won't even submit the second request until they've
received a result for the first one. It would be foolish to do otherwise,
since a failure on the first request would preclude the second from having
any chance of success.

I note that when test008 is run with replication logging, the resulting
replog with this patch is processed by slurpd without any rejects. Without
this patch there are large numbers of rejects.

> >slurpd has been modified to attempt to sort its queue when
> it reads the
> >replog. With this patch, if operations execute at close to
> the same time and
> >are logged at close to the same time, slurpd will replicate
> them correctly.
> >However, when the log is out of order, there is no check for missing
> >operations in a sequence. An operation that is significantly
> delayed (due to
> >database contention) may still be replicated out of order if
> it is not logged
> >by the time slurpd reads the replog. I.e., if all changes
> have been written
> >to the replog, then slurpd will send them all in correct
> order. If any
> >changes are outstanding/missing when slurpd reads the log,
> then the order is
> >not guaranteed.
> >
> >  -- Howard Chu
> >  Chief Architect, Symas Corp.       Director, Highland Sun
> >  http://www.symas.com               http://highlandsun.com/hyc
> >  Symas: Premier OpenSource Development and Support
>
>
>