Re: (ITS#6406) back-ldif fails test043-delta-syncrepl

[hyc@symas.com]

h.b.furuseth@usit.uio.no wrote:
> I wrote:
>> This often fails (producer and consumer databases differ):
>>      SLAPD_DEBUG=0x4105 ./run -b ldif test043-delta-syncrepl
>> when configured with --enable-accesslog.
>
> Apparently syncprov sends changes in the order it gets them from an
> accesslog search, so it needs accesslog to return change records ordered
> by change CSN.  Otherwise the consumer discards changes, complaining:
>
> do_syncrep2: rid=001 CSN too old, ignoring 20091130202958.564835Z#000000#000#000000
>
> This works with hdb/bdb under normal operation:  They return entries
> ordered by entry ID, which always increases as new entries are added.
> It works out since something (syncrepl or accesslog) serializes writes
> so change records are added to the log database in the same order as
> the changes are applied to the main database.
>
> With back-ndb, I imagine somthing like an SQL 'ORDER BY<reqStart or
> entryCSN>' will do the trick, if it isn't already handled.

back-ndb also orders by entryID.

> back-ldif does not support this: It returns changes memcmp-ordered
> by normalized reqStart.  Normalization "<t>.210000Z" ->  "<t>.21Z"
> makes that value sort after "<t>.219999Z".

> Anyway, if I have understood this correctly, this requirement needs to
> be documented, and it looks rather fragile anyway.  Only databases
> supporing this use of accesslog.  The admin must not slapcat the
> accesslog, rearrange it (maybe he's looking for changes to something
> specific), and then slapadd it back.  Not something which would happen
> every day, but there's no reason people _wouldn't_ do it either.  LDAP
> defines elements as unordered, after all.

I thought it was obvious; a "log" is an ordered transcript of events. If you 
rearrange the sequence of events then you invalidate it.

> What happens if a consumer does a full refresh from a provider without
> accesslog, and is also running syncprov with accesslog?  Do these
> changes end up in the consumer accesslog?  I don't suppose the refresh
> can send entries by entryCSN order, since there may be children that
> were changed before parents.

True, the changes will be written out of order. But while a consumer is in 
refresh mode, it won't be updating its contextCSN. And while the contextCSN 
doesn't change, the provider won't be sending CSNs in the cookie it sends with 
any changes. (And the cookie CSN is what matters, not any entryCSN in the 
entry's attributes.) So any downstream consumer will accept these changes 
implicitly, since there is no CSN to check.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/