OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Documentation/6406
Full headers

From: h.b.furuseth@usit.uio.no
Subject: back-ldif fails test043-delta-syncrepl
Compose comment
Download message
State:
0 replies:
3 followups: 1 2 3

Major security issue: yes  no

Notes:

Notification:


Date: Mon, 30 Nov 2009 20:40:04 +0000
From: h.b.furuseth@usit.uio.no
To: openldap-its@OpenLDAP.org
Subject: back-ldif fails test043-delta-syncrepl
Full_Name: Hallvard B Furuseth
Version: 2.4.20
OS: Linux x86_64
URL: http://folk.uio.no/hbf/test043-ldif.tgz
Submission from: (NULL) (129.240.6.233)
Submitted by: hallvard


This often fails (producer and consumer databases differ):
    SLAPD_DEBUG=0x4105 ./run -b ldif test043-delta-syncrepl
when configured with --enable-accesslog.

The error has been the same several times:  server2's object
   cn=Some Staff,ou=Groups,dc=example,dc=com
has the extra attribute
   description: Everyone in the sample data

Test output + testrun/ URL enclosed

Followup 1

Download message
From: Hallvard B Furuseth <h.b.furuseth@usit.uio.no>
Date: Sat, 5 Dec 2009 18:11:36 +0100
To: openldap-its@openldap.org
Subject: Re: (ITS#6406) back-ldif fails test043-delta-syncrepl
I wrote:
> This often fails (producer and consumer databases differ):
>     SLAPD_DEBUG=0x4105 ./run -b ldif test043-delta-syncrepl
> when configured with --enable-accesslog.

Apparently syncprov sends changes in the order it gets them from an
accesslog search, so it needs accesslog to return change records ordered
by change CSN.  Otherwise the consumer discards changes, complaining:

do_syncrep2: rid=001 CSN too old, ignoring
20091130202958.564835Z#000000#000#000000

This works with hdb/bdb under normal operation:  They return entries
ordered by entry ID, which always increases as new entries are added.
It works out since something (syncrepl or accesslog) serializes writes
so change records are added to the log database in the same order as
the changes are applied to the main database.

With back-ndb, I imagine somthing like an SQL 'ORDER BY <reqStart or
entryCSN>' will do the trick, if it isn't already handled.

back-ldif does not support this: It returns changes memcmp-ordered
by normalized reqStart.  Normalization "<t>.210000Z" -> "<t>.21Z"
makes that value sort after "<t>.219999Z".


Anyway, if I have understood this correctly, this requirement needs to
be documented, and it looks rather fragile anyway.  Only databases
supporing this use of accesslog.  The admin must not slapcat the
accesslog, rearrange it (maybe he's looking for changes to something
specific), and then slapadd it back.  Not something which would happen
every day, but there's no reason people _wouldn't_ do it either.  LDAP
defines elements as unordered, after all.

What happens if a consumer does a full refresh from a provider without
accesslog, and is also running syncprov with accesslog?  Do these
changes end up in the consumer accesslog?  I don't suppose the refresh
can send entries by entryCSN order, since there may be children that
were changed before parents.

-- 
Hallvard



Followup 2

Download message
Date: Sat, 05 Dec 2009 12:03:33 -0800
From: Howard Chu <hyc@symas.com>
To: h.b.furuseth@usit.uio.no
CC: openldap-its@openldap.org
Subject: Re: (ITS#6406) back-ldif fails test043-delta-syncrepl
h.b.furuseth@usit.uio.no wrote:
> I wrote:
>> This often fails (producer and consumer databases differ):
>>      SLAPD_DEBUG=0x4105 ./run -b ldif test043-delta-syncrepl
>> when configured with --enable-accesslog.
>
> Apparently syncprov sends changes in the order it gets them from an
> accesslog search, so it needs accesslog to return change records ordered
> by change CSN.  Otherwise the consumer discards changes, complaining:
>
> do_syncrep2: rid=001 CSN too old, ignoring
20091130202958.564835Z#000000#000#000000
>
> This works with hdb/bdb under normal operation:  They return entries
> ordered by entry ID, which always increases as new entries are added.
> It works out since something (syncrepl or accesslog) serializes writes
> so change records are added to the log database in the same order as
> the changes are applied to the main database.
>
> With back-ndb, I imagine somthing like an SQL 'ORDER BY<reqStart or
> entryCSN>' will do the trick, if it isn't already handled.

back-ndb also orders by entryID.

> back-ldif does not support this: It returns changes memcmp-ordered
> by normalized reqStart.  Normalization "<t>.210000Z" -> 
"<t>.21Z"
> makes that value sort after "<t>.219999Z".

> Anyway, if I have understood this correctly, this requirement needs to
> be documented, and it looks rather fragile anyway.  Only databases
> supporing this use of accesslog.  The admin must not slapcat the
> accesslog, rearrange it (maybe he's looking for changes to something
> specific), and then slapadd it back.  Not something which would happen
> every day, but there's no reason people _wouldn't_ do it either.  LDAP
> defines elements as unordered, after all.

I thought it was obvious; a "log" is an ordered transcript of events. If you 
rearrange the sequence of events then you invalidate it.

> What happens if a consumer does a full refresh from a provider without
> accesslog, and is also running syncprov with accesslog?  Do these
> changes end up in the consumer accesslog?  I don't suppose the refresh
> can send entries by entryCSN order, since there may be children that
> were changed before parents.

True, the changes will be written out of order. But while a consumer is in 
refresh mode, it won't be updating its contextCSN. And while the contextCSN 
doesn't change, the provider won't be sending CSNs in the cookie it sends with 
any changes. (And the cookie CSN is what matters, not any entryCSN in the 
entry's attributes.) So any downstream consumer will accept these changes 
implicitly, since there is no CSN to check.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 3

Download message
From: Hallvard B Furuseth <h.b.furuseth@usit.uio.no>
Date: Sat, 5 Dec 2009 21:52:06 +0100
To: Howard Chu <hyc@symas.com>
Cc: openldap-its@openldap.org
Subject: Re: (ITS#6406) back-ldif fails test043-delta-syncrepl
Howard Chu writes:
>> With back-ndb, I imagine somthing like an SQL 'ORDER BY<reqStart or
>> entryCSN>' will do the trick, if it isn't already handled.
>
> back-ndb also orders by entryID.

Thanks, I'll note that for the test.

>> Anyway, if I have understood this correctly, this requirement needs to
>> be documented, and it looks rather fragile anyway.  Only databases
>> supporing this use of accesslog.  The admin must not slapcat the
>> accesslog, rearrange it (maybe he's looking for changes to something
>> specific), and then slapadd it back.  Not something which would happen
>> every day, but there's no reason people _wouldn't_ do it either.  LDAP
>> defines elements as unordered, after all.
> 
> I thought it was obvious; a "log" is an ordered transcript of events.
> If you rearrange the sequence of events then you invalidate it.

No, it's not obvious that the order of slapadding entires has any
influence on their order returned from search.  The LDAP spec doesn't
say so, nor man 5 slapd-bdb, nor do the syncprov or accesslog manpages
say that they need it.

Obviously *something* need to take care of ordering, but as far as
the user knows that could be accesslog or syncprov.  Actually, the
slapo-accesslog(5) suggestion to index reqStart may give a careful
reader a hint about how the ordering happens.

So anyway, syncprov needs to document that it doesn't ensure this
ordering but depends on the admin getting it right.

Long-term I'd also like some way for syncprov to catch errors if the
admin does get tings like this wrong.  E.g. let accesslog require an
attribute to be set in the log database which promises ordering.  slapd
startup would fail if the database doesn't confirm that it can honor
this flag, and slapadd of entries out of CSN order would also fail.
Unless that breaks what you say about refresh below... anyway, all that
will have to wait for RE25 or later.

-- 
Hallvard


Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org