Issue 7710 - contextCSN values not updated by internal non-replicated ops
Summary: contextCSN values not updated by internal non-replicated ops
Status: VERIFIED FIXED
Alias: None
Product: OpenLDAP
Classification: Unclassified
Component: slapd (show other issues)
Version: 2.4.36
Hardware: All All
: --- normal
Target Milestone: ---
Assignee: OpenLDAP project
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-09-26 15:48 UTC by Michael Ströder
Modified: 2014-08-01 21:04 UTC (History)
0 users

See Also:


Attachments
openldap-testbed-its7710.tar.bz2 (6.52 KB, application/x-tar)
2013-10-04 13:40 UTC, Michael Ströder
Details

Note You need to log in before you can comment on or make changes to this issue.
Description Michael Ströder 2013-09-26 15:48:02 UTC
Full_Name: 
Version: 2.4.36
OS: 
URL: 
Submission from: (NULL) (2001:8d8:1fe:7:d6be:d9ff:fe09:1b7c)


Configuration:
- MMR with 3 nodes and normal syncrepl.
- slapo-memberof used to populate/update attribute 'memberOf' in user entries.

If a group entry is modified and slapo-memberof updates the attribute 'memberOf'
in the member's entry the contextCSN value of the server where the LDAP request
was sent to is not updated on the other MMR replicas.

All changes are correctly done though.

After restart contextCSN values are all the same on all MMR replicas.
Comment 1 Howard Chu 2013-09-30 23:14:55 UTC
michael@stroeder.com wrote:
> Full_Name:
> Version: 2.4.36
> OS:
> URL:
> Submission from: (NULL) (2001:8d8:1fe:7:d6be:d9ff:fe09:1b7c)
>
>
> Configuration:
> - MMR with 3 nodes and normal syncrepl.
> - slapo-memberof used to populate/update attribute 'memberOf' in user entries.
>
> If a group entry is modified and slapo-memberof updates the attribute 'memberOf'
> in the member's entry the contextCSN value of the server where the LDAP request
> was sent to is not updated on the other MMR replicas.

I don't understand this sentence, can you rephrase?

Also see ITS#6915.
>
> All changes are correctly done though.
>
> After restart contextCSN values are all the same on all MMR replicas.
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 2 Howard Chu 2013-09-30 23:15:32 UTC
changed state Open to Feedback
Comment 3 Michael Ströder 2013-10-01 07:54:14 UTC
hyc@symas.com wrote:
>> Configuration:
>> - MMR with 3 nodes and normal syncrepl.
>> - slapo-memberof used to populate/update attribute 'memberOf' in user entries.
>>
>> If a group entry is modified and slapo-memberof updates the attribute 'memberOf'
>> in the member's entry the contextCSN value of the server where the LDAP request
>> was sent to is not updated on the other MMR replicas.
> 
> I don't understand this sentence, can you rephrase?

I looked into this a bit more:

We have three MMR nodes, let's simply enumerate them as 1, 2 and 3.

When modifying a group entry on 1 'memberOf' gets *locally* updated and the
contextCSN value for server 1 gets also updated. So far so good.

Now the group modification are pulled by 2 and 3 and slapo-memberof installed
there *locally* updates attr memberOf and the *local* contextCSN value for 2
and 3 are updated with new timestamp. But the updated contextCSN values of 2
and 3 never get replicated to the other instances! So my monitoring check
shows a long-lasting replication gap. After next server restart all contextCSN
values get updated.

> Also see ITS#6915.

Sorry, glancing over ITS#6915 I don't understand what it means in this context.

Ciao, Michael.

Comment 4 Howard Chu 2013-10-01 08:01:48 UTC
Michael Ströder wrote:
> hyc@symas.com wrote:
>>> Configuration:
>>> - MMR with 3 nodes and normal syncrepl.
>>> - slapo-memberof used to populate/update attribute 'memberOf' in user entries.
>>>
>>> If a group entry is modified and slapo-memberof updates the attribute 'memberOf'
>>> in the member's entry the contextCSN value of the server where the LDAP request
>>> was sent to is not updated on the other MMR replicas.
>>
>> I don't understand this sentence, can you rephrase?
>
> I looked into this a bit more:
>
> We have three MMR nodes, let's simply enumerate them as 1, 2 and 3.
>
> When modifying a group entry on 1 'memberOf' gets *locally* updated and the
> contextCSN value for server 1 gets also updated. So far so good.
>
> Now the group modification are pulled by 2 and 3 and slapo-memberof installed
> there *locally* updates attr memberOf and the *local* contextCSN value for 2
> and 3 are updated with new timestamp. But the updated contextCSN values of 2
> and 3 never get replicated to the other instances! So my monitoring check
> shows a long-lasting replication gap. After next server restart all contextCSN
> values get updated.
>
>> Also see ITS#6915.
>
> Sorry, glancing over ITS#6915 I don't understand what it means in this context.

The point of #6915 is that internal ops are not supposed to affect the 
contextCSN. If you're seeing new contextCSNs on 2 and 3 solely due to 
slapo-memberof then some part of #6915 is still broken. We'll need a duplicate 
setup to reproduce this...

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 5 marco.pizzoli@gmail.com 2013-10-01 11:02:27 UTC
Hi, this *could* be also the root cause of a problem I found some times ago
joking with a particular OL cluster scenario in which I failed to obtain
all entries correctly populated. Please tell me if this could be related.

Long story short:
4-way multimaster cluster --> cluster A
3-way multimaster cluster --> cluster B

one of the members of cluster A has also configured, as provider, one of
the member of cluster B.

By modifying data on any of the members of cluster A I should be able to
see the modification also on any member of cluster B. Correct?
Well, this was failing sometimes on 1 or 2 members.

I had load balancer health-checks continuously polling all of members from
both A and B:
- bind
- search
- unbind

All the members of the cluster were using the contrib/slapo-lastbind
overlay. So the internal authTimestamp attribute populated by an internal
operation.
Could it be that the contextCSN of one node of the cluster were newer than
the one of the providers?

I'm not too expert, just trying to be of help by sharing experiences.
Thanks for reading
Marco
Comment 6 Michael Ströder 2013-10-01 17:36:27 UTC
Marco Pizzoli wrote:
> Hi, this *could* be also the root cause of a problem I found some times ago
> joking with a particular OL cluster scenario in which I failed to obtain
> all entries correctly populated. Please tell me if this could be related.
> [..]
> All the members of the cluster were using the contrib/slapo-lastbind
> overlay.

We're also using slapo-lastbind but deactivating does not make a difference
when modifying group entries (tested today).

Are you using slapo-memberof at all?

Ciao, Michael.

Comment 7 marco.pizzoli@gmail.com 2013-10-01 17:39:21 UTC
Hi Michael,
yes I was, but I'm quite sure I didn't have any memberOf-operation running
in the meantime.

Marco


On Tue, Oct 1, 2013 at 7:36 PM, Michael Ströder <michael@stroeder.com>wrote:

> Marco Pizzoli wrote:
> > Hi, this *could* be also the root cause of a problem I found some times
> ago
> > joking with a particular OL cluster scenario in which I failed to obtain
> > all entries correctly populated. Please tell me if this could be related.
> > [..]
> > All the members of the cluster were using the contrib/slapo-lastbind
> > overlay.
>
> We're also using slapo-lastbind but deactivating does not make a difference
> when modifying group entries (tested today).
>
> Are you using slapo-memberof at all?
>
> Ciao, Michael.
>
Comment 8 Michael Ströder 2013-10-04 13:40:55 UTC
See attached a canned MMR config.

The behaviour regarding contextCSN is really strange. You have to play around
with some more modify requests changing the group membership of e.g.
uid=michael,ou=users,dc=example,dc=com

If contextCSN values are out-of-sync restarting one slapd brings them up to
same values again, but the older(!) ones.

Ciao, Michael.

Comment 9 Michael Ströder 2013-10-04 17:03:43 UTC
Hmmpf, the attachment did not come through.

Download config here:
http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2

Comment 10 Michael Ströder 2013-10-06 15:52:22 UTC
I've updated the test data with LDIF files and a README describing the LDAP
operations to perform:

http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2

I consider this to be a serious issue which could be the cause for other
replication issues including dead-locks we're experiencing. (Maybe there are
similar issues in slapo-refint since dead-locks in our deployment arised when
moving entries into different subtree.)

Ciao, Michael.

The README copied here for direct access:

----------------------------------------------------------------------------

0. Start both servers by invoking start-slapd1.sh and start-slapd2.sh

----------------------------------------------------------------------------

1. Add test entries on first server:

$ ldapadd -H ldap://localhost:2071 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 1_ldapadd.ldif
adding new entry "dc=example,dc=com"
[..]
adding new entry "cn=replicas,ou=groups,dc=example,dc=com"

You can now see contextCSN value of first server on both servers (as expected):

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000

----------------------------------------------------------------------------

2. Send a simple modification to second server:

$ ldapmodify -H ldap://localhost:2072 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 2_ldapmodify.ldif
modifying entry "uid=michael,ou=users,dc=example,dc=com"

You can now see contextCSN value of second server on both servers (as expected):

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

----------------------------------------------------------------------------

3. Modification of group membership on first server:

$ ldapmodify -H ldap://localhost:2071 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 3_ldapmodify.ldif
modifying entry "cn=testgroup1,ou=groups,dc=example,dc=com"

Now the contextCSN values differ:

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154449.514135Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

Comment 11 Christian Kratzer 2013-10-06 20:20:26 UTC
Hi Michael,

On Sun, 6 Oct 2013, michael@stroeder.com wrote:
> I've updated the test data with LDIF files and a README describing the LDAP
> operations to perform:
>
> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>
> I consider this to be a serious issue which could be the cause for other
> replication issues including dead-locks we're experiencing. (Maybe there are
> similar issues in slapo-refint since dead-locks in our deployment arised when
> moving entries into different subtree.)

I can confirm that I can see the same bug with your testcase.

It seems that restarting the second slapd instance fixes the contextCSN
issue just because it scans all it's entries and does the right thing.

Even a slapcat of both directories shows the issue.

Things that I find disturbing:

After applying the group membership the member dn: gets it's memberOf 
attribute updated accordingly. But the entryCSN stays unchanged.

Entry before applying group membership:

     # michael, users, example.com
     dn: uid=michael,ou=users,dc=example,dc=com
     uid: michael
     objectClass: account
     objectClass: simpleSecurityObject
     userPassword:: dGVzdHNlY3JldA==
     description: test
     entryCSN: 20131006193757.912740Z#000000#002#000000

Entry after applying group membership is obviously updated but still has
the same entryCSN:

     # michael, users, example.com
     dn: uid=michael,ou=users,dc=example,dc=com
     uid: michael
     objectClass: account
     objectClass: simpleSecurityObject
     userPassword:: dGVzdHNlY3JldA==
     memberOf: cn=admins,ou=groups,dc=example,dc=com
     memberOf: cn=testgroup1,ou=groups,dc=example,dc=com
     description: test
     entryCSN: 20131006193757.912740Z#000000#002#000000

I can also confirm the subsequent contextCSN mismatch.

I too consider this a serious issue.

If I read Howards reply correctly and from what I see in the source code
of the memberOf overlay the strategy here seems to be not to replicate the
side effect of adding the memberOf: entry.

OpenLDAP relies on the side effect happening on the replicas through the
same codepath when the member: attribute in the group gets updated
through replication.

This sounds very similar to the difference between:

1. statement replication in sql databases ala mysql

2. log based replication ala postgresql

OpenLDAP is doing 1. here while 2. would perhaps have been the better option.

I understand turning this around is propably too late at this point in time.

Looks like we will just have to work out the bugs the hard way.

I am using slapo-lastbind and slapo-ppolicy extensively in my current
project and will keep my eyes open for similar issues resulting from not
replicating the side effects that these overlays produce.

Greetings
Christian

-- 
Christian Kratzer                      CK Software GmbH
Email:   ck@cksoft.de                  Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0          D-71126 Gaeufelden
Fax:     +49 7032 893 997 - 9          HRB 245288, Amtsgericht Stuttgart
Web:     http://www.cksoft.de/         Geschaeftsfuehrer: Christian Kratzer

Comment 12 Michael Ströder 2013-10-06 22:22:21 UTC
Christian Kratzer wrote:
> This sounds very similar to the difference between:
> 
> 1. statement replication in sql databases ala mysql
> 
> 2. log based replication ala postgresql
> 
> OpenLDAP is doing 1. here while 2. would perhaps have been the better option.

From my understanding 2. is OpenLDAP's delta-syncrepl.

Ciao, Michael.

Comment 13 Howard Chu 2013-10-09 05:03:17 UTC
changed notes
changed state Feedback to Test
moved from Incoming to Software Bugs
Comment 14 Quanah Gibson-Mount 2013-10-09 09:27:34 UTC
changed notes
changed state Test to Release
Comment 15 Howard Chu 2013-10-09 12:01:40 UTC
michael@stroeder.com wrote:
> I've updated the test data with LDIF files and a README describing the LDAP
> operations to perform:
>
> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>
> I consider this to be a serious issue which could be the cause for other
> replication issues including dead-locks we're experiencing. (Maybe there are
> similar issues in slapo-refint since dead-locks in our deployment arised when
> moving entries into different subtree.)

Thanks, fixed now in master. Looks like refint may have run into the same 
issue, also patched in master.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 16 Michael Ströder 2013-10-09 13:11:40 UTC
On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com> wrote

> michael@stroeder.com wrote:
> > I've updated the test data with LDIF files and a README describing the LDAP
> > operations to perform:
> >
> > http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
> >
> > I consider this to be a serious issue which could be the cause for other
> > replication issues including dead-locks we're experiencing. (Maybe there
> > are similar issues in slapo-refint since dead-locks in our deployment
> > arised when moving entries into different subtree.)
> 
> Thanks, fixed now in master. Looks like refint may have run into the same 
> issue, also patched in master.

contextCSN values are correct now.

But now I see this during initial refresh phase of second server:

send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
already exists"

Is that expected?

Ciao, Michael.


52555576 syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD)
52555576 syncrepl_entry: rid=002 inserted UUID
b5736b9e-c52f-1032-99c4-31a68f008587
52555576 mdb_idl_fetch_key: [3bec70f1]
52555576 send_ldap_result: err=0 matched="" text=""
52555576 syncrepl_entry: rid=002 be_search (0)
52555576 syncrepl_entry: rid=002 cn=replicas,ou=groups,dc=example,dc=com
52555576 ==> mdb_add: cn=replicas,ou=groups,dc=example,dc=com
52555576 mdb_idl_insert_keys: b [f4f2ead9]
52555576 mdb_idl_insert_keys: b [8f2b53e8]
52555576 mdb_idl_insert_keys: b [cb13035d]
52555576 mdb_idl_insert_keys: b [c14a3e76]
52555576 mdb_idl_insert_keys: b [3bec70f1]
52555576 mdb_idl_insert_keys: b 
52555576 send_ldap_result: err=0 matched="" text=""
52555576 mdb_modify: cn=slapd-1,ou=sys,dc=example,dc=com
52555576 mdb_modify_internal: add memberOf
52555576 dnMatch 0
	"cn=replicas,ou=groups,dc=example,dc=com"
	"cn=replicas,ou=groups,dc=example,dc=com"
52555576 mdb_modify_internal: 20 modify/add: memberOf: value #0 already exists
52555576 send_ldap_result: err=20 matched="" text="modify/add: memberOf: value
#0 already exists"
52555576 conn=-1 op=0: memberof_value_modify
DN="cn=slapd-1,ou=sys,dc=example,dc=com" add
memberOf="cn=replicas,ou=groups,dc=example,dc=com" failed err=20
52555576 mdb_modify: cn=slapd-2,ou=sys,dc=example,dc=com
52555576 mdb_modify_internal: add memberOf
52555576 dnMatch 0
	"cn=replicas,ou=groups,dc=example,dc=com"
	"cn=replicas,ou=groups,dc=example,dc=com"
52555576 mdb_modify_internal: 20 modify/add: memberOf: value #0 already exists
52555576 send_ldap_result: err=20 matched="" text="modify/add: memberOf: value
#0 already exists"
52555576 conn=-1 op=0: memberof_value_modify
DN="cn=slapd-2,ou=sys,dc=example,dc=com" add
memberOf="cn=replicas,ou=groups,dc=example,dc=com" failed err=20
52555576 syncrepl_entry: rid=002 be_add cn=replicas,ou=groups,dc=example,dc=com
(0)
52555576 do_syncrep2: rid=002 LDAP_RES_INTERMEDIATE - REFRESH_DELETE
52555576 do_syncrep2: rid=002
cookie=rid=002,sid=001,csn=20131009130900.890770Z#000000#001#000000
52555576 slap_queue_csn: queing 0x7fd79c10c200
20131009130900.890770Z#000000#001#000000
52555576 base_candidates: base: "dc=example,dc=com" (0x00000001)
52555576 send_ldap_result: err=0 matched="" text=""
52555576 mdb_modify: dc=example,dc=com
52555576 mdb_modify_internal: replace contextCSN
52555576 send_ldap_result: err=0 matched="" text=""
52555576 slap_graduate_commit_csn: removing 0x7fd79c10c170
20131009130900.890770Z#000000#001#000000


Comment 17 Howard Chu 2013-10-09 13:56:24 UTC
Michael Ströder wrote:
> On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com> wrote
>
>> michael@stroeder.com wrote:
>>> I've updated the test data with LDIF files and a README describing the LDAP
>>> operations to perform:
>>>
>>> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>>>
>>> I consider this to be a serious issue which could be the cause for other
>>> replication issues including dead-locks we're experiencing. (Maybe there
>>> are similar issues in slapo-refint since dead-locks in our deployment
>>> arised when moving entries into different subtree.)
>>
>> Thanks, fixed now in master. Looks like refint may have run into the same
>> issue, also patched in master.
>
> contextCSN values are correct now.
>
> But now I see this during initial refresh phase of second server:
>
> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
> already exists"
>
> Is that expected?

Not seeing that with your testcase.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 18 Howard Chu 2013-10-09 14:05:43 UTC
Howard Chu wrote:
> Michael Ströder wrote:
>> On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com> wrote
>>
>>> michael@stroeder.com wrote:
>>>> I've updated the test data with LDIF files and a README describing the LDAP
>>>> operations to perform:
>>>>
>>>> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>>>>
>>>> I consider this to be a serious issue which could be the cause for other
>>>> replication issues including dead-locks we're experiencing. (Maybe there
>>>> are similar issues in slapo-refint since dead-locks in our deployment
>>>> arised when moving entries into different subtree.)
>>>
>>> Thanks, fixed now in master. Looks like refint may have run into the same
>>> issue, also patched in master.
>>
>> contextCSN values are correct now.
>>
>> But now I see this during initial refresh phase of second server:
>>
>> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
>> already exists"
>>
>> Is that expected?
>
> Not seeing that with your testcase.
>
Ah, I see it now. Yes, it's normal; memberOf on the provider already added the 
relevant values. The consumer receives a group entry and performs the same set 
of memberof updates, which are redundant at that point. It's harmless.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 19 Michael Ströder 2013-10-09 14:33:33 UTC
On Wed, 9 Oct 2013 14:05:52 GMT hyc@symas.com wrote
> Howard Chu wrote:
> > Michael Ströder wrote:
> >> But now I see this during initial refresh phase of second server:
> >>
> >> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
> >> already exists"
> >>
> >> Is that expected?
> >
> > Not seeing that with your testcase.
> >
> Ah, I see it now. Yes, it's normal; memberOf on the provider already added
> the  relevant values. The consumer receives a group entry and performs the
> same set  of memberof updates, which are redundant at that point. It's
> harmless. 

Hmm, wouldn't it be reasonable to strip those attributes marked as
non-replication attrs when generating syncrepl search results at the provider?
(Even if consumer asks for attrs=*,+)

Ciao, Michael.


Comment 20 Howard Chu 2013-10-09 15:13:24 UTC
Michael Ströder wrote:
> On Wed, 9 Oct 2013 14:05:52 GMT hyc@symas.com wrote
>> Howard Chu wrote:
>>> Michael Ströder wrote:
>>>> But now I see this during initial refresh phase of second server:
>>>>
>>>> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
>>>> already exists"
>>>>
>>>> Is that expected?
>>>
>>> Not seeing that with your testcase.
>>>
>> Ah, I see it now. Yes, it's normal; memberOf on the provider already added
>> the  relevant values. The consumer receives a group entry and performs the
>> same set  of memberof updates, which are redundant at that point. It's
>> harmless.
>
> Hmm, wouldn't it be reasonable to strip those attributes marked as
> non-replication attrs when generating syncrepl search results at the provider?
> (Even if consumer asks for attrs=*,+)

slapd already strips DSA-specific attributes before sending a syncrepl entry. 
memberOf is not marked in the schema as DSA-specific. This is working as designed.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 21 Michael Ströder 2013-10-09 15:43:13 UTC
On Wed, 09 Oct 2013 08:13:24 -0700 Howard Chu <hyc@symas.com> wrote
> slapd already strips DSA-specific attributes before sending a syncrepl entry.
> memberOf is not marked in the schema as DSA-specific. This is working as
> designed. 

IIRC attribute 'memberOf' was replicated in former releases. So it was not
DSA-specific back then.

Then the behaviour was changed in a more recent OpenLDAP release. Nowadays each
replica has to be configured with slapo-memberof performing *local* operations.
Therefore I'd argue that 'memberOf' should be marked DSA-specific now since the
*local* configuration is significant for its content.
Note that there is no formal specification for attribute 'memberOf' at all.

I have deployments where most users are member of more than 10 groups,
sometimes more than 20. So not sending 'memberOf' could save quite a lot of
network traffic.

What are your objections against marking 'memberOf' as DSA-specific?

(I vaguely remember this being discussed before without result though.)

Ciao, Michael.


Comment 22 Michael Ströder 2013-10-09 17:42:41 UTC
michael@stroeder.com wrote:
> On Wed, 09 Oct 2013 08:13:24 -0700 Howard Chu <hyc@symas.com> wrote
>> slapd already strips DSA-specific attributes before sending a syncrepl entry.
>> memberOf is not marked in the schema as DSA-specific. This is working as
>> designed. 
> 
> IIRC attribute 'memberOf' was replicated in former releases. So it was not
> DSA-specific back then.
> 
> Then the behaviour was changed in a more recent OpenLDAP release. Nowadays each
> replica has to be configured with slapo-memberof performing *local* operations.
> Therefore I'd argue that 'memberOf' should be marked DSA-specific now since the
> *local* configuration is significant for its content.
> Note that there is no formal specification for attribute 'memberOf' at all.
> 
> I have deployments where most users are member of more than 10 groups,
> sometimes more than 20. So not sending 'memberOf' could save quite a lot of
> network traffic.
> 
> What are your objections against marking 'memberOf' as DSA-specific?
> 
> (I vaguely remember this being discussed before without result though.)

Additionally consider partial replication where only a subset of group entries
is present on a certain consumer. One would not want to have 'memberOf' point
to group entries not really existing on that consumer.

=> 'memberOf' is definitely DSA-specific.

Ciao, Michael.

Comment 23 Michael Ströder 2013-10-10 10:32:52 UTC
With current master and re24 which have the fix for this ITS I now see seg
faults in test023-refint test057-memberof-refint.


Comment 24 Howard Chu 2013-10-10 11:05:20 UTC
michael@stroeder.com wrote:
> With current master and re24 which have the fix for this ITS I now see seg
> faults in test023-refint test057-memberof-refint.

Fixed now in master.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 25 Michael Ströder 2013-10-10 11:22:07 UTC
On Thu, 10 Oct 2013 11:05:28 GMT hyc@symas.com wrote
> michael@stroeder.com wrote:
> > With current master and re24 which have the fix for this ITS I now see seg
> > faults in test023-refint test057-memberof-refint.
> 
> Fixed now in master.

Thanks. These tests are passing now.

@Quanah: Would be nice if this last fix could be also ported to RE24. Thanks in
advance.

Ciao, Michael.


Comment 26 Quanah Gibson-Mount 2013-10-10 16:54:15 UTC
--On Thursday, October 10, 2013 11:22 AM +0000 michael@stroeder.com wrote:

> On Thu, 10 Oct 2013 11:05:28 GMT hyc@symas.com wrote
>> michael@stroeder.com wrote:
>> > With current master and re24 which have the fix for this ITS I now see
>> > seg faults in test023-refint test057-memberof-refint.
>>
>> Fixed now in master.
>
> Thanks. These tests are passing now.
>
> @Quanah: Would be nice if this last fix could be also ported to RE24.
> Thanks in advance.

Not particularly necessary to make requests like this.  I read all ITS mail 
and commit traffic.

--Quanah



--

Quanah Gibson-Mount
Architect - Server
Zimbra Software, LLC
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Comment 27 Quanah Gibson-Mount 2013-10-28 09:28:09 UTC
changed notes
changed state Release to Closed
Comment 28 OpenLDAP project 2014-08-01 21:04:49 UTC
fixed in master
fixed in RE24
fixed in RE25