OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Software Bugs/7710
Full headers

From: michael@stroeder.com
Subject: contextCSN values not updated by internal non-replicated ops
Compose comment
Download message
State:
0 replies:
23 followups: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Major security issue: yes  no

Notes:

Notification:


Date: Thu, 26 Sep 2013 15:48:02 +0000
From: michael@stroeder.com
To: openldap-its@OpenLDAP.org
Subject: contextCSN values not updated by internal non-replicated ops
Full_Name: 
Version: 2.4.36
OS: 
URL: 
Submission from: (NULL) (2001:8d8:1fe:7:d6be:d9ff:fe09:1b7c)


Configuration:
- MMR with 3 nodes and normal syncrepl.
- slapo-memberof used to populate/update attribute 'memberOf' in user entries.

If a group entry is modified and slapo-memberof updates the attribute 'memberOf'
in the member's entry the contextCSN value of the server where the LDAP request
was sent to is not updated on the other MMR replicas.

All changes are correctly done though.

After restart contextCSN values are all the same on all MMR replicas.

Followup 1

Download message
Date: Mon, 30 Sep 2013 16:14:55 -0700
From: Howard Chu <hyc@symas.com>
To: michael@stroeder.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
michael@stroeder.com wrote:
> Full_Name:
> Version: 2.4.36
> OS:
> URL:
> Submission from: (NULL) (2001:8d8:1fe:7:d6be:d9ff:fe09:1b7c)
>
>
> Configuration:
> - MMR with 3 nodes and normal syncrepl.
> - slapo-memberof used to populate/update attribute 'memberOf' in user
entries.
>
> If a group entry is modified and slapo-memberof updates the attribute
'memberOf'
> in the member's entry the contextCSN value of the server where the LDAP
request
> was sent to is not updated on the other MMR replicas.

I don't understand this sentence, can you rephrase?

Also see ITS#6915.
>
> All changes are correctly done though.
>
> After restart contextCSN values are all the same on all MMR replicas.
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 2

Download message
Date: Tue, 01 Oct 2013 09:54:14 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: hyc@symas.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
hyc@symas.com wrote:
>> Configuration:
>> - MMR with 3 nodes and normal syncrepl.
>> - slapo-memberof used to populate/update attribute 'memberOf' in user
entries.
>>
>> If a group entry is modified and slapo-memberof updates the attribute
'memberOf'
>> in the member's entry the contextCSN value of the server where the LDAP
request
>> was sent to is not updated on the other MMR replicas.
> 
> I don't understand this sentence, can you rephrase?

I looked into this a bit more:

We have three MMR nodes, let's simply enumerate them as 1, 2 and 3.

When modifying a group entry on 1 'memberOf' gets *locally* updated and the
contextCSN value for server 1 gets also updated. So far so good.

Now the group modification are pulled by 2 and 3 and slapo-memberof installed
there *locally* updates attr memberOf and the *local* contextCSN value for 2
and 3 are updated with new timestamp. But the updated contextCSN values of 2
and 3 never get replicated to the other instances! So my monitoring check
shows a long-lasting replication gap. After next server restart all contextCSN
values get updated.

> Also see ITS#6915.

Sorry, glancing over ITS#6915 I don't understand what it means in this context.

Ciao, Michael.



Followup 3

Download message
Date: Tue, 01 Oct 2013 01:01:48 -0700
From: Howard Chu <hyc@symas.com>
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>,
        openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Michael Str.der wrote:
> hyc@symas.com wrote:
>>> Configuration:
>>> - MMR with 3 nodes and normal syncrepl.
>>> - slapo-memberof used to populate/update attribute 'memberOf' in
user entries.
>>>
>>> If a group entry is modified and slapo-memberof updates the
attribute 'memberOf'
>>> in the member's entry the contextCSN value of the server where the
LDAP request
>>> was sent to is not updated on the other MMR replicas.
>>
>> I don't understand this sentence, can you rephrase?
>
> I looked into this a bit more:
>
> We have three MMR nodes, let's simply enumerate them as 1, 2 and 3.
>
> When modifying a group entry on 1 'memberOf' gets *locally* updated and the
> contextCSN value for server 1 gets also updated. So far so good.
>
> Now the group modification are pulled by 2 and 3 and slapo-memberof
installed
> there *locally* updates attr memberOf and the *local* contextCSN value for
2
> and 3 are updated with new timestamp. But the updated contextCSN values of
2
> and 3 never get replicated to the other instances! So my monitoring check
> shows a long-lasting replication gap. After next server restart all
contextCSN
> values get updated.
>
>> Also see ITS#6915.
>
> Sorry, glancing over ITS#6915 I don't understand what it means in this
context.

The point of #6915 is that internal ops are not supposed to affect the 
contextCSN. If you're seeing new contextCSNs on 2 and 3 solely due to 
slapo-memberof then some part of #6915 is still broken. We'll need a duplicate 
setup to reproduce this...

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 4

Download message
From: Marco Pizzoli <marco.pizzoli@gmail.com>
Date: Tue, 1 Oct 2013 13:02:27 +0200
Subject: Re: (ITS#7710) contextCSN values not updated by internal
 non-replicated ops
To: openldap-its@openldap.org
Cc: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
--20cf303ea4b89c8b7504e7abe440
Content-Type: text/plain; charset=ISO-8859-1

Hi, this *could* be also the root cause of a problem I found some times ago
joking with a particular OL cluster scenario in which I failed to obtain
all entries correctly populated. Please tell me if this could be related.

Long story short:
4-way multimaster cluster --> cluster A
3-way multimaster cluster --> cluster B

one of the members of cluster A has also configured, as provider, one of
the member of cluster B.

By modifying data on any of the members of cluster A I should be able to
see the modification also on any member of cluster B. Correct?
Well, this was failing sometimes on 1 or 2 members.

I had load balancer health-checks continuously polling all of members from
both A and B:
- bind
- search
- unbind

All the members of the cluster were using the contrib/slapo-lastbind
overlay. So the internal authTimestamp attribute populated by an internal
operation.
Could it be that the contextCSN of one node of the cluster were newer than
the one of the providers?

I'm not too expert, just trying to be of help by sharing experiences.
Thanks for reading
Marco

--20cf303ea4b89c8b7504e7abe440
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, this *could* be also the root cause of a problem I
fou=
nd some times ago joking with a particular OL cluster scenario in which I f=
ailed to obtain all entries correctly populated. Please tell me if this cou=
ld be related.<div>

<br></div><div>Long story short:</div><div>4-way
multimaster cluster --&gt;=
 cluster A</div><div>3-way multimaster cluster --&gt; cluster
B</div><div><=
br></div><div>one of the members of cluster A
has=A0also=A0configured, as p=
rovider, one of the member of cluster B.</div>

<div><br></div><div>By modifying data on any of the
members of cluster A I =
should be able to see the modification also on any member of cluster B. Cor=
rect?</div><div>Well, this was failing sometimes on 1 or 2
members.<br>

</div><div><br></div><div>I had load balancer
health-checks continuously po=
lling all of members from both A and B:</div><div>-
bind</div><div>- search=
</div><div>-
unbind</div><div><br></div><div>All the members of
the cluster=
 were using the contrib/slapo-lastbind overlay. So the internal authTimesta=
mp attribute populated by an internal operation.</div>

<div>Could it be that the contextCSN of one node of the cluster were newer
=
than the one of the providers?</div><div><br></div><div>I&#39;m
not too exp=
ert, just trying to be of help by sharing
experiences.</div><div>Thanks for=
 reading</div>

<div>Marco</div></div>

--20cf303ea4b89c8b7504e7abe440--



Followup 5

Download message
Date: Tue, 01 Oct 2013 19:36:27 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: Marco Pizzoli <marco.pizzoli@gmail.com>, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Marco Pizzoli wrote:
> Hi, this *could* be also the root cause of a problem I found some times ago
> joking with a particular OL cluster scenario in which I failed to obtain
> all entries correctly populated. Please tell me if this could be related.
> [..]
> All the members of the cluster were using the contrib/slapo-lastbind
> overlay.

We're also using slapo-lastbind but deactivating does not make a difference
when modifying group entries (tested today).

Are you using slapo-memberof at all?

Ciao, Michael.



Followup 6

Download message
From: Marco Pizzoli <marco.pizzoli@gmail.com>
Date: Tue, 1 Oct 2013 19:39:21 +0200
Subject: Re: (ITS#7710) contextCSN values not updated by internal
 non-replicated ops
To: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
Cc: openldap-its@openldap.org
--047d7bdc109e14c94f04e7b17008
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Michael,
yes I was, but I'm quite sure I didn't have any memberOf-operation running
in the meantime.

Marco


On Tue, Oct 1, 2013 at 7:36 PM, Michael Str=F6der
<michael@stroeder.com>wro=
te:

> Marco Pizzoli wrote:
> > Hi, this *could* be also the root cause of a problem I found some
times
> ago
> > joking with a particular OL cluster scenario in which I failed to
obtai=
n
> > all entries correctly populated. Please tell me if this could be
relate=
d.
> > [..]
> > All the members of the cluster were using the contrib/slapo-lastbind
> > overlay.
>
> We're also using slapo-lastbind but deactivating does not make a differen=
ce
> when modifying group entries (tested today).
>
> Are you using slapo-memberof at all?
>
> Ciao, Michael.
>

--047d7bdc109e14c94f04e7b17008
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div>Hi
Michael,<br></div>yes I was, but I&#39;m quit=
e sure I didn&#39;t have any memberOf-operation running in the
meantime.<br=
><br></div>Marco<br></div><div
class=3D"gmail_extra"><br><br><div class=3D"=
gmail_quote">

On Tue, Oct 1, 2013 at 7:36 PM, Michael Str=F6der <span
dir=3D"ltr">&lt;<a =
href=3D"mailto:michael@stroeder.com" target=3D"_blank">michael@stroeder.com=
</a>&gt;</span> wrote:<br><blockquote
class=3D"gmail_quote" style=3D"margin=
:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div class=3D"im">Marco Pizzoli wrote:<br>
&gt; Hi, this *could* be also the root cause of a problem I found some time=
s ago<br>
&gt; joking with a particular OL cluster scenario in which I failed to obta=
in<br>
&gt; all entries correctly populated. Please tell me if this could be relat=
ed.<br>
</div>&gt; [..]<br>
<div class=3D"im">&gt; All the members of the cluster were using the
contri=
b/slapo-lastbind<br>
&gt; overlay.<br>
<br>
</div>We&#39;re also using slapo-lastbind but deactivating does not
make a =
difference<br>
when modifying group entries (tested today).<br>
<br>
Are you using slapo-memberof at all?<br>
<br>
Ciao, Michael.<br>
</blockquote></div><br></div>

--047d7bdc109e14c94f04e7b17008--



Followup 7

Download message
To: <openldap-its@openldap.org>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Fri, 04 Oct 2013 15:40:55 +0200
----==62f1203288d98ad3817191f9b713342b
Content-Type: text/plain; charset=UTF-8; format=fixed
Content-Transfer-Encoding: 8bit

See attached a canned MMR config.

The behaviour regarding contextCSN is really strange. You have to play around
with some more modify requests changing the group membership of e.g.
uid=michael,ou=users,dc=example,dc=com

If contextCSN values are out-of-sync restarting one slapd brings them up to
same values again, but the older(!) ones.

Ciao, Michael.


----==62f1203288d98ad3817191f9b713342b
Content-Type: application/octet-stream; name="openldap-testbed-its7710.tar.bz2"
Content-Disposition: attachment; filename="openldap-testbed-its7710.tar.bz2"
Content-Transfer-Encoding: base64

QlpoOTFBWSZTWS/IcOYADFR/tf7wKIR9/////+///v////6AQAwAGABAAAAIYCFvtXi4dW2a669a
8s1s1Rhe5yWddW7l1yqUkKAAbndwwp7MoLnuA0NGQQiklFraDW2yGgstttdd3rg600iUBSNY9UNd
wkkITRpponop4mCamyNJpk0AaaExppqbU9RtT01GRkehqaaAaCRIAQhEYmTKp+TUyJ7U2qeaT1R6
nqPU00NAAAGgyAAAJSNEUPSHqPUDQAAAGgA0AAAAAAADQCTUiTKCeU8gT1JtE9IMgGjJoNAaDEBo
A9IyAABocAAA0DQ0NDTIANAABoBoaAABkAABUkIICMQI000Enk0Jpkn6kw1Mpo08oeoNAepoAAaN
A3uEH6CD7VzHMcwP95Q4z5/a/W+muViqEBA9LoELQ+yG5pE6QKpVVFoqkKJK5SAA91fdED90pwcj
ETpHul0pWt3UnWxSVZXEfwDXYhmQxOBCoGLpYKVKk0rJDc2ZnDIqQ1GnDh6nMcQQEjCaq/CsnDpe
UrShWFkREW4QMQZAEWQA55AZJCMqJJIIEDGqhBCCESHHIJzoHAnHqALsFkiISUw87bDPpM9dPSXQ
y7v3nuQvnO5desdcbdm/ClPly8QLLcIX4EMjg8h/F8IDO24uMg2w/jVN4wlUfHBS1j0ECUvmUqVP
QVDEXTUGGPgT0QsesuFc5zQaRpCLjDMEwNZ4BT/Jp/g+x0Wo7kxeHqYJoZrB4yBnDnDUJaS+Th9Q
4xhAypWBkMyAwptA9lq3M19qIP0Ra9xtGiPblilklhUlrmqMYOo54VKqSqqU0z+z9h9bvKfIeLcw
e9r2K9bDkfDp0GoZNkWFkhi3S6x0Vc92K0oWN+jdf8ycsSz7RsESk2X1sLMLeqz6zZY43M17VXVq
1pdiXfUtZ3qUHm7JHmbTLDoaiHU7ivVlqqFYPqPmOZMUCYV+nad+OR1kJmm9MoXTaHGxnMesLYbL
E7YZoYg2YDi62GjP4tqcyjk3+d0esUKmTJ5YPQdiw8tpZHpq/xkseLVTCnBuPJsdfZv4PPJ7B9wP
rMHI5Z1nTG0zaq62mMr3+j3VXxFToMe5w5HpZIu9azi6ts7H54hREOR8bNKEkBIHYiSSCh6BJw2D
S90wlAnbNgKAVDUon5lI+NnqeazJPpUu53rh+XV76v9ntq/L8XRjfuxJWuuilkxxR6zo6g7ssFPf
IN10C4u6mGEJ1IpUxE6Sv1H+0qeS8hhK7XTcw7/sz9OefSkg+SwRqYZGoyJrGOHOo5jrm8QeKQDH
wo2ex0aS5cpUt9PjkonyYPci5mmx8oeVfD6vKQ4foVmSqk8x3cMsUA3KyCgPYZBxQPJzsZjrHgNT
Qasr/CMJ0HK76MW9Tm8V5K+D4LnlZZvV62RsLx/v3Z+HzZCkeRk8iDgplhScJ1gUz5bHh/ScFKr7
zDQxa0X5ky9fcbnL1yO8z4C6n7zmR28l3mxIZ7PSrEWZo2+Nn238PuM2TfRuN/he+/wp70fnqTHF
weVXK2OqIlL816UcywkMAWxzB28o2mE5rG4i3bnE0MhdkMA5Iay+SuUp4Ul4ifpWykvCvAsVUt4A
KNI/F16kei0SZc5Qd5Q6/6yQHu85v5g+qm+qZvm+O/p3dA1y0WhjKTF+eaILxbdQWU4fqKvAwERa
JsS8KqBjxXfZucXYOuL4rKd9nm9Tg47a7uW/t5JlOztW2KldoIDIdXBf+Sxt1dR1s8Yi8Q5czzTV
z0nefZem/iBYMHQ6HHUUiZMkaML1JLrI2fLY9vQXWZ0ekcPouOg/1lbMWdJYyXTfgUyIMK2QyQrk
UQxE9Jhi28oQyfHdg47lDXkfIUqdhrGlhniPUu8qVx+USNyTs3A1hi8ZApK2nVqsZ5jExSVe812B
OwaxUMRVtNYvIkbqDz9nrJlD12I4VhsO5IGxR4S/QimGZWxuWKLb8njUn3cFV7QoS8usdnk5h1HM
0fKzkYgNJiL9BSKXJLsK3+0cygeO5iJQ00+ZevPw9F6LU2eza8TqZvmh/pfk+7XXi+pc6HiUWet6
L1VP4LLKbNvm62G+jgj0UooUoUk/sE/+KRVIlUiRSpFUI6xE/C0ksRKGCRzzS0u97Xv2vX4V5393
oszeWvbkx5tmjb2eSCVIjaQ4H92x4H5hol43mrlI2t+TVu36sGezuBoADuG9qzbLuZVuJus6BbLd
yceDbnLIxENweE4ygKCA4AXT9k1V1ie4WTzyiG4OuebNNnW1csjzsDcMcheIKmKD9508/BJPY2T5
DFZjxGkJYTKnHQJ3GW0LDvGn0d6mTvXWehTySR9ck/Goefn8W50YaKKU+tZub2i+x58bQq1UrNqb
W4zND+lY6jR1u+hLqpLm5eWI5PXBcsZHhxhMCM4rJ6i8Ub1I/xBVPNeOQA85weMEekxAgtBg2RwT
0oBbSzCSvyfBm3SfhliBXv5P/J1ON21NkE6nxeDNtVOXW5lq2eR/fMk1nRN/p3PGvSk6kgVKQij1
AIVTaEKCrhCs1jCQGpASQvG9M89owOrReLSJ0gBEl1iClRk0hFG3DwxIiVgLZAQtVhzSLI2a+kq+
cOUetrAwwVdw3zHefbgzUySYOGo5zk7caSlq0c5E6pKzTqML5hMEUcVPCL1gkTXyscVzGS+3RKZe
6+uWLWXVgXANgK0gN5DMI6r42TJqVGcxUIXEJjXLxUuIUHiLoG5sBuUskbIYNDH2GRyN4IIiEUXi
hQbt/Dic8kN8Pdd099yGUGKqymFXIkPE0aa4PVMDWZiISGAdjrn7BbObBQis4WBk2Zla80XkNEGR
ZEWMFIKqqjzyG9LopmRI83WWbNLESKaEP534V0h5G3+OpxDyrZHlTn57VXbO0e/k4zpepraSD3pT
CBkhyQefKdCFvDorirqw9DVDT0vzWmvqas0wIGixYlihRBUJECUDgcGeIKHjawys7tDlbftNhOhM
30VFD30YGnTJJFgdnAUyZUhMpLm5SGmjN8NcbrbkpvYbGnrPD53xy3nmPZjRowe3bNf4pRn416zZ
mJYsuC/LqSCEkHFXEUSOVQm6m/WOTnJfVsNdOvGTgwRAkh6OxR1GCg5kMJfiKYyYvMEWP7hmJWrg
kkFBCzyWOqNtjOfR+MLkQkWJGxZmKoBX3EMnHhIYrOQ1edltrDqrDFMLdmnoxwU1aEHDUhmxAumC
DTodOvHWYCjbFxce8a80+VKWpj0c129vQd6y7jbu+kL3V7Ajl5pHUNgYKo4dJOuanVVtuqyymxXI
qdjdIWG+hno8fV3RGXcztqiaouiYYuutfLKrv25DfLrIJzGxBV5oFFZUo8lihTZwO0T4FItqETw6
JSaUMIt+0lKjtMqa7W5jyx2wTtqhqXYVQIz9Juv5k0kNJbT8qq/JLRAaipJMkQkckIggqIGOfLnH
O4FpXZYmYJjIcjpTJOlultcmGuJE4KbtS21iyFzBq0Ql2VAm11XW5nG94MpIVAY0aktVxTLcdRZA
jO2WWNkE4HSmwTMMqDKgMm9piUKgyRsCDZIJKuM0K4apahMkymWkEj0mLVxLOJk5+dA9bN1VKBfj
aeBTrfEylY4X3lUmkHCEEHUZUg50VDQjRuWgzsCIUuMEgMsApE3G5orRWktTCvKhIkGzCnAj7aaE
csHoDFeNiCnFM9IdNuR0JdCwCLIRzGlnoKCxyFQyVIbOKrd7FCk5YiqqTd0gvaVuDWcbsqrJBc0k
+CgyzIrBYCTdsVnOLmRZleouMWuINgW+GSA1SpMvJAKhGZOlCD9tVglWICsViSkCBhfjMwskyp4o
BdlrUL7EHO+rJU5pVS1WnOJ830NByKLcWA5oBP1M344CXDLWJ1hkyALvROFqlxYnNtAoaSKWr24v
yPVjayXOutHMpPNtkm9yshCkkxEwaE7Vyr9BQjkFVSeD2lYI8QuPm9PSDtt3OAHDh6YYYHdvdoPj
5xe0wlhqJdLZqt0Qmg0hydn1OtIpdd2ruK0URUeacJ7HESHVvc1HJOmfYZFzaBE1FE6NBS7sf3V3
8zOQmUd1SHJJGS0skhkkK5KZ98oYCxCLUIFWZNKRQhchfwglTk0jZC6HMD5LnEPPcoH1982w6aS9
oA8Qj4CnSqCPkSxrtnXXgFD3JI9STrPKfcPHHIJxdNo8sF0yIKBN3uuONRCZZxlXyArYJXMHQc4P
DxG6fqHVIsm4JFJXDTKUlWGlS/ZFJ3DMsaDIMhMy3+kTUVWioq

Message of length 9777 truncated


Followup 8

Download message
Date: Fri, 04 Oct 2013 19:03:43 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Hmmpf, the attachment did not come through.

Download config here:
http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2



Followup 9

Download message
Date: Sun, 06 Oct 2013 17:52:22 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
I've updated the test data with LDIF files and a README describing the LDAP
operations to perform:

http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2

I consider this to be a serious issue which could be the cause for other
replication issues including dead-locks we're experiencing. (Maybe there are
similar issues in slapo-refint since dead-locks in our deployment arised when
moving entries into different subtree.)

Ciao, Michael.

The README copied here for direct access:

----------------------------------------------------------------------------

0. Start both servers by invoking start-slapd1.sh and start-slapd2.sh

----------------------------------------------------------------------------

1. Add test entries on first server:

$ ldapadd -H ldap://localhost:2071 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 1_ldapadd.ldif
adding new entry "dc=example,dc=com"
[..]
adding new entry "cn=replicas,ou=groups,dc=example,dc=com"

You can now see contextCSN value of first server on both servers (as expected):

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000

----------------------------------------------------------------------------

2. Send a simple modification to second server:

$ ldapmodify -H ldap://localhost:2072 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 2_ldapmodify.ldif
modifying entry "uid=michael,ou=users,dc=example,dc=com"

You can now see contextCSN value of second server on both servers (as expected):

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

----------------------------------------------------------------------------

3. Modification of group membership on first server:

$ ldapmodify -H ldap://localhost:2071 -D "uid=diradm,dc=example,dc=com" -w
testsecret -f 3_ldapmodify.ldif
modifying entry "cn=testgroup1,ou=groups,dc=example,dc=com"

Now the contextCSN values differ:

$ for I in 1 2 ; do (echo ldap://localhost:207$I ;  ldapsearch -LLL -H
ldap://localhost:207$I -D "uid=diradm,dc=example,dc=com" -w testsecret -b
"dc=example,dc=com" -s base "(objectClass=*)" contextCSN ) ; done
ldap://localhost:2071
dn: dc=example,dc=com
contextCSN: 20131006154449.514135Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000

ldap://localhost:2072
dn: dc=example,dc=com
contextCSN: 20131006154300.921415Z#000000#001#000000
contextCSN: 20131006154406.940154Z#000000#002#000000



Followup 10

Download message
Date: Mon, 07 Oct 2013 00:22:21 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: Christian Kratzer <ck@cksoft.de>
CC: openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Christian Kratzer wrote:
> This sounds very similar to the difference between:
> 
> 1. statement replication in sql databases ala mysql
> 
> 2. log based replication ala postgresql
> 
> OpenLDAP is doing 1. here while 2. would perhaps have been the better
option.

From my understanding 2. is OpenLDAP's delta-syncrepl.

Ciao, Michael.



Followup 11

Download message
Date: Sun, 6 Oct 2013 22:20:26 +0200 (CEST)
From: Christian Kratzer <ck@cksoft.de>
To: michael@stroeder.com
cc: openldap-its@openldap.org, Christian Kratzer <ck@cksoft.de>
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Hi Michael,

On Sun, 6 Oct 2013, michael@stroeder.com wrote:
> I've updated the test data with LDIF files and a README describing the LDAP
> operations to perform:
>
> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>
> I consider this to be a serious issue which could be the cause for other
> replication issues including dead-locks we're experiencing. (Maybe there
are
> similar issues in slapo-refint since dead-locks in our deployment arised
when
> moving entries into different subtree.)

I can confirm that I can see the same bug with your testcase.

It seems that restarting the second slapd instance fixes the contextCSN
issue just because it scans all it's entries and does the right thing.

Even a slapcat of both directories shows the issue.

Things that I find disturbing:

After applying the group membership the member dn: gets it's memberOf 
attribute updated accordingly. But the entryCSN stays unchanged.

Entry before applying group membership:

     # michael, users, example.com
     dn: uid=michael,ou=users,dc=example,dc=com
     uid: michael
     objectClass: account
     objectClass: simpleSecurityObject
     userPassword:: dGVzdHNlY3JldA==
     description: test
     entryCSN: 20131006193757.912740Z#000000#002#000000

Entry after applying group membership is obviously updated but still has
the same entryCSN:

     # michael, users, example.com
     dn: uid=michael,ou=users,dc=example,dc=com
     uid: michael
     objectClass: account
     objectClass: simpleSecurityObject
     userPassword:: dGVzdHNlY3JldA==
     memberOf: cn=admins,ou=groups,dc=example,dc=com
     memberOf: cn=testgroup1,ou=groups,dc=example,dc=com
     description: test
     entryCSN: 20131006193757.912740Z#000000#002#000000

I can also confirm the subsequent contextCSN mismatch.

I too consider this a serious issue.

If I read Howards reply correctly and from what I see in the source code
of the memberOf overlay the strategy here seems to be not to replicate the
side effect of adding the memberOf: entry.

OpenLDAP relies on the side effect happening on the replicas through the
same codepath when the member: attribute in the group gets updated
through replication.

This sounds very similar to the difference between:

1. statement replication in sql databases ala mysql

2. log based replication ala postgresql

OpenLDAP is doing 1. here while 2. would perhaps have been the better option.

I understand turning this around is propably too late at this point in time.

Looks like we will just have to work out the bugs the hard way.

I am using slapo-lastbind and slapo-ppolicy extensively in my current
project and will keep my eyes open for similar issues resulting from not
replicating the side effects that these overlays produce.

Greetings
Christian

-- 
Christian Kratzer                      CK Software GmbH
Email:   ck@cksoft.de                  Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0          D-71126 Gaeufelden
Fax:     +49 7032 893 997 - 9          HRB 245288, Amtsgericht Stuttgart
Web:     http://www.cksoft.de/         Geschaeftsfuehrer: Christian Kratzer



Followup 12

Download message
Date: Wed, 09 Oct 2013 05:01:40 -0700
From: Howard Chu <hyc@symas.com>
To: michael@stroeder.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
michael@stroeder.com wrote:
> I've updated the test data with LDIF files and a README describing the LDAP
> operations to perform:
>
> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>
> I consider this to be a serious issue which could be the cause for other
> replication issues including dead-locks we're experiencing. (Maybe there
are
> similar issues in slapo-refint since dead-locks in our deployment arised
when
> moving entries into different subtree.)

Thanks, fixed now in master. Looks like refint may have run into the same 
issue, also patched in master.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 13

Download message
To: openldap-its@openldap.org, Howard Chu <hyc@symas.com>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Wed, 09 Oct 2013 15:11:40 +0200
On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com> wrote

> michael@stroeder.com wrote:
> > I've updated the test data with LDIF files and a README describing the
LDAP
> > operations to perform:
> >
> > http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
> >
> > I consider this to be a serious issue which could be the cause for
other
> > replication issues including dead-locks we're experiencing. (Maybe
there
> > are similar issues in slapo-refint since dead-locks in our deployment
> > arised when moving entries into different subtree.)
> 
> Thanks, fixed now in master. Looks like refint may have run into the same 
> issue, also patched in master.

contextCSN values are correct now.

But now I see this during initial refresh phase of second server:

send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
already exists"

Is that expected?

Ciao, Michael.


52555576 syncrepl_entry: rid=002 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD)
52555576 syncrepl_entry: rid=002 inserted UUID
b5736b9e-c52f-1032-99c4-31a68f008587
52555576 mdb_idl_fetch_key: [3bec70f1]
52555576 send_ldap_result: err=0 matched="" text=""
52555576 syncrepl_entry: rid=002 be_search (0)
52555576 syncrepl_entry: rid=002 cn=replicas,ou=groups,dc=example,dc=com
52555576 ==> mdb_add: cn=replicas,ou=groups,dc=example,dc=com
52555576 mdb_idl_insert_keys: b [f4f2ead9]
52555576 mdb_idl_insert_keys: b [8f2b53e8]
52555576 mdb_idl_insert_keys: b [cb13035d]
52555576 mdb_idl_insert_keys: b [c14a3e76]
52555576 mdb_idl_insert_keys: b [3bec70f1]
52555576 mdb_idl_insert_keys: b 
52555576 send_ldap_result: err=0 matched="" text=""
52555576 mdb_modify: cn=slapd-1,ou=sys,dc=example,dc=com
52555576 mdb_modify_internal: add memberOf
52555576 dnMatch 0
	"cn=replicas,ou=groups,dc=example,dc=com"
	"cn=replicas,ou=groups,dc=example,dc=com"
52555576 mdb_modify_internal: 20 modify/add: memberOf: value #0 already exists
52555576 send_ldap_result: err=20 matched="" text="modify/add: memberOf: value
#0 already exists"
52555576 conn=-1 op=0: memberof_value_modify
DN="cn=slapd-1,ou=sys,dc=example,dc=com" add
memberOf="cn=replicas,ou=groups,dc=example,dc=com" failed err=20
52555576 mdb_modify: cn=slapd-2,ou=sys,dc=example,dc=com
52555576 mdb_modify_internal: add memberOf
52555576 dnMatch 0
	"cn=replicas,ou=groups,dc=example,dc=com"
	"cn=replicas,ou=groups,dc=example,dc=com"
52555576 mdb_modify_internal: 20 modify/add: memberOf: value #0 already exists
52555576 send_ldap_result: err=20 matched="" text="modify/add: memberOf: value
#0 already exists"
52555576 conn=-1 op=0: memberof_value_modify
DN="cn=slapd-2,ou=sys,dc=example,dc=com" add
memberOf="cn=replicas,ou=groups,dc=example,dc=com" failed err=20
52555576 syncrepl_entry: rid=002 be_add cn=replicas,ou=groups,dc=example,dc=com
(0)
52555576 do_syncrep2: rid=002 LDAP_RES_INTERMEDIATE - REFRESH_DELETE
52555576 do_syncrep2: rid=002
cookie=rid=002,sid=001,csn=20131009130900.890770Z#000000#001#000000
52555576 slap_queue_csn: queing 0x7fd79c10c200
20131009130900.890770Z#000000#001#000000
52555576 base_candidates: base: "dc=example,dc=com" (0x00000001)
52555576 send_ldap_result: err=0 matched="" text=""
52555576 mdb_modify: dc=example,dc=com
52555576 mdb_modify_internal: replace contextCSN
52555576 send_ldap_result: err=0 matched="" text=""
52555576 slap_graduate_commit_csn: removing 0x7fd79c10c170
20131009130900.890770Z#000000#001#000000




Followup 14

Download message
Date: Wed, 09 Oct 2013 06:56:24 -0700
From: Howard Chu <hyc@symas.com>
To: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>,
        openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Michael Str..der wrote:
> On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com> wrote
>
>> michael@stroeder.com wrote:
>>> I've updated the test data with LDIF files and a README describing
the LDAP
>>> operations to perform:
>>>
>>> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>>>
>>> I consider this to be a serious issue which could be the cause for
other
>>> replication issues including dead-locks we're experiencing. (Maybe
there
>>> are similar issues in slapo-refint since dead-locks in our
deployment
>>> arised when moving entries into different subtree.)
>>
>> Thanks, fixed now in master. Looks like refint may have run into the
same
>> issue, also patched in master.
>
> contextCSN values are correct now.
>
> But now I see this during initial refresh phase of second server:
>
> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value #0
> already exists"
>
> Is that expected?

Not seeing that with your testcase.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 15

Download message
Date: Wed, 09 Oct 2013 07:05:43 -0700
From: Howard Chu <hyc@symas.com>
To: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>,
        openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Howard Chu wrote:
> Michael Str..der wrote:
>> On Wed, 09 Oct 2013 05:01:40 -0700 Howard Chu <hyc@symas.com>
wrote
>>
>>> michael@stroeder.com wrote:
>>>> I've updated the test data with LDIF files and a README
describing the LDAP
>>>> operations to perform:
>>>>
>>>> http://www.stroeder.com/temp/openldap-testbed-its7710.tar.bz2
>>>>
>>>> I consider this to be a serious issue which could be the cause
for other
>>>> replication issues including dead-locks we're experiencing.
(Maybe there
>>>> are similar issues in slapo-refint since dead-locks in our
deployment
>>>> arised when moving entries into different subtree.)
>>>
>>> Thanks, fixed now in master. Looks like refint may have run into
the same
>>> issue, also patched in master.
>>
>> contextCSN values are correct now.
>>
>> But now I see this during initial refresh phase of second server:
>>
>> send_ldap_result: err=20 matched="" text="modify/add: memberOf: value
#0
>> already exists"
>>
>> Is that expected?
>
> Not seeing that with your testcase.
>
Ah, I see it now. Yes, it's normal; memberOf on the provider already added the 
relevant values. The consumer receives a group entry and performs the same set 
of memberof updates, which are redundant at that point. It's harmless.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 16

Download message
To: openldap-its@openldap.org, <hyc@symas.com>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Wed, 09 Oct 2013 16:33:33 +0200
On Wed, 9 Oct 2013 14:05:52 GMT hyc@symas.com wrote
> Howard Chu wrote:
> > Michael Str..der wrote:
> >> But now I see this during initial refresh phase of second server:
> >>
> >> send_ldap_result: err=20 matched="" text="modify/add: memberOf:
value #0
> >> already exists"
> >>
> >> Is that expected?
> >
> > Not seeing that with your testcase.
> >
> Ah, I see it now. Yes, it's normal; memberOf on the provider already added
> the  relevant values. The consumer receives a group entry and performs the
> same set  of memberof updates, which are redundant at that point. It's
> harmless. 

Hmm, wouldn't it be reasonable to strip those attributes marked as
non-replication attrs when generating syncrepl search results at the provider?
(Even if consumer asks for attrs=*,+)

Ciao, Michael.




Followup 17

Download message
Date: Wed, 09 Oct 2013 08:13:24 -0700
From: Howard Chu <hyc@symas.com>
To: =?UTF-8?B?TWljaGFlbCBTdHLDtmRlcg==?= <michael@stroeder.com>,
        openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
Michael Str..der wrote:
> On Wed, 9 Oct 2013 14:05:52 GMT hyc@symas.com wrote
>> Howard Chu wrote:
>>> Michael Str..der wrote:
>>>> But now I see this during initial refresh phase of second
server:
>>>>
>>>> send_ldap_result: err=20 matched="" text="modify/add: memberOf:
value #0
>>>> already exists"
>>>>
>>>> Is that expected?
>>>
>>> Not seeing that with your testcase.
>>>
>> Ah, I see it now. Yes, it's normal; memberOf on the provider already
added
>> the  relevant values. The consumer receives a group entry and performs
the
>> same set  of memberof updates, which are redundant at that point. It's
>> harmless.
>
> Hmm, wouldn't it be reasonable to strip those attributes marked as
> non-replication attrs when generating syncrepl search results at the
provider?
> (Even if consumer asks for attrs=*,+)

slapd already strips DSA-specific attributes before sending a syncrepl entry. 
memberOf is not marked in the schema as DSA-specific. This is working as
designed.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 18

Download message
To: openldap-its@openldap.org, Howard Chu <hyc@symas.com>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Wed, 09 Oct 2013 17:43:13 +0200
On Wed, 09 Oct 2013 08:13:24 -0700 Howard Chu <hyc@symas.com> wrote
> slapd already strips DSA-specific attributes before sending a syncrepl
entry.
> memberOf is not marked in the schema as DSA-specific. This is working as
> designed. 

IIRC attribute 'memberOf' was replicated in former releases. So it was not
DSA-specific back then.

Then the behaviour was changed in a more recent OpenLDAP release. Nowadays each
replica has to be configured with slapo-memberof performing *local* operations.
Therefore I'd argue that 'memberOf' should be marked DSA-specific now since the
*local* configuration is significant for its content.
Note that there is no formal specification for attribute 'memberOf' at all.

I have deployments where most users are member of more than 10 groups,
sometimes more than 20. So not sending 'memberOf' could save quite a lot of
network traffic.

What are your objections against marking 'memberOf' as DSA-specific?

(I vaguely remember this being discussed before without result though.)

Ciao, Michael.




Followup 19

Download message
Date: Wed, 09 Oct 2013 19:42:41 +0200
From: =?ISO-8859-1?Q?Michael_Str=F6der?= <michael@stroeder.com>
To: hyc@symas.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
michael@stroeder.com wrote:
> On Wed, 09 Oct 2013 08:13:24 -0700 Howard Chu <hyc@symas.com> wrote
>> slapd already strips DSA-specific attributes before sending a syncrepl
entry.
>> memberOf is not marked in the schema as DSA-specific. This is working
as
>> designed. 
> 
> IIRC attribute 'memberOf' was replicated in former releases. So it was not
> DSA-specific back then.
> 
> Then the behaviour was changed in a more recent OpenLDAP release. Nowadays
each
> replica has to be configured with slapo-memberof performing *local*
operations.
> Therefore I'd argue that 'memberOf' should be marked DSA-specific now since
the
> *local* configuration is significant for its content.
> Note that there is no formal specification for attribute 'memberOf' at all.
> 
> I have deployments where most users are member of more than 10 groups,
> sometimes more than 20. So not sending 'memberOf' could save quite a lot of
> network traffic.
> 
> What are your objections against marking 'memberOf' as DSA-specific?
> 
> (I vaguely remember this being discussed before without result though.)

Additionally consider partial replication where only a subset of group entries
is present on a certain consumer. One would not want to have 'memberOf' point
to group entries not really existing on that consumer.

=> 'memberOf' is definitely DSA-specific.

Ciao, Michael.



Followup 20

Download message
To: <openldap-its@openldap.org>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Thu, 10 Oct 2013 12:32:52 +0200
With current master and re24 which have the fix for this ITS I now see seg
faults in test023-refint test057-memberof-refint.




Followup 21

Download message
Date: Thu, 10 Oct 2013 04:05:20 -0700
From: Howard Chu <hyc@symas.com>
To: michael@stroeder.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated
 ops
michael@stroeder.com wrote:
> With current master and re24 which have the fix for this ITS I now see seg
> faults in test023-refint test057-memberof-refint.

Fixed now in master.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 22

Download message
To: openldap-its@openldap.org, <hyc@symas.com>
From: "Michael =?UTF-8?B?U3Ryw7ZkZXI=?=" <michael@stroeder.com>
Subject: Re: (ITS#7710) contextCSN values not updated by internal non-replicated ops
Date: Thu, 10 Oct 2013 13:22:07 +0200
On Thu, 10 Oct 2013 11:05:28 GMT hyc@symas.com wrote
> michael@stroeder.com wrote:
> > With current master and re24 which have the fix for this ITS I now see
seg
> > faults in test023-refint test057-memberof-refint.
> 
> Fixed now in master.

Thanks. These tests are passing now.

@Quanah: Would be nice if this last fix could be also ported to RE24. Thanks in
advance.

Ciao, Michael.




Followup 23

Download message
Date: Thu, 10 Oct 2013 09:54:15 -0700
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: michael@stroeder.com, openldap-its@openldap.org
Subject: Re: (ITS#7710) contextCSN values not updated by internal
 non-replicated ops
--On Thursday, October 10, 2013 11:22 AM +0000 michael@stroeder.com wrote:

> On Thu, 10 Oct 2013 11:05:28 GMT hyc@symas.com wrote
>> michael@stroeder.com wrote:
>> > With current master and re24 which have the fix for this ITS I now
see
>> > seg faults in test023-refint test057-memberof-refint.
>>
>> Fixed now in master.
>
> Thanks. These tests are passing now.
>
> @Quanah: Would be nice if this last fix could be also ported to RE24.
> Thanks in advance.

Not particularly necessary to make requests like this.  I read all ITS mail 
and commit traffic.

--Quanah



--

Quanah Gibson-Mount
Architect - Server
Zimbra Software, LLC
--------------------
Zimbra ::  the leader in open source messaging and collaboration


Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org