OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Incoming/7400
Full headers

From: arunkumar_1123@yahoo.com
Subject: Memberof and Syncrepl incompatibility
Compose comment
Download message
State:
0 replies:
5 followups: 1 2 3 4 5

Major security issue: yes  no

Notes:

Notification:


Date: Tue, 25 Sep 2012 11:45:21 +0000
From: arunkumar_1123@yahoo.com
To: openldap-its@OpenLDAP.org
Subject: Memberof and Syncrepl incompatibility
Full_Name: Arunkumar shanmugam
Version: 2.4.29
OS: rhel5
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (203.83.248.32)


Hi,

I'm currently using Openldap 2.4.29 to model an Authorization platform. I
noticed some inconsistent behavior with syncrepl and memberof overlays.

The issue happens as follows:

If I Create groups with a large number of members and delete them in quick
succession on the writemaster, the data replicated to the readslave is
incorrect, in particular, the memberof fields of the User objects.

This seems to happen because the memberof field is getting replicated to the
slave nodes, although the documentation states that it shouldn't. While
replicating, the User object is replicated inclusive of the memberof fields, but
by the time the syncrepl search comes to the group object, it has already been
deleted, and hence not replicated. This leaves a dangling memberof field in the
read slave instance.

I was wondering if anyone has faced this issues (I did not see any ITS related
to this), and has a workaround.

Thanks,
Arunkumar

Followup 1

Download message
Date: Sun, 30 Sep 2012 08:21:54 -0700
From: Howard Chu <hyc@symas.com>
To: arunkumar_1123@yahoo.com
CC: openldap-its@openldap.org
Subject: Re: (ITS#7400) Memberof and Syncrepl incompatibility
arunkumar_1123@yahoo.com wrote:
> Full_Name: Arunkumar shanmugam
> Version: 2.4.29
> OS: rhel5
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (203.83.248.32)
>
>
> Hi,
>
> I'm currently using Openldap 2.4.29 to model an Authorization platform. I
> noticed some inconsistent behavior with syncrepl and memberof overlays.

Does this issue occur with the current release, 2.4.32?
>
> The issue happens as follows:
>
> If I Create groups with a large number of members and delete them in quick
> succession on the writemaster, the data replicated to the readslave is
> incorrect, in particular, the memberof fields of the User objects.
>
> This seems to happen because the memberof field is getting replicated to
the
> slave nodes, although the documentation states that it shouldn't.

Indeed. Do you have debug logs showing the replication traffic, and showing 
that the memberof attribute got replicated?

> While
> replicating, the User object is replicated inclusive of the memberof
fields, but
> by the time the syncrepl search comes to the group object, it has already
been
> deleted, and hence not replicated. This leaves a dangling memberof field in
the
> read slave instance.
>
> I was wondering if anyone has faced this issues (I did not see any ITS
related
> to this), and has a workaround.
>
> Thanks,
> Arunkumar
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 2

Download message
Date: Mon, 1 Oct 2012 19:51:44 +0800 (SGT)
From: arun s <arunkumar_1123@yahoo.com>
Subject: Re: (ITS#7400) Memberof and Syncrepl incompatibility
To: Howard Chu <hyc@symas.com>
Cc: "openldap-its@openldap.org" <openldap-its@openldap.org>
---872237384-2137566121-1349092304=:39209
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,=0AYes, I am able to reproduce the issue with 2.4.32=0A=0A=0AMaking sens=
e of the logs for the exact reproduction is hard since it needs a lot of op=
erations in a short time. But this will probably help:=0A=0A1. At the start=
 of the test, the group temp_group existed.=0A=0A2. I created a user temp_u=
ser and added temp_user to temp_group.=0A=0A3. During replication, the grou=
p was replicated first and I got an error 32 (NO_SUCH_OBJECT) when it tried=
 to modify the memberOf of the temp_user object (This does not exist in the=
 readslave yet).=0A=0A4. The temp_user object was replicated next, and as y=
ou see, querying it does show a memberOf attribute, proving that this field=
 was replicated. (Note that I have run OpenLDAP with debug as -1 and verifi=
ed that the replicated data has the memberOf field in it. I can provide thi=
s too if needed.)=0A=0A5. The more serious problem occurs when the sequence=
 is reversed and the group has been deleted as a last operation - The user =
is replicated first, but since the group is deleted, it is never replicated=
 and a stale memberOf entry stays with the user.=0A=0A=0ALOGS:=0A=0A5069157=
c syncrepl_message_to_entry: rid=3D179 DN: ou=3Dtemp_group,ou=3Dgroup,dc=3D=
example,dc=3Dcom, UUID: 31090ab1-f8a7-4363-83c2-c0ac0d3918d4=0A5069157c syn=
crepl_entry: rid=3D179 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD)=0A5069157c sync=
repl_entry: rid=3D179 inserted UUID 31090ab1-f8a7-4363-83c2-c0ac0d3918d4=0A=
5069157c syncrepl_entry: rid=3D179 be_search (0)=0A5069157c syncrepl_entry:=
 rid=3D179 ou=3Dtemp_group,ou=3Dgroup,dc=3Dexample,dc=3Dcom=0A5069157c slap=
_queue_csn: queing 0x4270c730 20121001040100.779862Z#000000#000#000000=0A50=
69157c slap_graduate_commit_csn: removing 0xfa035c0 20121001040100.779862Z#=
000000#000#000000=0A5069157c conn=3D-1 op=3D0: memberof_value_modify DN=3D"=
uid=3Dtemp_user,dc=3Dexample,dc=3Dcom" add memberOf=3D"ou=3Dtemp_group,ou=
=3Dgroup,dc=3Dexample,dc=3Dcom" failed err=3D32=0A5069157c syncrepl_entry: =
rid=3D179 be_modify ou=3Dtemp_group,ou=3Dgroup,dc=3Dexample,dc=3Dcom (0)=0A=
5069157c syncrepl_message_to_entry: rid=3D179 DN: uid=3Dtemp_user,dc=3Dexam=
ple,dc=3Dcom, UUID: 748bd1a9-6be3-450b-809c-5ea692aa073c=0A5069157c syncrep=
l_entry: rid=3D179 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD)=0A5069157c syncrepl=
_entry: rid=3D179 inserted UUID 748bd1a9-6be3-450b-809c-5ea692aa073c=0A5069=
157c syncrepl_entry: rid=3D179 be_search (0)=0A5069157c syncrepl_entry: rid=
=3D179 uid=3Dtemp_user,dc=3Dexample,dc=3Dcom=0A5069157c syncrepl_entry: rid=
=3D179 be_add uid=3Dtemp_user,dc=3Dexample,dc=3Dcom (0)=0A=0AThe object tem=
p_user:=0A=0Adn: uid=3Dtemp_user,dc=3Dexample,dc=3Dcom=0AmemberOf: ou=3Dtem=
p_group,ou=3Dgroup,dc=3Dexample,dc=3Dcom=0A=0A=0AWhat is interesting is tha=
t in this case, the memberOf field being replicated actually protects the s=
lave from incorrect data as the temp_user entry was not present at the time=
 the group got replicated (The user entry was the second entry in the repli=
cation order). On the other hand, a reversed path during replication causes=
 the mentioned bug.=0A=0AThanks=0A=0A=0A=0A________________________________=
=0A From: Howard Chu <hyc@symas.com>=0ATo: arunkumar_1123@yahoo.com =0ACc:
=
openldap-its@openldap.org =0ASent: Sunday, 30 September 2012 8:51 PM=0ASubj=
ect: Re: (ITS#7400) Memberof and Syncrepl incompatibility=0A =0Aarunkumar_1=
123@yahoo.com wrote:=0A> Full_Name: Arunkumar shanmugam=0A> Version:
2.4.29=
=0A> OS: rhel5=0A> URL: ftp://ftp.openldap.org/incoming/=0A> Submission
fro=
m: (NULL) (203.83.248.32)=0A>=0A>=0A> Hi,=0A>=0A> I'm currently
using Openl=
dap 2.4.29 to model an Authorization platform. I=0A> noticed some inconsist=
ent behavior with syncrepl and memberof overlays.=0A=0ADoes this issue occu=
r with the current release, 2.4.32?=0A>=0A> The issue happens as follows:=
=0A>=0A> If I Create groups with a large number of members and delete them
=
in quick=0A> succession on the writemaster, the data replicated to the read=
slave is=0A> incorrect, in particular, the memberof fields of the User obje=
cts.=0A>=0A> This seems to happen because the memberof field is getting
rep=
licated to the=0A> slave nodes, although the documentation states that it s=
houldn't.=0A=0AIndeed. Do you have debug logs showing the replication traff=
ic, and showing =0Athat the memberof attribute got replicated?=0A=0A> While=
=0A> replicating, the User object is replicated inclusive of the memberof f=
ields, but=0A> by the time the syncrepl search comes to the group object, i=
t has already been=0A> deleted, and hence not replicated. This leaves a dan=
gling memberof field in the=0A> read slave instance.=0A>=0A> I was
wonderin=
g if anyone has faced this issues (I did not see any ITS related=0A> to thi=
s), and 

Message of length 20887 truncated


Followup 3

Download message
Date: Mon, 01 Oct 2012 06:29:33 -0700
From: Howard Chu <hyc@symas.com>
To: arun s <arunkumar_1123@yahoo.com>
CC: "openldap-its@openldap.org" <openldap-its@openldap.org>
Subject: Re: (ITS#7400) Memberof and Syncrepl incompatibility
arun s wrote:
> Hi,
> Yes, I am able to reproduce the issue with 2.4.32
>
> Making sense of the logs for the exact reproduction is hard since it needs
a
> lot of operations in a short time. But this will probably help:
>
> 1. At the start of the test, the group temp_group existed.
>
> 2. I created a user temp_user and added temp_user to temp_group.
>
> 3. During replication, the group was replicated first and I got an error 32
> (NO_SUCH_OBJECT) when it tried to modify the memberOf of the temp_user
object
> (This does not exist in the readslave yet).
>
> 4. The temp_user object was replicated next, and as you see, querying it
does
> show a memberOf attribute, proving that this field was replicated. (Note
that
> I have run OpenLDAP with debug as -1 and verified that the replicated data
has
> the memberOf field in it. I can provide this too if needed.)

I see. The current code drops the memberOf attribute if it was not explicitly 
requested in the search. However, by default the consumer requests "+" which 
means "all operational attributes" and so slapd considers memberOf to have 
been requested. We need to reconsider this aspect of the design.
>
> 5. The more serious problem occurs when the sequence is reversed and the
group
> has been deleted as a last operation - The user is replicated first, but
since
> the group is deleted, it is never replicated and a stale memberOf entry
stays
> with the user.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 4

Download message
Date: Tue, 9 Oct 2012 19:48:32 +0800 (SGT)
From: arun s <arunkumar_1123@yahoo.com>
Subject: Re: (ITS#7400) Memberof and Syncrepl incompatibility
To: Howard Chu <hyc@symas.com>
Cc: "openldap-its@openldap.org" <openldap-its@openldap.org>
--1251447850-1386810138-1349783312=:99375
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hi,=0ATo try and overcome this issue, we tried two fixes:=0A=0A1. Every tim=
e a user was deleted from a group, we force-updated the user=A0=0Aobject ma=
nually to make sure its entryCSN got updated and it got=A0=0Areplicated pro=
perly. This is an expensive operation and did not scale=A0=0Awell for big g=
roup sizes (10-20k), and did not work out.=0A=0A2. We then tried to do the =
same thing in OpenLDAP. We noticed in the=A0=0Amemberof.c commits that ther=
e were a couple of patches to force the=A0=0AentryCSN of the user object to=
 get updated. (http://tinyurl.com/8k4qrdj=A0=0Aand http://tinyurl.com/9akqg=
fl)These have since been reverted because of=A0=0Aaccess log and some repli=
cation issues, but for us, speed was a higher=A0=0Apriority. I reapplied th=
ese patches back to the code. This solved the=A0=0Amember-of replication is=
sue, but we noticed that occasionally under a=A0=0Aheavy load, there was a =
sudden surge in OpenLDAP's memory usage going up=A0=0Ato whatever memory wa=
s available and finally crashing.=0A=0AWe have gone back to option (1) thou=
gh (2) would be the preferred option.=0A=0AAny help on figuring out why (2)=
 caused the memory bloat would be really=A0=0Agreat. I can provide more det=
ails/memory traces if needed.=0A=0AWe will be glad to contribute any fixes =
once we are able to nail down=A0=0Athe issue.=0A=0AThanks,=0AArunkumar=0A=
=0A________________________________=0A From: Howard Chu
<hyc@symas.com>=0AT=
o: arun s <arunkumar_1123@yahoo.com> =0ACc: "openldap-its@openldap.org"
<op=
enldap-its@openldap.org> =0ASent: Monday, 1 October 2012 6:59 PM=0ASubject:=
 Re: (ITS#7400) Memberof and Syncrepl incompatibility=0A =0Aarun s wrote:=
=0A> Hi,=0A> Yes, I am able to reproduce the issue with 2.4.32=0A>
=0A> Mak=
ing sense of the logs for the exact reproduction is hard since it needs a=
=0A> lot of operations in a short time. But this will probably help:=0A> =
=0A> 1. At the start of the test, the group temp_group existed.=0A>
=0A> 2.=
 I created a user temp_user and added temp_user to temp_group.=0A> =0A> 3.
=
During replication, the group was replicated first and I got an error 32=0A=
> (NO_SUCH_OBJECT) when it tried to modify the memberOf of the temp_user ob=
ject=0A> (This does not exist in the readslave yet).=0A> =0A> 4. The
temp_u=
ser object was replicated next, and as you see, querying it does=0A> show a=
 memberOf attribute, proving that this field was replicated. (Note that=0A>=
 I have run OpenLDAP with debug as -1 and verified that the replicated data=
 has=0A> the memberOf field in it. I can provide this too if needed.)=0A=0A=
I see. The current code drops the memberOf attribute if it was not explicit=
ly requested in the search. However, by default the consumer requests "+" w=
hich means "all operational attributes" and so slapd considers memberOf to =
have been requested. We need to reconsider this aspect of the design.=0A> =
=0A> 5. The more serious problem occurs when the sequence is reversed and t=
he group=0A> has been deleted as a last operation - The user is replicated =
first, but since=0A> the group is deleted, it is never replicated and a sta=
le memberOf entry stays=0A> with the user.=0A=0A--=A0  -- Howard Chu=0A=A0 =
CTO, Symas Corp.=A0 =A0 =A0 =A0 =A0 http://www.symas.com=0A=A0 Director, Hi=
ghland Sun=A0 =A0 http://highlandsun.com/hyc/=0A=A0 Chief Architect, OpenLD=
AP=A0 http://www.openldap.org/project/
--1251447850-1386810138-1349783312=:99375
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"color:#000; background-color:#fff;
font-family:ti=
mes new roman, new york, times, serif;font-size:12pt"><div
style=3D"font-fa=
mily: 'times new roman', 'new york', times, serif; font-size: 12pt;
"><span=
>=0A=0A=0A=0A=0A=0A=0A=0A<div
class=3D"p1">Hi,</div></span></div><div><div>=
To try and overcome this issue, we tried two
fixes:</div><div><br></div><di=
v>1. Every time a user was deleted from a group, we force-updated the
user&=
nbsp;</div><div>object manually to make sure its entryCSN got
updated and i=
t got&nbsp;</div><div>replicated properly. This is an expensive
operation a=
nd did not scale&nbsp;</div><div>well for big group sizes
(10-20k), and did=
 not work out.</div><div><br></div><div>2. We then
tried to do the same thi=
ng in OpenLDAP. We noticed in the&nbsp;</div><div>memberof.c
commits that t=
here were a couple of patches to force
the&nbsp;</div><div>entryCSN of the =
user object to get updated.
(http://tinyurl.com/8k4qrdj&nbsp;</div><div>and=
 http://t

Message of length 9816 truncated


Followup 5

Download message
Date: Tue, 09 Oct 2012 07:32:02 -0700
From: Howard Chu <hyc@symas.com>
To: arun s <arunkumar_1123@yahoo.com>
CC: "openldap-its@openldap.org" <openldap-its@openldap.org>
Subject: Re: (ITS#7400) Memberof and Syncrepl incompatibility
arun s wrote:
> Hi,
> To try and overcome this issue, we tried two fixes:
>
> 1. Every time a user was deleted from a group, we force-updated the user
> object manually to make sure its entryCSN got updated and it got
> replicated properly. This is an expensive operation and did not scale
> well for big group sizes (10-20k), and did not work out.

This is also one of the reasons the code in memberof.c was reverted. Working 
as you suggest *cannot* scale. It makes no difference whether you do it in 
your external client or inside slapd, the amount of actual work required to 
replicate everything quickly grows out of control. This is why the 
documentation states that memberof must be configured on each replica - the 
only way to successfully execute the amount of work is to distribute the work 
evenly to each replica.

> 2. We then tried to do the same thing in OpenLDAP. We noticed in the
> memberof.c commits that there were a couple of patches to force the
> entryCSN of the user object to get updated. (http://tinyurl.com/8k4qrdj
> and http://tinyurl.com/9akqgfl)These have since been reverted because of
> access log and some replication issues, but for us, speed was a higher
> priority. I reapplied these patches back to the code. This solved the
> member-of replication issue, but we noticed that occasionally under a
> heavy load, there was a sudden surge in OpenLDAP's memory usage going up
> to whatever memory was available and finally crashing.
>
> We have gone back to option (1) though (2) would be the preferred option.
>
> Any help on figuring out why (2) caused the memory bloat would be really
> great. I can provide more details/memory traces if needed.
>
> We will be glad to contribute any fixes once we are able to nail down
> the issue.

Probably this conversation should continue on the openldap-devel mailing list. 
It needs some new design work; it is not a simple bugfix.

>
> Thanks,
> Arunkumar
> ------------------------------------------------------------------------------
> *From:* Howard Chu <hyc@symas.com>
> *To:* arun s <arunkumar_1123@yahoo.com>
> *Cc:* "openldap-its@openldap.org" <openldap-its@openldap.org>
> *Sent:* Monday, 1 October 2012 6:59 PM
> *Subject:* Re: (ITS#7400) Memberof and Syncrepl incompatibility
>
> arun s wrote:
>  > Hi,
>  > Yes, I am able to reproduce the issue with 2.4.32
>  >
>  > Making sense of the logs for the exact reproduction is hard since it
needs a
>  > lot of operations in a short time. But this will probably help:
>  >
>  > 1. At the start of the test, the group temp_group existed.
>  >
>  > 2. I created a user temp_user and added temp_user to temp_group.
>  >
>  > 3. During replication, the group was replicated first and I got an
error 32
>  > (NO_SUCH_OBJECT) when it tried to modify the memberOf of the
temp_user object
>  > (This does not exist in the readslave yet).
>  >
>  > 4. The temp_user object was replicated next, and as you see, querying
it does
>  > show a memberOf attribute, proving that this field was replicated.
(Note that
>  > I have run OpenLDAP with debug as -1 and verified that the replicated
data has
>  > the memberOf field in it. I can provide this too if needed.)
>
> I see. The current code drops the memberOf attribute if it was not
explicitly
> requested in the search. However, by default the consumer requests "+"
which
> means "all operational attributes" and so slapd considers memberOf to have
> been requested. We need to reconsider this aspect of the design.
>  >
>  > 5. The more serious problem occurs when the sequence is reversed and
the group
>  > has been deleted as a last operation - The user is replicated first,
but since
>  > the group is deleted, it is never replicated and a stale memberOf
entry stays
>  > with the user.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/


Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org