[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#7400) Memberof and Syncrepl incompatibility
--1251447850-1386810138-1349783312=:99375
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Hi,=0ATo try and overcome this issue, we tried two fixes:=0A=0A1. Every tim=
e a user was deleted from a group, we force-updated the user=A0=0Aobject ma=
nually to make sure its entryCSN got updated and it got=A0=0Areplicated pro=
perly. This is an expensive operation and did not scale=A0=0Awell for big g=
roup sizes (10-20k), and did not work out.=0A=0A2. We then tried to do the =
same thing in OpenLDAP. We noticed in the=A0=0Amemberof.c commits that ther=
e were a couple of patches to force the=A0=0AentryCSN of the user object to=
get updated. (http://tinyurl.com/8k4qrdj=A0=0Aand http://tinyurl.com/9akqg=
fl)These have since been reverted because of=A0=0Aaccess log and some repli=
cation issues, but for us, speed was a higher=A0=0Apriority. I reapplied th=
ese patches back to the code. This solved the=A0=0Amember-of replication is=
sue, but we noticed that occasionally under a=A0=0Aheavy load, there was a =
sudden surge in OpenLDAP's memory usage going up=A0=0Ato whatever memory wa=
s available and finally crashing.=0A=0AWe have gone back to option (1) thou=
gh (2) would be the preferred option.=0A=0AAny help on figuring out why (2)=
caused the memory bloat would be really=A0=0Agreat. I can provide more det=
ails/memory traces if needed.=0A=0AWe will be glad to contribute any fixes =
once we are able to nail down=A0=0Athe issue.=0A=0AThanks,=0AArunkumar=0A=
=0A________________________________=0A From: Howard Chu <hyc@symas.com>=0AT=
o: arun s <arunkumar_1123@yahoo.com> =0ACc: "openldap-its@openldap.org" <op=
enldap-its@openldap.org> =0ASent: Monday, 1 October 2012 6:59 PM=0ASubject:=
Re: (ITS#7400) Memberof and Syncrepl incompatibility=0A =0Aarun s wrote:=
=0A> Hi,=0A> Yes, I am able to reproduce the issue with 2.4.32=0A> =0A> Mak=
ing sense of the logs for the exact reproduction is hard since it needs a=
=0A> lot of operations in a short time. But this will probably help:=0A> =
=0A> 1. At the start of the test, the group temp_group existed.=0A> =0A> 2.=
I created a user temp_user and added temp_user to temp_group.=0A> =0A> 3. =
During replication, the group was replicated first and I got an error 32=0A=
> (NO_SUCH_OBJECT) when it tried to modify the memberOf of the temp_user ob=
ject=0A> (This does not exist in the readslave yet).=0A> =0A> 4. The temp_u=
ser object was replicated next, and as you see, querying it does=0A> show a=
memberOf attribute, proving that this field was replicated. (Note that=0A>=
I have run OpenLDAP with debug as -1 and verified that the replicated data=
has=0A> the memberOf field in it. I can provide this too if needed.)=0A=0A=
I see. The current code drops the memberOf attribute if it was not explicit=
ly requested in the search. However, by default the consumer requests "+" w=
hich means "all operational attributes" and so slapd considers memberOf to =
have been requested. We need to reconsider this aspect of the design.=0A> =
=0A> 5. The more serious problem occurs when the sequence is reversed and t=
he group=0A> has been deleted as a last operation - The user is replicated =
first, but since=0A> the group is deleted, it is never replicated and a sta=
le memberOf entry stays=0A> with the user.=0A=0A--=A0 -- Howard Chu=0A=A0 =
CTO, Symas Corp.=A0 =A0 =A0 =A0 =A0 http://www.symas.com=0A=A0 Director, Hi=
ghland Sun=A0 =A0 http://highlandsun.com/hyc/=0A=A0 Chief Architect, OpenLD=
AP=A0 http://www.openldap.org/project/
--1251447850-1386810138-1349783312=:99375
Content-Type: text/html; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"color:#000; background-color:#fff; font-family:ti=
mes new roman, new york, times, serif;font-size:12pt"><div style=3D"font-fa=
mily: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span=
>=0A=0A=0A=0A=0A=0A=0A=0A<div class=3D"p1">Hi,</div></span></div><div><div>=
To try and overcome this issue, we tried two fixes:</div><div><br></div><di=
v>1. Every time a user was deleted from a group, we force-updated the user&=
nbsp;</div><div>object manually to make sure its entryCSN got updated and i=
t got </div><div>replicated properly. This is an expensive operation a=
nd did not scale </div><div>well for big group sizes (10-20k), and did=
not work out.</div><div><br></div><div>2. We then tried to do the same thi=
ng in OpenLDAP. We noticed in the </div><div>memberof.c commits that t=
here were a couple of patches to force the </div><div>entryCSN of the =
user object to get updated. (http://tinyurl.com/8k4qrdj </div><div>and=
http://tinyurl.com/9akqgfl)These have since been reverted because of =
</div><div>access log and some replication issues, but for us, speed was a =
higher </div><div>priority. I reapplied these patches back to the code=
. This solved
the </div><div>member-of replication issue, but we noticed that occas=
ionally under a </div><div>heavy load, there was a sudden surge in Ope=
nLDAP's memory usage going up </div><div>to whatever memory was availa=
ble and finally crashing.</div><div><br></div><div>We have gone back to opt=
ion (1) though (2) would be the preferred option.</div><div><br></div><div>=
Any help on figuring out why (2) caused the memory bloat would be really&nb=
sp;</div><div>great. I can provide more details/memory traces if needed.</d=
iv><div><br></div><div>We will be glad to contribute any fixes once we are =
able to nail down </div><div>the issue.</div><div><br></div><div>Thank=
s,</div><div>Arunkumar</div></div> <div style=3D"font-family: 'times new r=
oman', 'new york', times, serif; font-size: 12pt; "> <div style=3D"font-fam=
ily: 'times new roman', 'new york', times, serif; font-size: 12pt; "> <div =
dir=3D"ltr"> <font size=3D"2" face=3D"Arial"> <hr size=3D"1"> <b><span
style=3D"font-weight:bold;">From:</span></b> Howard Chu <hyc@symas.com&=
gt;<br> <b><span style=3D"font-weight: bold;">To:</span></b> arun s <aru=
nkumar_1123@yahoo.com> <br><b><span style=3D"font-weight: bold;">Cc:</sp=
an></b> "openldap-its@openldap.org" <openldap-its@openldap.org> <br> =
<b><span style=3D"font-weight: bold;">Sent:</span></b> Monday, 1 October 20=
12 6:59 PM<br> <b><span style=3D"font-weight: bold;">Subject:</span></b> Re=
: (ITS#7400) Memberof and Syncrepl incompatibility<br> </font> </div> <br>a=
run s wrote:<br>> Hi,<br>> Yes, I am able to reproduce the issue with=
2.4.32<br>> <br>> Making sense of the logs for the exact reproductio=
n is hard since it needs a<br>> lot of operations in a short time. But t=
his will probably help:<br>> <br>> 1. At the start of the test, the g=
roup temp_group existed.<br>> <br>> 2. I created a user temp_user and=
added temp_user to temp_group.<br>> <br>> 3. During replication, the
group was replicated first and I got an error 32<br>> (NO_SUCH_OBJECT) =
when it tried to modify the memberOf of the temp_user object<br>> (This =
does not exist in the readslave yet).<br>> <br>> 4. The temp_user obj=
ect was replicated next, and as you see, querying it does<br>> show a me=
mberOf attribute, proving that this field was replicated. (Note that<br>>=
; I have run OpenLDAP with debug as -1 and verified that the replicated dat=
a has<br>> the memberOf field in it. I can provide this too if needed.)<=
br><br>I see. The current code drops the memberOf attribute if it was not e=
xplicitly requested in the search. However, by default the consumer request=
s "+" which means "all operational attributes" and so slapd considers membe=
rOf to have been requested. We need to reconsider this aspect of the design=
.<br>> <br>> 5. The more serious problem occurs when the sequence is =
reversed and the group<br>> has been deleted as a last operation
- The user is replicated first, but since<br>> the group is deleted, it=
is never replicated and a stale memberOf entry stays<br>> with the user=
.<br><br>-- -- Howard Chu<br> CTO, Symas Corp. &n=
bsp; <a href=3D"http://www.symas.com/"; target=3D"_blank">htt=
p://www.symas.com</a><br> Director, Highland Sun <a hre=
f=3D"http://highlandsun.com/hyc/"; target=3D"_blank">http://highlandsun.com/=
hyc/</a><br> Chief Architect, OpenLDAP <a href=3D"http://www.op=
enldap.org/project/" target=3D"_blank">http://www.openldap.org/project/</a>=
<br><br><br> </div> </div> </div></body></html>
--1251447850-1386810138-1349783312=:99375--