[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#5860) slapd memeory leak under openldap 2.4

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
From: rlvcosta@yahoo.com
Date: Tue, 27 Jan 2009 10:27:22 GMT
--0-1644073040-1233052012=:88983
Content-Type: text/plain; charset=us-ascii

Howard,

Initially the search behaves normally but after some time something changes and memory starts to be consumed again until it reaches another stable condition, even very large(around 2G) and the search get stuck as before.

Once search get stuck even I stop both searches and start a new one it get stuck. So in my test I stopped my 2 searches and started a new one. Then I saw the new search with only 240 returns and then get stuck entering in the same model as before with 16 entrances in bursts in order of minutes to enter in these bursts.

As I tried to explain this appears to happen when the seconds ldapsearch, started at a later time than the first ldapsearch, reaches the dn cachesize. This is what appears to happen.

I beleive for you to reproduce you need to have a DB large enough and a slapd.conf where dn cachesize will overpass the memory boundary imposed. Then after the second ldapsearch memory boundary is reached the issue should appear.

The commands I runned were :

time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret  -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*' > /backup/temp.ldif

and

time ldapsearch -LLL -x -D "cn=admin,ou=INDEXES,o=alcatel,c=fr" -w secret  -b "ou=INDEXES,o=alcatel,c=fr" -H ldap://10.142.15.42:389 'pnnumber=*'| egrep -e '^pnnumber*' |wc -l


With the first command, redirecting to a file, I can tail or wc and see it's progress. After the second ldapsearch process starts to make slapd consume memory in large chunks until the new 2G boundary is reached. Once this happen the search get stuck as before with the single process returning 16 records per minute(s).

Once this condition is reached even stopping the old searches and starting new ones the rate will be very slow, like 16 reports per minute(s). It get stuck as before.

Please see attached a single entrance in my DB where you can change DB keys and load it as needed.

Thanks,

Rodrigo.

--- On Mon, 1/26/09, Howard Chu <hyc@symas.com> wrote:

> From: Howard Chu <hyc@symas.com>
> Subject: Re: (ITS#5860) slapd memeory leak under openldap 2.4
> To: rlvcosta@yahoo.com
> Cc: openldap-its@openldap.org
> Date: Monday, January 26, 2009, 11:16 PM
> rlvcosta@yahoo.com wrote:
> > Quanah,
> >
> > In this specific test I reported in my previous e-mail
> it was used glibc.
> >
> > Now I tested with tcmalloc and the memory again was
> much more consumed and never released.
> >
> >    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
> TIME+  COMMAND
> > 10652 ldap      18   0 2006m 1.8g  68m S  100 15.8  
> 8:36.75 slapd
> >
> > Once it get this much memory it then stabilize but it
> is much more than before that was 385m.
> 
> We may need to have sample data from you to reproduce this
> situation. I 
> haven't seen anything like it on my tests. Running two
> searches simultaneously 
> still behaves normally for me.
> >
> > The scenario was :
> 
> Likewise, the exact commands and arguments will be
> needed...
> >
> > 1) Start a full ldapsearch for one specific dn(the DB
> has 4 dn's);
> > 2) After sometime start a new full ldapsearch to
> behave as queries to DB when replication for LDIF backup is
> being done;
> > 3) Export one of the queries to a file so speed
> verification can be done with grep and wc.
> >
> > After sometime the memory starts to be consumed in
> large chunks until it stopped in around 2G.
> >
> > Since one of the queries is being redirect to a file I
> can tail to have a visual rate of the entrances being
> returned by slapd. As the issue before after sometime the
> rate appears to get stuck and come in small 16 entrances
> chunks but only after some minutes.
> >
> > This behavior above appears to happen after the second
> ldapsearch fills its own dn cache even monitor always shows
> only one information as I could see.
> >
> > [root@brtldp11 ~]# date; grep -e '^pnnum*'
> /backup/temp.ldif |wc -l
> > Mon Jan 26 21:43:31 BRST 2009
> > 482432
> > [root@brtldp11 ~]# date; grep -e '^pnnum*'
> /backup/temp.ldif |wc -l
> > Mon Jan 26 21:44:56 BRST 2009
> > 482448
> >
> > This behavior is similar as before with the initial
> issue with a single process.
> >
> > I just notice when slapd starts there are these
> threads :
> >
> > [root@brtldp12 ~]# ps -mu ldap
> >    PID TTY          TIME CMD
> > 10638 pts/2    00:00:00 slapd
> >      - -        00:00:00 -
> >      - -        00:00:00 -
> >
> > Then after the 2 queries are started I have :
> >
> > [root@brtldp12 ~]# ps -mu ldap
> >    PID TTY          TIME CMD
> > 10195 pts/2    00:44:22 slapd
> >      - -        00:00:00 -
> >      - -        00:00:00 -
> >      - -        00:26:09 -
> >      - -        00:18:13 -
> >      - -        00:00:00 -
> >
> > So 3 new threads. Looks like for some reason with
> multiples queries the issue with response getting stuck
> repeats.
> >
> > Even one ldapsearch is stopped the number of threads
> continues the same, memory continues allocated and the other
> search get with this slow rate.
> >
> > These outputs are using now tcmalloc but similar as
> with glibc.
> 
> -- 
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com
>    Director, Highland Sun     http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP 
> http://www.openldap.org/project/


      
--0-1644073040-1233052012=:88983
Content-Type: text/plain; name="one_INDEXES_entrance.txt"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="one_INDEXES_entrance.txt"

ZG46IHBubnVtYmVyPVwyQjU1NDE4NDAxMTQ5NixvdT1yYW5nZTk2LG91PW1z
aXNkbixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQpzdWJwbmlkOiAwDQpo
b3N0OiBicnRsZHAxMmUNCm9iamVjdENsYXNzOiBwaG9uZWluZm8NCm9iamVj
dENsYXNzOiBpbmRleG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wy
QjU1NDE4NDAxMTQ5NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNh
dGVsLGM9Zg0KIHINCnBubnVtYmVyOiArNTU0MTg0MDExNDk2DQoNCmRuOiB3
ZWJsb2dpbj1cMkI1NTQxODQwMTE0OTYsb3U9cmFuZ2U2NyxvdT13ZWJsb2dp
bixvdT1JTkRFWEVTLG89YWxjYXRlbCxjPWZyDQp3ZWJsb2dpbjogKzU1NDE4
NDAxMTQ5Ng0KaG9zdDogYnJ0bGRwMTJlDQphY2NvdW50U3RhdHVzOiAxDQpv
YmplY3RDbGFzczogY29tbW9udXNlcg0Kb2JqZWN0Q2xhc3M6IGluZGV4bm9k
ZQ0KcmVmZG46IHVpZD0xMjE5ODQzNzk2NjU3XDJCNTU0MTg0MDExNDk2LG91
PVJFUE9TSVRPUlksb3U9Q09OVEVOVCxvPWFsY2F0ZWwsYz1mDQogcg0KY246
ICs1NTQxODQwMTE0OTYNCg0KZG46IG1haWw9XDJCNTU0MTg0MDExNDk2L3R5
cGVcM0RQTE1OQG1tcy5icmFzaWx0ZWxlY29tLmNvbS5icixvdT1yYW5nZTg0
LG91PW1hDQogaWwsb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0KbWFpbDog
KzU1NDE4NDAxMTQ5Ni90eXBlPVBMTU5AbW1zLmJyYXNpbHRlbGVjb20uY29t
LmJyDQpob3N0OiBicnRsZHAxMmUNCmFjY291bnRTdGF0dXM6IDENCm9iamVj
dENsYXNzOiBjb21tb251c2VyDQpvYmplY3RDbGFzczogaW5kZXhub2RlDQpy
ZWZkbjogdWlkPTEyMTk4NDM3OTY2NTdcMkI1NTQxODQwMTE0OTYsb3U9UkVQ
T1NJVE9SWSxvdT1DT05URU5ULG89YWxjYXRlbCxjPWYNCiByDQpjbjogKzU1
NDE4NDAxMTQ5Ng0KDQpkbjogbWFpbGxvZ2luPVwyQjU1NDE4NDAxMTQ5Ni90
eXBlXDNEUExNTkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnIsb3U9cmFuZ2U4
NCwNCiBvdT1pbWFwbG9naW4sb3U9SU5ERVhFUyxvPWFsY2F0ZWwsYz1mcg0K
c3VibXhpZDogMA0KbWFpbGxvZ2luOiArNTU0MTg0MDExNDk2L3R5cGU9UExN
TkBtbXMuYnJhc2lsdGVsZWNvbS5jb20uYnINCmhvc3Q6IGJydGxkcDEyZQ0K
b2JqZWN0Q2xhc3M6IGNvbW1vbm1haWxib3gNCm9iamVjdENsYXNzOiBpbmRl
eG5vZGUNCnJlZmRuOiB1aWQ9MTIxOTg0Mzc5NjY1N1wyQjU1NDE4NDAxMTQ5
NixvdT1SRVBPU0lUT1JZLG91PUNPTlRFTlQsbz1hbGNhdGVsLGM9Zg0KIHIN
Cg==

--0-1644073040-1233052012=:88983
Content-Type: application/x-zip; name="index_simplex.zip"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="index_simplex.zip"

UEsDBBQAAgAIAC9DOzrp93S+DBEAADUEAQASABUAaW5kZXhfc2ltcGxleC5s
ZGlmVVQJAAMa4X5JGuF+SVV4BADuAwEC7Z1tb+JIFoW/8yv4PlEPVS6/pRVp
e9IZKVJ3Opukd2dWI43AdtKeBZsF0y+z2f++NraBlE+RgA+ku4eWZkNwcc71
fVy36jrAhslxN52dnF+8Pvvl7PooPekPg34WDY+Ck9tJJx38EQVZMOxPp/mw
yV0/if/sZ3Ga9IfvkzjrpLPjbvXSTicspd68fnV5+u7i56MWsqvHh2F/PJ0N
psEkHhdjptXh0/LwNM6iOLlN57HU3p1R//OD1ySD467TK/51Gs/nT+UaSX8U
HXeTNPs9jG7jJArLEwrjrDhy8vr8RhxtcXb56+fRVTrH3UKoU5zTh3RaPTWY
ZMNwLGQ0f74fhpOoeKnovRBKvhD2CyXnR8bpJDvuWp5fqGVfxlGV/E4/yOKP
eQDhcffm6v1Z/nuQKwRpkk3S4XH3n1fnN2f5ed/exp/X0Z6kaVacc5Cc9MNR
nBytHzr+lGc/CiZRlmvPg4vC0nmUhlF+AqNpPA2To0/RYJje5XKjfjw8ikf9
8fzXRn4lK7+Sl988lJuzi5tGhvOznl+w6TiazB88MfOV3lMyv2boSubrWXd+
3XrSFXO5VClVo4LXNJp8jCZxeNI72txlRaHzQK2YdCtPlJhGX7pBOpvE0aTb
fA2mNh1lFTVpd4orq/xFKKsznqQf42l+enFyV8HJJv0kZxP276J/R1+Ouz8W
jE8u37y9WKTx8urdz+dvztqnckVoIX79/qfr06vzy5vzdxfV8esjpikyKN3z
bNzGw6jmSIxktZxWLp2FW8G5+qViPJoW9bU/G2adLAo+JHHQH6bjcgaJl72X
wcuX9+KleBm+fNkcIPMDxQCZD4QDrGqAZRqgKguVD/ww/ymrn1b1U1U/7eqn
U/10C8H8p1f83hS25zELcMSpLJ15bKJ4IPOhFhjq5kOzfKhrSoBvOiB6xiOL
rFZpLR7IwgaMlfVYaZSz6iGWcYgy5kLYJZh7YdfZyB/J+SMw2qlHl7mTaIxX
x+OZXcu0oZfLXvVyaUygrJMijUmRdVKkMSlSGY/Y9Ytt4xC3HmK8MqRf5Ur6
5RwAk6NnnDbmCWWcSVYxZQpDq5gz2NA2vrgmazn1jHXml8H8gWWQc+sXuaZT
VLWuqnQ7wzQ/HA3jab4sRMlRXr3y7cg4r1yf0klRnoodYZjm60yyKFAvBpP+
NB7m5S4K0tGL4r9BvhZ9HsflUp/MRoNokk/5fOM4X4TzJSWqpIoFa6lerdHF
k/+ZpVm/mKfKs11HL8piv0VZNIvyXToMt6rI1YVpGlDPDOPEOFTkQ0U+VGRS
RVZ1Ra6KqqqLqjIWVXqNrsa6xi2YU131qpqH30iNlvut0fJhjR5Gd/3gy672
zIfKfKjMi8pc6fhGnUOJblGipeHIYa/81L1yj3YL4/Rpu+WedhNj9thNDONW
2bhF/poKcFYV4LAuwNl2BdgznddXUpnbF2RzGTYUX3zEr+N9ZFcc1iU3MJbc
sC65gbHkmutpWNfTwFgsg7pYblj4VF3e9ldjTvN/5hrz94c1Jk6y4jZ0eTt1
PInm9+t/sKz5HebVu9bFTebV3xvFSey7OIlmcbobpoP+0Fib5pvFYE2Nujde
QdVSfV+t3c9Qs+7rlb/cCz1Sw7KWm8hq0t57poQcdpmmQlfuMosxxSOj4lOL
3yP7zTqnUhhP9akF8pE96RN3nmG98zTsoeu9uPTKLWHxoEXlrdcHyzi3rfpq
s4Rp8tb1+96yjIGv2cI+3vtb9VlbXkW7eGS6NlTNXZXcfR8Nqk9LGU+rXnzu
1eIaV4tr/LtfkOS+FyTZXJCKAOMgQiuS2Hi3vOZ2Rbjpbtd0pJ7B7nwN+eY2
t+Z1wLQHpextlyW/xY2Fp+1yt63dot7l9tbU2sPm1lBLrH3XEqtZSz7Gk2y2
ZnfL6rwNtcRwYKPG+TtrmJ+9pjxfv7zdPcVDJempfVcS1awk+RM5gSwKD50y
rVM+dMiHDvnQIX+rHfL3v+7Y+153bHB71vBeq6rUGVuTunkxXpx/uSVHVDdn
e9/xzdmgXnrCx5eeqmIIx5QR/S0AX9fS0/4NA5u/LeDbWnqCeukJDzdnv4fl
yNn3cuSAm7Px+LAaHVajw2r0F1mN/spLjSyvW1WTWSw63/9S4+57qXG1T5kE
yXfxtjl5eNvc1/K2ua/53XJr6vk2f2KsL5f9lalXr56hTEnb3vPHLOzGDZrg
93rcDj7uRvnsA3taf9OfNNjbJwEe/eAVfO++ZdwkoTuuvNlstX3HfT6/Xr1+
e37x/vrsivs1BUvZ0mn+dSDzbwFZ/EWu/B6NI0YM8y/UmOU15+HT84oVRHMc
nUUE81q0GkJ5aJ60PGX/Pb369fLmf3FydWtnP/05ev+P9G5ZTUqIxecTircB
F++8Kt4xUfyts7jvXDT7xaao+LKZcuTye0kKBoP0890knY3LmhRPVwPsVt9g
sXiuLLH50563fHLJu1vMe0utjq9i7JZ/C9Xynl91QVDkSJBTvpLYpQfMaf+P
oPfhlx8v78L0X28fz+mm+UQnPJ2No0kREeWkF2qNk9aOrD1xdDZa7PPsfpqO
dxd17fAw3kpEDyevSfnQHV44lQEMRuhXSn6dHEkcotx1iBKGKDcI0dp1iBYM
0dogRLXrEBUMUW0Qor3rEG0Yor1BiM6uQ3RgiM4GIbq7DtGFIbobhOjtOkQP
huhtEKK/6xB9GKK/JsTcs/pmtla7uFJjIbn4lrdWorXKQnbxdXHtdLVvnStS
UHwZXbsE5AoLuUk/uYvKT6Yyktut9JA+J9PrHEhJX2fRPv0GdUFGIJA+FQF0
4CKAFjQEurokI5BIn4oAOnARQAsaAl3dIiOwkD4VAXTgIoAWNAS6uiIjUEif
igA6cBFACxoCXd0mI7CRPhUBdOAigBY0BLq6Q0bgIH0qAujARQAtaAh0dZeM
wEX6VATQgYsAWtAQ6OoeGYGH9KkIoAMXAbSgIdDVfTICH+lTEUAHLgJoQUOg
qQtydyx6SJ+JADtQEWALFoKGOrk7FgLpUxFABy4CaEFDoKuTu2MhkT4VAXTg
IoAWNAS6Ork7FhbSpyKADlwE0IKGQFcnd8dCIX0qAujARQAtaAh0dXJ3LGyk
T0UAHbgIoAUNga5O7o6Fg/SpCKADFwG0oCHQ1cndsXCRPhUBdOAigBY0BLo6
uTsWHtKnIoAOXATQgoZAVyd3x8JH+lQE0IGLAFrQEGjqktwdyx7SZyLADlQE
2IKFoKFO7o6lQPpUBNCBiwBa0BDo6uTuWEqkT0UAHbgIoAUNga5O7o6lhfSp
CKADFwG0oCHQ1cndsVRIn4oAOnARQAsaAl2d3B1LG+lTEUAHLgJoQUOgq5O7
Y+kgfSoC6MBFAC1oCHR1cncsXaRPRQAduAigBQ2Brk7ujqWH9KkIoAMXAbSg
IdDVyd2x9JE+FQF04CKAFjQEmrpF7o6tHtJnIsAOVATYgoWgoU7uji2B9KkI
oAMXAbSgIdDVyd2xJZE+FQF04CKAFjQEujq5O7YspE9FAB24CKAFDYGuTu6O
LYX0qQigAxcBtKAh0NXJ3bFlI30qAujARQAtaAh0dXJ3bDlIn4oAOnARQAsa
Al2d3B1bLtKnIoAOXATQgoZAVyd3x5aH9KkIoAMXAbSgIdDVyd2x5SN9KgLo
wEUALWgINHVF7o5VD+kzEWAHKgJswULQUCd3x0ogfSoC6MBFAC1oCHR1cnes
JNKnIoAOXATQgoZAVyd3x8pC+lQE0IGLAFrQEOjq5O5YKaRPRQAduAigBQ2B
rk7ujpWN9KkIoAMXAbSgIdDVyd2xcpA+FQF04CKAFjQEujq5O1Yu0qcigA5c
BNCChkBXJ3fHykP6VATQgYsAWtAQ6Ork7lj5SJ+KADpwEUALGgJN3SZ3x3YP
6TMRYAcqAmzBQtBQJ3fHtkD6VATQgYsAWtAQ6Ork7tiWSJ+KADpwEUALGgJd
ndwd2xbSpyKADlwE0IKGQFcnd8e2QvpUBNCBiwBa0BDo6uTu2LaRPhUBdOAi
gBY0BLo6uTu2HaRPRQAduAigBQ2Brk7ujm0X6VMRQAcuAmhBQ6Crk7tj20P6
VATQgYsAWtAQ6Ork7tj2kT4VAXTgIoAWNASaukPujp0e0mciwA5UBNiChaCh
Tu6OHYH0qQigAxcBtKAh0NXJ3bEjkT4VAXTgIoAWNAS6Ork7diykT0UAHbgI
oAUNga5O7o4dhfSpCKADFwG0oCHQ1cndsWMjfSoC6MBFAC1oCHR1cnfsOEif
igA6cBFACxoCXZ3cHTsu0qcigA5cBNCChkBXJ3fHjof0qQigAxcBtKAh0NXJ
3bHjI30qAujARQAtaAg0dZfcHbs9pM9EgB2oCLAFC0FDndwduwLpUxFABy4C
aEFDoKuTu2NXIn0qAujARQAtaAh0dXJ37FpIn4oAOnARQAsaAl2d3B27CulT
EUAHLgJoQUOgq5O7Y9dG+lQE0IGLAFrQEOjq5O7YdZA+FQF04CKAFjQEujq5
O3ZdpE9FAB24CKAFDYGuTu6OXQ/pUxFABy4CaEFDoKuTu2PXR/pUBNCBiwBa
0BBo6h65O/Z6SJ+JADtQEWALFoKGOrk79gTSpyKADlwE0IKGQFcnd8eeRPpU
BNCBiwBa0BDo6uTu2LOQPhUBdOAigBY0BLo6uTv2FNKnIoAOXATQgoZAVyd3
x56N9KkIoAMXAbSgIdDVyd2x5yB9KgLowEUALWgIdHVyd+y5SJ+KADpwEUAL
GgJdndwdex7SpyKADlwE0IKGQFcnd8eej/SpCKADFwG0oCHQ1H1yd+z3kD4T
AXagIsAWLAQNdXJ37AukT0UAHbgIoAUNga5O7o59ifSpCKADFwG0oCHQ1cnd
sW8hfSoC6MBFAC1oCHR1cnfsK6RPRQAduAigBQ2Brk7ujn0b6VMRQAcuAmhB
Q6Crk7tj30H6VATQgYsAWtAQ6Ork7th3kT4VAXTgIoAWNAS6Ork79j2kT0UA
HbgIoAUNga5O7o59H+lTEUAHLgJoQUNQqY+TZDYaRJOT337oLf4d1Z69LXrm
8Yc0ieLkNn34dJyE0eckDaNObXncXbHsTGeDcRKHx91e50M6zY67g0k2DMdC
Rp1JdFuEOovDk3k8V2eX767Pb95d/Vr8dvru4ubs4kaPaX52RbZORqNp8Lf8
f14MJv1pPMxHREE6elH8N5gcbf934VxglCazaTQxnWmhlgexzr8T5GFOv0yz
aNTpB0E6S7Jp1s9muY4gJqK+9jHmzf/vUx8/99rxIeX9nG1xJuXpbsL+6bP3
dDUJhdkg/fzw0DIP+XU9+lxe14uwHrsmOJnQ5rf8qaYgtp3hp3iGo3Ou5zKY
7Ln9ujOUvZ4npHBkT0mh8uHNyB87/04XXfk4ARu+2f3UcP2fPuH6F9UEePBE
NQ2unzINyKmZV8eHEj9mX8bRbyeXb95ePGHagBWp022buLJo/qAHtT6mry6v
6JrbIrmLmtTpUqvSSinaNNMrBW0/KV1Tx8SyjolnqGMbXVSyGXn7OrZMwIaf
GGlXx4Q+30Sr+dYyNc06JjaaauiPHjuoY2KLOvbseUXX3BbJ3W8dE7urYy1S
uqaOyWUdk89Qx+QmGbCakbevY8sEbPixq3Z1TOrzTbaaby1T06xjcrOpJvZS
x+QWdezZ84quuS2Su986JndXx1qkdE0ds5Z1zHqGOmZtkgHVjLx9HVsmYMPP
LrarY5Y+36xW861lapp1zNpsqsm91DFrizr27HlF19wWyd1vHbN2V8dapHRN
HVPLOqaeoY6pTTJgNyNvX8eWCdjwA8Dt6pjS55tqNd9apqZZx9RmU83aSx1T
W9SxZ88ruua2SO5+65jaXR1rkdLO/wFQSwECFwMUAAIACAAvQzs66fd0vgwR
AAA1BAEAEgANAAAAAAABAAAApIEAAAAAaW5kZXhfc2ltcGxleC5sZGlmVVQF
AAMa4X5JVXgAAFBLBQYAAAAAAQABAE0AAABREQAAAAA=

--0-1644073040-1233052012=:88983--
Prev by Date: Re: (ITS#5540) Normalization assertion in attr.c
Next by Date: Re: (ITS#5860) slapd memeory leak under openldap 2.4
Index(es):
- Chronological
- Thread