[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#6919) "Weekly" crashes of OpenLDAP 2.4.22 using syncrepl master/master
- To: openldap-its@OpenLDAP.org
- Subject: Re: (ITS#6919) "Weekly" crashes of OpenLDAP 2.4.22 using syncrepl master/master
- From: jarl@dallur.com
- Date: Sun, 1 May 2011 22:46:28 GMT
- Auto-submitted: auto-generated (OpenLDAP-ITS)
--000e0cd48c428f954504a23eaa22
Content-Type: text/plain; charset=UTF-8
First of all let me apologize for the state of the config.
As you say the logs don't show any crash yet the OpenLDAP server is not
accepting any new connections after 20:36:37, there should be hundreds of
connections being logged there but instead the server hangs completely and
nothing is logged, even on log level 1.
Perhaps someone could suggest a way of getting more information than what
loglevel 1 shows?
Would the only other step be a debug build?
Jarl
On Sun, May 1, 2011 at 4:01 AM, Howard Chu <hyc@symas.com> wrote:
> jarl@dallur.com wrote:
>
>> Full_Name: Jarl Stefansson
>> Version: 2.4.22
>> OS: Centos 5.5
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (81.15.35.75)
>>
>>
>> Two servers running OpenLDAP in master/master using syncrepl, roughly once
>> per
>> week one of the servers stops responding to new connections, I tried
>> logging
>> with loglevel=1 and this is all I got, all suggestions and comments
>> appreciated.
>>
>> This is most likely a "load" related problem since we never experience
>> this
>> problem in the testlab, only production.:
>>
>
> Your log doesn't indicate any kind of crash. It only shows that a shutdown
> was requested, and apparently your startup scripts didn't wait for the
> shutdown to complete before starting again.
>
>
> Apr 23 20:36:37 ldap01 slapd2.4[32233]:
>> bdb_dn2entry("macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx")
>> Apr 23 20:36:37 ldap01 slapd2.4[32233]: =>
>> bdb_dn2id("macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx")
>> Apr 23 20:36:37 ldap01 slapd2.4[32233]:<= bdb_dn2id: get failed:
>> DB_NOTFOUND:
>> No matching key/data pair found (-30988)
>> Apr 23 20:36:37 ldap01 slapd2.4[32233]: => bdb_dn2id_add 0x2fed7:
>> "macaddress=1\2C6\2C00:00:00:00:00:01,ou=xxx,ou=xx,o=xxxx"
>> Apr 23 20:37:44 ldap01 slapd2.4[32233]: slap_listener_activate(8):
>> Apr 23 21:35:42 ldap01 slapd2.4[32233]: connection_close: conn=1025 sd=16
>> Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1022 sd=17
>> Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1037 sd=18
>> Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1038 sd=20
>> Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=1039 sd=25
>>
>> Apr 24 12:02:23 ldap01 slapd2.4[32233]: daemon: shutdown requested and
>> initiated. I have two Centos 5 servers running Openldap 2.4.22 with
>> master/master syncrepl setup,
>> Apr 24 12:02:23 ldap01 slapd2.4[32233]: connection_close: conn=1005 sd=21
>> Apr 24 12:02:23 ldap01 slapd2.4[32233]: slapd shutdown: waiting for 59
>> operations/tasks to finish
>> Apr 24 12:02:34 ldap01 slapd2.4[26503]: bdb_db_open: database "": unclean
>> shutdown detected; attempting recovery.
>> Apr 24 12:02:35 ldap01 slapd2.4[26503]: slapd starting
>>
>
> Your config is a mess, with global directives and DB-specific directives
> interleaved. That may not be the cause of any specific problems, but it
> shows sloppiness on the part of the sysadmins.
>
>
> ---------------------- Relevant Config
>> ------------------------------------------------
>> moduleload syncprov.la
>> TLSCertificateFile /etc/pki/tls/private/ldap.pem
>> TLSCertificateKeyFile /etc/pki/tls/private/ldap.pem
>> TLSCACertificateFile /etc/pki/tls/private/ldap.pem
>>
>> serverID 001
>> database bdb
>> sizelimit 50000
>> cachesize 10000
>> checkpoint 256 5
>>
>>
>> syncrepl rid=001
>> provider=ldap://10.10.10.10:389
>> bindmethod=simple
>> binddn="uid=xxxxrep,ou=xxxxxx,o=xxxxxx"
>> credentials=MyPassword
>> searchbase="o=xxxx"
>> schemachecking=on
>> sizelimit="unlimited"
>> timelimit="unlimited"
>> type=refreshAndPersist
>> interval=00:00:00:10
>> retry="5 5 60 +"
>> attrs="*,+"
>>
>> mirrormode on
>> overlay syncprov
>> syncprov-checkpoint 100 10
>> syncprov-sessionlog 100
>>
>> idletimeout 3600
>> threads 32
>>
>>
>> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--
Regards
Jarl
jarl@dallur.com
--000e0cd48c428f954504a23eaa22
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
First of all let me=C2=A0apologize=C2=A0for the state of the config.<div><b=
r></div><div>As you say the logs don't show any crash yet the OpenLDAP =
server is not accepting any new connections after=C2=A0<meta http-equiv=3D"=
content-type" content=3D"text/html; charset=3Dutf-8">20:36:37, there should=
be hundreds of connections being logged there but instead the server hangs=
completely and nothing is logged, even on log level 1.</div>
<div><br></div><div>Perhaps someone could suggest a way of getting more inf=
ormation than what loglevel 1 shows?</div><div><br></div><div>Would the onl=
y other step be a debug build?</div><div><br></div><div>Jarl<br><br><div cl=
ass=3D"gmail_quote">
On Sun, May 1, 2011 at 4:01 AM, Howard Chu <span dir=3D"ltr"><<a href=3D=
"mailto:hyc@symas.com";>hyc@symas.com</a>></span> wrote:<br><blockquote c=
lass=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;=
padding-left:1ex;">
<div class=3D"im"><a href=3D"mailto:jarl@dallur.com"; target=3D"_blank">jarl=
@dallur.com</a> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Full_Name: Jarl Stefansson<br>
Version: =C2=A02.4.22<br>
OS: Centos 5.5<br>
URL: <a href=3D"ftp://ftp.openldap.org/incoming/"; target=3D"_blank">ftp://f=
tp.openldap.org/incoming/</a><br>
Submission from: (NULL) (81.15.35.75)<br>
<br>
<br>
Two servers running OpenLDAP in master/master using syncrepl, roughly once =
per<br>
week one of the servers stops responding to new connections, I tried loggin=
g<br>
with loglevel=3D1 and this is all I got, all suggestions and comments<br>
appreciated.<br>
<br>
This is most likely a "load" related problem since we never exper=
ience this<br>
problem in the testlab, only production.:<br>
</blockquote>
<br></div>
Your log doesn't indicate any kind of crash. It only shows that a shutd=
own was requested, and apparently your startup scripts didn't wait for =
the shutdown to complete before starting again.<div class=3D"im"><br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
Apr 23 20:36:37 ldap01 slapd2.4[32233]:<br>
bdb_dn2entry("macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,=
o=3Dxxxx")<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]: =3D><br>
bdb_dn2id("macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,o=
=3Dxxxx")<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]:<=3D bdb_dn2id: get failed: DB_NO=
TFOUND:<br>
No matching key/data pair found (-30988)<br>
Apr 23 20:36:37 ldap01 slapd2.4[32233]: =3D> =C2=A0bdb_dn2id_add 0x2fed7=
:<br>
"macaddress=3D1\2C6\2C00:00:00:00:00:01,ou=3Dxxx,ou=3Dxx,o=3Dxxxx"=
;<br>
Apr 23 20:37:44 ldap01 slapd2.4[32233]: slap_listener_activate(8):<br>
Apr 23 21:35:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1025 sd=3D=
16<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1022 sd=3D=
17<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1037 sd=3D=
18<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1038 sd=3D=
20<br>
Apr 23 21:55:42 ldap01 slapd2.4[32233]: connection_close: conn=3D1039 sd=3D=
25<br>
<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: daemon: shutdown requested and<br>
initiated. I have two Centos 5 servers running Openldap 2.4.22 with<br>
master/master syncrepl setup,<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: connection_close: conn=3D1005 sd=3D=
21<br>
Apr 24 12:02:23 ldap01 slapd2.4[32233]: slapd shutdown: waiting for 59<br>
operations/tasks to finish<br>
Apr 24 12:02:34 ldap01 slapd2.4[26503]: bdb_db_open: database "":=
unclean<br>
shutdown detected; attempting recovery.<br>
Apr 24 12:02:35 ldap01 slapd2.4[26503]: slapd starting<br>
</blockquote>
<br></div>
Your config is a mess, with global directives and DB-specific directives in=
terleaved. That may not be the cause of any specific problems, but it shows=
sloppiness on the part of the sysadmins.<div><div></div><div class=3D"h5">
<br>
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">
---------------------- Relevant Config<br>
------------------------------------------------<br>
moduleload =C2=A0 =C2=A0 <a href=3D"http://syncprov.la"; target=3D"_blank">s=
yncprov.la</a><br>
TLSCertificateFile =C2=A0 =C2=A0 =C2=A0/etc/pki/tls/private/ldap.pem<br>
TLSCertificateKeyFile =C2=A0 /etc/pki/tls/private/ldap.pem<br>
TLSCACertificateFile =C2=A0 =C2=A0/etc/pki/tls/private/ldap.pem<br>
<br>
serverID =C2=A0 =C2=A0 =C2=A0 =C2=A0001<br>
database =C2=A0 =C2=A0 =C2=A0 =C2=A0bdb<br>
sizelimit =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A050000<br>
cachesize 10000<br>
checkpoint 256 5<br>
<br>
<br>
syncrepl =C2=A0 =C2=A0 =C2=A0 =C2=A0rid=3D001<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 provider=3Dldap://=
<a href=3D"http://10.10.10.10:389"; target=3D"_blank">10.10.10.10:389</a><br=
>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bindmethod=3Dsimpl=
e<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 binddn=3D"uid=
=3Dxxxxrep,ou=3Dxxxxxx,o=3Dxxxxxx"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 credentials=3DMyPa=
ssword<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 searchbase=3D"=
;o=3Dxxxx"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 schemachecking=3Do=
n<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 sizelimit=3D"=
unlimited"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 timelimit=3D"=
unlimited"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 type=3DrefreshAndP=
ersist<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 interval=3D00:00:0=
0:10<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 retry=3D"5 5 =
60 +"<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 attrs=3D"*,+&=
quot;<br>
<br>
mirrormode on<br>
overlay syncprov<br>
syncprov-checkpoint =C2=A0 =C2=A0 100 =C2=A0 =C2=A0 10<br>
syncprov-sessionlog =C2=A0 =C2=A0 100<br>
<br>
idletimeout 3600<br>
threads 32<br>
<br>
<br>
</blockquote></div></div><font color=3D"#888888">
-- <br>
=C2=A0-- Howard Chu<br>
=C2=A0CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:=
//www.symas.com" target=3D"_blank">http://www.symas.com</a><br>
=C2=A0Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"http://highlandsun.c=
om/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a><br>
=C2=A0Chief Architect, OpenLDAP =C2=A0<a href=3D"http://www.openldap.org/p=
roject/" target=3D"_blank">http://www.openldap.org/project/</a><br>
</font></blockquote></div><br><br clear=3D"all"><br>-- <br>Regards<br><br>J=
arl<br><a href=3D"mailto:jarl@dallur.com";>jarl@dallur.com</a><br>
</div>
--000e0cd48c428f954504a23eaa22--