[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Fwd: (ITS#7862) slapd crash on modifications after host hard reset (in mdb)
- To: openldap-its@OpenLDAP.org
- Subject: Fwd: (ITS#7862) slapd crash on modifications after host hard reset (in mdb)
- From: vadius@vadius.ru
- Date: Wed, 28 May 2014 10:45:30 GMT
- Auto-submitted: auto-generated (OpenLDAP-ITS)
--047d7b5d285813733e04fa73805f
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
2014-05-28 13:47 GMT+04:00 Howard Chu <hyc@symas.com>:
vadius@vadius.ru wrote:
>
>> Full_Name: Vadim Prozorov
>> Version: 2.4.38
>> OS: RHEL 6.3
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (91.210.4.1)
>>
>>
>> Hello,
>>
>> I'm using slapd (2.4.38 release) built from sources on RHEL 6.3 64bit.
>> Build params:
>> LD_LIBRARY_PATH=3D"/usr/lib:/usr/local/lib:/opt/local/BerkeleyDB.5.3/lib=
"
>> LDFLAGS=3D"-L/usr/local/lib -L/opt/local/BerkeleyDB.5.3/lib"
>> CPPFLAGS=3D"-I/usr/local/include -I/opt/local/BerkeleyDB.5.3/include"
>> ./configure
>> --prefix=3D/opt/local/OpenLDAP --enable-dynacl --enable-ldap
>> =E2=80=94enable-overlays
>> --disable-static --enable-dynamic
>>
>> slapd works in master-master sync mode on host with big amount of RAM
>> (whole DB
>> can fit into RAM).
>> One time host was hardly restarted. After restart slapd has started, but
>> crash
>> on any modification (at least, from ext application; may be form replica=
,
>> haven't check yet). There is always the same crash stack:
>>
>
> Did you have dbnosync configured on this backend? Looks like garbage data
> got written to the freeDB.
>
No, DB config section is:
# define MDB parameters
backend mdb
database mdb
maxsize 20000000000
envflags writemap mapasync
# end define
>
> (gdb) bt
>> #0 mdb_xcursor_init1 (mc=3D0x7f3e1d6bbd40, node=3D0x7f3f99f46f82) at
>> ./../../../libraries/liblmdb/mdb.c:6585
>> #1 0x000000000049f8eb in mdb_cursor_set (mc=3D0x7f3e1d6bbd40,
>> key=3D0x7f3e1d6bbef0,
>> data=3D0x0, op=3DMDB_SET_RANGE, exactp=3D<value optimized out>) at
>> ./../../../libraries/liblmdb/mdb.c:5329
>> #2 0x00000000004a019f in mdb_cursor_get (mc=3D0x7f3e1d6bbd40,
>> key=3D0x7f3e1d6bbef0,
>> data=3D0x0, op=3D<value optimized out>) at ./../../../libraries/liblmdb/
>> mdb.c:5529
>> #3 0x000000000049e0ed in mdb_page_alloc (mc=3D<value optimized out>, nu=
m=3D1,
>> mp=3D0x7f3e1d6bbf68) at ./../../../libraries/liblmdb/mdb.c:1727
>> #4 0x000000000049e469 in mdb_page_touch (mc=3D0x7f3e1d6bc1d0) at
>> ./../../../libraries/liblmdb/mdb.c:1911
>> #5 0x000000000049eb3a in mdb_page_search (mc=3D0x7f3e1d6bc1d0, key=3D0x=
0,
>> flags=3D5)
>> at ./../../../libraries/liblmdb/mdb.c:4830
>> #6 0x00000000004a7404 in mdb_freelist_save (txn=3D0x7f3e10115130) at
>> ./../../../libraries/liblmdb/mdb.c:2522
>> #7 mdb_txn_commit (txn=3D0x7f3e10115130) at
>> ./../../../libraries/liblmdb/mdb.c:2972
>> #8 0x00000000004a9e4e in mdb_modify (op=3D0x7f3e100028f0,
>> rs=3D0x7f3e1d6bd910) at
>> modify.c:664
>> #9 0x000000000043b4cb in fe_op_modify (op=3D0x7f3e100028f0,
>> rs=3D0x7f3e1d6bd910) at
>> modify.c:303
>> #10 0x000000000043bdf6 in do_modify (op=3D0x7f3e100028f0,
>> rs=3D0x7f3e1d6bd910) at
>> modify.c:177
>> #11 0x0000000000423c99 in connection_operation (ctx=3D0x7f3e1d6bda70,
>> arg_v=3D0x7f3e100028f0) at connection.c:1155
>> #12 0x0000000000424475 in connection_read_thread (ctx=3D0x7f3e1d6bda70,
>> argv=3D<value optimized out>) at connection.c:1291
>> #13 0x00007f3f9b6f04e8 in ldap_int_thread_pool_wrapper (xpool=3D0x16f36d=
0)
>> at
>> tpool.c:688
>> #14 0x0000003b6f2079d1 in start_thread () from /lib64/libpthread.so.0
>> #15 0x0000003b6eee8b6d in clone () from /lib64/libc.so.6
>>
>> Crash can be reproduced by using data.mdb file on any host with the same
>> slapd
>> binaries.
>> Also note contextCSN of DB became 'old' - it was actual on May 22 for bo=
th
>> instances, when reset occured, but after start context CSN shows May 21
>> and 20.
>>
>> Also mdb_stat utility crashes on attempt to show FreeDB data:
>> $ mdb_stat -f ./data
>> Freelist Status
>> Tree depth: 3
>> Branch pages: 7
>> Leaf pages: 817
>> Overflow pages: 0
>> Entries: 16995
>> Segmentation fault (core dumped)
>>
>>
>>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--
WBR,
Vadim Prozorov
--047d7b5d285813733e04fa73805f
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><br><br><div class=3D"gmail_quote"><div dir=3D"ltr"><div><=
div class=3D"h5"><div class=3D"gmail_quote"><div dir=3D"ltr"><div class=3D"=
gmail_extra"><div class=3D"gmail_quote">2014-05-28 13:47 GMT+04:00 Howard C=
hu <span dir=3D"ltr"><<a href=3D"mailto:hyc@symas.com"; target=3D"_blank"=
>hyc@symas.com</a>></span>:<div>
<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:sol=
id;padding-left:1ex">
<a href=3D"mailto:vadius@vadius.ru"; target=3D"_blank">vadius@vadius.ru</a> =
wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex">
Full_Name: Vadim Prozorov<br>
Version: 2.4.38<br>
OS: RHEL 6.3<br>
URL: <a href=3D"ftp://ftp.openldap.org/incoming/"; target=3D"_blank">ftp://f=
tp.openldap.org/<u></u>incoming/</a><br>
Submission from: (NULL) (91.210.4.1)<br>
<br>
<br>
Hello,<br>
<br>
I'm using slapd (2.4.38 release) built from sources on RHEL 6.3 64bit.<=
br>
Build params:<br>
LD_LIBRARY_PATH=3D"/usr/lib:/<u></u>usr/local/lib:/opt/local/<u></u>Be=
rkeleyDB.5.3/lib"<br>
LDFLAGS=3D"-L/usr/local/lib -L/opt/local/BerkeleyDB.5.3/<u></u>lib&quo=
t;<br>
CPPFLAGS=3D"-I/usr/local/include -I/opt/local/BerkeleyDB.5.3/<u></u>in=
clude" ./configure<br>
--prefix=3D/opt/local/OpenLDAP --enable-dynacl --enable-ldap =E2=80=94enabl=
e-overlays<br>
--disable-static --enable-dynamic<br>
<br>
slapd works in master-master sync mode on host with big amount of RAM (whol=
e DB<br>
can fit into RAM).<br>
One time host was hardly restarted. After restart slapd has started, but cr=
ash<br>
on any modification (at least, from ext application; may be form replica,<b=
r>
haven't check yet). There is always the same crash stack:<br>
</blockquote>
<br>
Did you have dbnosync configured on this backend? Looks like garbage data g=
ot written to the freeDB.<br></blockquote><div><br></div></div><div>No, DB =
config section is:</div><div># define MDB parameters =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</div>
<div>backend<span style=3D"white-space:pre-wrap"> </span>mdb</div><div>dat=
abase<span style=3D"white-space:pre-wrap"> </span>mdb</div><div>max=
size<span style=3D"white-space:pre-wrap"> 20</span>000000000</div>
<div>envflags<span style=3D"white-space:pre-wrap"> </span>writemap mapasync=
</div><div># end define</div><div><div><div>=C2=A0</div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo=
rder-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex">
(gdb) bt<br>
#0 =C2=A0mdb_xcursor_init1 (mc=3D0x7f3e1d6bbd40, node=3D0x7f3f99f46f82) at<=
br>
./../../../libraries/liblmdb/<u></u>mdb.c:6585<br>
#1 =C2=A00x000000000049f8eb in mdb_cursor_set (mc=3D0x7f3e1d6bbd40, key=3D0=
x7f3e1d6bbef0,<br>
data=3D0x0, op=3DMDB_SET_RANGE, exactp=3D<value optimized out>) at<br=
>
./../../../libraries/liblmdb/<u></u>mdb.c:5329<br>
#2 =C2=A00x00000000004a019f in mdb_cursor_get (mc=3D0x7f3e1d6bbd40, key=3D0=
x7f3e1d6bbef0,<br>
data=3D0x0, op=3D<value optimized out>) at ./../../../libraries/liblm=
db/<u></u>mdb.c:5529<br>
#3 =C2=A00x000000000049e0ed in mdb_page_alloc (mc=3D<value optimized out=
>, num=3D1,<br>
mp=3D0x7f3e1d6bbf68) at ./../../../libraries/liblmdb/<u></u>mdb.c:1727<br>
#4 =C2=A00x000000000049e469 in mdb_page_touch (mc=3D0x7f3e1d6bc1d0) at<br>
./../../../libraries/liblmdb/<u></u>mdb.c:1911<br>
#5 =C2=A00x000000000049eb3a in mdb_page_search (mc=3D0x7f3e1d6bc1d0, key=3D=
0x0, flags=3D5)<br>
at ./../../../libraries/liblmdb/<u></u>mdb.c:4830<br>
#6 =C2=A00x00000000004a7404 in mdb_freelist_save (txn=3D0x7f3e10115130) at<=
br>
./../../../libraries/liblmdb/<u></u>mdb.c:2522<br>
#7 =C2=A0mdb_txn_commit (txn=3D0x7f3e10115130) at<br>
./../../../libraries/liblmdb/<u></u>mdb.c:2972<br>
#8 =C2=A00x00000000004a9e4e in mdb_modify (op=3D0x7f3e100028f0, rs=3D0x7f3e=
1d6bd910) at<br>
modify.c:664<br>
#9 =C2=A00x000000000043b4cb in fe_op_modify (op=3D0x7f3e100028f0, rs=3D0x7f=
3e1d6bd910) at<br>
modify.c:303<br>
#10 0x000000000043bdf6 in do_modify (op=3D0x7f3e100028f0, rs=3D0x7f3e1d6bd9=
10) at<br>
modify.c:177<br>
#11 0x0000000000423c99 in connection_operation (ctx=3D0x7f3e1d6bda70,<br>
arg_v=3D0x7f3e100028f0) at connection.c:1155<br>
#12 0x0000000000424475 in connection_read_thread (ctx=3D0x7f3e1d6bda70,<br>
argv=3D<value optimized out>) at connection.c:1291<br>
#13 0x00007f3f9b6f04e8 in ldap_int_thread_pool_wrapper (xpool=3D0x16f36d0) =
at<br>
tpool.c:688<br>
#14 0x0000003b6f2079d1 in start_thread () from /lib64/libpthread.so.0<br>
#15 0x0000003b6eee8b6d in clone () from /lib64/libc.so.6<br>
<br>
Crash can be reproduced by using data.mdb file on any host with the same sl=
apd<br>
binaries.<br>
Also note contextCSN of DB became 'old' - it was actual on May 22 f=
or both<br>
instances, when reset occured, but after start context CSN shows May 21 and=
20.<br>
<br>
Also mdb_stat utility crashes on attempt to show FreeDB data:<br>
$ mdb_stat -f ./data<br>
Freelist Status<br>
=C2=A0 =C2=A0Tree depth: 3<br>
=C2=A0 =C2=A0Branch pages: 7<br>
=C2=A0 =C2=A0Leaf pages: 817<br>
=C2=A0 =C2=A0Overflow pages: 0<br>
=C2=A0 =C2=A0Entries: 16995<br>
Segmentation fault (core dumped)<br>
<br>
<br><span><font color=3D"#888888">
</font></span></blockquote><span><font color=3D"#888888">
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:=
//www.symas.com" target=3D"_blank">http://www.symas.com</a><br>
=C2=A0 Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"http://highlandsun.c=
om/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a><br>
=C2=A0 Chief Architect, OpenLDAP =C2=A0<a href=3D"http://www.openldap.org/p=
roject/" target=3D"_blank">http://www.openldap.org/<u></u>project/</a><br>
</font></span></blockquote></div></div></div><span><font color=3D"#888888">=
<br></font></span></div></div></div>--</div></div><div>WBR,</div><div>Vadim=
Prozorov</div></div></div>
</div>
--047d7b5d285813733e04fa73805f--