Re: (ITS#7862) slapd crash on modifications after host hard reset (in mdb)

[hyc@symas.com]

vadius@vadius.ru wrote:
> Full_Name: Vadim Prozorov
> Version: 2.4.38
> OS: RHEL 6.3
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (91.210.4.1)
>
>
> Hello,
>
> I'm using slapd (2.4.38 release) built from sources on RHEL 6.3 64bit.
> Build params:
> LD_LIBRARY_PATH="/usr/lib:/usr/local/lib:/opt/local/BerkeleyDB.5.3/lib"
> LDFLAGS="-L/usr/local/lib -L/opt/local/BerkeleyDB.5.3/lib"
> CPPFLAGS="-I/usr/local/include -I/opt/local/BerkeleyDB.5.3/include" ./configure
> --prefix=/opt/local/OpenLDAP --enable-dynacl --enable-ldap ?enable-overlays
> --disable-static --enable-dynamic
>
> slapd works in master-master sync mode on host with big amount of RAM (whole DB
> can fit into RAM).
> One time host was hardly restarted. After restart slapd has started, but crash
> on any modification (at least, from ext application; may be form replica,
> haven't check yet). There is always the same crash stack:

Did you have dbnosync configured on this backend? Looks like garbage data got 
written to the freeDB.

> (gdb) bt
> #0  mdb_xcursor_init1 (mc=0x7f3e1d6bbd40, node=0x7f3f99f46f82) at
> ./../../../libraries/liblmdb/mdb.c:6585
> #1  0x000000000049f8eb in mdb_cursor_set (mc=0x7f3e1d6bbd40, key=0x7f3e1d6bbef0,
> data=0x0, op=MDB_SET_RANGE, exactp=<value optimized out>) at
> ./../../../libraries/liblmdb/mdb.c:5329
> #2  0x00000000004a019f in mdb_cursor_get (mc=0x7f3e1d6bbd40, key=0x7f3e1d6bbef0,
> data=0x0, op=<value optimized out>) at ./../../../libraries/liblmdb/mdb.c:5529
> #3  0x000000000049e0ed in mdb_page_alloc (mc=<value optimized out>, num=1,
> mp=0x7f3e1d6bbf68) at ./../../../libraries/liblmdb/mdb.c:1727
> #4  0x000000000049e469 in mdb_page_touch (mc=0x7f3e1d6bc1d0) at
> ./../../../libraries/liblmdb/mdb.c:1911
> #5  0x000000000049eb3a in mdb_page_search (mc=0x7f3e1d6bc1d0, key=0x0, flags=5)
> at ./../../../libraries/liblmdb/mdb.c:4830
> #6  0x00000000004a7404 in mdb_freelist_save (txn=0x7f3e10115130) at
> ./../../../libraries/liblmdb/mdb.c:2522
> #7  mdb_txn_commit (txn=0x7f3e10115130) at
> ./../../../libraries/liblmdb/mdb.c:2972
> #8  0x00000000004a9e4e in mdb_modify (op=0x7f3e100028f0, rs=0x7f3e1d6bd910) at
> modify.c:664
> #9  0x000000000043b4cb in fe_op_modify (op=0x7f3e100028f0, rs=0x7f3e1d6bd910) at
> modify.c:303
> #10 0x000000000043bdf6 in do_modify (op=0x7f3e100028f0, rs=0x7f3e1d6bd910) at
> modify.c:177
> #11 0x0000000000423c99 in connection_operation (ctx=0x7f3e1d6bda70,
> arg_v=0x7f3e100028f0) at connection.c:1155
> #12 0x0000000000424475 in connection_read_thread (ctx=0x7f3e1d6bda70,
> argv=<value optimized out>) at connection.c:1291
> #13 0x00007f3f9b6f04e8 in ldap_int_thread_pool_wrapper (xpool=0x16f36d0) at
> tpool.c:688
> #14 0x0000003b6f2079d1 in start_thread () from /lib64/libpthread.so.0
> #15 0x0000003b6eee8b6d in clone () from /lib64/libc.so.6
>
> Crash can be reproduced by using data.mdb file on any host with the same slapd
> binaries.
> Also note contextCSN of DB became 'old' - it was actual on May 22 for both
> instances, when reset occured, but after start context CSN shows May 21 and 20.
>
> Also mdb_stat utility crashes on attempt to show FreeDB data:
> $ mdb_stat -f ./data
> Freelist Status
>    Tree depth: 3
>    Branch pages: 7
>    Leaf pages: 817
>    Overflow pages: 0
>    Entries: 16995
> Segmentation fault (core dumped)
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/