[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Server Hangs

To: Quanah Gibson-Mount <quanah@zimbra.com>
Subject: Re: Server Hangs
From: David Engeset <davidke@whidbey.net>
Date: Wed, 17 Aug 2011 10:58:35 -0700
Cc: openldap-technical@openldap.org
Dkim-signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=whidbey.net; s=whidbey01; h=Content-Type:In-Reply-To:References:Subject:CC:To:MIME-Version:From:Date:Message-ID; bh=JDVeMjzzcbUMXR46Xop+dhcDg/hJx6+Tex8X4x0Jaug=; b=AM39gcP7ooW6qZbGvRiWmXGm04c6F0dJ1ZhW1sE613Ho7nW34q6Zy/o+6VZpJ6G2eY6m601fc70yfmlEyyy+j/ivsYwfRibglfd+eEKYYb/sWhOFG+osk9Sfce0TcO5416j/2IVypwKTrAY/Dpa6lQG6+xFyShFPU0ML0T+i1ro=;
In-reply-to: <355B6731854B7E5A6CEA7A84@[192.168.1.2]>
References: <4E45B0A7.4020903@whidbey.net> <12063A463C6341AFE0EAAD30@[192.168.1.2]> <4E4AF48C.1040801@whidbey.net> <355B6731854B7E5A6CEA7A84@[192.168.1.2]>
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0) Gecko/20110624 Thunderbird/5.0

On 8/16/11 4:32 PM, Quanah Gibson-Mount wrote:

--On Tuesday, August 16, 2011 3:51 PM -0700 David Engeset<davidke@whidbey.net> wrote:
     Below is how I generally configure BDB and OpenLDAP for normal
operational use.
     I configured BDB with no parameters for all versions from 4.2 to
5.1, so I ran:

../dist/configure
make&&  make install
For building BDB on *nix systems, you should specify:

--enable-posixmutexes --with-mutex=POSIX/pthreads
     For the debugging I did the following for BDB:

env CFLAGS=-O2 ../dist/configure --enable-debug
You need to use "-g" as I stated previously.
CFLAGS="-g -O2 -fPIC" are the flags I pass to BDB. What we arelooking for is the gcc debugging symbols. The --enable-debug flaglooks like it does some internal debugging stuff in the BDB code whichisn't really what we're asking for.
make&&  make install

     For OpenLDAP:

export LD_LIBRARY_PATH="/usr/local/BerkeleyDB.4.8/lib"
env CPPFLAGS="-I/usr/local/BerkeleyDB.4.8/include" CFLAGS="-g -O0" \
LDFLAGS="-L/usr/local/BerkeleyDB.4.8/lib" ./configure--enable-wrappers \
--enable-crypt --with-cyrus-sasl --with-tls --enable-debug
make depend&&  make&&  make install STRIP=''
This looks good.
Can you please re-generate your stack trace with a correctly build BDBbehind openldap?
Thanks!

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Quanah,

Attached are the two new debug output files and below is what Iused to compile BDB for debug.

env CFPLAGS="-g -O2 -fPIC" ../dist/configure --enable-posixmutexes--with-mutex=POSIX/pthreads

make && make install

Thank you,

--
David
Whidbey Telecom Internet and Broadband
Software Engineer

Default locking region information:
83	Last allocated locker ID
0x7fffffff	Current maximum unused locker ID
9	Number of lock modes
1000	Maximum number of locks possible
1000	Maximum number of lockers possible
1000	Maximum number of lock objects possible
20	Number of lock object partitions
18	Number of current locks
253	Maximum number of locks at any one time
6	Maximum number of locks in any one bucket
0	Maximum number of locks stolen by for an empty partition
0	Maximum number of locks stolen for any one partition
63	Number of current lockers
63	Maximum number of lockers at any one time
14	Number of current lock objects
138	Maximum number of lock objects at any one time
3	Maximum number of lock objects in any one bucket
0	Maximum number of objects stolen by for an empty partition
0	Maximum number of objects stolen for any one partition
1204217	Total number of locks requested
1203962	Total number of locks released
0	Total number of locks upgraded
16	Total number of locks downgraded
234	Lock requests not available due to conflicts, for which we waited
231	Lock requests not available due to conflicts, for which we did not wait
0	Number of deadlocks
0	Lock timeout value
0	Number of locks that have timed out
0	Transaction timeout value
0	Number of transactions that have timed out
536KB	The size of the lock region
34	The number of partition locks that required waiting (0%)
24	The maximum number of times any partition lock was waited for (0%)
0	The number of object queue operations that required waiting (0%)
1	The number of locker allocations that required waiting (0%)
0	The number of region locks that required waiting (0%)
3	Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock	Region type
5	Region ID
__db.005	Region name
0x95731000	Region address
0x957310c4	Region primary address
0	Region maximum allocation
0	Region allocated
Region allocations: 3006 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
  1KB	3002
  2KB	1
  4KB	0
  8KB	0
 16KB	2
 32KB	0
 64KB	0
128KB	1
256KB	0
512KB	0
1024KB	0
REGION_JOIN_OK	Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
65567	Lock region region mutex [0/1042 0% 2531/3078736720]
1031	locker table size
1031	object table size
716	obj_off
96708	locker_off
1	need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker   Mode      Count Status  ----------------- Object ---------------
      25 dd=61 locks held 1    write locks 0    pid/thread 1502/3078571888
      25 READ          1 HELD    id2entry.bdb              handle        0
      26 dd=60 locks held 0    write locks 0    pid/thread 1502/3078571888
      27 dd=59 locks held 1    write locks 0    pid/thread 1502/3078571888
      27 READ          1 HELD    dn2id.bdb                 handle        0
      28 dd=58 locks held 0    write locks 0    pid/thread 1502/3078571888
      29 dd=57 locks held 0    write locks 0    pid/thread 1502/3078571888
      2a dd=54 locks held 0    write locks 0    pid/thread 1502/2483026800
      2b dd=53 locks held 0    write locks 0    pid/thread 1502/2483026800
      2c dd=52 locks held 0    write locks 0    pid/thread 1502/2483026800
      2d dd=51 locks held 1    write locks 0    pid/thread 1502/2483026800
      2d READ          1 HELD    objectClass.bdb           handle        0
      2e dd=50 locks held 0    write locks 0    pid/thread 1502/2483026800
      2f dd=49 locks held 0    write locks 0    pid/thread 1502/2483026800
      30 dd=48 locks held 1    write locks 0    pid/thread 1502/2483026800
      30 READ          1 HELD    uid.bdb                   handle        0
      31 dd=47 locks held 0    write locks 0    pid/thread 1502/2483026800
      32 dd=46 locks held 0    write locks 0    pid/thread 1502/2483026800
      33 dd=43 locks held 0    write locks 0    pid/thread 1502/2451553136
      34 dd=42 locks held 0    write locks 0    pid/thread 1502/2451553136
      35 dd=41 locks held 1    write locks 0    pid/thread 1502/2451553136
      35 READ          1 HELD    cn.bdb                    handle        0
      36 dd=40 locks held 0    write locks 0    pid/thread 1502/2451553136
      37 dd=39 locks held 0    write locks 0    pid/thread 1502/2451553136
      38 dd=38 locks held 0    write locks 0    pid/thread 1502/2451553136
      39 dd=37 locks held 0    write locks 0    pid/thread 1502/2451553136
      3a dd=36 locks held 1    write locks 0    pid/thread 1502/2451553136
      3a READ          1 HELD    sn.bdb                    handle        0
      3b dd=35 locks held 0    write locks 0    pid/thread 1502/2451553136
      3c dd=34 locks held 0    write locks 0    pid/thread 1502/2451553136
      3d dd=33 locks held 0    write locks 0    pid/thread 1502/2451553136
      3e dd=32 locks held 0    write locks 0    pid/thread 1502/2451553136
      3f dd=31 locks held 1    write locks 0    pid/thread 1502/2451553136
      3f READ          1 HELD    gidNumber.bdb             handle        0
      40 dd=30 locks held 0    write locks 0    pid/thread 1502/2451553136
      41 dd=29 locks held 0    write locks 0    pid/thread 1502/2451553136
      42 dd=28 locks held 1    write locks 0    pid/thread 1502/2451553136
      42 READ          1 HELD    uidNumber.bdb             handle        0
      43 dd=27 locks held 0    write locks 0    pid/thread 1502/2451553136
      44 dd=26 locks held 0    write locks 0    pid/thread 1502/2451553136
      45 dd=25 locks held 0    write locks 0    pid/thread 1502/2451553136
      46 dd=24 locks held 0    write locks 0    pid/thread 1502/2451553136
      47 dd=23 locks held 0    write locks 0    pid/thread 1502/2451553136
      48 dd=22 locks held 0    write locks 0    pid/thread 1502/2451553136
      49 dd=21 locks held 0    write locks 0    pid/thread 1502/2476735344
      4a dd=18 locks held 0    write locks 0    pid/thread 1502/2483026800
      4b dd=16 locks held 0    write locks 0    pid/thread 1502/2433710960
      4c dd=15 locks held 0    write locks 0    pid/thread 1502/2433710960
      4d dd=14 locks held 0    write locks 0    pid/thread 1502/2437909360
      4e dd=13 locks held 0    write locks 0    pid/thread 1502/2437909360
      4f dd=12 locks held 0    write locks 0    pid/thread 1502/2437909360
      50 dd=11 locks held 0    write locks 0    pid/thread 1502/2410621808
      51 dd= 4 locks held 0    write locks 0    pid/thread 1502/2386738032
      52 dd= 2 locks held 0    write locks 0    pid/thread 1502/2347895664
      53 dd= 1 locks held 0    write locks 0    pid/thread 1502/2347895664
80005423 dd=10 locks held 0    write locks 0    pid/thread 1502/2396187504
80005426 dd= 9 locks held 0    write locks 0    pid/thread 1502/2386738032
80005427 dd= 8 locks held 0    write locks 0    pid/thread 1502/2391989104
8000542a dd= 7 locks held 0    write locks 0    pid/thread 1502/2373094256
8000542d dd= 6 locks held 1    write locks 1    pid/thread 1502/2366790512
8000542d WRITE         1 HELD    dn2id.bdb                 page          4
80005434 dd= 5 locks held 0    write locks 0    pid/thread 1502/2361539440
80005441 dd= 3 locks held 0    write locks 0    pid/thread 1502/2347895664
80004e4a dd=56 locks held 0    write locks 0    pid/thread 1502/3078571888
80004e4b dd=55 locks held 0    write locks 0    pid/thread 1502/2483026800
80004e4e dd=45 locks held 0    write locks 0    pid/thread 1502/2476735344
80004e4f dd=44 locks held 0    write locks 0    pid/thread 1502/2451553136
80005666 dd= 0 locks held 2    write locks 0    pid/thread 1502/2483026800
80005666 READ          1 HELD    0x2c1bc len:   5 data: 0x0300000000
80005666 READ          1 HELD    0x29f4c len:   5 data: 0x020x52000000
80005667 dd= 0 locks held 5    write locks 3    pid/thread 1502/2483026800
80005667 READ          1 WAIT    dn2id.bdb                 page          4
80005667 WRITE         2 HELD    dn2id.bdb                 page          2
80005667 READ          2 HELD    dn2id.bdb                 page          2
80005667 WRITE         2 HELD    dn2id.bdb                 page       1300
80005667 READ          1 HELD    dn2id.bdb                 page       1300
80005667 WRITE         4 HELD    dn2id.bdb                 page       1301
80004e6c dd=20 locks held 0    write locks 0    pid/thread 1502/2433710960
80004e6f dd=19 locks held 0    write locks 0    pid/thread 1502/2437909360
80004e6f READ          1 WAIT    dn2id.bdb                 page          4
80004eb2 dd=17 locks held 0    write locks 0    pid/thread 1502/2410621808
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker   Mode      Count Status  ----------------- Object ---------------
80005666 READ          1 HELD    0x2c1bc len:   5 data: 0x0300000000

      2d READ          1 HELD    objectClass.bdb           handle        0

80005667 WRITE         4 HELD    dn2id.bdb                 page       1301

80005667 READ          1 HELD    dn2id.bdb                 page       1300
80005667 WRITE         2 HELD    dn2id.bdb                 page       1300

      25 READ          1 HELD    id2entry.bdb              handle        0

      30 READ          1 HELD    uid.bdb                   handle        0

80005666 READ          1 HELD    0x29f4c len:   5 data: 0x020x52000000

      3f READ          1 HELD    gidNumber.bdb             handle        0

      42 READ          1 HELD    uidNumber.bdb             handle        0

      35 READ          1 HELD    cn.bdb                    handle        0

8000542d WRITE         1 HELD    dn2id.bdb                 page          4
80005667 READ          1 WAIT    dn2id.bdb                 page          4
80004e6f READ          1 WAIT    dn2id.bdb                 page          4

80005667 READ          2 HELD    dn2id.bdb                 page          2
80005667 WRITE         2 HELD    dn2id.bdb                 page          2

      27 READ          1 HELD    dn2id.bdb                 handle        0

      3a READ          1 HELD    sn.bdb                    handle        0

GNU gdb (GDB) Fedora (7.3.50.20110722-1.fc16)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) attach 1502
Attaching to process 1502
Reading symbols from /usr/local/libexec/slapd...done.
Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so...done.
Loaded symbols for /usr/local/BerkeleyDB.4.8/lib/libdb-4.8.so
Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
[New Thread 0x8bf20b70 (LWP 2463)]
[New Thread 0x8cc23b70 (LWP 2458)]
[New Thread 0x8d125b70 (LWP 2457)]
[New Thread 0x8d728b70 (LWP 2455)]
[New Thread 0x8e42bb70 (LWP 2454)]
[New Thread 0x8e92db70 (LWP 2452)]
[New Thread 0x8ed2eb70 (LWP 2451)]
[New Thread 0x8faf2b70 (LWP 2316)]
[New Thread 0x910f7b70 (LWP 2307)]
[New Thread 0x914f8b70 (LWP 2306)]
[New Thread 0x921fbb70 (LWP 1880)]
[New Thread 0x939ffb70 (LWP 1543)]
[New Thread 0x93fffb70 (LWP 1506)]
[New Thread 0x9455db70 (LWP 1505)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl.so.10
Reading symbols from /lib/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypto.so.10
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libwrap.so.0
Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libgssapi_krb5.so.2
Reading symbols from /lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libk5crypto.so.3
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libfreebl3.so
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /lib/libnss_myhostname.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libnss_myhostname.so.2
Reading symbols from /usr/lib/sasl2/libcrammd5.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libcrammd5.so
Reading symbols from /usr/lib/sasl2/libanonymous.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libanonymous.so
Reading symbols from /usr/lib/sasl2/libgssapiv2.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libgssapiv2.so
Reading symbols from /usr/lib/sasl2/libdigestmd5.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libdigestmd5.so
Reading symbols from /usr/lib/sasl2/liblogin.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/liblogin.so
Reading symbols from /usr/lib/sasl2/libsasldb.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libsasldb.so
Reading symbols from /lib/libdb-5.2.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libdb-5.2.so
Reading symbols from /usr/lib/sasl2/libplain.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/sasl2/libplain.so
0x00d3a424 in __kernel_vsyscall ()
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gssapi-2.1.23-25.fc16.i686 cyrus-sasl-lib-2.1.23-25.fc16.i686 cyrus-sasl-md5-2.1.23-25.fc16.i686 cyrus-sasl-plain-2.1.23-25.fc16.i686 glibc-2.14.90-4.i686 keyutils-libs-1.5.2-1.fc16.i686 krb5-libs-1.9.1-9.fc17.i686 libcom_err-1.41.14-2.fc15.i686 libdb-5.2.28-2.fc16.i686 libselinux-2.1.0-1.fc17.i686 libuuid-2.20-0.1.fc17.i686 nss-myhostname-0.3-1.fc16.i686 nss-softokn-freebl-3.12.10-4.fc16.i686 openssl-1.0.0d-7.fc16.i686 tcp_wrappers-libs-7.6-65.fc17.i686 zlib-1.2.5-4.fc16.i686
(gdb) bt full
#0  0x00d3a424 in __kernel_vsyscall ()
No symbol table info available.
#1  0x477a8d95 in pthread_join () from /lib/libpthread.so.0
No symbol table info available.
#2  0x0816e38b in ldap_pvt_thread_join (thread=2488654704, thread_return=0x0) at thr_posix.c:197
No locals.
#3  0x0806b7cf in slapd_daemon () at daemon.c:2922
        i = 0
        rc = 0
        listener_tid = 0x946a570
#4  0x0804e603 in main (argc=3, argv=0xbfe18d64) at main.c:983
        i = 3
        no_detach = 0
        rc = -12
        urls = 0x93d3008 "ldap://";
        username = 0x0
        groupname = 0x0
        sandbox = 0x0
        syslogUser = 160
        g_argc = 3
        g_argv = 0xbfe18d64
        configfile = 0x0
        configdir = 0x0
        serverName = 0xbfe19f47 "slapd"
        serverMode = 1
        scp = 0x0
        scp_entry = 0x0
        debug_unknowns = 0x0
        syslog_unknowns = 0x0
        serverNamePrefix = 0x81a20c4 ""
        l = 1221099641
        slapd_pid_file_unlink = 1
        slapd_args_file_unlink = 1
        firstopt = 0
        __PRETTY_FUNCTION__ = "main"
(gdb) thread apply all bt

Thread 15 (Thread 0x9455db70 (LWP 1505)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x476e7156 in epoll_wait () from /lib/libc.so.6
#2  0x0806a849 in slapd_daemon_task (ptr=0x0) at daemon.c:2528
#3  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#4  0x476e674e in clone () from /lib/libc.so.6

Thread 14 (Thread 0x93fffb70 (LWP 1506)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0062e993 in __db_pthread_mutex_lock (env=0x94684d0, mutex=65677)
    at ../dist/../mutex/mut_pthread.c:318
#3  0x006c9158 in __lock_get_internal (lt=0x94687b8, sh_locker=0x94fb6044, flags=0, obj=0x94a1e70, 
    lock_mode=DB_LOCK_READ, timeout=0, lock=0x93ffeab0) at ../dist/../lock/lock.c:953
#4  0x006c9737 in __lock_get (env=0x94684d0, locker=0x94fb6044, flags=0, obj=0x94a1e70, 
    lock_mode=DB_LOCK_READ, lock=0x93ffeab0) at ../dist/../lock/lock.c:459
#5  0x006f78bf in __db_lget (dbc=0x94a1de8, action=0, pgno=4, mode=DB_LOCK_READ, lkflags=0, 
    lockp=0x93ffeab0) at ../dist/../db/db_meta.c:1203
#6  0x0064e514 in __bam_search (dbc=0x94a1de8, root_pgno=1, key=0x93ffede4, flags=1409, slevel=1, 
    recnop=0x0, exactp=0x93ffebc4) at ../dist/../btree/bt_search.c:702
#7  0x006376ee in __bamc_search (dbc=0x94a1de8, root_pgno=0, key=0x93ffede4, flags=27, 
    exactp=0x93ffebc4) at ../dist/../btree/bt_cursor.c:2785
#8  0x006398fc in __bamc_get (dbc=0x94a1de8, key=0x93ffede4, data=0x93ffed48, flags=27, 
    pgnop=0x93ffec5c) at ../dist/../btree/bt_cursor.c:1088
#9  0x006e78fd in __dbc_iget (dbc=0x93457fc0, key=0x93ffede4, data=0x93ffed48, flags=27)
    at ../dist/../db/db_cam.c:934
#10 0x006e82c5 in __dbc_get (dbc=0x93457fc0, key=0x93ffede4, data=0x93ffed48, flags=27)
    at ../dist/../db/db_cam.c:755
#11 0x006f38d0 in __dbc_get_pp (dbc=0x93457fc0, key=0x93ffede4, data=0x93ffed48, flags=27)
    at ../dist/../db/db_iface.c:2201
#12 0x08133ba3 in bdb_idl_delete_key (be=0x941b608, db=0x94697b0, tid=0x8b11b7c8, key=0x93ffede4, 
    id=20994) at idl.c:949
#13 0x0812c86e in bdb_dn2id_delete (op=0x8b10c7f8, txn=0x8b11b7c8, eip=0x93a026c8, e=0x94e98b1c)
    at dn2id.c:222
#14 0x0812b1f2 in bdb_delete (op=0x8b10c7f8, rs=0x93fff064) at delete.c:402
#15 0x0808d57f in fe_op_delete (op=0x8b10c7f8, rs=0x93fff064) at delete.c:174
#16 0x0808d2dd in do_delete (op=0x8b10c7f8, rs=0x93fff064) at delete.c:95
#17 0x0806e606 in connection_operation (ctx=0x93fff138, arg_v=0x8b10c7f8) at connection.c:1138
#18 0x0806eb27 in connection_read_thread (ctx=0x93fff138, argv=0x15) at connection.c:1274
#19 0x0816d399 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:685
#20 0x477a7c8e in start_thread () from /lib/libpthread.so.0
#21 0x476e674e in clone () from /lib/libc.so.6

Thread 13 (Thread 0x939ffb70 (LWP 1543)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 12 (Thread 0x921fbb70 (LWP 1880)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 11 (Thread 0x914f8b70 (LWP 2306)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0062e993 in __db_pthread_mutex_lock (env=0x94684d0, mutex=65776)
    at ../dist/../mutex/mut_pthread.c:318
#3  0x006c9158 in __lock_get_internal (lt=0x94687b8, sh_locker=0x94fb755c, flags=0, 
    obj=0x93a02e30, lock_mode=DB_LOCK_READ, timeout=0, lock=0x913f69f0)
    at ../dist/../lock/lock.c:953
#4  0x006c9737 in __lock_get (env=0x94684d0, locker=0x94fb755c, flags=0, obj=0x93a02e30, 
    lock_mode=DB_LOCK_READ, lock=0x913f69f0) at ../dist/../lock/lock.c:459
#5  0x006f78bf in __db_lget (dbc=0x93a02da8, action=0, pgno=4, mode=DB_LOCK_READ, lkflags=0, 
    lockp=0x913f69f0) at ../dist/../db/db_meta.c:1203
#6  0x0064e514 in __bam_search (dbc=0x93a02da8, root_pgno=1, key=0x91436d24, flags=1409, slevel=1, 
    recnop=0x0, exactp=0x913f6b04) at ../dist/../btree/bt_search.c:702
#7  0x006376ee in __bamc_search (dbc=0x93a02da8, root_pgno=0, key=0x91436d24, flags=27, 
    exactp=0x913f6b04) at ../dist/../btree/bt_cursor.c:2785
#8  0x006398fc in __bamc_get (dbc=0x93a02da8, key=0x91436d24, data=0x91436c8c, flags=27, 
    pgnop=0x913f6b9c) at ../dist/../btree/bt_cursor.c:1088
#9  0x006e78fd in __dbc_iget (dbc=0x9469ce8, key=0x91436d24, data=0x91436c8c, flags=27)
    at ../dist/../db/db_cam.c:934
#10 0x006e82c5 in __dbc_get (dbc=0x9469ce8, key=0x91436d24, data=0x91436c8c, flags=2075)
    at ../dist/../db/db_cam.c:755
#11 0x006f38d0 in __dbc_get_pp (dbc=0x9469ce8, key=0x91436d24, data=0x91436c8c, flags=2075)
    at ../dist/../db/db_iface.c:2201
#12 0x081329ae in bdb_idl_fetch_key (be=0x941b608, db=0x94697b0, txn=0x934568c8, key=0x91436d24, 
    ids=0x91476f00, saved_cursor=0x0, get_flag=0) at idl.c:605
#13 0x0812d210 in bdb_dn2idl (op=0x8b100b80, txn=0x934568c8, ndn=0x91436ee4, ei=0x93a026c8, 
    ids=0x91476f00, stack=0x8faf3008) at dn2id.c:376
#14 0x08101fe8 in search_candidates (op=0x8b100b80, rs=0x914f8064, e=0x91436ed8, txn=0x934568c8, 
    ids=0x91476f00, scopes=0x91436f00) at search.c:1257
#15 0x08100826 in bdb_search (op=0x8b100b80, rs=0x914f8064) at search.c:616
#16 0x08071f34 in fe_op_search (op=0x8b100b80, rs=0x914f8064) at search.c:402
#17 0x080718a6 in do_search (op=0x8b100b80, rs=0x914f8064) at search.c:247
#18 0x0806e606 in connection_operation (ctx=0x914f8138, arg_v=0x8b100b80) at connection.c:1138
#19 0x0806eb27 in connection_read_thread (ctx=0x914f8138, argv=0xe) at connection.c:1274
#20 0x0816d399 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:685
#21 0x477a7c8e in start_thread () from /lib/libpthread.so.0
#22 0x476e674e in clone () from /lib/libc.so.6

Thread 10 (Thread 0x910f7b70 (LWP 2307)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
---Type <return> to continue, or q <return> to quit---
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 9 (Thread 0x8faf2b70 (LWP 2316)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 8 (Thread 0x8ed2eb70 (LWP 2451)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 7 (Thread 0x8e92db70 (LWP 2452)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 6 (Thread 0x8e42bb70 (LWP 2454)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 5 (Thread 0x8d728b70 (LWP 2455)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 4 (Thread 0x8d125b70 (LWP 2457)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 3 (Thread 0x8cc23b70 (LWP 2458)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 2 (Thread 0x8bf20b70 (LWP 2463)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477ab80c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0816e420 in ldap_pvt_thread_cond_wait (cond=0x93ea454, mutex=0x93ea43c) at thr_posix.c:277
#3  0x0816d311 in ldap_int_thread_pool_wrapper (xpool=0x93ea438) at tpool.c:672
#4  0x477a7c8e in start_thread () from /lib/libpthread.so.0
#5  0x476e674e in clone () from /lib/libc.so.6

Thread 1 (Thread 0xb77f4770 (LWP 1502)):
#0  0x00d3a424 in __kernel_vsyscall ()
#1  0x477a8d95 in pthread_join () from /lib/libpthread.so.0
#2  0x0816e38b in ldap_pvt_thread_join (thread=2488654704, thread_return=0x0) at thr_posix.c:197
#3  0x0806b7cf in slapd_daemon () at daemon.c:2922
#4  0x0804e603 in main (argc=3, argv=0xbfe18d64) at main.c:983
(gdb)

Follow-Ups:
- Re: Server Hangs
  - From: Howard Chu <hyc@symas.com>

References:
- Server Hangs
  - From: David Engeset <davidke@whidbey.net>
- Re: Server Hangs
  - From: Quanah Gibson-Mount <quanah@zimbra.com>
- Re: Server Hangs
  - From: Quanah Gibson-Mount <quanah@zimbra.com>

Prev by Date: Re: migrating contents from openldap-2.1 to openldap-2.3.37
Next by Date: Re: How to design an architecture of OpenLDAP for satisfy our needs?
Index(es):
- Chronological
- Thread