[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#5469) Assertion failure causes slapd crash

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#5469) Assertion failure causes slapd crash
From: jwm@horde.net
Date: Tue, 22 Apr 2008 19:48:13 GMT
On Mon, Apr 21, 2008 at 09:54:31AM -0700, Quanah Gibson-Mount wrote:
> --On April 17, 2008 8:11:11 PM +0000 jwm@horde.net wrote:
> >On Tue, Apr 15, 2008 at 01:20:58PM -0700, Howard Chu wrote:
> > > Howard Chu wrote:
> > > Please try the patch in slapd/connection.c 1.416 and report your
> > > results,  thanks.
> >
> > I patched our local slapd yesterday; I'll let you know in a couple of
> > weeks, once we can be certain that slapd is no longer crashing.
> 
> I'm working on putting together OpenLDAP 2.3.42 release, so I was wondering 
> if you've seen any ill effects from this patch.

After being up for about a day, our NOC reported that our patched slapd
wasn't responding and restarted it. I'm not sure if it really *was* wedged
or if they were just confused or trigger-happy, since they didn't do a whole
lot of troubleshooting first. The backtrace shows that slapd got a SIGTERM,
and that every thread was waiting except for one in epoll_wait(); I'm not
sure if that's normal state for an idle slapd.

Other than that questionable problem, it's been fine.

john

Program received signal SIGTERM, Terminated.
[Switching to Thread -1212565280 (LWP 26178)]
0xb7ce9183 in pthread_join () from /lib/tls/libpthread.so.0

Thread 14 (Thread 2141186992 (LWP 5848)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x9562048, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0xf5, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x7dbfa008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x42, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2141186992
        i = 697
        keyslot = 697
        hash = 697
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 13 (Thread 2145381296 (LWP 5847)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x9a428f0, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0xf2, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x7e3fb008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x43, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2145381296
        i = 761
        keyslot = 761
        hash = 761
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 12 (Thread -2144339024 (LWP 5846)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x8f497f8, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0xf1, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x7ebfc008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x41, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2150628272
        i = 602
        keyslot = 602
        hash = 602
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 11 (Thread -2125464656 (LWP 5356)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0xcc7c050, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0xed, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x808ff008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x40, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2169502640
        i = 635
        keyslot = 635
        hash = 635
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 10 (Thread -1967129680 (LWP 26656)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x9401ac8, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8238260, ltk_data = 0x3d, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x8247af0, ltk_data = 0xbe, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x89efe008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2327837616
        i = 756
        keyslot = 756
        hash = 756
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 9 (Thread -1936733264 (LWP 26210)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x844f890, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0x83, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8b3fa008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x3c, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2358234032
        i = 662
        keyslot = 662
        hash = 662
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 8 (Thread -1931486288 (LWP 26209)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x846cae8, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8238260, ltk_data = 0x3a, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8bcfc008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8247af0, ltk_data = 0x82, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2363481008
        i = 758
        keyslot = 758
        hash = 758
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 7 (Thread -1917846608 (LWP 26208)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x82606a8, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0x7c, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8cdfe008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x3b, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2377120688
        i = 743
        keyslot = 743
        hash = 743
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 6 (Thread -1884959824 (LWP 26205)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x825cf38, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0x4a, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8e459008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x34, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2410007472
        i = 687
        keyslot = 687
        hash = 687
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 5 (Thread -1879712848 (LWP 26204)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x825c578, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8247af0, ltk_data = 0x51, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8dc58008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8238260, ltk_data = 0x38, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2415254448
        i = 783
        keyslot = 783
        hash = 783
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 4 (Thread -1875518544 (LWP 26203)):
#0  0xb7ceac01 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f590a0 in ldap_pvt_thread_cond_wait (cond=0x816f374, mutex=0x816f35c) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:299
No locals.
#2  0xb7f5806a in ldap_int_thread_pool_wrapper (xpool=0x816f358) at /var/jwm/o2/openldap/libraries/libldap_r/tpool.c:466
        pool = (struct ldap_int_thread_pool_s *) 0x816f358
        ctx = (ldap_int_thread_ctx_t *) 0x0
        ltc_key = {{ltk_key = 0x80cb11d, ltk_data = 0x8259f38, ltk_free = 0x80caf58 <slap_sl_mem_destroy>}, {ltk_key = 0x8238260, ltk_data = 0x1c, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0xb7a06e55, ltk_data = 0x8ee5c008, ltk_free = 0xb7a06e2a <search_stack_free>}, {ltk_key = 0x8247af0, ltk_data = 0x7d, ltk_free = 0xb7a1564d <bdb_locker_id_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 28 times>}
        tid = 2419448752
        i = 592
        keyslot = 592
        hash = 592
#3  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#4  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 3 (Thread -1538761808 (LWP 26202)):
#0  0xb7c7d679 in epoll_wait () from /lib/tls/libc.so.6
No symbol table info available.
#1  0x0806f7fa in slapd_daemon_task (ptr=0x0) at /var/jwm/o2/openldap/servers/slapd/daemon.c:2174
        ns = 1
        at = 0
        nfds = 12
        revents = (struct epoll_event *) 0xb7b37008
        tvp = (struct timeval *) 0xa44853a4
        cat = {tv_sec = 1208360230, tv_usec = 0}
        i = 0
        nwriters = 0
        now = 1208360042
        tv = {tv_sec = 188, tv_usec = 0}
        tdelta = 1
        rtask = (struct re_s *) 0x81bea68
        l = 4
        last_idle_check = 0
        idle = {tv_sec = 0, tv_usec = 0}
        ebadf = 0
#2  0xb7ce80bd in start_thread () from /lib/tls/libpthread.so.0
No symbol table info available.
#3  0xb7c7d01e in clone () from /lib/tls/libc.so.6
No symbol table info available.

Thread 1 (Thread -1212565280 (LWP 26178)):
#0  0xb7ce9183 in pthread_join () from /lib/tls/libpthread.so.0
No symbol table info available.
#1  0xb7f58f9b in ldap_pvt_thread_join (thread=2756205488, thread_return=0x0) at /var/jwm/o2/openldap/libraries/libldap_r/thr_posix.c:193
No locals.
#2  0x08070524 in slapd_daemon () at /var/jwm/o2/openldap/servers/slapd/daemon.c:2579
        listener_tid = 2756205488
        rc = 0
#3  0x0805a58d in main (argc=9, argv=0xbff9b084) at /var/jwm/o2/openldap/servers/slapd/main.c:859
        i = 9
        no_detach = 0
        rc = 0
        urls = 0x8162008 "ldap:/// ldaps:///"
        username = 0x8162030 "gidNumber"
        groupname = 0x8162020 "\202.Í·°K\235·"
        sandbox = 0x0
        syslogUser = 160
        g_argc = 9
        g_argv = (char **) 0xbff9b084
        configfile = 0x8162040 "/etc/ldap/slapd.conf.master"
        configdir = 0x0
        serverName = 0xbff9be9b "slapd"
        serverMode = 1
        scp = (struct sync_cookie *) 0x0
        scp_entry = (struct sync_cookie *) 0x0
        debug_unknowns = (char **) 0x0
        syslog_unknowns = (char **) 0x0
        serverNamePrefix = 0x81112bb ""
        l = 3086212344
        slapd_pid_file_unlink = 1
        slapd_args_file_unlink = 1
        __PRETTY_FUNCTION__ = "main"

-- 
John Morrissey          _o            /\         ----  __o
jwm@horde.net        _-< \_          /  \       ----  <  \,
www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__
Prev by Date: (ITS#5484) Remove runqueue task when freeing syncops bugfix
Next by Date: Re: (ITS#5483) "syncprov_qplay: failed to get [dn] ..." Bad?
Index(es):
- Chronological
- Thread