[Date Prev][Date Next] [Chronological] [Thread] [Top]
(ITS#4338) assertion in connection manager

To: openldap-its@OpenLDAP.org
Subject: (ITS#4338) assertion in connection manager
From: bgmilne@staff.telkomsa.net
Date: Fri, 13 Jan 2006 10:41:44 GMT
Full_Name: Buchan Milne
Version: 2.3.17
OS: Linux 2.6 (RHEL 4)
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (196.25.124.11)


With the same setup as ITS 4333, slapd is crashing with an assertion:

slapd: daemon.c:1357: slap_listener: Assertion `((slap_daemon.sd_index[(s)]) ==
-1)' failed.

"Real" connections are run from another host (using approx 10 concurrent
ldapsearch's running), soemthing like:

ldapsearch -D cn=qmail,dc=cybertrade,dc=co,dc=za -w $password -x -h ldapmaster
(&(objectClass=qmailUser)(|(mail=$someaddres)(mailAlternateAddress=$someaddress)))
-l 5 -LLL dn

The contents of connections[39] shows that this connection is from localhost,
and using the dn cn=qmail,dc=cybertrade,dc=co,dc=za,dc=isp, thus it is the
connection from back-meta that has triggered the crash.

This build is 2.3.17 + Pierangelo's fixes in CVS (of yesterday's HEAD) to
back-meta, specifically back-meta/{back-meta.h,conn.c,bind.c}, and
back-ldap/back-ldap.h. AFAICR, the assertion was present (yesterday) without
these changes, I just hadn't caught it under gdb.

Running under gdb, I got the following (with Howard's help on irc):


(gdb) handle SIGPIPE nostop
Signal        Stop      Print   Pass to program Description
SIGPIPE       No        Yes     Yes             Broken pipe
(gdb) r -g ldap -u ldap -d0
Starting program: /home/bgmilne/rpm/BUILD/openldap-2.3.13/servers/slapd/.libs/slap
d -g ldap -u ldap -d0
[Thread debugging using libthread_db enabled]
[New Thread -1208011072 (LWP 25627)]
[New Thread -1475880016 (LWP 25630)]
[New Thread -1486369872 (LWP 25631)]
[New Thread -1496859728 (LWP 25632)]
[New Thread -1507349584 (LWP 25633)]
[New Thread -1522541648 (LWP 25634)]
[New Thread -1544557648 (LWP 25635)]
[New Thread -1589670992 (LWP 25636)]
[New Thread -1609606224 (LWP 25637)]
[New Thread -1629545552 (LWP 25638)]
[New Thread -1640035408 (LWP 25639)]
[New Thread -1650525264 (LWP 25640)]
[New Thread -1661015120 (LWP 25641)]
[New Thread -1717568592 (LWP 25664)]
[New Thread -1741685840 (LWP 25689)]

Program received signal SIGPIPE, Broken pipe.
slapd: daemon.c:1357: slap_listener: Assertion `((slap_daemon.sd_index[(s)]) ==
-1)' failed.

Program received signal SIGABRT, Aborted.
[Switching to Thread -1475880016 (LWP 25630)]
0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) thread apply all bt                                                      


Thread 15 (Thread -1741685840 (LWP 25689)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 14 (Thread -1717568592 (LWP 25664)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 13 (Thread -1661015120 (LWP 25641)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 12 (Thread -1650525264 (LWP 25640)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 11 (Thread -1640035408 (LWP 25639)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 10 (Thread -1629545552 (LWP 25638)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 9 (Thread -1609606224 (LWP 25637)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 8 (Thread -1589670992 (LWP 25636)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 7 (Thread -1544557648 (LWP 25635)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 6 (Thread -1522541648 (LWP 25634)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 5 (Thread -1507349584 (LWP 25633)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00b64ef4 in poll () from /lib/tls/libc.so.6
#2  0x00a267f7 in ldap_int_select (ld=0xfffffffc, timeout=0x2710) at
os-ip.c:922
#3  0x00a11d12 in ldap_result (ld=0x99a80cf0, msgid=1, all=0,
    timeout=0xa6279ed8, result=0xa6279ed4) at result.c:299
#4  0x00f782f4 in meta_back_single_bind (op=0xa6279f80, rs=0xa627a230,
    mc=0x99a80698, candidate=0, massage=1) at bind.c:348
#5  0x00f7891b in meta_back_bind (op=0x9c103498, rs=0xa627a230) at bind.c:186
#6  0x0808c3e2 in fe_op_bind (op=0x9c103498, rs=0xa627a230) at bind.c:405
#7  0x0808bc51 in do_bind (op=0x9c103498, rs=0xa627a230) at bind.c:200
#8  0x0806f0ab in connection_operation (ctx=0xa627a2c0, arg_v=0x9c103498)
    at connection.c:1307
#9  0x00a0e917 in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:481
#10 0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#11 0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 4 (Thread -1496859728 (LWP 25632)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 3 (Thread -1486369872 (LWP 25631)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c45b36 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#2  0x00a0f614 in ldap_pvt_thread_cond_wait (cond=0xfffffffc, mutex=0xfffffffc)
    at thr_posix.c:282
#3  0x00a0ea4e in ldap_int_thread_pool_wrapper (xpool=0x9ad9ce0) at tpool.c:469
#4  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#5  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 2 (Thread -1475880016 (LWP 25630)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00acf955 in raise () from /lib/tls/libc.so.6
#2  0x00ad1319 in abort () from /lib/tls/libc.so.6
#3  0x00ac8f41 in __assert_fail () from /lib/tls/libc.so.6
#4  0x0806ad17 in slap_listener (sl=0x9ac41c0) at daemon.c:1357
#5  0x0806c8eb in slapd_daemon_task (ptr=0x0) at daemon.c:2092
#6  0x00c433ae in start_thread () from /lib/tls/libpthread.so.0
#7  0x00b6eaee in clone () from /lib/tls/libc.so.6

Thread 1 (Thread -1208011072 (LWP 25627)):
#0  0x00a8f7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x00c440f8 in pthread_join () from /lib/tls/libpthread.so.0
#2  0x00a0f4d4 in ldap_pvt_thread_join (thread=4294967292,
    thread_return=0xfffffffc) at thr_posix.c:186
#3  0x0806cdb1 in slapd_daemon () at daemon.c:2219
#4  0x08059608 in main (argc=6, argv=0xbfffb104) at main.c:786

gdb) thread 2
[Switching to thread 2 (Thread -1475880016 (LWP 25630))]#0  0x00a8f7a2 in
_dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) frame 5
#5  0x0806c8eb in slapd_daemon_task (ptr=0x0) at daemon.c:2092
2092                                    rc = slap_listener( SLAP_EVENT_LISTENER(
i ));
(gdb) print slap_daemon
$1 = {sd_mutex = {__m_reserved = 1, __m_count = 0, __m_owner = 0x641e,
    __m_kind = 0, __m_lock = {__status = 1, __spinlock = 0}}, tcpd_mutex = {
    __m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 0, __m_lock =
{
      __status = 0, __spinlock = 0}}, sd_nactives = 13, sd_nwriters = 0,
  sd_epolls = 0x9ac4dd8, sd_nepolls = 0, sd_index = 0x9acade0, sd_epfd = 10,
  sd_nfds = 18}

(gdb) print connections[39]
$4 = {c_struct_state = 2, c_conn_state = 2, c_conn_idx = 39,
  c_close_reason = 0x8135600 "connection lost", c_mutex = {__m_reserved = 0,
    __m_count = 0, __m_owner = 0x0, __m_kind = 0, __m_lock = {__status = 0,
      __spinlock = 0}}, c_sb = 0x99de9730, c_starttime = 1137144581,
  c_activitytime = 1137144581, c_connid = 269658, c_peer_domain = {bv_len = 7,
    bv_val = 0x98e388d0 "unknown"}, c_peer_name = {bv_len = 18,
    bv_val = 0x98e387a0 "IP=127.0.0.1:45824"}, c_listener = 0x9ac41c0,
  c_sasl_bind_in_progress = 0, c_sasl_bind_mech = {bv_len = 0, bv_val = 0x0},
  c_sasl_dn = {bv_len = 0, bv_val = 0x0}, c_sasl_authz_dn = {bv_len = 0,
    bv_val = 0x0}, c_authz_backend = 0x9b2b208, c_authz_cookie = 0x0, c_authz =
{
    sai_method = 128, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {
      bv_len = 41,
      bv_val = 0x99cf7df0 "cn=qmail,dc=cybertrade,dc=co,dc=za,dc=isp"},
    sai_ndn = {bv_len = 41,
      bv_val = 0x99b4e1e0 "cn=qmail,dc=cybertrade,dc=co,dc=za,dc=isp"},
    sai_ssf = 0, sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0},
  c_protocol = 3, c_ops = {stqh_first = 0x0, stqh_last = 0xa80818b8},
  c_pending_ops = {stqh_first = 0x0, stqh_last = 0xa80818c0}, c_write_mutex = {
    __m_reserved = 0, __m_count = 0, __m_owner = 0x0, __m_kind = 0, __m_lock =
{
      __status = 0, __spinlock = 0}}, c_write_cv = {__c_lock = {__status = 0,
      __spinlock = 0}, __c_waiting = 0x0, __padding = '\0' <repeats 27 times>,
    __align = 0}, c_currentber = 0x0, c_writewaiter = 0, c_is_tls = 0,
  c_needs_tls_accept = 0, c_sasl_layers = 0, c_sasl_done = 0,
  c_sasl_authctx = 0x98e38918, c_sasl_sockctx = 0x0, c_sasl_extra = 0x98e40728,
  c_sasl_bindop = 0x0, c_pagedresults_state = {ps_be = 0x0, ps_size = 0,
    ps_cookie = 0, ps_count = 0}, c_n_ops_received = 1, c_n_ops_executing = 0, 
  c_n_ops_pending = 0, c_n_ops_completed = 1, c_n_get = 1, c_n_read = 1,
  c_n_write = 0, c_pb = 0x0, c_extensions = 0x0, c_clientfunc = 0,
  c_clientarg = 0x0, c_send_ldap_result = 0x807e699 <slap_send_ldap_result>,
  c_send_search_entry = 0x807f124 <slap_send_search_entry>,
  c_send_search_reference = 0x8080e3b <slap_send_search_reference>,
  c_send_ldap_extended = 0x807ed5e <slap_send_ldap_extended>,
  c_send_ldap_intermediate = 0x807ef75 <slap_send_ldap_intermediate>}
Prev by Date: (ITS#4337) inserting a bdb instance
Next by Date: (ITS#4339) "attributeoptions" not migrated from slapd.conf to back-config
Index(es):
- Chronological
- Thread