Re: RE24 connection code round 2

[Howard Chu <hyc@symas.com>]

Michael StrÃder wrote:
> Quanah Gibson-Mount wrote:
> > --On January 28, 2009 4:33:53 PM +0100 Michael StrÃder
> > <michael@stroeder.com>  wrote:
> >
> > > (gdb) bt all
> > > No symbol "all" in current context.
> > Generally:
> >
> > thr apply all bt
> >
> > I think is what you are going for (i.e., get a full backtrace on all
> > threads).
>
> Now gdb backtrace of hanging test008 with HEAD and compiled with only -O2:

Thanks. Seems like this is a race condition due to the edge-triggered epoll - 
if the socket becomes writable before we wait for it, we won't wake up from 
the wait. Apparently we'll have to go back to level-triggered like before. Too 
bad we can't set different epoll behaviors for different events.

(Thread 15 is waiting to write again...)

> Thread 18 (Thread 0x40d14b90 (LWP 26740)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x40579c26 in epoll_wait () from /lib/libc.so.6
> #2  0x0806617f in slapd_daemon_task (ptr=0x0) at daemon.c:2291
> #3  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #4  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 17 (Thread 0x41115b90 (LWP 26741)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 16 (Thread 0x41e18b90 (LWP 27131)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 15 (Thread 0x42b1bb90 (LWP 27136)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x4073a288,
> mutex=0x4073a270) at thr_posix.c:277
> #3  0x0807a418 in send_ldap_ber (conn=0x4073a178, ber=0x42b19de4) at
> result.c:217
> #4  0x0807c43b in slap_send_search_entry (op=0x8381ce0, rs=0x42b1b0c8)
> at result.c:1246
> #5  0x0810b2ba in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8,
> e_parent=0x835f3e4, sub=1) at search.c:115
> #6  0x0810b1cc in monitor_send_children (op=0x8381ce0, rs=0x42b1b0c8,
> e_parent=<value optimized out>, sub=1) at search.c:123
> #7  0x0810b5dd in monitor_back_search (op=0x8381ce0, rs=0x42b1b0c8) at
> search.c:245
> #8  0x0806bb73 in fe_op_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:366
> #9  0x0806c3e8 in do_search (op=0x8381ce0, rs=0x42b1b0c8) at search.c:217
> #10 0x080696bb in connection_operation (ctx=0x42b1b1cc, arg_v=0x8381ce0)
> at connection.c:1100
> #11 0x0806a117 in connection_read_thread (ctx=0x42b1b1cc, argv=0x1c) at
> connection.c:1226
> #12 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:663
> #13 0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #14 0x4057938e in clone () from /lib/libc.so.6
>
> Thread 14 (Thread 0x4381eb90 (LWP 27138)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 13 (Thread 0x43c1fb90 (LWP 27144)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 12 (Thread 0x44020b90 (LWP 27145)):
> ---Type<return>  to continue, or q<return>  to quit---
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 11 (Thread 0x44421b90 (LWP 27146)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0
> #2  0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0
> #3  0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0
> #4  0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at
> thr_posix.c:296
> #5  0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117
> #6  0x0810b155 in monitor_send_children (op=0x83891a8, rs=0x444210c8,
> e_parent=<value optimized out>, sub=1) at search.c:144
> #7  0x0810b1cc in monitor_send_children (op=0x83891a8, rs=0x444210c8,
> e_parent=<value optimized out>, sub=1) at search.c:123
> #8  0x0810b5dd in monitor_back_search (op=0x83891a8, rs=0x444210c8) at
> search.c:245
> #9  0x0806bb73 in fe_op_search (op=0x83891a8, rs=0x444210c8) at search.c:366
> #10 0x0806c3e8 in do_search (op=0x83891a8, rs=0x444210c8) at search.c:217
> #11 0x080696bb in connection_operation (ctx=0x444211cc, arg_v=0x83891a8)
> at connection.c:1100
> #12 0x0806a117 in connection_read_thread (ctx=0x444211cc, argv=0x18) at
> connection.c:1226
> #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:663
> #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #15 0x4057938e in clone () from /lib/libc.so.6
>
> Thread 10 (Thread 0x45427b90 (LWP 27150)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 9 (Thread 0x45929b90 (LWP 27154)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 8 (Thread 0x45e2bb90 (LWP 27160)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 7 (Thread 0x4632db90 (LWP 27163)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x402507b9 in __lll_lock_wait () from /lib/libpthread.so.0
> #2  0x4024bcd1 in _L_lock_89 () from /lib/libpthread.so.0
> #3  0x4024b5b2 in pthread_mutex_lock () from /lib/libpthread.so.0
> #4  0x4002fffd in ldap_pvt_thread_mutex_lock (mutex=0x837a8c8) at
> thr_posix.c:296
> #5  0x0815f27c in monitor_cache_lock (e=0x835f3e4) at cache.c:117
> #6  0x0810b155 in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8,
> e_parent=<value optimized out>, sub=1) at search.c:144
> #7  0x0810b1cc in monitor_send_children (op=0x839b3a0, rs=0x4632d0c8,
> e_parent=<value optimized out>, sub=1) at search.c:123
> #8  0x0810b5dd in monitor_back_search (op=0x839b3a0, rs=0x4632d0c8) at
> search.c:245
> #9  0x0806bb73 in fe_op_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:366
> #10 0x0806c3e8 in do_search (op=0x839b3a0, rs=0x4632d0c8) at search.c:217
> #11 0x080696bb in connection_operation (ctx=0x4632d1cc, arg_v=0x839b3a0)
> at connection.c:1100
> #12 0x0806a117 in connection_read_thread (ctx=0x4632d1cc, argv=0x1f) at
> connection.c:1226
> #13 0x4002f400 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:663
> #14 0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #15 0x4057938e in clone () from /lib/libc.so.6
>
> Thread 6 (Thread 0x4682fb90 (LWP 27166)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 5 (Thread 0x46d31b90 (LWP 27169)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 4 (Thread 0x47233b90 (LWP 27170)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 3 (Thread 0x47634b90 (LWP 27173)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 2 (Thread 0x47a35b90 (LWP 27174)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024dc35 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #2  0x40030094 in ldap_pvt_thread_cond_wait (cond=0x82ba4f4,
> mutex=0x82ba4dc) at thr_posix.c:277
> #3  0x4002f472 in ldap_int_thread_pool_wrapper (xpool=0x82ba4d8) at
> tpool.c:654
> #4  0x4024a1b5 in start_thread () from /lib/libpthread.so.0
> #5  0x4057938e in clone () from /lib/libc.so.6
>
> Thread 1 (Thread 0x406f1930 (LWP 26725)):
> #0  0xffffe430 in __kernel_vsyscall ()
> #1  0x4024a747 in pthread_join () from /lib/libpthread.so.0
> #2  0x400301d4 in ldap_pvt_thread_join (thread=1087458192,
> thread_return=0x0) at thr_posix.c:197
> #3  0x08063693 in slapd_daemon () at daemon.c:2658
> #4  0x08050ee6 in main (argc=8, argv=0xbf90dc64) at main.c:948


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/