Re: (ITS#6029) slapadd lockup

[hyc@symas.com]

quanah@zimbra.com wrote:
> Full_Name: Quanah Gibson-Mount
> Version: RE24 3/19/2009
> OS: Linux 2.6
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (75.111.29.239)
>
>
> While testing various load scenarios, slapadd deadlocked on me.

The trace for thread 1 and 2 look identical, so I'm assuming they're the same 
thread. In that case I see 3 tool threads running. How many did you have 
configured? If 1 and 2 are really different threads (certainly their thread 
IDs look quite different), then there's something very wrong in the thread 
library...

print bdb_tool_index_tcount

It looks like one thread has exited early. I'm not sure how that can happen 
since the only way for any index thread to exit is if slapd_shutdown is set. 
Is it set?

>
> (gdb) thr apply all bt
>
> Thread 5 (Thread 1106889024 (LWP 9361)):
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized out>,
> ptr=<value optimized out>) at tools.c:1124
> #2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at
> ../../../libraries/libldap_r/tpool.c:663
> #3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
> #4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>
> Thread 4 (Thread 1115281728 (LWP 9362)):
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>,
> ptr=<value optimized out>) at tools.c:1147
> #2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at
> ../../../libraries/libldap_r/tpool.c:663
> #3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
> #4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>
> Thread 3 (Thread 1123674432 (LWP 9363)):
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>,
> ptr=<value optimized out>) at tools.c:1147
> #2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) at
> ../../../libraries/libldap_r/tpool.c:663
> #3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
> #4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>
> Thread 2 (Thread 1132067136 (LWP 9360)):
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value
> optimized out>, e=0x2b7b521d2068) at tools.c:475
> #2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068,
> text=0x7fff93960cc0) at tools.c:545
> #3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
> #4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
> ../../../servers/slapd/main.c:391
>
> Thread 1 (Thread 47794824997056 (LWP 9360)):
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value
> optimized out>, e=0x2b7b521d2068) at tools.c:475
> #2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, e=0x2b7b521d2068,
> text=0x7fff93960cc0) at tools.c:545
> #3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
> #4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
> ../../../servers/slapd/main.c:391
> #0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/