[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#6029) slapadd lockup

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6029) slapadd lockup
From: quanah@zimbra.com
Date: Sun, 22 Mar 2009 02:41:30 GMT
Auto-submitted: auto-generated (OpenLDAP-ITS)
After around 24 hours, backtrace is the same:

(gdb) thr apply all bt

Thread 5 (Thread 1106889024 (LWP 9361)):
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized out>, 
ptr=<value optimized out>) at tools.c:1124
#2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) 
at ../../../libraries/libldap_r/tpool.c:663
#3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
#4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6

Thread 4 (Thread 1115281728 (LWP 9362)):
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, 
ptr=<value optimized out>) at tools.c:1147
#2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) 
at ../../../libraries/libldap_r/tpool.c:663
#3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
#4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6

Thread 3 (Thread 1123674432 (LWP 9363)):
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized out>, 
ptr=<value optimized out>) at tools.c:1147
#2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper (xpool=0x17e07fd0) 
at ../../../libraries/libldap_r/tpool.c:663
#3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
#4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6

Thread 2 (Thread 1132067136 (LWP 9360)):
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value 
optimized out>, e=0x2b7b521d2068) at tools.c:475
#2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, 
e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545
#3  0x00000000004906f9 in slapadd (argc=<value optimized out>, 
argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
#4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at 
../../../servers/slapd/main.c:391

Thread 1 (Thread 47794824997056 (LWP 9360)):
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0
#1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310, txn=<value 
optimized out>, e=0x2b7b521d2068) at tools.c:475
#2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240, 
e=0x2b7b521d2068, text=0x7fff93960cc0) at tools.c:545
#3  0x00000000004906f9 in slapadd (argc=<value optimized out>, 
argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
#4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at 
../../../servers/slapd/main.c:391
#0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from 
/lib64/libpthread.so.0

--On Sunday, March 22, 2009 2:37 AM +0000 quanah@zimbra.com wrote:

> --On Saturday, March 21, 2009 7:09 PM -0700 Howard Chu <hyc@symas.com>
> wrote:
>
>> quanah@zimbra.com wrote:
>>> Full_Name: Quanah Gibson-Mount
>>> Version: RE24 3/19/2009
>>> OS: Linux 2.6
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (75.111.29.239)
>>>
>>>
>>> While testing various load scenarios, slapadd deadlocked on me.
>>
>> The trace for thread 1 and 2 look identical, so I'm assuming they're the
>> same thread. In that case I see 3 tool threads running. How many did you
>> have configured? If 1 and 2 are really different threads (certainly their
>> thread IDs look quite different), then there's something very wrong in
>> the thread library...
>>
>> print bdb_tool_index_tcount
>>
>
> (gdb) print bdb_tool_index_tcount
> $1 = 1
>
>
>> It looks like one thread has exited early. I'm not sure how that can
>> happen since the only way for any index thread to exit is if
>> slapd_shutdown is set. Is it set?
>
> (gdb) print slapd_shutdown
> $2 = 0
>
>
> It's configured with 4 tool threads.
>
> --Quanah
>
>>> (gdb) thr apply all bt
>>>
>>> Thread 5 (Thread 1106889024 (LWP 9361)):
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>> # 1  0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized
>>> # out>,
>>> ptr=<value optimized out>) at tools.c:1124
>>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>>> # (xpool=0x17e07fd0) at
>>> ../../../libraries/libldap_r/tpool.c:663
>>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>>
>>> Thread 4 (Thread 1115281728 (LWP 9362)):
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>> # 1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized
>>> # out>,
>>> ptr=<value optimized out>) at tools.c:1147
>>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>>> # (xpool=0x17e07fd0) at
>>> ../../../libraries/libldap_r/tpool.c:663
>>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>>
>>> Thread 3 (Thread 1123674432 (LWP 9363)):
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>> # 1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized
>>> # out>,
>>> ptr=<value optimized out>) at tools.c:1147
>>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>>> # (xpool=0x17e07fd0) at
>>> ../../../libraries/libldap_r/tpool.c:663
>>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>>
>>> Thread 2 (Thread 1132067136 (LWP 9360)):
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>> # 1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310,
>>> # txn=<value
>>> optimized out>, e=0x2b7b521d2068) at tools.c:475
>>> # 2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240,
>>> # e=0x2b7b521d2068,
>>> text=0x7fff93960cc0) at tools.c:545
>>> # 3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
>>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
>>> # 4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
>>> ../../../servers/slapd/main.c:391
>>>
>>> Thread 1 (Thread 47794824997056 (LWP 9360)):
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>> # 1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310,
>>> # txn=<value
>>> optimized out>, e=0x2b7b521d2068) at tools.c:475
>>> # 2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240,
>>> # e=0x2b7b521d2068,
>>> text=0x7fff93960cc0) at tools.c:545
>>> # 3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
>>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
>>> # 4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
>>> ../../../servers/slapd/main.c:391
>>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>>> /lib64/libpthread.so.0
>>
>>
>> --
>>    -- Howard Chu
>>    CTO, Symas Corp.           http://www.symas.com
>>    Director, Highland Sun     http://highlandsun.com/hyc/
>>    Chief Architect, OpenLDAP  http://www.openldap.org/project/
>
>
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration
>
>



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Prev by Date: Re: (ITS#6029) slapadd lockup
Next by Date: Re: patch filed (ITS#5366)
Index(es):
- Chronological
- Thread