[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#3709) slapd deadlock on DEL
If you can reproduce this situation, it would help to see the output of
db_stat -CA on the database environment at this point.
richton@nbcs.rutgers.edu wrote:
>Full_Name: Aaron Richton
>Version: 2.2.25+patches
>OS: Solaris 9
>URL:
>Submission from: (NULL) (128.6.31.135)
>
>
>OL 2.2.25 with patches for ITS#3685 and ITS#3688 (essentially 2.2.26). An
>ldapdelete process with a small amount of changes was running for excessive time
>and was caught by our systems monitoring software. The stack trace showed a
>likely deadlock condition in slapd. ldapdelete was stuck in poll(), waiting for
>return from slapd. I sent a SIGINT resulting in
>
>[local4.debug] slapd shutdown: waiting for 3 threads to terminate
>
>slapd was continuously yield()ing. I waited a few minutes and used gcore before
>sending SIGKILL. The backtrace looks like something may be deadlocked:
>
>Thread 5 (LWP 2):
>#0 0xfede59b4 in lwp_yield () from /usr/lib/libthread.so.1
>#1 0x0016c954 in ldap_pvt_thread_yield () at thr_posix.c:184
>#2 0x0016b20c in ldap_pvt_thread_pool_destroy (tpool=0x230934, run_pending=1)
> at tpool.c:376
>#3 0x00058284 in slapd_daemon_task (ptr=0x0) at daemon.c:2007
>
>Thread 4 (LWP 4):
>#0 0xfee9f418 in ___lwp_cond_wait () from /usr/lib/libc.so.1
>#1 0xfee96428 in _lwp_cond_wait () from /usr/lib/libc.so.1
>#2 0xff2943a4 in __db_pthread_mutex_lock () from /usr/local/lib/libdb-4.2.so
>#3 0xff317d6c in __lock_get_internal () from /usr/local/lib/libdb-4.2.so
>#4 0xff316780 in __lock_vec () from /usr/local/lib/libdb-4.2.so
>#5 0xff2fa2c0 in __db_lget () from /usr/local/lib/libdb-4.2.so
>#6 0xff2a69cc in __bam_search () from /usr/local/lib/libdb-4.2.so
>#7 0xff29bb94 in __bam_c_search () from /usr/local/lib/libdb-4.2.so
>#8 0xff298214 in __bam_c_get () from /usr/local/lib/libdb-4.2.so
>#9 0xff2edad8 in __db_c_get () from /usr/local/lib/libdb-4.2.so
>#10 0xff2f7800 in __db_c_get_pp () from /usr/local/lib/libdb-4.2.so
>#11 0x00130770 in hdb_dn2id_parent (op=0xd4b678, txn=0x0, ei=0xd9f3f5f0,
> idp=0xd9f3f5a4) at dn2id.c:827
>#12 0x00127c8c in hdb_cache_find_parent (op=0xd4b678, txn=0x0, id=1,
> res=0xd9f3f968) at cache.c:383
>#13 0x001287a0 in hdb_cache_find_id (op=0xd4b678, tid=0x0, id=1,
> eip=0xd9f3f968, islocked=0, locker=54, lock=0xd9f3f7fc) at cache.c:659
>#14 0x000f7ca4 in hdb_do_search () at tools.c:283
>#15 0x000f5714 in hdb_search () at tools.c:283
>#16 0x0005f668 in do_search (op=0xd4b678, rs=0xd9fffd58) at search.c:412
>#17 0x0005bb34 in _end ()
>#18 0x0016b650 in ldap_int_thread_pool_wrapper (xpool=0x241638) at tpool.c:467
>
>Thread 3 (LWP 5):
>#0 0xfee9f418 in ___lwp_cond_wait () from /usr/lib/libc.so.1
>#1 0xfee96428 in _lwp_cond_wait () from /usr/lib/libc.so.1
>#2 0xff2943a4 in __db_pthread_mutex_lock () from /usr/local/lib/libdb-4.2.so
>#3 0xff317d6c in __lock_get_internal () from /usr/local/lib/libdb-4.2.so
>#4 0xff316780 in __lock_vec () from /usr/local/lib/libdb-4.2.so
>#5 0xff2fa2c0 in __db_lget () from /usr/local/lib/libdb-4.2.so
>#6 0xff2a69cc in __bam_search () from /usr/local/lib/libdb-4.2.so
>#7 0xff29bb94 in __bam_c_search () from /usr/local/lib/libdb-4.2.so
>#8 0xff298214 in __bam_c_get () from /usr/local/lib/libdb-4.2.so
>#9 0xff2edad8 in __db_c_get () from /usr/local/lib/libdb-4.2.so
>#10 0xff2f7800 in __db_c_get_pp () from /usr/local/lib/libdb-4.2.so
>#11 0x00130770 in hdb_dn2id_parent (op=0x284b3c0, txn=0x0, ei=0xd973f5f0,
> idp=0xd973f5a4) at dn2id.c:827
>#12 0x00127c8c in hdb_cache_find_parent (op=0x284b3c0, txn=0x0, id=1,
> res=0xd973f968) at cache.c:383
>#13 0x001287a0 in hdb_cache_find_id (op=0x284b3c0, tid=0x0, id=1,
> eip=0xd973f968, islocked=0, locker=55, lock=0xd973f7fc) at cache.c:659
>#14 0x000f7ca4 in hdb_do_search () at tools.c:283
>#15 0x000f5714 in hdb_search () at tools.c:283
>#16 0x0005f668 in do_search (op=0x284b3c0, rs=0xd97ffd58) at search.c:412
>#17 0x0005bb34 in _end ()
>#18 0x0016b650 in ldap_int_thread_pool_wrapper (xpool=0x241638) at tpool.c:467
>
>Thread 2 (LWP 15):
>#0 0xfee9f418 in ___lwp_cond_wait () from /usr/lib/libc.so.1
>#1 0xfee96428 in _lwp_cond_wait () from /usr/lib/libc.so.1
>#2 0xff2943a4 in __db_pthread_mutex_lock () from /usr/local/lib/libdb-4.2.so
>#3 0xff317d6c in __lock_get_internal () from /usr/local/lib/libdb-4.2.so
>#4 0xff316780 in __lock_vec () from /usr/local/lib/libdb-4.2.so
>#5 0xff2fa2c0 in __db_lget () from /usr/local/lib/libdb-4.2.so
>#6 0xff2a69cc in __bam_search () from /usr/local/lib/libdb-4.2.so
>#7 0xff29bb94 in __bam_c_search () from /usr/local/lib/libdb-4.2.so
>#8 0xff298214 in __bam_c_get () from /usr/local/lib/libdb-4.2.so
>#9 0xff2edad8 in __db_c_get () from /usr/local/lib/libdb-4.2.so
>#10 0xff2e6270 in __db_del () from /usr/local/lib/libdb-4.2.so
>#11 0xff2f50a0 in __db_del_pp () from /usr/local/lib/libdb-4.2.so
>#12 0x00134f90 in hdb_id2entry_delete (be=0x28b400, tid=0x3340c88, e=0xa824e40)
> at id2entry.c:148
>#13 0x0012e7a0 in hdb_delete (op=0x29be028, rs=0xd47ffd58) at delete.c:398
>#14 0x0007db44 in do_delete (op=0x29be028, rs=0xd47ffd58) at delete.c:230
>#15 0x0005ba14 in _end ()
>#16 0x0016b650 in ldap_int_thread_pool_wrapper (xpool=0x241638) at tpool.c:467
>
>Thread 1 (LWP 1):
>#0 0xfee9f330 in _lwp_wait () from /usr/lib/libc.so.1
>#1 0xfeddd6bc in lwp_wait () from /usr/lib/libthread.so.1
>#2 0xfedd90f0 in _thrp_join () from /usr/lib/libthread.so.1
>#3 0x0016c8d8 in ldap_pvt_thread_join (thread=2, thread_return=0x0)
> at thr_posix.c:151
>#4 0x00058394 in slapd_daemon () at daemon.c:2041
>#5 0x00046ae4 in main (argc=4, argv=0xffbffd94) at main.c:713
>
>Logs and core files available upon request.
>
>
>
>
--
-- Howard Chu
Chief Architect, Symas Corp. Director, Highland Sun
http://www.symas.com http://highlandsun.com/hyc
Symas: Premier OpenSource Development and Support