(ITS#3933) slapd failes to shut down

[quanah@stanford.edu]

Full_Name: Quanah Gibson-mount
Version: REL_ENG_23
OS: Solaris 8
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (171.66.182.82)


Yesterday I set up 4 systems, 1 a master, 3 replicas.  I have a nightly job that
stops the server and does various things and then restarts the server.  The job
waits until the server stops to do anything.  If the server hasn't exited within
a certain amount of time, it throws an error and stops anything else from
happening.  Last night, the job went to stop the master, sending it a signal
around 4:00 a.m.:

Aug 13 04:09:15 ldap-dev0.Stanford.EDU slapd[606]: [ID 543694 local4.debug]
daemon: shutdown requested and initiated.

This process has never exited.  A backtrace shows:

(gdb) thr apply all bt

Thread 10 (LWP 3):
#0  0xfed64e48 in lwp_mutex_lock () from /usr/lib/lwp/libthread.so.1
#1  0xfed5ff90 in mutex_lock_kernel () from /usr/lib/lwp/libthread.so.1
#2  0xfed609b0 in stall () from /usr/lib/lwp/libthread.so.1
#3  0xfed60f7c in mutex_lock_internal () from /usr/lib/lwp/libthread.so.1

Thread 9 (LWP 4):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 8 (LWP 5):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 7 (LWP 6):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 6 (LWP 7):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 5 (LWP 8):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 4 (LWP 9):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 3 (LWP 10):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 2 (LWP 11):
#0  0xfed64d7c in __lwp_park () from /usr/lib/lwp/libthread.so.1
#1  0xfed61f74 in cond_wait_queue () from /usr/lib/lwp/libthread.so.1
#2  0xfed626e8 in cond_wait () from /usr/lib/lwp/libthread.so.1
#3  0xfed62724 in pthread_cond_wait () from /usr/lib/lwp/libthread.so.1

Thread 1 (LWP 1):
#0  0xfee1f368 in _lwp_wait () from /usr/lib/libc.so.1
#1  0xfed5ca88 in lwp_wait () from /usr/lib/lwp/libthread.so.1
#2  0xfed58370 in _thrp_join () from /usr/lib/lwp/libthread.so.1
#3  0x000245b0 in slapd_daemon () at daemon.c:2045
(gdb) thr 1
[Switching to thread 1 (LWP 1)]#0  0xfee1f368 in _lwp_wait () from
/usr/lib/libc.so.1
(gdb) l
2045                    ldap_pvt_thread_join( listener_tid, (void *) NULL );
2046            }
2047    #else
2048            /* experimental code */
2049            slapd_daemon_task( NULL );
2050    #endif
2051
2052            return 0;
2053
2054    }


--Quanah