ghola@rebelbase.com wrote:
> Full_Name: Duncan Idaho
> Version: 2.4.25
> OS: Centos 5.5
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (216.148.0.72)
>
>
> We are running a cluster of "synchronization hosts" that act as Producers for a
> large number of Consumers and additionally Consume from a single global
> read/write master.  After several weeks of stable operation synchronization came
> to a halt on both nodes in the Producer/Consumer cluster simultaneously.

Can you also provide the output of db_stat -CA on the database when this 
lockup occurs? It seems that a database operation is preventing syncprov from 
making any progress.
>
> Stack-trace:
>
> Thread 15 (Thread 10745):
> #0  0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
> #1  0x000000000041929d in slapd_daemon () at daemon.c:2922
> #2  0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
> at main.c:983
>
> Thread 14 (Thread 8866):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 14876):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 14875):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 14874):
> #0  0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
> #1  0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
> #2  0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
> at syncprov.c:2471
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 14873):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:217
>
> ...back 1 page
>
> Thread 15 (Thread 10745):
> #0  0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
> #1  0x000000000041929d in slapd_daemon () at daemon.c:2922
> #2  0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
> at main.c:983
>
> Thread 14 (Thread 8866):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 14876):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 14875):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 14874):
> #0  0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
> #1  0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
> #2  0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
> at syncprov.c:2471
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 14873):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:217
> [root@su1-admin-syncrepl01 ldap]# cat gdb.txt
>
> Thread 15 (Thread 10745):
> #0  0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
> #1  0x000000000041929d in slapd_daemon () at daemon.c:2922
> #2  0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
> at main.c:983
>
> Thread 14 (Thread 8866):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 14876):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 14875):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:672
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 14874):
> #0  0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
> #1  0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
> #2  0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
> at syncprov.c:2471
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 14873):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x459a6d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x459a6d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 9 (Thread 14872):
> #0  0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
> #1  0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
> #2  0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e48 in syncprov_op_search (op=0x2aaab89e58c0, rs=0x451a5c10)
> at syncprov.c:2476
> #4  0x000000000047e64a in overlay_op_walk (op=0x2aaab89e58c0, rs=0x451a5c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2aaab89e58c0, rs=0x451a5c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2aaab89e58c0, rs=0x451a5c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2aaab89e58c0, rs=0x451a5c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x451a5d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x451a5d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 8 (Thread 16034):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x000000000042db1c in send_ldap_ber (op=<value optimized out>,
> ber=0x449a43f0) at result.c:372
> #2  0x00000000004308d8 in slap_send_search_entry (op=0x449a45a0, rs=0x449a4af0)
> at result.c:1410
> #3  0x00000000004da80d in syncprov_qplay (ctx=<value optimized out>, arg=<value
> optimized out>) at syncprov.c:884
> #4  syncprov_qtask (ctx=<value optimized out>, arg=<value optimized out>) at
> syncprov.c:1006
> #5  0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #6  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #7  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #8  0x0000000000000000 in ?? ()
>
> Thread 7 (Thread 16033):
> #0  0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
> #1  0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
> #2  0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
> #3  0x00000000004d7e5e in syncprov_op_search (op=0x2d554b70, rs=0x441a3c10) at
> syncprov.c:2471
> #4  0x000000000047e64a in overlay_op_walk (op=0x2d554b70, rs=0x441a3c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #5  0x000000000047ec27 in over_op_func (op=0x2d554b70, rs=0x441a3c10,
> which=op_search) at backover.c:721
> #6  0x0000000000420a79 in fe_op_search (op=0x2d554b70, rs=0x441a3c10) at
> search.c:372
> #7  0x0000000000421217 in do_search (op=0x2d554b70, rs=0x441a3c10) at
> search.c:217
> #8  0x000000000041e965 in connection_operation (ctx=0x441a3d60, arg_v=<value
> optimized out>) at connection.c:1113
> #9  0x000000000041f23c in connection_read_thread (ctx=0x441a3d60, argv=<value
> optimized out>) at connection.c:1249
> #10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #13 0x0000000000000000 in ?? ()
>
> Thread 6 (Thread 11937):
> #0  0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
> #1  0x00000000004d7e7f in syncprov_op_search (op=0x2aaab802af20, rs=0x418f7c10)
> at syncprov.c:2475
> #2  0x000000000047e64a in overlay_op_walk (op=0x2aaab802af20, rs=0x418f7c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #3  0x000000000047ec27 in over_op_func (op=0x2aaab802af20, rs=0x418f7c10,
> which=op_search) at backover.c:721
> #4  0x0000000000420a79 in fe_op_search (op=0x2aaab802af20, rs=0x418f7c10) at
> search.c:372
> #5  0x0000000000421217 in do_search (op=0x2aaab802af20, rs=0x418f7c10) at
> search.c:217
> #6  0x000000000041e965 in connection_operation (ctx=0x418f7d60, arg_v=<value
> optimized out>) at connection.c:1113
> #7  0x000000000041f23c in connection_read_thread (ctx=0x418f7d60, argv=<value
> optimized out>) at connection.c:1249
> #8  0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #9  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #11 0x0000000000000000 in ?? ()
>
> Thread 5 (Thread 10755):
> #0  0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
> #1  0x00002b1d8ae85bd1 in __db_pthread_mutex_lock () from
> /usr/local/openldap/lib/libdb-5.1.so
> #2  0x00002b1d8ae8528a in __db_tas_mutex_lock () from
> /usr/local/openldap/lib/libdb-5.1.so
> #3  0x00002b1d8af1804c in __lock_get_internal () from
> /usr/local/openldap/lib/libdb-5.1.so
> #4  0x00002b1d8af189ea in __lock_vec () from
> /usr/local/openldap/lib/libdb-5.1.so
> #5  0x00002b1d8af1969b in __lock_vec_pp () from
> /usr/local/openldap/lib/libdb-5.1.so
> #6  0x00000000004bc6c1 in hdb_cache_entry_db_relock (bdb=0xe2bc780, txn=<value
> optimized out>, ei=0x2aaadaefc6a0, rw=1, tryOnly=<value optimized out>,
> lock=0x439a1640) at cache.c:198
> #7  0x00000000004bcb70 in hdb_cache_modify (bdb=0x2b1d8f0fa00c,
> e=0x2b1df490e138, newAttrs=0x2b1e071d6918, txn=0x0, lock=0x0) at cache.c:1231
> #8  0x0000000000495d13 in hdb_modify (op=0x439a2120, rs=0x439a1cb0) at
> modify.c:662
> #9  0x000000000047e6c2 in overlay_op_walk (op=0x439a2120, rs=0x439a1cb0,
> which=op_modify, oi=0xe2be010, on=0x0) at backover.c:669
> #10 0x000000000047ec27 in over_op_func (op=0x439a2120, rs=0x439a1cb0,
> which=op_modify) at backover.c:721
> #11 0x000000000047240f in syncrepl_entry (si=0xe2c2800, op=0x439a2120,
> entry=0x2b1df490e408, modlist=0x439a2c68, syncstate=2, syncUUID=<value optimized
> out>, syncCSN=0x2aaade6b6ab0) at syncrepl.c:2694
> #12 0x0000000000479cf7 in do_syncrep2 (ctx=<value optimized out>, arg=<value
> optimized out>) at syncrepl.c:955
> #13 do_syncrepl (ctx=<value optimized out>, arg=<value optimized out>) at
> syncrepl.c:1455
> #14 0x000000000041f7ca in connection_read_thread (ctx=0x439a2d60, argv=<value
> optimized out>) at connection.c:1251
> #15 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #16 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #17 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #18 0x0000000000000000 in ?? ()
>
> Thread 4 (Thread 10752):
> #0  0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
> #1  0x00000000004d7e7f in syncprov_op_search (op=0xe6f0990, rs=0x410f6c10) at
> syncprov.c:2475
> #2  0x000000000047e64a in overlay_op_walk (op=0xe6f0990, rs=0x410f6c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #3  0x000000000047ec27 in over_op_func (op=0xe6f0990, rs=0x410f6c10,
> which=op_search) at backover.c:721
> #4  0x0000000000420a79 in fe_op_search (op=0xe6f0990, rs=0x410f6c10) at
> search.c:372
> #5  0x0000000000421217 in do_search (op=0xe6f0990, rs=0x410f6c10) at
> search.c:217
> #6  0x000000000041e965 in connection_operation (ctx=0x410f6d60, arg_v=<value
> optimized out>) at connection.c:1113
> #7  0x000000000041f23c in connection_read_thread (ctx=0x410f6d60, argv=<value
> optimized out>) at connection.c:1249
> #8  0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #9  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #11 0x0000000000000000 in ?? ()
>
> Thread 3 (Thread 10749):
> #0  0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
> #1  0x00000000004d7e7f in syncprov_op_search (op=0xe5eefa0, rs=0x431a1c10) at
> syncprov.c:2475
> #2  0x000000000047e64a in overlay_op_walk (op=0xe5eefa0, rs=0x431a1c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #3  0x000000000047ec27 in over_op_func (op=0xe5eefa0, rs=0x431a1c10,
> which=op_search) at backover.c:721
> #4  0x0000000000420a79 in fe_op_search (op=0xe5eefa0, rs=0x431a1c10) at
> search.c:372
> #5  0x0000000000421217 in do_search (op=0xe5eefa0, rs=0x431a1c10) at
> search.c:217
> #6  0x000000000041e965 in connection_operation (ctx=0x431a1d60, arg_v=<value
> optimized out>) at connection.c:1113
> #7  0x000000000041f23c in connection_read_thread (ctx=0x431a1d60, argv=<value
> optimized out>) at connection.c:1249
> #8  0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #9  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #11 0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 10748):
> #0  0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
> #1  0x00000000004d7e7f in syncprov_op_search (op=0xe5ebbf0, rs=0x429a0c10) at
> syncprov.c:2475
> #2  0x000000000047e64a in overlay_op_walk (op=0xe5ebbf0, rs=0x429a0c10,
> which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
> #3  0x000000000047ec27 in over_op_func (op=0xe5ebbf0, rs=0x429a0c10,
> which=op_search) at backover.c:721
> #4  0x0000000000420a79 in fe_op_search (op=0xe5ebbf0, rs=0x429a0c10) at
> search.c:372
> #5  0x0000000000421217 in do_search (op=0xe5ebbf0, rs=0x429a0c10) at
> search.c:217
> #6  0x000000000041e965 in connection_operation (ctx=0x429a0d60, arg_v=<value
> optimized out>) at connection.c:1113
> #7  0x000000000041f23c in connection_read_thread (ctx=0x429a0d60, argv=<value
> optimized out>) at connection.c:1249
> #8  0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
> tpool.c:685
> #9  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #11 0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 10747):
> #0  0x000000322e2d4108 in epoll_wait () from /lib64/libc.so.6
> #1  0x000000000041bc8b in slapd_daemon_task (ptr=<value optimized out>) at
> daemon.c:2528
> #2  0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
> #3  0x000000322e2d3d1d in clone () from /lib64/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Logs at the time of the lock-up:
>
> Producer:
> Jul 28 22:27:08 admin02 slapd[5178]: conn=422733 op=92 MOD
> dn="thing=thing1,ou=things,dc=example,dc=net"
> Jul 28 22:27:08 admin02 slapd[5178]: conn=422733 op=92 MOD attr=stuff
> Jul 28 22:27:08 admin02 slapd[5178]: slap_queue_csn: queing 0x4b0883f0
>   20110728222708.058275Z#000000#000#000000
> Jul 28 22:27:08 admin02 slapd[5178]: syncprov_sendresp: cookie=rid=001
> ,csn=20110728222708.058275Z#000000#000#000000
> Jul 28 22:27:08 admin02 slapd[5178]: syncprov_sendresp: cookie=rid=001
>
> Producer later:
> Jul 29 07:04:55 admin02 slapd[5178]: Entry
> thing=thing1,ou=things,dc=example,dc=net CSN
> 20110728222708.058275Z#000000#000#000000 older or equal to ctx
> 20110728222708.058275Z#000000#000#000000
>
> Producer/Consumer:
> Jul 28 22:27:08 su1-admin-syncrepl01 slapd[10745]: do_syncrep2: rid=001
> cookie=rid=001,csn=20110728222708.058275Z#000000#000#000000
> Jul 28 22:27:08 su1-admin-syncrepl01 slapd[10745]: slap_queue_csn: queing
> 0x2aaadecf4b00 20110728222708.058275Z#000000#000#000000
>
> ^^ No more syncrepl logs after this
>
> Help!
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

changed state Open to Feedback

Here is the data requested from the most recent lock-up.

Default locking region information:
14	Last allocated locker ID
0x7fffffff	Current maximum unused locker ID
9	Number of lock modes
1000	Maximum number of locks possible
1000	Maximum number of lockers possible
1000	Maximum number of lock objects possible
160	Number of lock object partitions
27	Number of current locks
682	Maximum number of locks at any one time
5	Maximum number of locks in any one bucket
0	Maximum number of locks stolen by for an empty partition
0	Maximum number of locks stolen for any one partition
63	Number of current lockers
63	Maximum number of lockers at any one time
21	Number of current lock objects
377	Maximum number of lock objects at any one time
3	Maximum number of lock objects in any one bucket
0	Maximum number of objects stolen by for an empty partition
0	Maximum number of objects stolen for any one partition
13M	Total number of locks requested (13581842)
13M	Total number of locks released (13581804)
0	Total number of locks upgraded
13	Total number of locks downgraded
6	Lock requests not available due to conflicts, for which we waited
1	Lock requests not available due to conflicts, for which we did not wait
0	Number of deadlocks
0	Lock timeout value
0	Number of locks that have timed out
0	Transaction timeout value
0	Number of transactions that have timed out
808KB	The size of the lock region
15280	The number of partition locks that required waiting (0%)
10921	The maximum number of times any partition lock was waited for (0%)
0	The number of object queue operations that required waiting (0%)
4742	The number of locker allocations that required waiting (0%)
0	The number of region locks that required waiting (0%)
3	Maximum hash bucket length
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock REGINFO information:
Lock	Region type
5	Region ID
__db.005	Region name
0x2aeb3285c000	Region address
0x2aeb3285c138	Region primary address
0	Region maximum allocation
0	Region allocated
Region allocations: 327 allocations, 0 failures, 0 frees, 1 longest
Allocations by power-of-two sizes:
  1KB	322
  2KB	0
  4KB	0
  8KB	0
 16KB	1
 32KB	2
 64KB	0
128KB	1
256KB	1
512KB	0
1024KB	0
REGION_JOIN_OK	Region flags
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock region parameters:
262177	Lock region region mutex [0/450861 0% 15746/47187942583984]
1031	locker table size
1031	object table size
952	obj_off
122400	locker_off
1	need_dd
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Lock conflict matrix:
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by lockers:
Locker   Mode      Count Status  ----------------- Object ---------------
       1 dd=22 locks held 1    write locks 0    pid/thread
7123/47021676422512 priority 100
       1 READ          1 HELD    id2entry.bdb              handle        0
       2 dd=21 locks held 1    write locks 0    pid/thread
7123/47021676422512 priority 100
       2 READ          1 HELD    dn2id.bdb                 handle        0
       3 dd=20 locks held 0    write locks 0    pid/thread
7123/47021676422512 priority 100
       4 dd=16 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       4 READ          1 HELD    entryUUID.bdb             handle        0
       5 dd=15 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       5 READ          1 HELD    modifyTimestamp.bdb       handle        0
       6 dd=12 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       6 READ          1 HELD    cn.bdb                    handle        0
       7 dd=11 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       7 READ          1 HELD    givenName.bdb             handle        0
       8 dd=10 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       8 READ          1 HELD    sn.bdb                    handle        0
       9 dd= 9 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       9 READ          1 HELD    objectClass.bdb           handle        0
       a dd= 6 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       a READ          1 HELD    uid.bdb                   handle        0
       b dd= 5 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       b READ          1 HELD    uidNumber.bdb             handle        0
       c dd= 4 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
       c READ          1 HELD    gidNumber.bdb             handle        0
       d dd= 3 locks held 1    write locks 0    pid/thread
7123/1121872192 priority 100
       d READ          1 HELD    sambaSID.bdb              handle        0
       e dd= 2 locks held 1    write locks 0    pid/thread
7123/1121872192 priority 100
       e READ          1 HELD    sambaPrimaryGroupSID.bdb  handle        0
80000380 dd=14 locks held 0    write locks 0    pid/thread
7123/1121872192 priority 100
80000381 dd=13 locks held 0    write locks 0    pid/thread
7123/1092884800 priority 100
80000003 dd=19 locks held 0    write locks 0    pid/thread
7123/47021676422512 priority 100
80000004 dd=18 locks held 0    write locks 0    pid/thread
7123/47021676422512 priority 100
80000005 dd=17 locks held 1    write locks 0    pid/thread
7123/1113479488 priority 100
80000005 READ          1 HELD    0x3dc78 len:   9 data: a2b007000000000000
80000938 dd= 8 locks held 0    write locks 0    pid/thread
7123/1138657600 priority 100
80000939 dd= 7 locks held 0    write locks 0    pid/thread
7123/1130264896 priority 100
80048a47 dd= 1 locks held 0    write locks 0    pid/thread
7123/1147050304 priority 100
800dc266 dd= 0 locks held 11   write locks 5    pid/thread
7123/1138657600 priority 100
800dc266 WRITE         1 WAIT    0x3dc78 len:   9 data: a2b007000000000000
800dc266 READ          2 HELD    dn2id.bdb                 page      11658
800dc266 WRITE         4 HELD    dn2id.bdb                 page      11658
800dc266 READ          2 HELD    objectClass.bdb           page        115
800dc266 WRITE         2 HELD    objectClass.bdb           page        115
800dc266 READ          1 HELD    objectClass.bdb           page          3
800dc266 READ          1 HELD    entryUUID.bdb             page       2839
800dc266 WRITE         2 HELD    entryUUID.bdb             page       2839
800dc266 READ          1 HELD    modifyTimestamp.bdb       page       3168
800dc266 WRITE         2 HELD    modifyTimestamp.bdb       page       3168
800dc266 WRITE         3 HELD    id2entry.bdb              page      14475
800dc266 READ          1 HELD    0x36718 len:   9 data: 9cb007000000000000
800dc268 dd= 0 locks held 0    write locks 0    pid/thread
7123/1163835712 priority 100
800dc269 dd= 0 locks held 0    write locks 0    pid/thread
7123/1155443008 priority 100
800dc26a dd= 0 locks held 0    write locks 0    pid/thread
7123/1172228416 priority 100
800dc26b dd= 0 locks held 0    write locks 0    pid/thread
7123/1180621120 priority 100
800dc26c dd= 0 locks held 0    write locks 0    pid/thread
7123/1189013824 priority 100
800dc26d dd= 0 locks held 0    write locks 0    pid/thread
7123/1197406528 priority 100
800dc26e dd= 0 locks held 0    write locks 0    pid/thread
7123/1205799232 priority 100
800dc26f dd= 0 locks held 0    write locks 0    pid/thread
7123/1214191936 priority 100
800dc270 dd= 0 locks held 0    write locks 0    pid/thread
7123/1222584640 priority 100
800dc271 dd= 0 locks held 0    write locks 0    pid/thread
7123/1230977344 priority 100
800dc272 dd= 0 locks held 0    write locks 0    pid/thread
7123/1239370048 priority 100
800dc273 dd= 0 locks held 0    write locks 0    pid/thread
7123/1247762752 priority 100
800dc274 dd= 0 locks held 0    write locks 0    pid/thread
7123/1256155456 priority 100
800dc275 dd= 0 locks held 0    write locks 0    pid/thread
7123/1264548160 priority 100
800dc276 dd= 0 locks held 0    write locks 0    pid/thread
7123/1272940864 priority 100
800dc277 dd= 0 locks held 0    write locks 0    pid/thread
7123/1289726272 priority 100
800dc278 dd= 0 locks held 0    write locks 0    pid/thread
7123/1281333568 priority 100
800dc279 dd= 0 locks held 0    write locks 0    pid/thread
7123/1298118976 priority 100
800dc27a dd= 0 locks held 0    write locks 0    pid/thread
7123/1306511680 priority 100
800dc27b dd= 0 locks held 0    write locks 0    pid/thread
7123/1314904384 priority 100
800dc27c dd= 0 locks held 0    write locks 0    pid/thread
7123/1331689792 priority 100
800dc27d dd= 0 locks held 0    write locks 0    pid/thread
7123/1340082496 priority 100
800dc27e dd= 0 locks held 0    write locks 0    pid/thread
7123/1348475200 priority 100
800dc27f dd= 0 locks held 0    write locks 0    pid/thread
7123/1356867904 priority 100
800dc280 dd= 0 locks held 0    write locks 0    pid/thread
7123/1365260608 priority 100
800dc281 dd= 0 locks held 0    write locks 0    pid/thread
7123/1373653312 priority 100
800dc282 dd= 0 locks held 0    write locks 0    pid/thread
7123/1382046016 priority 100
800dc283 dd= 0 locks held 0    write locks 0    pid/thread
7123/1390438720 priority 100
800dc284 dd= 0 locks held 0    write locks 0    pid/thread
7123/1407224128 priority 100
800dc285 dd= 0 locks held 0    write locks 0    pid/thread
7123/1398831424 priority 100
800dc286 dd= 0 locks held 0    write locks 0    pid/thread
7123/1415616832 priority 100
800dc287 dd= 0 locks held 0    write locks 0    pid/thread
7123/1424009536 priority 100
800dc288 dd= 0 locks held 0    write locks 0    pid/thread
7123/1432402240 priority 100
800dc289 dd= 0 locks held 0    write locks 0    pid/thread
7123/1440794944 priority 100
800dc28a dd= 0 locks held 0    write locks 0    pid/thread
7123/1449187648 priority 100
800dc28b dd= 0 locks held 0    write locks 0    pid/thread
7123/1457580352 priority 100
800dc28c dd= 0 locks held 0    write locks 0    pid/thread
7123/1474365760 priority 100
800dc28d dd= 0 locks held 0    write locks 0    pid/thread
7123/1465973056 priority 100
800dc28e dd= 0 locks held 0    write locks 0    pid/thread
7123/1482758464 priority 100
800dc28e READ          1 WAIT    dn2id.bdb                 page      11658
800dc28f dd= 0 locks held 0    write locks 0    pid/thread
7123/1491151168 priority 100
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker   Mode      Count Status  ----------------- Object ---------------
800dc266 READ          2 HELD    objectClass.bdb           page        115
800dc266 WRITE         2 HELD    objectClass.bdb           page        115

800dc266 WRITE         3 HELD    id2entry.bdb              page      14475

       4 READ          1 HELD    entryUUID.bdb             handle        0

       1 READ          1 HELD    id2entry.bdb              handle        0

       b READ          1 HELD    uidNumber.bdb             handle        0

80000005 READ          1 HELD    0x3dc78 len:   9 data: a2b007000000000000
800dc266 WRITE         1 WAIT    0x3dc78 len:   9 data: a2b007000000000000

       e READ          1 HELD    sambaPrimaryGroupSID.bdb  handle        0

       2 READ          1 HELD    dn2id.bdb                 handle        0

800dc266 READ          1 HELD    entryUUID.bdb             page       2839
800dc266 WRITE         2 HELD    entryUUID.bdb             page       2839

       a READ          1 HELD    uid.bdb                   handle        0

       6 READ          1 HELD    cn.bdb                    handle        0

800dc266 READ          1 HELD    0x36718 len:   9 data: 9cb007000000000000

800dc266 READ          1 HELD    modifyTimestamp.bdb       page       3168
800dc266 WRITE         2 HELD    modifyTimestamp.bdb       page       3168

       d READ          1 HELD    sambaSID.bdb              handle        0

       c READ          1 HELD    gidNumber.bdb             handle        0

       5 READ          1 HELD    modifyTimestamp.bdb       handle        0

       8 READ          1 HELD    sn.bdb                    handle        0

       7 READ          1 HELD    givenName.bdb             handle        0

800dc266 READ          1 HELD    objectClass.bdb           page          3

       9 READ          1 HELD    objectClass.bdb           handle        0

800dc266 READ          2 HELD    dn2id.bdb                 page      11658
800dc266 WRITE         4 HELD    dn2id.bdb                 page      11658
800dc28e READ          1 WAIT    dn2id.bdb                 page      11658

changed state Feedback to Open

Hello,

Sorry for the delay in response.  I believe this issue has been resolved 
since the 2.4.25 release.  If you can reproduce this issue with the current 
OpenLDAP release, please follow up.

Thanks,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

Should be fixed in later releases.

changed notes
changed state Open to Closed
moved from Incoming to Software Bugs