[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapd syncrepl consumer having permanent high CPU load

To: Howard Chu <hyc@symas.com>
Subject: Re: slapd syncrepl consumer having permanent high CPU load
From: Rodrigo Costa <rlvcosta@yahoo.com>
Date: Tue, 24 Mar 2009 12:22:23 -0700 (PDT)
Cc: openldap-software@openldap.org, John Morrissey <jwm@horde.net>

Howard and all,

I made more tests and looks like problem persists. I saw some changes but only in the memory consumption in "consumer(slave)" syncrepl.

Let me try to explain better. I have a pair of provider/consumer machines where one machine will always receive all read/writes and the other is just for High Availability(HA) purposes, so it is better have the more close as possible the DBs.

I start the provider(master) and then just after start the consumer(slave). The configuration doesn't appear to have problems since I have in my configuration 2 DBs, CONTENT and INDEX, and I see consumer doing 2 searches in these DBs when started(this is ok).

After this both consumer and provider CPU usage increases so as memory allocation by slapd process. After the HEAD changes the memory consumption in consumer increases in a much more fast rate, something like 10:1. In this way to reproduce the issue I needed to reduce the dncachesize directive in consumer to 1/10 of the provider value, or from 4,000,000 to 400,000. This avoid the process to consume all memory before the issue arises.

Let me try to summarize :

1) Start provider(mater) slapd process;
2) Start consumer(slave) slapd process;
3) Monitor memory and CPU usage in both provider and consumer;
4) Make sometimes a monitor check to see the cache information;
5) Before cache is full in provider(master) I made a gdb debug to check the consumer(slave) process threads;
6) Wait until the consumer(slave) process starts to use around 200% CPU and then collect again a gdb debug;
7) Wait a little more until the provider(master) CPU usage becomes 0% and then see that consumer(slave) CPU stay stable in 200%. Collect a gdb debug.
8) Wait some more time just for more gdb debug to see if something changed.

I re-compile the HEAD with GDB symbols for debugging. In this way I created the file attached where more than once I collect the debug information from the consumer slapd(includes the syncrepl thread). Please see file attached for details.

The item 7) is the issue I think is happening. The synchronization never ends, the responsiveness from consumer(slave) to queries is very slow, CPU usage becomes fixed in 200%, and then the logic appears never be working as expected, or in the end never synchronizing.

In the end appears that syncrepl still with some issue to synchronize the DBs.

Regards,

Rodrigo.

--- On Thu, 3/19/09, Howard Chu <hyc@symas.com> wrote:

> From: Howard Chu <hyc@symas.com>
> Subject: Re: slapd syncrepl consumer having permanent high CPU load
> To: rlvcosta@yahoo.com
> Cc: openldap-software@openldap.org, "John Morrissey" <jwm@horde.net>
> Date: Thursday, March 19, 2009, 2:04 PM
> Rodrigo Costa wrote:
> >
> > Folks,
> >
> > I was preparing openLDAP with GDB symbols but looks
> like the issue was
> identified and solved in HEAD. Just to identify this issue;
> was created any
> sort of ITS for verification in a new load?
> 
> No, the further work was just associated with ITS#5860.
> 
> > Sorry my late response but my baby daughter just born
> last week and I was
> having some work at home.
> 
> Congratulations!
> 
> > I will give a try in the HEAD load.
> 
> Try RE24 now, that's the current release candidate.
> >
> > Best Regards,
> >
> > Rodrigo.
> >
> > PS->  Just some link from my daughter
> > http://sites.google.com/site/lauramenina/laura_english
> >
> > --- On Wed, 3/18/09, Howard Chu<hyc@symas.com> 
> wrote:
> >
> >> From: Howard Chu<hyc@symas.com>
> >> Subject: Re: slapd syncrepl consumer having
> permanent high CPU load
> >> To: "John Morrissey"<jwm@horde.net>
> >> Cc: openldap-software@openldap.org
> >> Date: Wednesday, March 18, 2009, 5:21 AM
> >> John Morrissey wrote:
> >>> After ~16h uptime, slapd with this BDB had
> increased
> >> its DN cache to ~250k
> >>> entries after it previously appeared stable at
> the
> >> configured 20k entries,
> >>> and its entry cache had ballooned to ~480k
> entries.
> >> Its RSS was about 3.6GB
> >>> at this point, with a BDB cache size of 2GB.
> >>
> >> I was finally able to reproduce this (took several
> hours of
> >> searches. Fortunately I was at a St. Pat's party
> so I didn't
> >> have to wait around, just got home in time to see
> it start
> >> going bad...). A fix is now in HEAD.
> >>
> >> (And now we'll see if Guinness is Good For Your
> Code... ;)
> >> --   -- Howard Chu
> >>    CTO, Symas Corp.
> >>     http://www.symas.com
> >>    Director, Highland Sun 
>    http://highlandsun.com/hyc/
> >>    Chief Architect, OpenLDAP  http://www.openldap.org/project/
> >>
> >
> >
> >
> >
> >
> 
> 
> -- 
>    -- Howard Chu
>    CTO, Symas Corp.     
>      http://www.symas.com
>    Director, Highland Sun 
>    http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP  http://www.openldap.org/project/
>

[First moment after some minutes the consumer(slave) slapd start the search]
Thread 5 (Thread -1498268784 (LWP 22749)):
#0  0x0033f402 in __kernel_vsyscall ()
#1  0x004944d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x00493e5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502467184 (LWP 22750)):
#0  0x0033f402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00380230 in ldap_pvt_thread_cond_wait (cond=0x9da920c, mutex=0x9da91f4)
    at thr_posix.c:277
#3  0x0037eeeb in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x00493e5e in clone () from /lib/libc.so.6

Thread 3 (Thread -1506665584 (LWP 22751)):
#0  0x0033f402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00380230 in ldap_pvt_thread_cond_wait (cond=0x9da920c, mutex=0x9da91f4)
    at thr_posix.c:277
#3  0x0037eeeb in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x00493e5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1858077808 (LWP 22876)):
#0  0x005e6420 in bdb_cache_lru_purge (bdb=0x9df0308) at cache.c:716
#1  0x005e6d30 in bdb_cache_find_id (op=0x913fe774, tid=0x9eb7288, id=405592, 
    eip=0x9133e66c, flag=0, lock=0x9133e650) at cache.c:1053
#2  0x005d5df5 in bdb_search (op=0x913fe774, rs=0x913fe84c) at search.c:705
#3  0x080eb977 in cancel_extop ()
#4  0x080e71b1 in cancel_extop ()
#5  0x080e8c34 in cancel_extop ()
#6  0x0807752e in connection_client_stop ()
#7  0x0037ef5e in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:663
#8  0x0030b45b in start_thread () from /lib/libpthread.so.0
#9  0x00493e5e in clone () from /lib/libc.so.6

Thread 1 (Thread -1208891632 (LWP 22748)):
#0  0x0033f402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x0038012b in ldap_pvt_thread_join (thread=2796698512, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()

Thread 2 (Thread -1858077808 (LWP 22876)):
#0  0x005e6420 in bdb_cache_lru_purge (bdb=0x9df0308) at cache.c:716
#1  0x005e6d30 in bdb_cache_find_id (op=0x913fe774, tid=0x9eb7288, id=405592, 
    eip=0x9133e66c, flag=0, lock=0x9133e650) at cache.c:1053
#2  0x005d5df5 in bdb_search (op=0x913fe774, rs=0x913fe84c) at search.c:705
#3  0x080eb977 in cancel_extop ()
#4  0x080e71b1 in cancel_extop ()
#5  0x080e8c34 in cancel_extop ()
#6  0x0807752e in connection_client_stop ()
#7  0x0037ef5e in ldap_int_thread_pool_wrapper (xpool=0x9da91f0) at tpool.c:663
#8  0x0030b45b in start_thread () from /lib/libpthread.so.0
#9  0x00493e5e in clone () from /lib/libc.so.6

Thread 1 (Thread -1208891632 (LWP 22748)):
#0  0x0033f402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x0038012b in ldap_pvt_thread_join (thread=2796698512, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()

#0  0x0033f402 in __kernel_vsyscall ()



---------------------------------------------
[Second moment after more time passed the consumer(slave) slapd has around 188-200% CPU usage and search appears to continue since provider(master) slapd process continues to have CPU in use]
Thread 5 (Thread -1497973872 (LWP 28856)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x005204d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x0051fe5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502172272 (LWP 28857)):
#0  0x0019db2c in __lock_get_internal () from /usr/lib/libdb-4.7.so
#1  0xa71677a0 in ?? ()
#2  0x08a0c5e8 in ?? ()
#3  0x08a0c1e8 in ?? ()
#4  0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so
#5  0x00000000 in ?? ()

Thread 3 (Thread -1506370672 (LWP 28858)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x001e0b79 in __env_alloc () from /usr/lib/libdb-4.7.so
#1  0xb63221b4 in ?? ()
#2  0xafe6c000 in ?? ()
#3  0x0000000d in ?? ()
#4  0x00000000 in ?? ()

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()
#0  0x0044e402 in __kernel_vsyscall ()



------------------------------------------------------------------------------
[Third moment after more time passed the consumer(slave) slapd has around 188-200% CPU usage, and similar to second moment and search appears to continue since provider(master) slapd process continues to have CPU in use]
Thread 5 (Thread -1497973872 (LWP 28856)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x005204d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x0051fe5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502172272 (LWP 28857)):
#0  0x0019c56c in __lock_getobj () from /usr/lib/libdb-4.7.so
#1  0x0000023c in ?? ()
#2  0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0
#3  0x00000125 in ?? ()
#4  0x002148d2 in __os_id () from /usr/lib/libdb-4.7.so
#5  0x00118365 in __db_tas_mutex_lock () from /usr/lib/libdb-4.7.so
#6  0xa0a0bbf8 in ?? ()
#7  0xa66a953c in ?? ()
#8  0xaf1a80c0 in ?? ()
#9  0x0023c0c0 in ?? () from /usr/lib/libdb-4.7.so
#10 0x08a0c5e8 in ?? ()
#11 0xa66a9428 in ?? ()
#12 0xa71899e0 in ?? ()
#13 0x0019d7d4 in __lock_get_internal () from /usr/lib/libdb-4.7.so
#14 0xa714ec50 in ?? ()
#15 0x08a0c5e8 in ?? ()
#16 0xa714f648 in ?? ()
#17 0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so
#18 0x00000000 in ?? ()

Thread 3 (Thread -1506370672 (LWP 28858)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x004b9712 in malloc () from /lib/libc.so.6
#1  0x00a3a822 in ber_memalloc_x (s=8, ctx=0x0) at memory.c:226
#2  0x00a3b290 in ber_strndup_x (s=0x36b5d2ce "domain,c=fr", l=7, ctx=0x0)
    at memory.c:675
#3  0x00b73081 in str2strval (str=0x36b5d2ce "domain,c=fr", stoplen=12, 
    val=0x907fe674, next=0x907fe688, flags=0, retFlags=0x907fe684, ctx=0x0)
    at getdn.c:1531
#4  0x00b727b9 in ldap_bv2rdn_x (bv=0x907fe71c, rdn=0x907fe7ac, 
    n_in=0x907fe7b0, flags=0, ctx=0x0) at getdn.c:1250
#5  0x00b719d3 in ldap_bv2dn_x (bvin=0x36bc2f98, dn=0x907fe82c, flags=0, 
    ctx=0x0) at getdn.c:787
#6  0x08090386 in dnPretty ()
#7  0x08099af5 in ordered_value_pretty ()
#8  0x0809497d in slap_mods_check ()
#9  0x080eaa75 in cancel_extop ()
#10 0x080e7159 in cancel_extop ()
#11 0x080e8c34 in cancel_extop ()
#12 0x0807752e in connection_client_stop ()
#13 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663
#14 0x0030b45b in start_thread () from /lib/libpthread.so.0
#15 0x0051fe5e in clone () from /lib/libc.so.6

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()
#0  0x0044e402 in __kernel_vsyscall ()


----------------------------------------------------------------------------
[Forth moment after just a little more time passed the consumer(slave) slapd has exactly 200% CPU usage and search appears to finish since provider(master) slapd process has 0-2% CPU in use by top]
Thread 5 (Thread -1497973872 (LWP 28856)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x005204d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x0051fe5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502172272 (LWP 28857)):
#0  0x08108a4a in UTF8bvnormalize ()
#1  0x080bb416 in nameUIDPretty ()
#2  0x0808fae3 in rdnValidate ()
#3  0x0808fbdf in rdnValidate ()
#4  0x0808fdcb in dnNormalize ()
#5  0x08099d9a in ordered_value_normalize ()
#6  0x08094b7f in slap_mods_check ()
#7  0x080eaa75 in cancel_extop ()
#8  0x080e7159 in cancel_extop ()
#9  0x080e8c34 in cancel_extop ()
#10 0x0807752e in connection_client_stop ()
#11 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663
#12 0x0030b45b in start_thread () from /lib/libpthread.so.0
#13 0x0051fe5e in clone () from /lib/libc.so.6

Thread 3 (Thread -1506370672 (LWP 28858)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x0019e316 in __lock_get_internal () from /usr/lib/libdb-4.7.so
#1  0xafb0ce38 in ?? ()
#2  0x08a0a098 in ?? ()
#3  0x08a09db8 in ?? ()
#4  0x0000057c in ?? ()
#5  0x000004a4 in ?? ()
#6  0xafaf40c0 in ?? ()
#7  0x00000000 in ?? ()

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x0019e316 in __lock_get_internal () from /usr/lib/libdb-4.7.so
#1  0xafb0ce38 in ?? ()
#2  0x08a0a098 in ?? ()
#3  0x08a09db8 in ?? ()
#4  0x0000057c in ?? ()
#5  0x000004a4 in ?? ()
#6  0xafaf40c0 in ?? ()
#7  0x00000000 in ?? ()

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()
#0  0x0044e402 in __kernel_vsyscall ()


-----------------------------------------------------------------------------
[Fifth moment just one more gdb information in a posterior moment from forfth moment - still with 200% CPU in consumer(slave)]
Thread 5 (Thread -1497973872 (LWP 28856)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x005204d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x0051fe5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502172272 (LWP 28857)):
#0  0x0019c636 in __lock_getobj () from /usr/lib/libdb-4.7.so
#1  0x0000023c in ?? ()
#2  0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0
#3  0x000003c7 in ?? ()
#4  0xa66a9428 in ?? ()
#5  0x08a0c5e8 in ?? ()
#6  0x00000000 in ?? ()

Thread 3 (Thread -1506370672 (LWP 28858)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x0019dae8 in __lock_get_internal () from /usr/lib/libdb-4.7.so
#1  0xafb09800 in ?? ()
#2  0x08a0a098 in ?? ()
#3  0x08a09db8 in ?? ()
#4  0x0019cf28 in __lock_put_internal () from /usr/lib/libdb-4.7.so
#5  0x00000000 in ?? ()

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()
#0  0x0044e402 in __kernel_vsyscall ()


-----------------------------------------------------------------------------
[Sixth moment just one more gdb information in a posterior moment from forfth moment - still with 200% CPU in consumer(slave) and 0-2% CPU usage in provider(master)]
Thread 5 (Thread -1497973872 (LWP 28856)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x005204d6 in epoll_wait () from /lib/libc.so.6
#2  0x0807334f in slapd_daemon_destroy ()
#3  0x0030b45b in start_thread () from /lib/libpthread.so.0
#4  0x0051fe5e in clone () from /lib/libc.so.6

Thread 4 (Thread -1502172272 (LWP 28857)):
#0  0x0030e4ea in __pthread_mutex_unlock_usercnt () from /lib/libpthread.so.0
#1  0x0030e880 in pthread_mutex_unlock () from /lib/libpthread.so.0
#2  0x00b5e2e7 in ldap_pvt_thread_mutex_unlock (mutex=0x132ce5c0)
    at thr_posix.c:308
#3  0x003ef608 in bdb_cache_lru_purge (bdb=0x89a8308) at cache.c:785
#4  0x003efd30 in bdb_cache_find_id (op=0xa6769774, tid=0x8a2c900, id=620396, 
    eip=0xa66a966c, flag=0, lock=0xa66a9650) at cache.c:1053
#5  0x003dedf5 in bdb_search (op=0xa6769774, rs=0xa676984c) at search.c:705
#6  0x080eb977 in cancel_extop ()
#7  0x080e71b1 in cancel_extop ()
#8  0x080e8c34 in cancel_extop ()
#9  0x0807752e in connection_client_stop ()
#10 0x00b5cf5e in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:663
#11 0x0030b45b in start_thread () from /lib/libpthread.so.0
#12 0x0051fe5e in clone () from /lib/libc.so.6

Thread 3 (Thread -1506370672 (LWP 28858)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030f256 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x00b5e230 in ldap_pvt_thread_cond_wait (cond=0x896120c, mutex=0x89611f4)
    at thr_posix.c:277
#3  0x00b5ceeb in ldap_int_thread_pool_wrapper (xpool=0x89611f0) at tpool.c:654
#4  0x0030b45b in start_thread () from /lib/libpthread.so.0
#5  0x0051fe5e in clone () from /lib/libc.so.6

Thread 2 (Thread -1870660720 (LWP 29107)):
#0  0x0019c5a1 in __lock_getobj () from /usr/lib/libdb-4.7.so
#1  0x000006b4 in ?? ()
#2  0x00a3fdac in ?? () from /usr/lib/liblber-2-devel.so.0
#3  0x000000d4 in ?? ()
#4  0x9073e428 in ?? ()
#5  0x08a0a098 in ?? ()
#6  0xb7ecf0b4 in ?? ()
#7  0xafaf40c0 in ?? ()
#8  0x08a09db8 in ?? ()
#9  0x00000000 in ?? ()

Thread 1 (Thread -1208596720 (LWP 28855)):
#0  0x0044e402 in __kernel_vsyscall ()
#1  0x0030c557 in pthread_join () from /lib/libpthread.so.0
#2  0x00b5e12b in ldap_pvt_thread_join (thread=2796993424, thread_return=0x0)
    at thr_posix.c:197
#3  0x08074074 in slapd_daemon ()
#4  0x08058318 in main ()
#0  0x0044e402 in __kernel_vsyscall ()

-----------------------------------------------------------------------------



Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.3%sy,  0.0%ni, 99.1%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12217276k total, 11809900k used,   407376k free,    20144k buffers
Swap:  4736848k total,       76k used,  4736772k free, 10657464k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
23938 ldap      17   0 1320m 1.2g 261m S    0 10.6  23:01.42 slapd





Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s): 50.0%us,  0.4%sy,  0.0%ni, 49.4%id,  0.1%wa,  0.1%hi,  0.1%si,  0.0%st
Mem:  12217276k total, 11775852k used,   441424k free,   181572k buffers
Swap:  4149328k total,       76k used,  4149252k free,  8831924k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
28855 ldap      18   0 2844m 2.7g 261m S  200 23.2  62:59.07 slapd 



This openldap was a HEAD compilation as :

env CFLAGS="-ggdb" \
./configure --enable-syslog --enable-proctitle --with-cyrus-sasl=yes --with-cyrus-sasl=yes \
--with-threads --with-tls --enable-slapd --enable-cleartext --enable-crypt --enable-spasswd \
--enable-dynamic --enable-dynacl --enable-modules --enable-rewrite --enable-backends=mod \
--enable-bdb=mod --enable-monitor=mod --enable-relay=mod --enable-hdb=mod --enable-ndb=mod \
--disable-ldbm --enable-shared --enable-dyngroup=mod --enable-ndb=no \
--enable-overlays=mod --enable-seqmod=mod --enable-syncprov=mod \
--sysconfdir=/etc --prefix=/usr --mandir=/usr/man

make depend

make

make install

My machines are(see as OS and Kernel version) :

Linux brtldp12 2.6.18-92.1.22.el5PAE #1 SMP Tue Dec 16 12:36:25 EST 2008 i686 i686 i386 GNU/Linux

Follow-Ups:
- Re: slapd syncrepl consumer having permanent high CPU load
  - From: John Morrissey <jwm@horde.net>

Prev by Date: Re: slapd syncrepl consumer having permanent high CPU load
Next by Date: Re: Xenserver 5 + Centos 5.2 x64 DomU + DB 4.7.25 + Openldap 2.4.13 or 2.4.15 do not work
Index(es):
- Chronological
- Thread