Hello!
I am trying to use openLDAP to hold a small but continuously rebuilt
database with a hdb backend. Basically I build a directory under a
temporary node and move it into place when its ready (hence the hdb, I
want to move the hole tree into place in one go). I build a new
directory, move out the current to an other temporary node and move in
the new one. Lastly I delete the defunct tree and start over building a
new tree. In short the idea is to always (except I suppose between
moving the tree out and the new in, but I don't see any solution for
that) have a complete tree in place while continuously trying to have it
updated.
This thing works well for a couple of hours on the machine I am running
it (PIII 1 cpu, 1000MHz, 512 Mb ram, linux 2.6 kernel), but then slows
down by a factor 10-20.
Why is this and what can I do to stop it? (easy to ask...)
free shows:
# free
total used free shared buffers cached
Mem: 508104 501992 6112 0 31268 332556
-/+ buffers/cache: 138168 369936
Swap: 2104504 2904 2101600
This isn't brilliant of course but AFAIU not catastrophic either. I have
about the same when it isn't slowed down. vmstat with a sampling rate of
a few seconds show no swapping before or after slapd slows down.
top shows that slapd and the script populating it runs at about 2-3%
each and not much cpu consumption apart from that (consistent with a
system that slows down a factor 20 I guess). The script uses Net::LDAP
in perl (over local socket) so no external clients are invoked.
The really puzzling bit is that if a shut down slapd and the "directory
builder" - thus reclaiming memory and filedescriptors and such to the
system - and then restart them I almost immediately get the same slow
down. In fact, the time it takes to get the computer to slow down after
firing up slapd seems proportional to how long I let it "rest".
I have tried all sorts of things to analyze this and finally decided to
profile slapd. I rebuilt it with -g -pg in CFLAGS and --enable-debug to
configure (actually I have used that switch all along). I also
discovered that I had to replace 'strip = -s' with 'strip =' in all
makefiles even though --enable-debug was given (is this intentional or a
bug in configure?). Finally I had to get the gprof-helper (and confirm
that it was used) by Hocevar/Jönsson to be able to profile threaded
applications. The result doesn't however tell me much. The slapd process
seems to spend most (70-80%) of its time in the "at_next" routine.
Info about system:
I am running v2.3.24 of slapd.
built with:
./configure --program-prefix=jj4 --with-threads=yes --enable-dynamic
--enable-debug --enable-crypt --enable-lmpasswd --enable-spasswd --enable-
modules --enable-backends=mod --enable-sql=no --enable-ldap=mod
--enable-meta=mod --enable-monitor=mod --enable-null=mod
--enable-perl=no --ena
ble-relay=mod --enable-shell=mod --enable-overlays=mod
--enable-denyop=mod --enable-dyngroup=mod --enable-dynlist=mod
--enable-lastmod=mod --enable-proxycache=mod --enable-retcode=mod
--enable-rwm=mod --enable-dependency-tracking
(lots of modules are built but only hbd-backend is actually loaded when
I'm running)
These are the relevant and nonsensitive parts of the slapd.conf:
---
sizelimit 1000000
moduleload back_hdb.la
database hdb
suffix *removed*
rootdn *removed*
rootpw *removed*
directory /usr/local/lis/var/db
checkpoint 512 5
dirtyread
dbconfig set_cachesize 0 16777216 8
dbconfig set_lg_regionmax 262144
dbconfig set_lg_bsize 2097152
dbconfig set_lg_max 16777216
dbconfig set_flags DB_LOG_AUTOREMOVE
index objectClass eq
---
I use dirtyread because I want to be able to read while I'm writing
(which is almost always) while an occasional bad read is acceptable (it
should anyway be very rare since I don't do the modifications in the
"current" tree where I read).
Thanks in advance
Johan Jönemo