Re: slapadding a large bdb database

[John Morrissey <jwm@horde.net>]

First off, thanks for all your help, Howard. I really appreciate it.

On Wed, Jul 10, 2002 at 01:02:24PM -0700, Howard Chu wrote:
% Look again. http://www.sleepycat.com/docs/index.html Under "Additional
% Documentation"
% is a link to "Programmer's Tutorial and Reference Guide" (which is
% http://www.sleepycat.com/docs/reftoc.html) In that document, start with
% section 3 "Access Method Configuration", specifically 3.4.2 "Selecting a
% cache size".
% 
% Also read section 5.9 "Access Method Tuning".
% 
% Then read section 9 "Berkeley DB Transactional Data Store Applications" with
% special attention to section 9.21 "Transaction Tuning."
% 
% To actually set most of these options, read section 7 "The Berkeley DB
% Environment". While some of the BDB library options can be set in slapd.conf,
% all of them can be set thru the BDB environment.

I've given all this a thorough read. Here's what I've got in my DB_CONFIG
now:

set_cachesize 0 500000000 2
set_flags DB_TXN_NOSYNC
set_lg_regionmax 500000
set_tx_max 500000

slapadd speed has improved considerably with DB_TXN_NOSYNC, which takes care
of one problem.

However, the other remains. To recap, I've slapcatted an existing ldbm
database (~450k entries) and I'm slapadding it as a bdb database. slapadd
fails at entry 65545 with:

slapadd: could not add entry dn="uid=foo,dc=example,dc=com" (line=29):
txn_aborted! Cannot allocate memory (12)

At this point, /var/lib/ldap/ is about 950MB, about 770MB of which is BDB
transaction logs (log.*).

If it helps, here's the cache-summary output of db_stat -m:

596MB 47KB 576B Total cache size (625000000 bytes).
2       Number of caches.
312M    Pool individual cache size.
116     Requested pages found in the cache (60%).
0       Requested pages mapped into the process' address space.
76      Requested pages not found in the cache.
0       Pages created in the cache.
76      Pages read into the cache.
66      Pages written from the cache to the backing file.
0       Clean pages forced from the cache.
0       Dirty pages forced from the cache.
0       Dirty buffers written by trickle-sync thread.
76      Current clean buffer count.
0       Current dirty buffer count.
65542   Number of hash buckets used for page location.
116     Total number of times hash chains searched for a page.
1       The longest hash chain searched for a page.
116     Total number of hash buckets examined for page location.
1470    The number of region locks granted without waiting.
0       The number of region locks granted after waiting.

I'm at a loss as to what else to tune; this machine's got 768MB of RAM, of
which it's using all but ~100MB for the BDB cache. The log region is several
multiples of its default size (which, according to the docs, I shouldn't
have to increase in my situation, anyway). The docs don't say what the
default for tx_max is, but I doubt I'm hitting on a 500k simultaneous
transaction limit.

The rest of the BDB docs seem to apply more to performance tuning (BDB block
size, lookup method [hash vs. btree, etc.]) or seem related to initial BDB
creation, which I really have no control over.

I've searched everyplace I can think of and spent a lot of time reading the
BDB reference manual, but I'm still stuck. Is there something really simple
I'm missing here? BTW, this is with OpenLDAP 2.1.3.

thanks,
john
-- 
John Morrissey          _o            /\         ----  __o
jwm@horde.net        _-< \_          /  \       ----  <  \,
www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__