[Date Prev][Date Next]
Re: slapadding a large bdb database
- To: openldap-software@OpenLDAP.org
- Subject: Re: slapadding a large bdb database
- From: John Morrissey <firstname.lastname@example.org>
- Date: Fri, 2 Aug 2002 12:51:17 -0400
- Content-disposition: inline
- In-reply-to: <NMEFLNHODBAOPDKNNJALIEDHDEAA.email@example.com>
- References: <20020710181814.GA10839@boost.horde.net> <NMEFLNHODBAOPDKNNJALIEDHDEAA.firstname.lastname@example.org>
First off, thanks for all your help, Howard. I really appreciate it.
On Wed, Jul 10, 2002 at 01:02:24PM -0700, Howard Chu wrote:
% Look again. http://www.sleepycat.com/docs/index.html Under "Additional
% is a link to "Programmer's Tutorial and Reference Guide" (which is
% http://www.sleepycat.com/docs/reftoc.html) In that document, start with
% section 3 "Access Method Configuration", specifically 3.4.2 "Selecting a
% cache size".
% Also read section 5.9 "Access Method Tuning".
% Then read section 9 "Berkeley DB Transactional Data Store Applications" with
% special attention to section 9.21 "Transaction Tuning."
% To actually set most of these options, read section 7 "The Berkeley DB
% Environment". While some of the BDB library options can be set in slapd.conf,
% all of them can be set thru the BDB environment.
I've given all this a thorough read. Here's what I've got in my DB_CONFIG
set_cachesize 0 500000000 2
slapadd speed has improved considerably with DB_TXN_NOSYNC, which takes care
of one problem.
However, the other remains. To recap, I've slapcatted an existing ldbm
database (~450k entries) and I'm slapadding it as a bdb database. slapadd
fails at entry 65545 with:
slapadd: could not add entry dn="uid=foo,dc=example,dc=com" (line=29):
txn_aborted! Cannot allocate memory (12)
At this point, /var/lib/ldap/ is about 950MB, about 770MB of which is BDB
transaction logs (log.*).
If it helps, here's the cache-summary output of db_stat -m:
596MB 47KB 576B Total cache size (625000000 bytes).
2 Number of caches.
312M Pool individual cache size.
116 Requested pages found in the cache (60%).
0 Requested pages mapped into the process' address space.
76 Requested pages not found in the cache.
0 Pages created in the cache.
76 Pages read into the cache.
66 Pages written from the cache to the backing file.
0 Clean pages forced from the cache.
0 Dirty pages forced from the cache.
0 Dirty buffers written by trickle-sync thread.
76 Current clean buffer count.
0 Current dirty buffer count.
65542 Number of hash buckets used for page location.
116 Total number of times hash chains searched for a page.
1 The longest hash chain searched for a page.
116 Total number of hash buckets examined for page location.
1470 The number of region locks granted without waiting.
0 The number of region locks granted after waiting.
I'm at a loss as to what else to tune; this machine's got 768MB of RAM, of
which it's using all but ~100MB for the BDB cache. The log region is several
multiples of its default size (which, according to the docs, I shouldn't
have to increase in my situation, anyway). The docs don't say what the
default for tx_max is, but I doubt I'm hitting on a 500k simultaneous
The rest of the BDB docs seem to apply more to performance tuning (BDB block
size, lookup method [hash vs. btree, etc.]) or seem related to initial BDB
creation, which I really have no control over.
I've searched everyplace I can think of and spent a lot of time reading the
BDB reference manual, but I'm still stuck. Is there something really simple
I'm missing here? BTW, this is with OpenLDAP 2.1.3.
John Morrissey _o /\ ---- __o
email@example.com _-< \_ / \ ---- < \,
www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__