[Date Prev][Date Next]
- To: OpenLDAP Devel <openldap-devel@OpenLDAP.org>
- Subject: slapadd/slapindex
- From: Howard Chu <firstname.lastname@example.org>
- Date: Tue, 25 Oct 2005 01:53:16 -0700
- User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9a1) Gecko/20051020 SeaMonkey/1.1a
Some observations regarding slapadd performance... The ideal is to have
enough memory to configure a BDB cache large enough for all of the
database files. Failing that, it's best to run slapadd and slapindex
For my test database with 360MB input LDIF and 285,000 entries and 15
indexed attributes, using a 512MB BDB cache, slapadd -q with indexing
took 1 hour 20 minutes.
With the IDL caching patch in HEAD, and IDL cachesize at 50,000, this
dropped to 1 hour even.
Running slapadd -q with no indexing took only 1 minute 15 seconds.
The resulting id2entry database is about 800MB; with all indexing the
total size is around 2.1GB.
Running slapindex with this BDB environment is pretty slow. But, by
setting BDB to mmap files of 800MB or less, and deleting the environment
so that id2entry is mmap'd directly instead of being double-buffered
through the BDB cache, the slapindex -q time drops to 26 minutes without
IDL caching, and 20 minutes with caching.
There's a lot of disk waits toward the end of the run; BDB is flushing
pages out of the cache and re-reading older pages, etc. Bumping up the
BDB cache to 768MB reduces this quite a bit and the slapindex time drops
to 15 minutes. This behavior is about as expected - you need at least
about 50% of the total size in BDB cache. Considering there's about
1.3GB of index data here, the 512MB cache was too small to keep all of
the Btree internal pages in cache, so there was a lot of thrashing.
Bumping up to 768MB was enough to let the cache work without thrashing.
Setting the BDB cache to 1.25GB, slapadd with indexing took 30 minutes.
Lowering the cache priority of the id2entry DB pages with this same
setup brought it down to 21:52 minutes. Strangely, increasing the BDB
cache to 1.38GB caused more page thrashing and a slower load than for
1.25GB. I have no idea why that would be...
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/