[Date Prev][Date Next] [Chronological] [Thread] [Top]

slapadd/slapindex

Some observations regarding slapadd performance... The ideal is to have enough memory to configure a BDB cache large enough for all of the database files. Failing that, it's best to run slapadd and slapindex separately.

For my test database with 360MB input LDIF and 285,000 entries and 15 indexed attributes, using a 512MB BDB cache, slapadd -q with indexing took 1 hour 20 minutes.
With the IDL caching patch in HEAD, and IDL cachesize at 50,000, this dropped to 1 hour even.

Running slapadd -q with no indexing took only 1 minute 15 seconds.

The resulting id2entry database is about 800MB; with all indexing the total size is around 2.1GB.

Running slapindex with this BDB environment is pretty slow. But, by setting BDB to mmap files of 800MB or less, and deleting the environment so that id2entry is mmap'd directly instead of being double-buffered through the BDB cache, the slapindex -q time drops to 26 minutes without IDL caching, and 20 minutes with caching.

There's a lot of disk waits toward the end of the run; BDB is flushing pages out of the cache and re-reading older pages, etc. Bumping up the BDB cache to 768MB reduces this quite a bit and the slapindex time drops to 15 minutes. This behavior is about as expected - you need at least about 50% of the total size in BDB cache. Considering there's about 1.3GB of index data here, the 512MB cache was too small to keep all of the Btree internal pages in cache, so there was a lot of thrashing. Bumping up to 768MB was enough to let the cache work without thrashing.

Setting the BDB cache to 1.25GB, slapadd with indexing took 30 minutes. Lowering the cache priority of the id2entry DB pages with this same setup brought it down to 21:52 minutes. Strangely, increasing the BDB cache to 1.38GB caused more page thrashing and a slower load than for 1.25GB. I have no idea why that would be...

--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/