Re: slapadd *very* slow: tuning advice?

[Quanah Gibson-Mount <quanah@zimbra.com>]

--On Tuesday, August 21, 2012 4:03 PM +1000 Nick Urbanik 
<nick.urbanik@optusnet.com.au> wrote:

> Dear Folks,
>
> I'm upgrading a cluster of OpenLDAP servers from 2.3.43-25.el5 to
> 2.4.32 with BDB 4.8.30 on CentOS 5, x86_64, on HP BL460cG6 blades with
> two 4-core CPUs, and 12GB RAM.  These are slaves, and have eleven
> trees on them.  I have dumped and restored six of the LDAP databases
> in reasonable time, but the seventh is taking a long time.  Here are
> the sizes of the slapcatted LDIF files:

> It's the 2.6G LDIF file that's taking the time to slapadd:
> -############          63.34% eta 02h09m elapsed       03h43m30s spd
> 2.7 k/s 	
>
> As you can see, it has slowed to a crawl.
>
> # cat DB_CONFIG set_flags DB_LOG_AUTOREMOVE
> set_cachesize 0 286162472 0
>
> # egrep 'tool-threads|cachesize' /etc/openldap/slapd.conf
> tool-threads 8

cachesize and idlcachesize have no bearing on slapadd.  Only toolthreads 
does.

You do not mention if you are using slapadd -q or not.

> Any suggestions on how to optimise this a little more towards slapadd?

If the LDIF file is 2.6GB, I fully assume the BDB database will be 
significantly larger than 2.6 GB.  Probably closer to 6 or 8 GB, depending 
on your indexing.  What you really need to look at from your 2.3 
installation is the total size of that database (du -c -h *.bdb).

Right now, you only have a 272MB BDB cache set... which is definitely going 
to be too small. I would change DB_CONFIG For that database to be something 
like:

set_cachesize 6 0 0

I would strongly advise examining using a shared memory key for that DB as 
well.

Finally, look at using MDB, which has zero tuning bits required, and only 
requires a tool-threads of 2.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration