Re: slapadd *very* slow: tuning advice?

[Nick Urbanik <nicku@nicku.org>]

Dear Quanah,

Thank you for your generous help.

On 21/08/12 07:32 -0700, Quanah Gibson-Mount wrote:
> --On Tuesday, August 21, 2012 4:03 PM +1000 Nick Urbanik 
> <nick.urbanik@optusnet.com.au> wrote:
>
> > Dear Folks,
> >
> > I'm upgrading a cluster of OpenLDAP servers from 2.3.43-25.el5 to
> > 2.4.32 with BDB 4.8.30 on CentOS 5, x86_64, on HP BL460cG6 blades
> > with two 4-core CPUs, and 12GB RAM.  These are slaves, and have
> > eleven trees on them.  I have dumped and restored six of the LDAP
> > databases in reasonable time, but the seventh is taking a long time.
> > Here are the sizes of the slapcatted LDIF files:
>
> > It's the 2.6G LDIF file that's taking the time to slapadd:
> > -############          63.34% eta 02h09m elapsed       03h43m30s spd
> > 2.7 k/s 	
> >
> > As you can see, it has slowed to a crawl.
> >
> > # cat DB_CONFIG set_flags DB_LOG_AUTOREMOVE
> > set_cachesize 0 286162472 0
> >
> > # egrep 'tool-threads|cachesize' /etc/openldap/slapd.conf
> > tool-threads 8
>
> cachesize and idlcachesize have no bearing on slapadd.  Only
> toolthreads does.

Thank you.

> You do not mention if you are using slapadd -q or not.

Yes, this is with the options -q -w.

> > Any suggestions on how to optimise this a little more towards slapadd?
>
> If the LDIF file is 2.6GB, I fully assume the BDB database will be
> significantly larger than 2.6 GB.  Probably closer to 6 or 8 GB,
> depending on your indexing.  What you really need to look at from
> your 2.3 installation is the total size of that database (du -c -h
> *.bdb).

In the biggest LDAP directory on the (now finally converted) machine:
6.1G

Over all eleven LDAP directories on that same machine:
13G

> Right now, you only have a 272MB BDB cache set... which is definitely 
> going to be too small. I would change DB_CONFIG For that database to 
> be something like:
>
> set_cachesize 6 0 0

Okay; with 12G total RAM, and 6.9G of other directories, am I likely
to overcommit memory?

> I would strongly advise examining using a shared memory key for that
> DB as well.

Okay: I could create that with a command like
ipcmk -M 16777216

But what size would be appropriate with 12G of RAM, eleven LDAP
databases, and 13G of bdb files?

> Finally, look at using MDB, which has zero tuning bits required, and 
> only requires a tool-threads of 2.

So with eight cores (and 16 hyperthreaded siblings) I am better off
specifying tool-threads 2?

Okay, my next one I'll convert to mdb and see how that behaves.

slapd-mdb(5) seems to indicate that mdb needs no shared memory
configuration.   Is that correct?
--
Nick Urbanik             http://nicku.org           nicku@nicku.org
GPG: 7FFA CDC7 5A77 0558 DC7A 790A 16DF EC5B BB9D 2C24 ID: BB9D2C24