[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Speed of slapadd for BDB on a Linux machine

Ralf Narozny wrote: 
Hello,

I'm trying to migrate about 19 million entries from OpenLDAP 2.0 to the new 2.3. Conversion and such things are done, but inserting the data takes days.

I hope someone can point me to some helpful direction, because 22 Entries/sec is not too good, especially not for real life use...

I'm using the following to insert the data
slapadd -q -v -c

My DB_CONFIG looks like this:

set_cachesize 1 524288000 1
set_lg_regionmax 262144
set_lg_bsize 2097152


Our OpenLDAP version is 2.3.27.

The speed is like following (took that from the output of my insertion script):

The gradual slowdown you're seeing indicates that the BDB cache is too small.

Since you have two CPUs you should set "tool-threads 2" in your slapd.conf.

You should not use a presence index on objectClass, just equality. Every object is required to have an objectClass attribute, so presence indexing on it is just a waste of time.

As I noted just a few hours ago on this list
http://www.openldap.org/lists/openldap-software/200611/msg00051.html

you're unlikely to be able to configure sufficient cache to get good performance with a 19 million entry DB on a 32 bit server. With only ~3GB of working memory available to a process, your performance here is going to be limited to the speed of your disk drives.


Finished run at: 1162437948
Run took 620309 seconds for 18914119 entries
Avg.: 30,4914 entries/second
***********************************************************


We are using the following machine:

Linux ldaprep4 2.6.15.3 #1 SMP Mon Feb 13 09:18:43 CET 2006 i686 GNU/Linux
MemTotal:      5975412 kB
SwapTotal:     2150152 kB
2 * Intel(R) Pentium(R) III CPU family      1133MHz


The slapd.conf is as follows (Don't mind the /tmp as path, I changed that ;-)):

-------------------------------------------------------------------------------------------------- 

include         /tmp/etc/openldap/schema/core.schema
include         /tmp/etc/openldap/schema/freenet.schema
pidfile         /tmp/var/ldap/run/slapd.pid
argsfile        /tmp/var/ldap/run/slapd.args
modulepath      /tmp/lib
moduleload      back_bdb.la
access          to * by * write
loglevel        0
sizelimit       10000
timelimit       3600
cachesize       1000000
backend         bdb

#######################################################################
# BDB database definitions
#######################################################################

# first database definition & config directives
database        bdb

directory       /var/lib/ldap/
replogfile      /tmp/log/replica.log

rootdn          "cn=root,o=....."
rootpw          .....

suffix          "o=....."

#replica uri=ldap://ldaprep1:389 binddn="cn=root,o=..." bindmethod=simple credentials=...
#replica uri=ldap://ldaprep2:389 binddn="cn=root,o=..." bindmethod=simple credentials=...
#replica uri=ldap://ldaprep3:389 binddn="cn=root,o=..." bindmethod=simple credentials=...

#attribute homeDirectory ces
#attribute folderName ces
#attribute locked ces
#
index cid pres,eq
index cn pres,eq,sub
index objectClass pres,eq
index folderName pres,eq
index locked pres,eq
-------------------------------------------------------------------------------------------------- 



Thans in advance,
Ralf



--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/