[Date Prev][Date Next] [Chronological] [Thread] [Top]

Slow to add 1 million items



Hello all,

I've been Google'ing around and searching the archives, but I haven't quite been able to find an answer, so I wanted to ask the list.

I've been experimenting with OpenLDAP adds to see how quickly we can get data inserted into the DB.

I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39 using lmdb, both with relatively similar results.

The short version is: to insert 1 million records, it's taking about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems like a long time to me.

The insert method is to use a single big ldiff file like this:

### snip ###
dn: cn=newtest0,ou=customer,dc=test,dc=com
objectclass: inetOrgPerson
userPassword: 90sac90sudasdjcao0sdjtest
telephoneNumber: 61400000000
sn: none

dn: cn=newtest1,ou=customer,dc=test,dc=com
objectclass: inetOrgPerson
userPassword: 90sac90sudasdjcao0sdjtest
telephoneNumber: 61400000000
sn: none
### snip ###

Then add it with ldapadd:
ldapadd -x -D cn=admin,dc=test,dc=com -w test -f ./test

My configuration (for mdb in this instance) used with 2.4.39 is this:
### config ###
include         /usr/local/etc/openldap/schema/core.schema
include         /usr/local/etc/openldap/schema/cosine.schema
include         /usr/local/etc/openldap/schema/inetorgperson.schema

database  mdb
directory /usr/local/var/openldap-data
suffix    "dc=test,dc=com"
rootdn    "cn=admin,dc=test,dc=com"
rootpw    test
maxsize   1980732000

#Indexes
index cn,telephoneNumber eq
index   objectClass     eq
### config ###

Any ideas or things I could try?

Thank you!
Andrew