[Date Prev][Date Next] [Chronological] [Thread] [Top]

Openldap scalability issues

I'm looking for some configuration advice to make openldap scale better for databases that hold millions of records.

In the past we have been loading most of our large databases from ldif with slapadd.  We have been successful in loading databases with 10 to 15 million entries.  The largest ones usually take a day or two - but we can deal with that.  However, now I am trying to load a large database programmatically, from a java application.

If I start with a blank, clean database - here is the behavior I see.

The first ~200,000 entries or so (times about 5 attributes per entry) take about 2 minutes per 10,000 entries.

Then it starts getting slower.  By the time I get to entry 300,000 - it is taking ~45 minutes per 10,000 entries.

Then it starts jumping around - sometimes taking a 1/2 hour per 10,000 sometimes 3 hours per 10,000.

Finally, usually somewhere around 600,000 concepts, it just kicks the client out with some odd error message - it changes from run to run.  The most recent example was:

javax.naming.NamingException: [LDAP: error code 80 - entry store failed]

After this happened - I could no longer connect to the ldap server with any client - the server actually crashed when I tried to connect to it.  But then when I restart the server, it seems to run fine, at least for browsing.  I currently don't have the ability to resume my inserts - so I haven't been able to tell if the insert speed would be good again after a restart, or if it would stay as slow as it was at the end.

The database size was only 3.6 GB when it failed - I have loaded 16 GB databases with slapadd before.  Why does this work in slapadd - but not when it is done with slapd?

Config info:
Openldap 2.2.23
BerkeleyDB 4.2.52(.2)

bdb backend
set_flags       DB_TXN_NOSYNC
set_flags       DB_TXN_NOT_DURABLE
set_cachesize   1       0       1

Useful? bits from the conf file:

schemacheck     on
idletimeout     14400
threads         150
sizelimit       6000

database        bdb
suffix          "service=test,dc=LexGrid,dc=org"
directory       /localwork/ldap/database/dbdanTest
checkpoint      512     30

index           objectClass eq
index           conceptCode eq
index           language pres,eq
index           dc eq
index           sourceConcept,targetConcept,association,presentationId eq
index           text,entityDescription pres,eq,sub,subany

You can see the schema here if you desire: http://informatics.mayo.edu/index.php?page=102