[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapadd and attribute value length limit

Jon Roberts wrote:

Howard Chu wrote:

Judging from the error message you received, since the BDB library ran out of memory you may be able to resolve this by increasing your BDB cache size. Also you may get some more explicit information from BDB if you enable debugging (-d1) while running slapadd.

I tried this. Cache size wasn't the problem. The gotcha:

bdb(o=mentata.com): Lock table is out of available locks

So now with DB_CONFIG:

set_lg_dir      /var/log/openldap
set_lg_max      104857600
set_lg_bsize    26214400
set_lk_max_locks 2000
set_lk_max_lockers 2000
set_lk_max_objects 2000
# set_cachesize 0 20971520 1
set_cachesize 0 419430400 1
set_flags       DB_TXN_NOSYNC

Now the load completes fine in under an hour. Much better than loading just the english with slapadd and then using ldapmodify LDIFs for each additional language (hours plural).

I know Quanah might add that I should now recomment:

#set_flags       DB_TXN_NOSYNC

Already done, along with the cache switch. Would I need this many locks/lockers in a runtime scenario, or should I comment out those as well?

slapadd stores entries one at a time in the database, and back-bdb basically uses one transaction per entry. Within this one transaction, it will acquire locks for the entry itself and all of the indexing records that are needed to write the entry. If your indexing has generated so many records that you exhausted the default of 1000 locks, then I expect you will need many more locks in a runtime scenario. Of course, this assumes that you're adding/modifying entries like this at runtime. I.e., if it takes over 1000 locks to write just one entry, and if you have the possibility of writing two or more entries at runtime, then you will need far more locks at runtime.

I strongly doubt that you needed to raise the max_lockers.

As to why you need so many locks - the substring indexing you're performing, with the long strings, is definitely hitting a large number of records in all of the index databases, so this lock situation is not too surprising.

 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support