[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Problem loading 100K entries into OpenLDAP2.2.24



This is now ITS#3666 and fixed in CVS OPENLDAP_REL_ENG_2_2 if you wish to test it. As noted before, the bug is not present in the 2.3 release.

Howard Chu wrote:

I'm seeing something similar, although DB_LOG_AUTOREMOVE is working for me. The 2.2.24 slapd starts at ~300MB and keeps growing. The 2.3/HEAD slapd doesn't do this. Probably should file an ITS with the particulars.

Kurt Zeilenga wrote:



Hello,



I found a strange problem when populating my LDAP database with 100.000
entries (30 attrs. each).

I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
I should use slapadd?).



At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
start loading the memory

consumption of slapd process is about 3MB, but it keeps increasing for
about 20 min during the load and reaches
290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
process is killed by kernel after about 30K entries

(of 100K) inserted.
syslog says:

           ...

       kernel: Out of Memory: Killed process 15748 (slapd)

           ...

See configuration for BDB and slapd bellow.



The second strange problem is increasing number of transaction log files
in a DB_HOME directory which

aren't removed automaticaly after checkpoints (see configuration
bellow). I believe that the first problem is related to this one.
Even if I do a manual checkpoint (db_checkpoint) and check for log files
that can be removed, db_archive

says that all log files are in use. I can only remove log files
(db_archive -d) after stopping an restarting slapd server.
Can't log files be removed after checkpoints during loading? I think
that's why I'm getting memory problems.



I checked also db_stat -m during the loading, but I didn't find anything
strange. I'll db files are on 96%-100% cache.



What I'm doing wrong? How should I configure my slapd, DB_CONFIG?

Thanks for tips,

BR,

Domen









I'm working with:

OpenLDAP 2.2.24

BerkeleyDB 4.2.52 + 4 patches (BDB)



Server info:

4 x 1,6 Ghz CPU

1 GB RAM

OS: Monta Vista Linux CGE 3.1







slapd configuration:



cachesize 100000

sizelimit 150000

checkpoint      1000    3



index cn eq,pres

index itRelease eq,pres

index itNeNode eq,sub,pres

index itProductName eq,sub,pres

index itProductCategory eq,sub,pres

index itNeNodeId eq,sub,pres

index itSerialNumber eq,sub,pres

index itParentContainerId eq,sub,pres







BDB configuration



set_lk_detect DB_LOCK_DEFAULT

set_lg_max 10485760

set_cachesize   0       20000000        0

set_lg_regionmax 262144

set_lg_bsize 2097152



# Automatically remove log files that are no longer needed.

set_flags DB_LOG_AUTOREMOVE

# Just use these settings when doing slapadd...

#set_flags DB_TXN_NOSYNC










--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support