[Date Prev][Date Next]
Problem loading 100K entries into OpenLDAP2.2.24
- To: <openldap-software@OpenLDAP.org>
- Subject: Problem loading 100K entries into OpenLDAP2.2.24
- From: Kurt Zeilenga <kurt@OpenLDAP.org>
- Date: Fri, 15 Apr 2005 11:29:53 +0200
- Thread-index: AcVBna1XbeHCOI/GQuKVycL+6oDbSA=From: "Mavric Domen ITWEN5" <firstname.lastname@example.org>
- Thread-topic: Problem loading 100K entries into OpenLDAP2.2.24
I found a strange problem when populating my LDAP database with 100.000
entries (30 attrs. each).
I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
I should use slapadd?).
At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
start loading the memory
consumption of slapd process is about 3MB, but it keeps increasing for
about 20 min during the load and reaches
290 MB!!!. The total consumption of RAM is 1GB of 1GB. The slapd
process is killed by kernel after about 30K entries
(of 100K) inserted.
kernel: Out of Memory: Killed process 15748 (slapd)
See configuration for BDB and slapd bellow.
The second strange problem is increasing number of transaction log files
in a DB_HOME directory which
aren't removed automaticaly after checkpoints (see configuration
bellow). I believe that the first problem is related to this one.
Even if I do a manual checkpoint (db_checkpoint) and check for log files
that can be removed, db_archive
says that all log files are in use. I can only remove log files
(db_archive -d) after stopping an restarting slapd server.
Can't log files be removed after checkpoints during loading? I think
that's why I'm getting memory problems.
I checked also db_stat -m during the loading, but I didn't find anything
strange. I'll db files are on 96%-100% cache.
What I'm doing wrong? How should I configure my slapd, DB_CONFIG?
Thanks for tips,
I'm working with:
BerkeleyDB 4.2.52 + 4 patches (BDB)
4 x 1,6 Ghz CPU
1 GB RAM
OS: Monta Vista Linux CGE 3.1
checkpoint 1000 3
index cn eq,pres
index itRelease eq,pres
index itNeNode eq,sub,pres
index itProductName eq,sub,pres
index itProductCategory eq,sub,pres
index itNeNodeId eq,sub,pres
index itSerialNumber eq,sub,pres
index itParentContainerId eq,sub,pres
set_cachesize 0 20000000 0
# Automatically remove log files that are no longer needed.
# Just use these settings when doing slapadd...