[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#4702) out-of-memory on huge DB ?

--On Friday, October 13, 2006 1:57 PM +0000 Paolo.Rossi.con@h3g.it wrote:

>> No. As I already mentioned, there is no reason for slapcat to use
>> large amounts of RAM, it should use whatever size your BDB cache is
>> configured for and that's about it. It's possible there is a memory
>> leak, but since no such leak appears under Linux that would point
>> to either the Solaris C library or some other library specific to
>> your platform.
> Today I've finished a cross test on a suse 10.0 linux
> Athlon XP 2200+ 768MB RAM + 1GB swap
> loaded huge db with same config, shared mem and  set_cachesize 0
> 384000000 1
> slapcat -f slapd.conf

So you understand that without the -l flag to slapcat, you are writing to 
stdout instead of a file?  Is it possible that is what is causing slapcat 
to crash?

On my linux 2.6 system (64 bit), with a 2.5 million entry db, which is:

ldap00:/var/lib/ldap/slamd/64-2.5mil-db42# du -h *.bdb
84M     cn.bdb
618M    dn2id.bdb
85M     employeeNumber.bdb
3.2G    id2entry.bdb
3.7M    objectClass.bdb
85M     uid.bdb


set_cachesize 0 384000000 1
set_lg_regionmax 262144
set_lg_bsize 2097152
set_lg_dir /var/log/bdb/slamd/64-2.5mil
set_lk_max_locks 3000
# Automatically remove log files that are no longer needed.
set_flags       DB_LOG_AUTOREMOVE
#set_flags      DB_DIRECT_DB
# Setting set_tas_spins reduces resource contention from multiple clients 
on systems with multiple CPU's.
set_tas_spins 1

Using slapcat -l ldif_file, I then ran a while (1) loop to check the OSS 
and RSS sizes of slapcat.  It reached the following size fairly quickly:

1159740 1191916

and then never grew past that.  This means the resident memory maxed out at 
1.1 GB, and the virtual memory maxed out at 1.1GB as well.

So, that does mean one thing -- It definitely outgrew the 366MB of BDB 
cache defined.  On the other hand, once it reached its max size, it stayed 
steady there until completion was reached.


Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html