[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: ITS#5860 slapd in openldap-2.4.16 still doesn't respect memory bounds
- To: openldap-bugs@openldap.org
- Subject: Re: ITS#5860 slapd in openldap-2.4.16 still doesn't respect memory bounds
- From: sscdvp@gmail.com
- Date: Tue, 28 Apr 2009 14:01:58 +0300
- Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=1tFG84xFrmLCv6sXTtYz6qKyj8j6V2ahch6WdU7jnHg=; b=ImxFR99MTNV6zWEVfbXbz9SvxN+YAeR5VuLbMh/LEFvoKUXRyNIOh6y5nfHmgUn4D+ O0/Dnwq6yg2I0NdNMo94kKCs028ZtEHZ/dx3hR7z9LzjzUSiX/1QnR64uZ7a5SaM2LkM M4VQbXaoGpD0T8W8CUwBHOPG4ANxjYQmfIg+8=
- Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=u2N+BIIRs6JFdAlgD9oYQ4csdJtTXwvO3pmiYsXK9J7D+VYkpsoP/kTl+uQPdzJZwT +InCljJSAp/rGEjN+yLcOhVgXIJVwI/UMZII5+0IuYgDUXD8n8YYXA6vr3RstDXx7mkC anQEdbOYxA63wkjOJnG/IDrBUpflgzaJZpkgU=
- In-reply-to: <49E99389.60105@symas.com>
- References: <9f32056c0904170217x5cbd3f05n929ff7f4c7ada4eb@mail.gmail.com> <49E99389.60105@symas.com>
2009/4/18, Howard Chu <hyc@symas.com>:
> sscdvp@gmail.com wrote:
>
> >
> > I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from
> > Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10.
> >
>
> A system with 8 CPUs and only 4GB of RAM is seriously unbalanced. Heck,
> even my single-processor laptop has 4GB of RAM.
>
>
> > According to ITS#5860 slapd should not allocate more memory than it was
> > specified. In fact I have the same behaviour as it was with previous
> > version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for
> > set_cachesize 300 MB). But slapd process continuosly grows in memory until
> > 32-bit limit then crashes due to unavalable memory resources. I need to
> > mention bdb database size is less then 1 GB.
> > After restart it rapidly consumes 1.5 GB RSS in few minutes and then it
> > grows beyond the limit during 24 hours. The more queries are processing
> > the more memory it eats.
> >
>
>
> > I am using this database for logging which size grows.
> > The question is: is it possible to use 32-bit openldap with relatively
> > large database sizes that override 32-bit per-process memory space?
> >
>
> Yes, it's possible. But performance will always be poor since it will be
> continually paging data in and out of cache. You have AMD64 processors, why
> are you running a 32 bit OS?
>
>
> > My slapd.conf contains:
> >
>
>
> > threads 64
> >
>
> 256MB of thread call stack right there.
> Plus another 64MB for per-thread sl_malloc.
> Plus another 512MB for per-thread search stack.
>
> This setting is too high for the amount of memory you have in your machine.
>
>
> > concurrency 50
> >
>
>
> > cachesize 200000
> > dncachesize 400000
> > idlcachesize 600000
> >
>
> You should also set cachefree.
>
>
> > shm_key 10
> >
> > DB_CONFIG:
> >
> > set_cachesize 0 300097152 1
> >
>
>
> > set_lk_max_locks 1000000
> > set_lk_max_lockers 1000000
> > set_lk_max_objects 1000000
> >
>
> Those settings are too high for the amount of memory you have in your
> machine.
>
> Right away you've consumed about 2GB of RAM *before* taking any of
> back-bdb's cache sizes into account.
>
>
> > set_lg_regionmax 1048576
> > set_lg_max 10485760
> > set_lg_bsize 2097152
> >
> > ls -ls ./openldap-data
> > total 1212876
> >
> > -rw-r--r-- 1 slapd nobody 640 Apr 17 08:42 DB_CONFIG
> > -rw------- 1 slapd nobody 8 Apr 16 10:50 __db.001
> > -rw-r--r-- 1 slapd nobody 4096 Apr 16 10:50 alock
> > -rw------- 1 slapd nobody 6053888 Apr 17 09:50 cn.bdb
> >
> > -rw------- 1 slapd nobody 1024000 Apr 17 09:50 dhcpHWAddress.bdb
> > -rw------- 1 slapd nobody 96583680 Apr 17 09:50 dn2id.bdb
> > -rw------- 1 slapd nobody 515637248 Apr 17 09:50 id2entry.bdb
> > -rw------- 1 slapd nobody 1343488 Apr 17 09:50 objectClass.bdb
> >
> >
> > Any help would be much appreciated.
> >
>
> Buy more RAM and use a 64 bit OS. You're running out of memory, but not due
> to ITS#5860.
>
I have added more RAM (12 gigabytes) and switched to 64-bit openldap
version (DB_CONFIG and slapd.conf options were not changed excepting
of using "threads 2" now). But slapd process grows continuously in
memory size (the size is 4 gigabytes now). I observed that that growth
appears only when data in database is replaced with new one. My
database presents application's activity logs which are completely
overwritting every 12 hours by application. This process is done by
using LDAP_DELETE operation on some attributes withing ldap_modify_ext
call or periodically by purging the whole old entries by
ldap_delete_ext call. What is recommended to avoid the cache trashing
by already deleted information?