[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: ITS#5860 slapd in openldap-2.4.16 still doesn't respect memory bounds



2009/4/18, Howard Chu <hyc@symas.com>:
> sscdvp@gmail.com wrote:
>
> >
> > I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from
> > Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10.
> >
>
>  A system with 8 CPUs and only 4GB of RAM is seriously unbalanced. Heck,
> even my single-processor laptop has 4GB of RAM.
>
>
> > According to ITS#5860 slapd should not allocate more memory than it was
> > specified. In fact I have the same behaviour as it was with previous
> > version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for
> > set_cachesize 300 MB). But slapd process continuosly grows in memory until
> > 32-bit limit then crashes due to unavalable memory resources. I need to
> > mention bdb database size is less then 1 GB.
> > After restart it rapidly consumes 1.5 GB RSS in few minutes and then it
> > grows beyond the limit during 24 hours. The more queries are processing
> > the more memory it eats.
> >
>
>
> > I am using this database for logging which size grows.
> > The question is: is it possible to use 32-bit openldap with relatively
> > large database sizes that override 32-bit per-process memory space?
> >
>
>  Yes, it's possible. But performance will always be poor since it will be
> continually paging data in and out of cache. You have AMD64 processors, why
> are you running a 32 bit OS?
>
>
> > My slapd.conf contains:
> >
>
>
> > threads 64
> >
>
>  256MB of thread call stack right there.
>  Plus another 64MB for per-thread sl_malloc.
>  Plus another 512MB for per-thread search stack.
>
>  This setting is too high for the amount of memory you have in your machine.
>
>
> > concurrency 50
> >
>
>
> > cachesize 200000
> > dncachesize 400000
> > idlcachesize 600000
> >
>
>  You should also set cachefree.
>
>
> > shm_key 10
> >
> > DB_CONFIG:
> >
> > set_cachesize           0 300097152 1
> >
>
>
> > set_lk_max_locks        1000000
> > set_lk_max_lockers      1000000
> > set_lk_max_objects      1000000
> >
>
>  Those settings are too high for the amount of memory you have in your
> machine.
>
>  Right away you've consumed about 2GB of RAM *before* taking any of
> back-bdb's cache sizes into account.
>
>
> > set_lg_regionmax        1048576
> > set_lg_max              10485760
> > set_lg_bsize            2097152
> >
> > ls -ls ./openldap-data
> > total 1212876
> >
> > -rw-r--r--   1 slapd    nobody       640 Apr 17 08:42 DB_CONFIG
> > -rw-------   1 slapd    nobody         8 Apr 16 10:50 __db.001
> > -rw-r--r--   1 slapd    nobody      4096 Apr 16 10:50 alock
> > -rw-------   1 slapd    nobody   6053888 Apr 17 09:50 cn.bdb
> >
> > -rw-------   1 slapd    nobody   1024000 Apr 17 09:50 dhcpHWAddress.bdb
> > -rw-------   1 slapd    nobody   96583680 Apr 17 09:50 dn2id.bdb
> > -rw-------   1 slapd    nobody   515637248 Apr 17 09:50 id2entry.bdb
> > -rw-------   1 slapd    nobody   1343488 Apr 17 09:50 objectClass.bdb
> >
> >
> > Any help would be much appreciated.
> >
>
>  Buy more RAM and use a 64 bit OS. You're running out of memory, but not due
> to ITS#5860.
>

I have added more RAM (12 gigabytes) and switched to 64-bit openldap
version (DB_CONFIG and slapd.conf options were not changed excepting
of using "threads 2" now). But slapd process grows continuously in
memory size (the size is 4 gigabytes now). I observed that that growth
appears only when data in database is replaced with new one. My
database presents application's activity logs which are completely
overwritting every 12 hours by application. This process is done by
using LDAP_DELETE operation on some attributes withing ldap_modify_ext
call or periodically by purging the whole old entries by
ldap_delete_ext call. What is recommended to avoid the cache trashing
by already deleted information?