[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: ITS#5860 slapd in openldap-2.4.16 still doesn't respect memory bounds



sscdvp@gmail.com wrote:

I have tried openldap-2.4.16 (latest) BDB 4.7.25 with all patches from
Oracle on Dell AMD64 RAM 4Gb, 8 CPUs, OS Solaris 10.

A system with 8 CPUs and only 4GB of RAM is seriously unbalanced. Heck, even my single-processor laptop has 4GB of RAM.

According to ITS#5860 slapd should not allocate more memory than it was
specified. In fact I have the same behaviour as it was with previous
version (2.4.11). It correctly allocates 1.2 Gb via shared memory (for
set_cachesize 300 MB). But slapd process continuosly grows in memory until
32-bit limit then crashes due to unavalable memory resources. I need to
mention bdb database size is less then 1 GB.
After restart it rapidly consumes 1.5 GB RSS in few minutes and then it
grows beyond the limit during 24 hours. The more queries are processing
the more memory it eats.

I am using this database for logging which size grows.
The question is: is it possible to use 32-bit openldap with relatively
large database sizes that override 32-bit per-process memory space?

Yes, it's possible. But performance will always be poor since it will be continually paging data in and out of cache. You have AMD64 processors, why are you running a 32 bit OS?

My slapd.conf contains:

threads 64

256MB of thread call stack right there.
Plus another 64MB for per-thread sl_malloc.
Plus another 512MB for per-thread search stack.

This setting is too high for the amount of memory you have in your machine.

concurrency 50

cachesize 200000
dncachesize 400000
idlcachesize 600000

You should also set cachefree.

shm_key 10

DB_CONFIG:

set_cachesize           0 300097152 1

set_lk_max_locks        1000000
set_lk_max_lockers      1000000
set_lk_max_objects      1000000

Those settings are too high for the amount of memory you have in your machine.

Right away you've consumed about 2GB of RAM *before* taking any of back-bdb's cache sizes into account.

set_lg_regionmax        1048576
set_lg_max              10485760
set_lg_bsize            2097152

ls -ls ./openldap-data
total 1212876

-rw-r--r--   1 slapd    nobody       640 Apr 17 08:42 DB_CONFIG
-rw-------   1 slapd    nobody         8 Apr 16 10:50 __db.001
-rw-r--r--   1 slapd    nobody      4096 Apr 16 10:50 alock
-rw-------   1 slapd    nobody   6053888 Apr 17 09:50 cn.bdb

-rw-------   1 slapd    nobody   1024000 Apr 17 09:50 dhcpHWAddress.bdb
-rw-------   1 slapd    nobody   96583680 Apr 17 09:50 dn2id.bdb
-rw-------   1 slapd    nobody   515637248 Apr 17 09:50 id2entry.bdb
-rw-------   1 slapd    nobody   1343488 Apr 17 09:50 objectClass.bdb


Any help would be much appreciated.

Buy more RAM and use a 64 bit OS. You're running out of memory, but not due to ITS#5860.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/