[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Fwd: 2.4.40 memory leak?



Hi Ryan,

Here's my config of LDAP master:
---------------------------------------------------------------------------
# cat /etc/openldap/slapd.conf | grep -v ^# | grep -ve '^$'
include         /etc/openldap/schema/core.schema
include         /etc/openldap/schema/cosine.schema
include         /etc/openldap/schema/inetorgperson.schema
include         /etc/openldap/schema/nis.schema
allow bind_v2
pidfile         /var/run/openldap/slapd.pid
argsfile        /var/run/openldap/slapd.args
modulepath      /usr/lib64/openldap
moduleload accesslog.la
moduleload syncprov.la
TLSCACertificateFile /etc/openldap/certs/CA.pem
TLSCertificateFile /etc/openldap/certs/ldap-master.pem
TLSCertificateKeyFile /etc/openldap/certs/ldap-master.key
TLSVerifyClient allow

[ .. some limits here .. ]

[ .. some ACLs here .. ]

database config
access to *
        by dn.exact="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage
        by * none
database monitor

[ .. some ACLs here .. ]

[ .. some limits here .. ]

database        bdb
cachesize       380000
idlcachesize    700000
readonly        off
suffix          "dc=domain,dc=com"
rootdn          "cn=Manager,dc=domain,dc=com"
rootpw          {SSHA}XXXXXXXXXX
directory       /var/lib/ldap
index   uid     eq
index   mail    eq
index   objectClass eq
index entryCSN eq
index entryUUID eq
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100

[ .. some limits here .. ]

loglevel sync stats stats2 shell
checkpoint 5120 10
serverID    1
---------------------------------------------------------------------------


Here's what I have on replica server:

---------------------------------------------------------------------------
include         /etc/openldap/schema/core.schema
include         /etc/openldap/schema/cosine.schema
include         /etc/openldap/schema/nis.schema
include         /etc/openldap/schema/inetorgperson.schema
allow bind_v2
pidfile         /var/run/openldap/slapd.pid
argsfile        /var/run/openldap/slapd.args
threads 8
[ .. some ACLs here .. ]
database config
access to *
        by dn.exact="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" manage
        by * none
database monitor
[ .. some ACLs here .. ]
database        bdb
cachesize       380000
idlcachesize    700000
readonly        off
suffix          "dc=domain,dc=com"
rootdn          "cn=Manager,dc=jetbrains,dc=com"
rootpw          {SSHA}XXXXXXXXXXXXXXXXX
directory       /var/lib/ldap
index   uid     eq
index   mail    eq
index   objectClass eq
index entryCSN eq
index entryUUID eq
checkpoint 5120 10
syncrepl rid=34
 provider=ldaps://ldap-master.domain.net:636
 tls_reqcert=demand
 tls_cacert=/etc/openldap/certs/CA.pem
 type=refreshAndPersist
 schemachecking=off
 searchbase="dc=domain,dc=com"
 scope=sub
 bindmethod=simple
 binddn="cn=repluser,ou=Accounts,dc=domain,dc=com"
 credentials=XXXXXXXXXX
 retry="300 +"
updateref ldaps://ldap-master.domain.net
[ .. some limits here .. ]
loglevel stats sync stats2 shell

---------------------------------------------------------------------------


I restarted slapd with "LD_PRELOAD=/usr/lib64/libtcmalloc.so.4.1.0" to use a different memory allocator (tcmalloc) and now memory consumption is almost flat, please see http://i.imgur.com/brIvarB.png

I've also added "threads 8" directive into slapd.conf on LDAP master server but have not started the slapd process to make it active.

According to what I see from the OS (Linux) perspective, slapd is using 18 threads:

# ps -L -o pid= -p  `pgrep slapd` | wc -l
18


> Do your logs show what kind of client activity triggered the growth?

I have some logs but I nothing really special there. No unusual activity.


Regards,
Sergey


On Sun, Mar 29, 2015 at 10:16 AM, Ryan Tandy <ryan@nardis.ca> wrote:
Hi,

On Thu, Mar 26, 2015 at 01:50:27PM +0300, Sergey Esin wrote:
Hi all,

We're running OpenLDAP 2.4.40 (the latest available release) with just one
replica server (connected via TLS) and have the following picture -
http://i.imgur.com/om0lMiy.png

On the graph you can see memory consumption of the slapd process on the
host: in the beginngin it started without replica, then replica server was
connected (memory consumption became around 4 Gigs) and then OOM
(out-of-memory) killer on linux machine just killed the process.

I've seen a similar thing recently. The test case I posted to ITS#8081 causes very high memory usage on the host. (The crash bug is unrelated, it was a regression introduced after 2.4.40 was released.) Are you able to share your host config for comparison?

Howard wrote https://github.com/hyc/mleak while looking into it, but AFAIK we don't have a proven cause, only a suspicion that memory fragmentation may be involved.

Do your logs show what kind of client activity triggered the growth?

Do you use delta-syncrepl?


There are ~400 000 users in our ldap database.

OpenLDAP was compiled from sources using "./configure --prefix=/ldap2440
--with-tls --enable-slapd".

Are there any ways to understand what's is going wrong and how to fix it?

This server is really important for us, please share any ideas how to make
it stable!


My DB_CONFIG is like below:

set_flags DB_LOG_AUTOREMOVE

set_cachesize 0 524288000 5
set_lg_regionmax 1048576
set_lg_max 10485760
set_lg_bsize 2097512

set_lk_max_locks 23000
set_lk_max_lockers 2300
set_lk_max_objects 2300


--
Regards,
Sergey



--
Regards,
Sergey