OpenLDAP slave failure in case of master indisponibility

[Matthieu Cerda <matthieu.cerda@nbs-system.com>]

Greetings,

I am observing a rather strange issue in the following setup:

* 1 OpenLDAP master server (2.4.31)

* 4 OpenLDAP slave servers (2.4.40)

* The OpenLDAP slaves do forward any update attempt to the master using
the chain overlay / proxyauthz (mainly to update the pwdFailureTime
attribute for ppolicy)

If I try to shut the master down (for maintenance let's say), the slaves
behave properly, then begin to deadlock one after each other after a few
minutes (by deadlock I mean no log output anymore, and any ldapwhoami /
ldapsearch request connects and then times out)

On the attached image, I monitored at the same time one of the slaves
using collectd, to keep an eye on cn=monitor data (the period between
15:24:30 and 15:26:00 has been extrapolated by Grafana, no data is
available at this time since cn=monitor access also deadlocks)

I can see that backload / pending threads and waiters seem to increase
gradually until the server gets unresponsive.

I found nothing on the ML (except
https://www.openldap.org/lists/openldap-technical/200912/msg00112.html)
or searching for clues, Is this predictable behavior or and obvious
misconfiguration, or it is an interesting occastion to dig a bit deeper ?

Thanks in advance,

-- 
Matthieu Cerda
Infrastructure, BU Means @ NBS System

Attachment: ldap_failure.jpg
Description: JPEG image

#
# This file is handled using salt. Any manual change will be removed !
#

TLSCACertificateFile	(REDACTED)
TLSCertificateFile      (REDACTED)
TLSCertificateKeyFile   (REDACTED)

include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/nis.schema
include /etc/ldap/schema/inetorgperson.schema
include /etc/ldap/schema/dnszone.schema
include /etc/ldap/schema/mmc.schema
include /etc/ldap/schema/mail.schema
include /etc/ldap/schema/openssh-lpk_openldap.schema
include /etc/ldap/schema/ppolicy.schema

pidfile /var/run/slapd/slapd.pid
argsfile /var/run/slapd/slapd.args

loglevel none stats

modulepath /usr/lib/ldap
moduleload back_mdb
moduleload back_ldap
moduleload back_monitor

moduleload ppolicy
moduleload memberof
moduleload pw-sha2

# Default password hashing algorithm
password-hash {SSHA512}

overlay                    chain
chain-uri                  "ldaps://(REDACTED)/"

chain-idassert-bind        bindmethod="simple"
                           binddn="(REDACTED)"
                           credentials="(REDACTED)"
                           mode="self"
chain-return-error         TRUE

backend mdb
database mdb

# 4 threads per core http://www.openldap.org/doc/admin24/tuning.html
threads 4
# end tuning

# 1GB maximum LMDB size
maxsize 1073741824

suffix "(REDACTED)"

rootdn "(REDACTED)"
rootpw (REDACTED)

directory "/var/lib/ldap"

index objectClass eq
# also index mail and uid
index entryCSN,entryUUID eq
index mail,mailalias,uid eq

# ACL definitions
#
# Note: the rootdn is always granted full r/w access to
# the database and bypasses any ACLs.
#

(REDACTED)

# ACL definitions end

# Password policy overlay configuration
overlay ppolicy

ppolicy_default "(REDACTED)"
ppolicy_use_lockout
ppolicy_hash_cleartext
ppolicy_forward_updates

# MemberOf overlay configuration
overlay memberof

# Syncrepl configuration
syncrepl rid=123
        provider=ldap://(REDACTED)/
        starttls=yes
        tls_cacert=/etc/ssl/certs/ca-certificates.crt
        type=refreshAndPersist
        retry="10 +"
        searchbase="(REDACTED)"
        schemachecking=on
        bindmethod=simple
        binddn="(REDACTED)"
        credentials=(REDACTED)

updateref ldaps://(REDACTED)/

# Monitoring (cn=Monitor)
database monitor

# ACL definitions
#
# Admins have full access, monitoring dn read only access,
# other people are rejected.
#
(REDACTED)

Attachment: signature.asc
Description: OpenPGP digital signature