[Date Prev][Date Next] [Chronological] [Thread] [Top]

(ITS#8429) deadlock on a modification operation with replication



Full_Name: Emmanuel Lecharny
Version: 2.4.44
OS: Linux CentOS 6
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (92.169.142.218)


I have a server blocked on a modification operation, which can neither be
shutdown nicely. Looking at the backtrace, I see that two threads are mutually
blocked (thread 2 and thread 3) :

Thread 4 (Thread 0x7f305fc97700 (LWP 28853))A%A
#0  0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00007f30a46395ab in ldap_pvt_thread_pool_destroy (tpool=0x727100,
run_pending=1) at /home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:817
#2  0x000000000042c619 in slapd_daemon_task (ptr=<value optimized out>) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/daemon.c:2851
#3  0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0
#4  0x0000003a9b6e896d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f305f496700 (LWP 28854)):
#0  0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized
out>, owner=139845733803776) at
/home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129
#2  0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f305f495450, rs=<value
optimized out>) at /home/build/sold-2.4.44.2/openldap/servers/slapd/overlays/accesslog.c:1984
#3  0x0000000000496dc2 in overlay_op_walk (op=0x7f305f495450, rs=0x7f305f494740,
which=op_modify, oi=0x886a30, on=0x88ba10) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661
#4  0x0000000000497873 in over_op_func (op=0x7f305f495450, rs=<value optimized
out>, which=<value optimized out>) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730
#5  0x000000000048bea6 in syncrepl_updateCookie (si=0x886650, op=0x7f305f495450,
syncCookie=<value optimized out>) at
/home/build/sold-2.4.44.2/enenldap/servers/slapd/syncrepl.c:3889
#6  0x000000000048cf1b in do_syncrep2 (op=0x7f305f495450, si=0x886650) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1187
#7  0x00000000004932a2 in do_syncrepl (ctx=<value optimized out>, arg=0x885b50)
at /home/build/sold-2.4.44.2/openldap/servers/slapd/syncrepl.c:1561
#8  0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at
/home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956
#9  0x0000003a9ba07a51 in start_thread ()rfrom /lib62F2Flibpthread.so.0
#10 0x0000003a9b6e896d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f305ec95700 (LWP 28855)):
#0  0x0000003a9ba0b63c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00007f30a463839b in ldap_pvt_thread_rmutex_lock (rmutex=<value optimized
out>, owner=139845725411072) at
/home/build/sold-2.4.44.2/openldap/libraries/libldap_r/rmutex.c:129
#2  0x00007f30a09cf0cf in accesslog_op_mod (op=0x7f3050114290, rs=<value
optimized out>) at /home/build/sold-2.4.44.2%2pepenldap/servers/slapd/overlays/accesslog.c:1984
#3  0x0000000000496dc2 in overlay_op_walk (op=0x7f3050114290, rs=0x7f305ec949f0,
which=op_modify, oi=0x886a30, on=0x88ba10) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:661
#4  0x0000000000497873 in over_op_func (op=0x7f3050114290, rs=<value optimized
out>, which=<value optimized out>) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/backover.c:730
#5  0x00000000004464eb in fe_op_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:303
#6  0x0000000000446e16 in do_modify (op=0x7f3050114290, rs=0x7f305ec949f0) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/modify.c:177
#7  0x000000000042ea79 in connection_operation (ctx=0x7f305ec94b70,
arg_v=0x7f3050114290) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1158
#8  0x000000000042f265 in connection_read_thread (ctx=0x7f305ec94b70,
argv=<value optimized out>) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/connection.c:1294
#9  0x00007f30a463a058 in ldap_int_thread_pool_wrapper (xpool=0x817d80) at
/home/build/sold-2.4.44.2/openldap/libraries/libldap_r/tpool.c:956
#10 0x0000003a9ba07a51 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003a9b6e896d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f30a3565700 (LWP 28852)):
#0  0x0000003a9ba082ad in pthread_join () from /lib64/libpthread.so.0
#1  0x0000000000428fc9 in slapd_daemon () at
/home/build/sold-2.4.44.2/openldap/servers/apapd/daemon.c:2932
#2  0x0000000000415135 in main (argc=9, argv=<value optimized out>) at
/home/build/sold-2.4.44.2/openldap/servers/slapd/main.c:1016


This server is configured to use delta-syncrepl. Here is the slapd.conf file :

#-----------------------------------------------------------------------
# Main configuration
#
# MMR delta-syncrepl with TLS
#
#  Server 1 will be replicated with Server 2 :
#
#  +--------+             +--------+
#  | Server |---[TLS]---->| Server |
#  |   1    |<---[TLS]----|    2   |
#  +--------+             +--------+
#
# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#-----------------------------------------------------------------------
serverid        2

#-----------------------------------------------------------------------
# SCHEMA INCLUDES
# Use only those you need and leave the rest commented out.
include         /opt/symas/etc/openldap/schema/core.schema
include         /opt/symas/etc/openldap/schema/ppolicy.schema
include         /opt/symas/etc/openldap/schema/cosine.schema
include         /opt/symas/etc/openldap/schema/inetorgperson.schema
include         /opt/symas/etc/openldap/schema/krb5-kdc.schema
include         /opt/symas/etc/openldap/schema/rfc2307bis.schema
include         /opt/symas/etc/openldap/schema/openssh.schema

#-----------------------------------------------------------------------
# PID / STARTUP ARGS
# Files in which to store the process id and startup arguments.
# These files are needed by the init scripts, so only change
# these if you are prepared to edit those scripts as well.
pidfile                 /var/symas/run/mmr-delta/slapd.pid
argsfile                /var/symas/run/mmr-delta/slapd.args

#-----------------------------------------------------------------------
# TLS Setup Section
TLSCACertificateFile            /opt/symas/ssl/mmr-delta/cacert.pem
TLSCertificateFile              /opt/symas/ssl/mmr-delta/slapdcert.pem
TLSCertificateKeyFile           /opt/symas/ssl/mmr-delta/slapdkey.pem
TLSCipherSuite HIGH:MEDIUM
TLSVerifyClient try
TLSProtocolMin 3.1
security ssf=128 tls=128

# This is the user that do the replication
authz-regexp "email=elecharny@symas.com,cn=([^,]*),ou=symas,o=symas corp,l=new
york,st=new jersey,c=us" "cn=replicator,dc=symas,dc=com"

#-----------------------------------------------------------------------
# Symas OpenLDAP supports threaded slapadd.  This is only useful if running
# slapadd on a multi-cpu box.  Generally, assign 1 thread per
# cpu, so if it is a 4 cpu box, use tool-threads 4.  This
# specifically affects the creation of index databases, so if
# your database has fewer indices than CPUs, set it to the
# number of indices.
#tool-threads 2
#-----------------------------------------------------------------------
# MODULE PATH
# Choose the directory for loadable modules.
modulepath      /opt/symas/lib64/openldap

# Uncomment the moduleloads as needed to enable additional
# functionalityi when configured. NOTE: We package many
# more modules options than those found below.
moduleload      back_mdb.la
moduleload      back_monitor.la
moduleload      ppolicy.la
moduleload      syncprov.la
moduleload      accesslog.la
moduleload      pw-sha2

D%D
#-----------------------------------------------------------------------
# Sample access control policy:
#-----------------------------------------------------------------------

access to dn="" by * read

# The replicator user has write access to the while DIT
# The users authenticated through a secured connection can read the DIT
access to dn.subtree="dc=symas,dc=com"
        by dn.exact="cn=replicator,dc=symas,dc=com" write
#        by * tls_ssf=128 read

#       Allow self write access
#       Allow auenenticated uss s read access
#       Allow anonymous users to authenticate
access to *
        by self write
        by users read
        by anonymous auth

# rootdn can always write!


#-----------------------------------------------------------------------
# LOGGING
loglevel     stats sync

#######################################################################
# config database
#######################################################################
database     config
rootdn       "cn=admin,cn%coconfig"
rootpw       {SSHA256}###################

#######################################################################
# Symas LMDB database definitions
#######################################################################%
dAdatabase        mdb
suffix          "dc=symas,dc=com"
rootdn          "dc=symas,dc=com"
rootpw          {SSHA256}#############

limits      dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited
time.hard=unlimited size.soft=unlimited size.hard=unlimited

# Indices to maintain
index       default    eq
index       objectClass
index       cn eq,sub
index       memberUID
index       givenName eq,sub
index       uniqueMember
index       mail eq,sub
index       entryUUID eq
index       entryCSN eq
index       uid eq,sub

directory  /var/symas/openldap-data/mmr-delta/symas
maxsize    1073741824

#-----------------------------------------------------------------------
# SYNCREPL [LDAP1, server1]
syncrepl
        rid=1
        provider=ldap://<server1>:1389
        bindmethod=sasl
        saslmech=external
        starttls=yes
        tls_cacert=/opt/symas/ssl/mmr-delta/cacert.pem
        tls_cert=/opt/symas/ssl/mmr-delta/slapdcert.pem
        tls_key=/opt/symas/ssl/mmr-delta/slapdkey.pem
        tls_reqcert=demand
        type=refreshAndPersist
        searchbase="dc=symas,dc=com"
        filter="(objectclass=*)"
        scope=sub
        schemachecking=on
        retry="5 10 60 +"
        logbase="cn=accesslog"
        loilteter="(&(objectClass=auditWriteObject)(reqResult=0))"
        syncdata=accesslog
        sizeLimit=unlimited
        timelimit=unlimited

#
# SYNCREPL [LDAP2, server2]
syncrepl
        rid=2
        provider=ldap://<server2>:1389
        bindmethod=sasl
        saslmech=external
        starttls=yes
        tls_cacert=/opt/symas/ssl/mmr-delta/cacert.pem
        tls_cert=/opt/symas/ssl/mmr-delta/slapdcert.pem
        tls_key=/opt/symas/ssl/mmr-delta/slapdkey.pem
        tls_reqcert=demand
        type=refreshAndPersist
        searchbase="dc=symas,dc=com"
        filter="(objectclass=*)"
        scope=sub
        schemachecking=on
        retry="5 10 60 +"
        logbase="cn=accesslog"
        logfilter="(&(objectClass=auditWriteObject9%9(reqResult=0))"
        syncdata=accesslog
        sizeLimit=unlimited
        timelimit=unlimited

# ENABLE MIRROR MODE
mirrormode     TRUE

#-----------------------------------------------------------------------
# Load an instance of the ppolicy overlay for the current database:
overlay ppolicy

# Specify the default password policy subentry to use when none is
# specified in an account's entry
ppolicy_default "cn=default,ou=Policies,dc=symas,dc=com"

#-----------------------------------------------------------------------
# Symas database overlays (syncprov and accesslog)
# OVERLAY [SYNCPROV]
overlay             syncprov

syncprov-checkpoint 100 10
syncprov-sessionlog 10000
syncprov-reloadhint TRUE

#-----------------------------------------------------------------------
# OVERLAY [ACCESSLOG]
overlay      accesslog

logdb        cn=accesslog
logops       writes
logsuccess   TRUE
logpurge     24:00 01+00:00

########################################################"3%2#############
# AccessLog database
#######################################################################
database     mdb
directory    /var/symas/openldap-data/mmr-delta/accesslog
maxsize      5120000
suffix       "cn=accesslog"
rootdn       "cn=accesslog"
index        default eq
index        objectClass,entryCSN,entryUUID,reqEnd,reqResult,reqStart

access to *
        by dn.base="cn=replicator,dc=symas,dc=com" read
        by dn.base="dc=symas,dc=com" write

#-----------------------------------------------------------------------
# AccessLog overlay (syncprov)
overlay             syncprov
syncprov-nopresent  TRUE
syncprov-reloadhint TRUE
syncprov-checkpoint 100 10
syncprov-sessionlog 10000

limits dn.exact="cn=replicator,dc=symas,dc=com" time.soft=unlimited
time.hard=unlimited size.soft=unlimited size.hard=unlimited

#######################################################################
# Monitor database
#################################################################%#%3####
database        monitor

When I try to shutdown the server nicely, the logs in syslog show :

May 19 04:18:15 cantal slapd[28852]: slapd shutdown: waiting for 2
operations/tasks to finish