[Date Prev][Date Next] [Chronological] [Thread] [Top]
(ITS#7636) slapd crash when multi-master replication (syncrepl) enabled

To: openldap-its@OpenLDAP.org
Subject: (ITS#7636) slapd crash when multi-master replication (syncrepl) enabled
From: kb9vqf@pearsoncomputing.net
Date: Tue, 2 Jul 2013 19:16:11 GMT
Auto-submitted: auto-generated (OpenLDAP-ITS)
Full_Name: Timothy Pearson
Version: 2.4.31
OS: Debian Wheezy
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (131.156.2.26)


The setup:
Multi-master syncrepl on two servers
Identical hardware and software between servers
Self-signed TLS using common (private) CA certificate

The problem:
slapd on one server crashes repeatably within a minute of slapd starting on the
other server.  slapd works reliably if and only if the other server is not
running a slapd process.


Backtrace (does not change appreciably from crash to crash):

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe356c700 (LWP 10433)]
0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
#2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3  0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
"munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe3569fd0,
rs=<optimized out>, prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:313
#5  slapi_op_search_callback (op=0x7fffe3569fd0, rs=<optimized out>,
prc=<optimized out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:296
#6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe3569f60, op=<optimized out>)
at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
#7  slapi_op_func (op=0x7fffe3569fd0, rs=0x7fffe3569f60) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#8  0x00005555555ff18a in overlay_op_walk (op=op@entry=0x7fffe3569fd0,
rs=0x7fffe3569f60, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
../../../../servers/slapd/backover.c:661
#9  0x00005555555ff31b in over_op_func (op=0x7fffe3569fd0, rs=<optimized out>,
which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#10 0x00007ffff1c4870e in syncprov_findbase (op=op@entry=0x555555dd8560,
fc=fc@entry=0x7fffe356a280) at
../../../../../servers/slapd/overlays/syncprov.c:453
#11 0x00007ffff1c4b1ef in syncprov_op_search (op=0x555555dd8560,
rs=0x7fffe356ba50) at ../../../../../servers/slapd/overlays/syncprov.c:2465
#12 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
rs=rs@entry=0x7fffe356ba50, which=which@entry=op_search, oi=0x5555559e57f0,
on=0x5555559e15e0) at ../../../../servers/slapd/backover.c:661
#13 0x00007ffff63d3086 in slapi_op_func (rs=0x7fffe356ba50, op=<optimized out>)
at ../../../../../servers/slapd/slapi/slapi_overlay.c:647
#14 slapi_op_func (op=0x555555dd8560, rs=0x7fffe356ba50) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#15 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
rs=0x7fffe356ba50, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
../../../../servers/slapd/backover.c:661
#16 0x00005555555ff31b in over_op_func (op=0x555555dd8560, rs=<optimized out>,
which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#17 0x0000555555594641 in fe_op_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
../../../../servers/slapd/search.c:402
#18 0x0000555555593f06 in do_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
../../../../servers/slapd/search.c:247
#19 0x0000555555591961 in connection_operation (ctx=ctx@entry=0x7fffe356bba0,
arg_v=arg_v@entry=0x555555dd8560) at
../../../../servers/slapd/connection.c:1150
#20 0x0000555555591c84 in connection_read_thread (ctx=0x7fffe356bba0,
argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286
#21 0x00007ffff7b9dff3 in ?? () from
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
#22 0x00007ffff5d90b50 in start_thread (arg=<optimized out>) at
pthread_create.c:304
#23 0x00007ffff5adaa7d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()


Relevant configuration files:


olcDatabase={0}config.ldif:

dn: olcDatabase={0}config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcAccess: {0}to * by
group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
by dn.base="cn=admin,dc=<redacted>" write by sockurl.regex="^ldapi:///$" write
by dynacl/aci write by * none
olcAddContentAcl: TRUE
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=config
olcRootPW:: REDACTED
olcSyncUseSubentry: FALSE
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
creatorsName: cn=config
createTimestamp: 20130702170316Z
olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1
tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"lcSyncrepl:
{1}rid=002 provider=ldaps://ldap003.<redacted>/ binddn="cn=admin,dc=<redacted>"
bindmethod=simple credentials="ldapadmin001" searchbase="cn=config"
type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1 tls_reqcert=never
tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
olcMirrorMode: TRUE
entryCSN: 20130702180604.682031Z#000000#002#000000
modifiersName:
modifyTimestamp: 20130702180604Z


olcDatabase={1}hdb.ldif:

dn: olcDatabase={1}hdb
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /var/lib/ldap
olcSuffix: dc=<redacted>
olcAccess: {0}to attrs=userPassword,shadowLastChange,krb5Key,krb5PrincipalName,krb5KeyVersionNumber,krb5MaxLife,krb5MaxRenew,krb5KDCFlags,privateRootCertificateKey
by group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" by
sockurl.regex="^ldapi:///$" write by anonymous auth by self write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by
group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
by sockurl.regex="^ldapi:///$" write by dynacl/aci write
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=admin,dc=<redacted>
olcRootPW:: REDACTED
olcMonitoring: TRUE
olcDbCacheSize: 1000
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 67108864 1
olcDbConfig: {1}set_lg_regionmax 262144
olcDbConfig: {2}set_lg_bsize 2097152
olcDbNoSync: FALSE
olcDbDirtyRead: FALSE
olcDbIDLcacheSize: 0
olcDbIndex: objectClass eq
olcDbIndex: krb5PrincipalName eq,pres
olcDbIndex: cn eq,pres,subinitial
olcDbIndex: mail eq,pres
olcDbIndex: uid pres,eq
olcDbIndex: uidNumber eq
olcDbIndex: gidNumber eq
olcDbLinearIndex: FALSE
olcDbMode: 0600
olcDbSearchStack: 16
olcDbShmKey: 0
olcDbCacheFree: 1
olcDbDNcacheSize: 0
olcPlugin: postoperation /opt/trinity/lib/slapi-acl-manager.so plugin_init
admingroup-dn:=cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>
realm:=CEET.NIU.EDU aclfile:=/etc/heimdal-kdc/kadmind.acl  builtinadmin:=admin
structuralObjectClass: olcHdbConfig
creatorsName: cn=config
createTimestamp: 20130702170316Z
olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
olcSyncrepl: {1}rid=002 provider=ldaps://ldap003.<redacted>/
binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
olcMirrorMode: TRUE
entryCSN: 20130702170511.039863Z#000000#002#000000
modifiersName:
modifyTimestamp: 20130702170511Z


olcDatabase={0}config/olcOverlay={0}syncprov.ldif:

dn: olcOverlay={0}syncprov
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
structuralObjectClass: olcSyncProvConfig
entryUUID: 14f25934-7785-1032-93bd-0fe5d581c3b6
creatorsName:
createTimestamp: 20130702170337Z
entryCSN: 20130702170337.631532Z#000000#002#000000
modifiersName:
modifyTimestamp: 20130702170337Z


olcDatabase={1}hdb/olcOverlay={0}syncprov.ldif:

dn: olcOverlay={0}syncprov
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: {0}syncprov
structuralObjectClass: olcSyncProvConfig
entryUUID: bfa2e530-778d-1032-9b41-e7480f731418
creatorsName:
createTimestamp: 20130702180539Z
entryCSN: 20130702180539.975057Z#000000#002#000000
modifiersName:
modifyTimestamp: 20130702180539Z
Prev by Date: (ITS#7635) lmdb race condition in read vs write txns
Next by Date: Re: (ITS#7636) slapd crash when multi-master replication (syncrepl) enabled
Index(es):
- Chronological
- Thread