Re: (ITS#7636) slapd crash when multi-master replication (syncrepl) enabled

[hyc@symas.com]

kb9vqf@pearsoncomputing.net wrote:
> Full_Name: Timothy Pearson
> Version: 2.4.31
> OS: Debian Wheezy
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (131.156.2.26)
>
>
> The setup:
> Multi-master syncrepl on two servers
> Identical hardware and software between servers
> Self-signed TLS using common (private) CA certificate
>
> The problem:
> slapd on one server crashes repeatably within a minute of slapd starting on the
> other server.  slapd works reliably if and only if the other server is not
> running a slapd process.

1) 2.4.31 is ancient. Current is 2.4.35. Please provide a backtrace against a 
current OpenLDAP release.

2) Your backtrace shows a crash in a slapi plugin. If the bug is in your 
plugin there's nothing we can do about it.

3) Don't touch the individual files inside the slapd configuration database. 
Use "slapcat -n0".

>
> Backtrace (does not change appreciably from crash to crash):
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7fffe356c700 (LWP 10433)]
> 0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
> #2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
> fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> #3  0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
> "munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
> #4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe3569fd0,
> rs=<optimized out>, prc=<optimized out>) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:313
> #5  slapi_op_search_callback (op=0x7fffe3569fd0, rs=<optimized out>,
> prc=<optimized out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:296
> #6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe3569f60, op=<optimized out>)
> at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
> #7  slapi_op_func (op=0x7fffe3569fd0, rs=0x7fffe3569f60) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:556
> #8  0x00005555555ff18a in overlay_op_walk (op=op@entry=0x7fffe3569fd0,
> rs=0x7fffe3569f60, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
> ../../../../servers/slapd/backover.c:661
> #9  0x00005555555ff31b in over_op_func (op=0x7fffe3569fd0, rs=<optimized out>,
> which=<optimized out>) at ../../../../servers/slapd/backover.c:723
> #10 0x00007ffff1c4870e in syncprov_findbase (op=op@entry=0x555555dd8560,
> fc=fc@entry=0x7fffe356a280) at
> ../../../../../servers/slapd/overlays/syncprov.c:453
> #11 0x00007ffff1c4b1ef in syncprov_op_search (op=0x555555dd8560,
> rs=0x7fffe356ba50) at ../../../../../servers/slapd/overlays/syncprov.c:2465
> #12 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
> rs=rs@entry=0x7fffe356ba50, which=which@entry=op_search, oi=0x5555559e57f0,
> on=0x5555559e15e0) at ../../../../servers/slapd/backover.c:661
> #13 0x00007ffff63d3086 in slapi_op_func (rs=0x7fffe356ba50, op=<optimized out>)
> at ../../../../../servers/slapd/slapi/slapi_overlay.c:647
> #14 slapi_op_func (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:556
> #15 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
> rs=0x7fffe356ba50, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
> ../../../../servers/slapd/backover.c:661
> #16 0x00005555555ff31b in over_op_func (op=0x555555dd8560, rs=<optimized out>,
> which=<optimized out>) at ../../../../servers/slapd/backover.c:723
> #17 0x0000555555594641 in fe_op_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../servers/slapd/search.c:402
> #18 0x0000555555593f06 in do_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../servers/slapd/search.c:247
> #19 0x0000555555591961 in connection_operation (ctx=ctx@entry=0x7fffe356bba0,
> arg_v=arg_v@entry=0x555555dd8560) at
> ../../../../servers/slapd/connection.c:1150
> #20 0x0000555555591c84 in connection_read_thread (ctx=0x7fffe356bba0,
> argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286
> #21 0x00007ffff7b9dff3 in ?? () from
> /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
> #22 0x00007ffff5d90b50 in start_thread (arg=<optimized out>) at
> pthread_create.c:304
> #23 0x00007ffff5adaa7d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #24 0x0000000000000000 in ?? ()
>
>
> Relevant configuration files:
>
>
> olcDatabase={0}config.ldif:
>
> dn: olcDatabase={0}config
> objectClass: olcDatabaseConfig
> olcDatabase: {0}config
> olcAccess: {0}to * by
> group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
> by dn.base="cn=admin,dc=<redacted>" write by sockurl.regex="^ldapi:///$" write
> by dynacl/aci write by * none
> olcAddContentAcl: TRUE
> olcLastMod: TRUE
> olcMaxDerefDepth: 15
> olcReadOnly: FALSE
> olcRootDN: cn=config
> olcRootPW:: REDACTED
> olcSyncUseSubentry: FALSE
> olcMonitoring: FALSE
> structuralObjectClass: olcDatabaseConfig
> creatorsName: cn=config
> createTimestamp: 20130702170316Z
> olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"lcSyncrepl:
> {1}rid=002 provider=ldaps://ldap003.<redacted>/ binddn="cn=admin,dc=<redacted>"
> bindmethod=simple credentials="ldapadmin001" searchbase="cn=config"
> type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1 tls_reqcert=never
> tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcMirrorMode: TRUE
> entryCSN: 20130702180604.682031Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702180604Z
>
>
> olcDatabase={1}hdb.ldif:
>
> dn: olcDatabase={1}hdb
> objectClass: olcDatabaseConfig
> objectClass: olcHdbConfig
> olcDatabase: {1}hdb
> olcDbDirectory: /var/lib/ldap
> olcSuffix: dc=<redacted>
> olcAccess: {0}to attrs=userPassword,shadowLastChange,krb5Key,krb5PrincipalName,krb5KeyVersionNumber,krb5MaxLife,krb5MaxRenew,krb5KDCFlags,privateRootCertificateKey
> by group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" by
> sockurl.regex="^ldapi:///$" write by anonymous auth by self write by * none
> olcAccess: {1}to dn.base="" by * read
> olcAccess: {2}to * by
> group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
> by sockurl.regex="^ldapi:///$" write by dynacl/aci write
> olcAddContentAcl: FALSE
> olcLastMod: TRUE
> olcMaxDerefDepth: 15
> olcReadOnly: FALSE
> olcRootDN: cn=admin,dc=<redacted>
> olcRootPW:: REDACTED
> olcMonitoring: TRUE
> olcDbCacheSize: 1000
> olcDbCheckpoint: 512 30
> olcDbConfig: {0}set_cachesize 0 67108864 1
> olcDbConfig: {1}set_lg_regionmax 262144
> olcDbConfig: {2}set_lg_bsize 2097152
> olcDbNoSync: FALSE
> olcDbDirtyRead: FALSE
> olcDbIDLcacheSize: 0
> olcDbIndex: objectClass eq
> olcDbIndex: krb5PrincipalName eq,pres
> olcDbIndex: cn eq,pres,subinitial
> olcDbIndex: mail eq,pres
> olcDbIndex: uid pres,eq
> olcDbIndex: uidNumber eq
> olcDbIndex: gidNumber eq
> olcDbLinearIndex: FALSE
> olcDbMode: 0600
> olcDbSearchStack: 16
> olcDbShmKey: 0
> olcDbCacheFree: 1
> olcDbDNcacheSize: 0
> olcPlugin: postoperation /opt/trinity/lib/slapi-acl-manager.so plugin_init
> admingroup-dn:=cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>
> realm:=CEET.NIU.EDU aclfile:=/etc/heimdal-kdc/kadmind.acl  builtinadmin:=admin
> structuralObjectClass: olcHdbConfig
> creatorsName: cn=config
> createTimestamp: 20130702170316Z
> olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcSyncrepl: {1}rid=002 provider=ldaps://ldap003.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcMirrorMode: TRUE
> entryCSN: 20130702170511.039863Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702170511Z
>
>
> olcDatabase={0}config/olcOverlay={0}syncprov.ldif:
>
> dn: olcOverlay={0}syncprov
> objectClass: olcOverlayConfig
> objectClass: olcSyncProvConfig
> olcOverlay: {0}syncprov
> structuralObjectClass: olcSyncProvConfig
> entryUUID: 14f25934-7785-1032-93bd-0fe5d581c3b6
> creatorsName:
> createTimestamp: 20130702170337Z
> entryCSN: 20130702170337.631532Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702170337Z
>
>
> olcDatabase={1}hdb/olcOverlay={0}syncprov.ldif:
>
> dn: olcOverlay={0}syncprov
> objectClass: olcOverlayConfig
> objectClass: olcSyncProvConfig
> olcOverlay: {0}syncprov
> structuralObjectClass: olcSyncProvConfig
> entryUUID: bfa2e530-778d-1032-9b41-e7480f731418
> creatorsName:
> createTimestamp: 20130702180539Z
> entryCSN: 20130702180539.975057Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702180539Z
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/