Since my report above seems to have been somewhat mangled by the
submission process, here is a link to a plain-text version of the same
report:
http://www.ceet.niu.edu/pastebin/openldap_bug_report_7636.txt

kb9vqf@pearsoncomputing.net wrote:
> Full_Name: Timothy Pearson
> Version: 2.4.31
> OS: Debian Wheezy
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (131.156.2.26)
>
>
> The setup:
> Multi-master syncrepl on two servers
> Identical hardware and software between servers
> Self-signed TLS using common (private) CA certificate
>
> The problem:
> slapd on one server crashes repeatably within a minute of slapd starting on the
> other server.  slapd works reliably if and only if the other server is not
> running a slapd process.

1) 2.4.31 is ancient. Current is 2.4.35. Please provide a backtrace against a 
current OpenLDAP release.

2) Your backtrace shows a crash in a slapi plugin. If the bug is in your 
plugin there's nothing we can do about it.

3) Don't touch the individual files inside the slapd configuration database. 
Use "slapcat -n0".

>
> Backtrace (does not change appreciably from crash to crash):
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7fffe356c700 (LWP 10433)]
> 0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
> #2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
> fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> #3  0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
> "munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
> #4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe3569fd0,
> rs=<optimized out>, prc=<optimized out>) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:313
> #5  slapi_op_search_callback (op=0x7fffe3569fd0, rs=<optimized out>,
> prc=<optimized out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:296
> #6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe3569f60, op=<optimized out>)
> at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
> #7  slapi_op_func (op=0x7fffe3569fd0, rs=0x7fffe3569f60) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:556
> #8  0x00005555555ff18a in overlay_op_walk (op=op@entry=0x7fffe3569fd0,
> rs=0x7fffe3569f60, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
> ../../../../servers/slapd/backover.c:661
> #9  0x00005555555ff31b in over_op_func (op=0x7fffe3569fd0, rs=<optimized out>,
> which=<optimized out>) at ../../../../servers/slapd/backover.c:723
> #10 0x00007ffff1c4870e in syncprov_findbase (op=op@entry=0x555555dd8560,
> fc=fc@entry=0x7fffe356a280) at
> ../../../../../servers/slapd/overlays/syncprov.c:453
> #11 0x00007ffff1c4b1ef in syncprov_op_search (op=0x555555dd8560,
> rs=0x7fffe356ba50) at ../../../../../servers/slapd/overlays/syncprov.c:2465
> #12 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
> rs=rs@entry=0x7fffe356ba50, which=which@entry=op_search, oi=0x5555559e57f0,
> on=0x5555559e15e0) at ../../../../servers/slapd/backover.c:661
> #13 0x00007ffff63d3086 in slapi_op_func (rs=0x7fffe356ba50, op=<optimized out>)
> at ../../../../../servers/slapd/slapi/slapi_overlay.c:647
> #14 slapi_op_func (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../../servers/slapd/slapi/slapi_overlay.c:556
> #15 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555dd8560,
> rs=0x7fffe356ba50, which=op_search, oi=0x5555559e57f0, on=0x5555559e4510) at
> ../../../../servers/slapd/backover.c:661
> #16 0x00005555555ff31b in over_op_func (op=0x555555dd8560, rs=<optimized out>,
> which=<optimized out>) at ../../../../servers/slapd/backover.c:723
> #17 0x0000555555594641 in fe_op_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../servers/slapd/search.c:402
> #18 0x0000555555593f06 in do_search (op=0x555555dd8560, rs=0x7fffe356ba50) at
> ../../../../servers/slapd/search.c:247
> #19 0x0000555555591961 in connection_operation (ctx=ctx@entry=0x7fffe356bba0,
> arg_v=arg_v@entry=0x555555dd8560) at
> ../../../../servers/slapd/connection.c:1150
> #20 0x0000555555591c84 in connection_read_thread (ctx=0x7fffe356bba0,
> argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286
> #21 0x00007ffff7b9dff3 in ?? () from
> /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
> #22 0x00007ffff5d90b50 in start_thread (arg=<optimized out>) at
> pthread_create.c:304
> #23 0x00007ffff5adaa7d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #24 0x0000000000000000 in ?? ()
>
>
> Relevant configuration files:
>
>
> olcDatabase={0}config.ldif:
>
> dn: olcDatabase={0}config
> objectClass: olcDatabaseConfig
> olcDatabase: {0}config
> olcAccess: {0}to * by
> group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
> by dn.base="cn=admin,dc=<redacted>" write by sockurl.regex="^ldapi:///$" write
> by dynacl/aci write by * none
> olcAddContentAcl: TRUE
> olcLastMod: TRUE
> olcMaxDerefDepth: 15
> olcReadOnly: FALSE
> olcRootDN: cn=config
> olcRootPW:: REDACTED
> olcSyncUseSubentry: FALSE
> olcMonitoring: FALSE
> structuralObjectClass: olcDatabaseConfig
> creatorsName: cn=config
> createTimestamp: 20130702170316Z
> olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"lcSyncrepl:
> {1}rid=002 provider=ldaps://ldap003.<redacted>/ binddn="cn=admin,dc=<redacted>"
> bindmethod=simple credentials="ldapadmin001" searchbase="cn=config"
> type=refreshAndPersist retry="5 5 300 5 600 +" timeout=1 tls_reqcert=never
> tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcMirrorMode: TRUE
> entryCSN: 20130702180604.682031Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702180604Z
>
>
> olcDatabase={1}hdb.ldif:
>
> dn: olcDatabase={1}hdb
> objectClass: olcDatabaseConfig
> objectClass: olcHdbConfig
> olcDatabase: {1}hdb
> olcDbDirectory: /var/lib/ldap
> olcSuffix: dc=<redacted>
> olcAccess: {0}to attrs=userPassword,shadowLastChange,krb5Key,krb5PrincipalName,krb5KeyVersionNumber,krb5MaxLife,krb5MaxRenew,krb5KDCFlags,privateRootCertificateKey
> by group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" by
> sockurl.regex="^ldapi:///$" write by anonymous auth by self write by * none
> olcAccess: {1}to dn.base="" by * read
> olcAccess: {2}to * by
> group/groupOfNames/member.exact="cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>"
> write by dn.base="uid=ldapadmin,ou=users,ou=core,ou=realm,dc=<redacted>" write
> by sockurl.regex="^ldapi:///$" write by dynacl/aci write
> olcAddContentAcl: FALSE
> olcLastMod: TRUE
> olcMaxDerefDepth: 15
> olcReadOnly: FALSE
> olcRootDN: cn=admin,dc=<redacted>
> olcRootPW:: REDACTED
> olcMonitoring: TRUE
> olcDbCacheSize: 1000
> olcDbCheckpoint: 512 30
> olcDbConfig: {0}set_cachesize 0 67108864 1
> olcDbConfig: {1}set_lg_regionmax 262144
> olcDbConfig: {2}set_lg_bsize 2097152
> olcDbNoSync: FALSE
> olcDbDirtyRead: FALSE
> olcDbIDLcacheSize: 0
> olcDbIndex: objectClass eq
> olcDbIndex: krb5PrincipalName eq,pres
> olcDbIndex: cn eq,pres,subinitial
> olcDbIndex: mail eq,pres
> olcDbIndex: uid pres,eq
> olcDbIndex: uidNumber eq
> olcDbIndex: gidNumber eq
> olcDbLinearIndex: FALSE
> olcDbMode: 0600
> olcDbSearchStack: 16
> olcDbShmKey: 0
> olcDbCacheFree: 1
> olcDbDNcacheSize: 0
> olcPlugin: postoperation /opt/trinity/lib/slapi-acl-manager.so plugin_init
> admingroup-dn:=cn=realmadmins,ou=groups,ou=core,ou=realm,dc=<redacted>
> realm:=CEET.NIU.EDU aclfile:=/etc/heimdal-kdc/kadmind.acl  builtinadmin:=admin
> structuralObjectClass: olcHdbConfig
> creatorsName: cn=config
> createTimestamp: 20130702170316Z
> olcSyncrepl: {0}rid=001 provider=ldaps://ldap002.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcSyncrepl: {1}rid=002 provider=ldaps://ldap003.<redacted>/
> binddn="cn=admin,dc=<redacted>" bindmethod=simple credentials="ldapadmin001"
> searchbase="dc=<redacted>" type=refreshAndPersist retry="5 5 300 5" timeout=1
> tls_reqcert=never tls_cacert="/etc/trinity/ldap/tde-ca/anchors/tdeca.pem"
> olcMirrorMode: TRUE
> entryCSN: 20130702170511.039863Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702170511Z
>
>
> olcDatabase={0}config/olcOverlay={0}syncprov.ldif:
>
> dn: olcOverlay={0}syncprov
> objectClass: olcOverlayConfig
> objectClass: olcSyncProvConfig
> olcOverlay: {0}syncprov
> structuralObjectClass: olcSyncProvConfig
> entryUUID: 14f25934-7785-1032-93bd-0fe5d581c3b6
> creatorsName:
> createTimestamp: 20130702170337Z
> entryCSN: 20130702170337.631532Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702170337Z
>
>
> olcDatabase={1}hdb/olcOverlay={0}syncprov.ldif:
>
> dn: olcOverlay={0}syncprov
> objectClass: olcOverlayConfig
> objectClass: olcSyncProvConfig
> olcOverlay: {0}syncprov
> structuralObjectClass: olcSyncProvConfig
> entryUUID: bfa2e530-778d-1032-9b41-e7480f731418
> creatorsName:
> createTimestamp: 20130702180539Z
> entryCSN: 20130702180539.975057Z#000000#002#000000
> modifiersName:
> modifyTimestamp: 20130702180539Z
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

> kb9vqf@pearsoncomputing.net wrote:
>> Full_Name: Timothy Pearson
>> Version: 2.4.31
>> OS: Debian Wheezy
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (131.156.2.26)
>>
>>
>> The setup:
>> Multi-master syncrepl on two servers
>> Identical hardware and software between servers
>> Self-signed TLS using common (private) CA certificate
>>
>> The problem:
>> slapd on one server crashes repeatably within a minute of slapd
starting on the
>> other server.  slapd works reliably if and only if the other server is not
>> running a slapd process.
>
> 1) 2.4.31 is ancient. Current is 2.4.35. Please provide a backtrace
against a
> current OpenLDAP release.
>
> 2) Your backtrace shows a crash in a slapi plugin. If the bug is in your
plugin there's nothing we can do about it.
>
> 3) Don't touch the individual files inside the slapd configuration
database.
> Use "slapcat -n0".

Thank you for your prompt response.  In the backtrace I posted, which
portion suggests that the crash occurred in a plugin?  I would have
expected to see at least one frame in something other than
../../../../servers/slapd.

I am not directly modifying the config database text files, but did not
know the correct method to dump the existing files.  Thanks for the
command!

kb9vqf@pearsoncomputing.net wrote:
>> kb9vqf@pearsoncomputing.net wrote:
>>> Full_Name: Timothy Pearson
>>> Version: 2.4.31
>>> OS: Debian Wheezy
>>> URL: ftp://ftp.openldap.org/incoming/
>>> Submission from: (NULL) (131.156.2.26)
>>>
>>>
>>> The setup:
>>> Multi-master syncrepl on two servers
>>> Identical hardware and software between servers
>>> Self-signed TLS using common (private) CA certificate
>>>
>>> The problem:
>>> slapd on one server crashes repeatably within a minute of slapd
> starting on the
>>> other server.  slapd works reliably if and only if the other server is not
>>> running a slapd process.
>>
>> 1) 2.4.31 is ancient. Current is 2.4.35. Please provide a backtrace
> against a
>> current OpenLDAP release.
>>
>> 2) Your backtrace shows a crash in a slapi plugin. If the bug is in your
> plugin there's nothing we can do about it.
>>
>> 3) Don't touch the individual files inside the slapd configuration
> database.
>> Use "slapcat -n0".
>
> Thank you for your prompt response.  In the backtrace I posted, which
> portion suggests that the crash occurred in a plugin?  I would have
> expected to see at least one frame in something other than
> ../../../../servers/slapd.


Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe356c700 (LWP 10433)]
0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
#2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3  0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
"munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe3569fd0,
rs=<optimized out>, prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:313
#5  slapi_op_search_callback (op=0x7fffe3569fd0, rs=<optimized out>,
prc=<optimized out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:296
#6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe3569f60, op=<optimized out>)
at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
#7  slapi_op_func (op=0x7fffe3569fd0, rs=0x7fffe3569f60) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556

The slapi_* functions would not be present in the stack trace unless you had 
configured a slapi plugin on this database.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

I have done further testing, and it appears that simply enabling a dummy
plugin is enough to cause slapd to crash.  This plugin code alone is
sufficient to cause a crash when syncrepl is enabled:

#include "slapi-plugin.h"

int plugin_init (Slapi_PBlock *pb);
int internal_plugin_init (Slapi_PBlock *pb);

__attribute__ ((visibility ("default"))) int plugin_init (Slapi_PBlock *pb) {
    return 0;
}

Therefore, it would seem that there is actually a bug in the openldap
code, not in a third-party plugin as originally thought.

What do you need to investigate this bug further?

Thanks!

Here is the backtrace with the above dummy plugin on the latest OpenLDAP
(2.4.35):

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe34a3700 (LWP 15269)]
0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
#2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3  0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
"munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe34a0ff0,
rs=<optimized out>, prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:313
#5  slapi_op_search_callback (op=0x7fffe34a0ff0, rs=<optimized out>,
prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:296
#6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe34a0f80, op=<optimized
out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
#7  slapi_op_func (op=0x7fffe34a0ff0, rs=0x7fffe34a0f80) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#8  0x00005555555ff18a in overlay_op_walk (op=op@entry=0x7fffe34a0ff0,
rs=0x7fffe34a0f80, which=op_search, oi=0x5555559e5680, on=0x5555559e43a0)
at ../../../../servers/slapd/backover.c:661
#9  0x00005555555ff31b in over_op_func (op=0x7fffe34a0ff0, rs=<optimized
out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#10 0x00007ffff1c4a45b in syncprov_findbase (op=op@entry=0x555555f093c0,
fc=fc@entry=0x7fffe34a1290) at syncprov.c:465
#11 0x00007ffff1c4cd87 in syncprov_op_search (op=0x555555f093c0,
rs=0x7fffe34a2a50) at syncprov.c:2461
#12 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555f093c0,
rs=rs@entry=0x7fffe34a2a50, which=which@entry=op_search,
oi=0x5555559e5680, on=0x5555559bfbe0) at
../../../../servers/slapd/backover.c:661
#13 0x00007ffff63d3086 in slapi_op_func (rs=0x7fffe34a2a50, op=<optimized
out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:647
#14 slapi_op_func (op=0x555555f093c0, rs=0x7fffe34a2a50) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#15 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555f093c0,
rs=0x7fffe34a2a50, which=op_search, oi=0x5555559e5680, on=0x5555559e43a0)
at ../../../../servers/slapd/backover.c:661
#16 0x00005555555ff31b in over_op_func (op=0x555555f093c0, rs=<optimized
out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#17 0x0000555555594641 in fe_op_search (op=0x555555f093c0,
rs=0x7fffe34a2a50) at ../../../../servers/slapd/search.c:402
#18 0x0000555555593f06 in do_search (op=0x555555f093c0, rs=0x7fffe34a2a50)
at ../../../../servers/slapd/search.c:247
#19 0x0000555555591961 in connection_operation
(ctx=ctx@entry=0x7fffe34a2ba0, arg_v=arg_v@entry=0x555555f093c0) at
../../../../servers/slapd/connection.c:1150
#20 0x0000555555591c84 in connection_read_thread (ctx=0x7fffe34a2ba0,
argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286
#21 0x00007ffff7b9dff3 in ?? () from
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
#22 0x00007ffff5d90b50 in start_thread (arg=<optimized out>) at
pthread_create.c:304
#23 0x00007ffff5adaa7d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()
(gdb)

I have traced the fault into the syncrepl overlay, specifically it passes
a static global variable to be_search().  If a plugin is configured,
slapi_op_search_callback() is called, which then attempts to free the
static global search variable passed to be_search(), causing a crash.

The attached patch fixes the problem on my test system.

Unfortunately the previous patch only fixes some of the most obvious
(immediate) crashes.  There are other places in the code (e.g. syncprov.c
line 647) which pass non-malloc()ed variables to be_search() and thereby
crash slapd.

Has syncrepl even been tested with plugins before?  We really wanted to
use OpenLDAP for our directory server, however if replication cannot be
used with plugins then we may need to go with a Microsoft solution
instead.

Any hints?

kb9vqf@pearsoncomputing.net wrote:
> Unfortunately the previous patch only fixes some of the most obvious
> (immediate) crashes.  There are other places in the code (e.g. syncprov.c
> line 647) which pass non-malloc()ed variables to be_search() and thereby
> crash slapd.
>
> Has syncrepl even been tested with plugins before?  We really wanted to
> use OpenLDAP for our directory server, however if replication cannot be
> used with plugins then we may need to go with a Microsoft solution
> instead.
>
> Any hints?

Seems like the problem should be fixed in the slapi support code. And in 
general, the slapi code receives very little attention since slapd overlays 
are the native plugin mechanism. Patching syncprov.c is definitely the wrong 
way to go though.

What are the plugins you're trying to use? Are you sure there isn't already a 
slapd overlay that provides the feature?

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

kb9vqf@pearsoncomputing.net wrote:
> Unfortunately the previous patch only fixes some of the most obvious
> (immediate) crashes.  There are other places in the code (e.g. syncprov.c
> line 647) which pass non-malloc()ed variables to be_search() and thereby
> crash slapd.
>
> Has syncrepl even been tested with plugins before?  We really wanted to
> use OpenLDAP for our directory server, however if replication cannot be
> used with plugins then we may need to go with a Microsoft solution
> instead.
>
> Any hints?

Somewhat of a workaround - slapi should not be doing anything here unless the 
filter was actually changed.

syncprov is using (objectclass=*) which frankly no plugin should be rewriting, 
so this will cover it for the most part. Aside from that it looks like this 
area of the slapi interaction needs some redesign.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

> kb9vqf@pearsoncomputing.net wrote:
>> Unfortunately the previous patch only fixes some of the most obvious
(immediate) crashes.  There are other places in the code (e.g.
>> syncprov.c
>> line 647) which pass non-malloc()ed variables to be_search() and
thereby crash slapd.
>>
>> Has syncrepl even been tested with plugins before?  We really wanted to
use OpenLDAP for our directory server, however if replication cannot be
used with plugins then we may need to go with a Microsoft solution
instead.
>>
>> Any hints?
>
> Seems like the problem should be fixed in the slapi support code. And in
general, the slapi code receives very little attention since slapd
overlays
> are the native plugin mechanism. Patching syncprov.c is definitely the
wrong
> way to go though.

OK, thanks for the info.  I don't know much acout the internals of
OpenLDAP, but I can still see where non-malloc()ed memory is being passed
(eventually) to a free()-type function. ;-)

> What are the plugins you're trying to use? Are you sure there isn't
already a
> slapd overlay that provides the feature?

We use a custom plugin to keep Kerberos ACLs in sync with group
memberships.  The plugin only monitors for certain changes to the
directory, it does not modify the directory in any way.

> Somewhat of a workaround - slapi should not be doing anything here
unless the
> filter was actually changed.
>
> syncprov is using (objectclass=*) which frankly no plugin should be
rewriting,
> so this will cover it for the most part. Aside from that it looks like this
> area of the slapi interaction needs some redesign.
>
> --

Unfortunately the workaround didn't help:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe2ca2700 (LWP 29843)]
0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
directory. (gdb) bt
#0  0x00007ffff5a32475 in *__GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff5a356f0 in *__GI_abort () at abort.c:92
#2  0x00007ffff5a6d52b in __libc_message (do_abort=<optimized out>,
fmt=<optimized out>) at ../sysdeps/unix/sysv/linux/libc_fatal.c:189 #3 
0x00007ffff5a76d76 in malloc_printerr (action=3, str=0x7ffff5b4f170
"munmap_chunk(): invalid pointer", ptr=<optimized out>) at malloc.c:6283
#4  0x00007ffff63d214a in slapi_op_search_callback (op=0x7fffe2c9fff0,
rs=<optimized out>, prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:313
#5  slapi_op_search_callback (op=0x7fffe2c9fff0, rs=<optimized out>,
prc=<optimized out>) at
../../../../../servers/slapd/slapi/slapi_overlay.c:296
#6  0x00007ffff63d304f in slapi_op_func (rs=0x7fffe2c9ff80, op=<optimized
out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:631
#7  slapi_op_func (op=0x7fffe2c9fff0, rs=0x7fffe2c9ff80) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#8  0x00005555555ff18a in overlay_op_walk (op=op@entry=0x7fffe2c9fff0,
rs=0x7fffe2c9ff80, which=op_search, oi=0x5555559e5680, on=0x5555559e43a0)
at ../../../../servers/slapd/backover.c:661
#9  0x00005555555ff31b in over_op_func (op=0x7fffe2c9fff0, rs=<optimized
out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#10 0x00007ffff1c4a45b in syncprov_findbase (op=op@entry=0x555555eeff60,
fc=fc@entry=0x7fffe2ca0290) at syncprov.c:465
#11 0x00007ffff1c4cd87 in syncprov_op_search (op=0x555555eeff60,
rs=0x7fffe2ca1a50) at syncprov.c:2461
#12 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555eeff60,
rs=rs@entry=0x7fffe2ca1a50, which=which@entry=op_search,
oi=0x5555559e5680, on=0x5555559bfbe0) at
../../../../servers/slapd/backover.c:661
#13 0x00007ffff63d3086 in slapi_op_func (rs=0x7fffe2ca1a50, op=<optimized
out>) at ../../../../../servers/slapd/slapi/slapi_overlay.c:647
#14 slapi_op_func (op=0x555555eeff60, rs=0x7fffe2ca1a50) at
../../../../../servers/slapd/slapi/slapi_overlay.c:556
#15 0x00005555555ff18a in overlay_op_walk (op=op@entry=0x555555eeff60,
rs=0x7fffe2ca1a50, which=op_search, oi=0x5555559e5680, on=0x5555559e43a0)
at ../../../../servers/slapd/backover.c:661
#16 0x00005555555ff31b in over_op_func (op=0x555555eeff60, rs=<optimized
out>, which=<optimized out>) at ../../../../servers/slapd/backover.c:723
#17 0x0000555555594641 in fe_op_search (op=0x555555eeff60,
rs=0x7fffe2ca1a50) at ../../../../servers/slapd/search.c:402
#18 0x0000555555593f06 in do_search (op=0x555555eeff60, rs=0x7fffe2ca1a50)
at ../../../../servers/slapd/search.c:247
#19 0x0000555555591961 in connection_operation
(ctx=ctx@entry=0x7fffe2ca1ba0, arg_v=arg_v@entry=0x555555eeff60) at
../../../../servers/slapd/connection.c:1150
#20 0x0000555555591c84 in connection_read_thread (ctx=0x7fffe2ca1ba0,
argv=<optimized out>) at ../../../../servers/slapd/connection.c:1286 #21
0x00007ffff7b9dff3 in ?? () from
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
#22 0x00007ffff5d90b50 in start_thread (arg=<optimized out>) at
pthread_create.c:304
#23 0x00007ffff5adaa7d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#24 0x0000000000000000 in ?? ()
(gdb)

Valgrind doesn't turn up anything strange before the crash, so I guess it
comes down to design error in a relatively untested code path.

Tim

> Somewhat of a workaround - slapi should not be doing anything here unless
> the
> filter was actually changed.
>
> syncprov is using (objectclass=*) which frankly no plugin should be
> rewriting,
> so this will cover it for the most part. Aside from that it looks like
> this
> area of the slapi interaction needs some redesign.

Ignore my earlier noise please--an old (unpatched) libslapi library was
present on the system from earlier debugging attempts, this was what
caused the recent crash.

Your patch seems to work!  Thank you for your quick response; we now have
a shot at going forward with the OpenLDAP deployment.

Tim

changed notes
changed state Open to Test
moved from Incoming to Software Bugs

changed notes
changed state Test to Partial

partial fix in master
partial fix in RE24

It is one more relevant bug with patch
http://www.OpenLDAP.org/its/index.cgi?findid=7965