The code making the update is in perl, using Net::LDAP, and looks like:

$entry->replace(eduPersonAffiliation => \@eduPersonAffiliation);

For the case of the crash, the array @eduPersonAffiliation is empty,
effectively deleting the attribute. I'm not 100% sure what this
generates on the wire, I'd have to debug it. I can say it's the same
code that's been running for a decade or so with no issues.

The schema that defines the attribute is available at:

https://spaces.internet2.edu/display/macedir/OpenLDAP+eduPerson

Hi Paul,
I've tried to reproduce the issue you're seeing, setting up deltasync
replication and trying to send various combinations of modifies to the
master. However, none of that has crashed either the server receiving
the modification, nor the other servers it replicated to. No change
either when the second server has to replicate directly off the
accesslog database without the benefit of syncprov's sessionlog.

Is that something that you can replicate reliably? Would you be able to
share the configuration (especially overlays)?

Based on the traces I've taken while testing, I haven't seen anything
suspicious that the perl library would generate, but if you have the
packets or slapd ber dumps, they might help confirm this for your case
as well.

Regards,
Ondrej

> From: Ondřej Kuzník
> Sent: Thursday, March 9, 2017 5:32 AM
> 
> Is that something that you can replicate reliably? Would you be able to
> share the configuration (especially overlays)?

Unfortunately it is not deterministically reproducible, it just randomly
happens occasionally. I haven't noticed anything in particular that seems to
be a pattern or in common on each occurrence. I will shoot you a copy of my
configuration directly.

> Based on the traces I've taken while testing, I haven't seen anything
> suspicious that the perl library would generate, but if you have the
> packets or slapd ber dumps, they might help confirm this for your case
> as well.

I haven't been able to get a packet capture or leave debugging on long
enough to catch it since the crash is so sporadic and it only occurs on a
production system at widely interspersed intervals :(.

Thanks for looking at it.

moved from Incoming to Software Bugs

Hi Paul,
so far I've been unable to reproduce the issue, but going through the
traceback again, there is something odd. In frame #5, syncrepl.c:2394
reads "rc = op->o_bd->be_add( op, &rs );" as per re24[0], but in your
traceback, over_op_func calls overlay_op_walk with "which=op_modify"
(see frame #3) which doesn't really make sense to me.

[0]. http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=blob;f=servers/slapd/syncrepl.c;hb=refs/heads/OPENLDAP_REL_ENG_2_4#l2394

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

> From: Ondřej Kuzník
> Sent: Thursday, March 23, 2017 5:29 AM
>
> so far I've been unable to reproduce the issue, but going through the
> traceback again, there is something odd. In frame #5, syncrepl.c:2394
> reads "rc = op->o_bd->be_add( op, &rs );" as per re24[0], but in your
> traceback, over_op_func calls overlay_op_walk with "which=op_modify"
> (see frame #3) which doesn't really make sense to me.

As I mentioned in the report, this is 2.4.44 with the ITS 8432 patch applied to prevent the infinite replication issue from occurring in our production environment. With that patch applied, line 2394 of syncrepl.c is:

                        rc = op->o_bd->be_modify( op, &rs );

Sorry for any confusion that caused :(.

On Thu, Mar 23, 2017 at 05:51:46PM -0700, Paul B. Henson wrote:
>> From: Ondřej Kuzník
>> Sent: Thursday, March 23, 2017 5:29 AM
>>
>> so far I've been unable to reproduce the issue, but going through the
>> traceback again, there is something odd. In frame #5, syncrepl.c:2394
>> reads "rc = op->o_bd->be_add( op, &rs );" as per re24[0], but in your
>> traceback, over_op_func calls overlay_op_walk with "which=op_modify"
>> (see frame #3) which doesn't really make sense to me.
> 
> As I mentioned in the report, this is 2.4.44 with the ITS 8432 patch
> applied to prevent the infinite replication issue from occurring in
> our production environment. With that patch applied, line 2394 of
> syncrepl.c is:
> 
>                         rc = op->o_bd->be_modify( op, &rs );
> 
> Sorry for any confusion that caused :(.

Looks like your patch differs from what was applied to re24 at least on
that line (which calls be_add instead). Do you know why that is? Could
you try with the current re24 branch (OPENLDAP_REL_ENG_2_4) instead?

Thanks

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

> From: Ondřej Kuzník
> Sent: Friday, March 24, 2017 2:28 AM
> 
> Looks like your patch differs from what was applied to re24 at least on
> that line (which calls be_add instead). Do you know why that is? Could
> you try with the current re24 branch (OPENLDAP_REL_ENG_2_4) instead?

Ah, more confusion, sorry; I confirmed with Quanah the ITS 8432 patch I applied is identical to the one applied upstream, however, the line I referred to is in the patch applied to 2.4.44 release code, whereas I see you are referring to the current head of the 2.4 branch, which I'm sure has had numerous other changes applied to it that have probably shuffled things around a bit.

If I could reliably reproduce this issue :(, I would definitely test it on our dev systems with whatever code you requested. Unfortunately, it only randomly occurs every 2-4 weeks in production, and I don't think I could get management approval to run the unreleased development branch of code in production for that long to see what happens. The 2.4.45 testing call came out last month, is there a timeline yet on getting that out? I should be able to push that into production fairly soon after it is released and see what happens there.

Thanks much…

--On Friday, March 24, 2017 8:07 PM +0000 henson@acm.org wrote:

>> From: Ond=C5=99ej Kuzn=C3=ADk
>> Sent: Friday, March 24, 2017 2:28 AM
>> =20
>> Looks like your patch differs from what was applied to re24 at least =
> on
>> that line (which calls be_add instead). Do you know why that is? Could
>> you try with the current re24 branch (OPENLDAP_REL_ENG_2_4) instead?
>
> Ah, more confusion, sorry; I confirmed with Quanah the ITS 8432 patch I =
> applied is identical to the one applied upstream, however, the line I =
> referred to is in the patch applied to 2.4.44 release code, whereas I =
> see you are referring to the current head of the 2.4 branch, which I'm =
> sure has had numerous other changes applied to it that have probably =
> shuffled things around a bit.
>
> If I could reliably reproduce this issue :(, I would definitely test it =
> on our dev systems with whatever code you requested. Unfortunately, it =
> only randomly occurs every 2-4 weeks in production, and I don't think I =
> could get management approval to run the unreleased development branch =
> of code in production for that long to see what happens. The 2.4.45 =
> testing call came out last month, is there a timeline yet on getting =
> that out? I should be able to push that into production fairly soon =
> after it is released and see what happens there.

There's too many outstanding issues to release 2.4.45, including this one.

The remaining changes to syncrepl.c between 2.4.44 and current RE25 are:

<http://www.openldap.org/its/index.cgi/?findid=8435>
<http://www.openldap.org/its/index.cgi/?findid=8413>

I would consider at least the patch for ITS#8413 as mandatory as well. 
Since ITS#8435 has been around since 2.4.40, it's unlikely to be related.

<https://github.com/Zimbra/packages/blob/master/thirdparty/openldap/patches/ITS8413.patch> 
is an easy spot to grab it from.

--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

On Fri, Mar 24, 2017 at 12:07:32PM -0700, Paul B. Henson wrote:
> Ah, more confusion, sorry; I confirmed with Quanah the ITS 8432 patch
> I applied is identical to the one applied upstream, however, the line
> I referred to is in the patch applied to 2.4.44 release code, whereas
> I see you are referring to the current head of the 2.4 branch, which
> I'm sure has had numerous other changes applied to it that have
> probably shuffled things around a bit.

Hi Paul,
yes it did, I should have checked for that and would have if the wrong
line didn't line up so well.

> If I could reliably reproduce this issue :(, I would definitely test
> it on our dev systems with whatever code you requested. Unfortunately,
> it only randomly occurs every 2-4 weeks in production, and I don't
> think I could get management approval to run the unreleased
> development branch of code in production for that long to see what
> happens. The 2.4.45 testing call came out last month, is there a
> timeline yet on getting that out? I should be able to push that into
> production fairly soon after it is released and see what happens
> there.

I've had a look whether I could reproduce the issue somehow and there is
a potential crasher if the accesslog entry contained "reqmod:
eduPersonAffiliation:+". Can you confirm whether you have entries like
this in your logdb? There shouldn't be a way to have this kind of entry
generated that I could see but processing one can sometimes cause
a crash like the one you've been seeing.

The following patch should fix an issue where an invalid accesslog entry
of this sort is encountered. It should help you if that is the issue.

ftp://ftp.openldap.org/incoming/Ondrej-Kuznik-20170327-ITS8609-ignore-invalid-accesslog-ops.patch

Regards,
Ondrej

> From: Quanah Gibson-Mount
> Sent: Monday, March 27, 2017 8:09 AM
> 
> There's too many outstanding issues to release 2.4.45, including this one.

Well, are those new issues that don't exist in 2.4.44, or simply existing
bugs already in the wild that haven't been fixed? If the former, then
certainly you don't want to release fresh bugs 8-/, but if the latter, it
seems to me that pushing a release that improves the state of production by
fixing some known bugs, even if other existing known bugs still remain to be
quashed, is preferable than leaving a larger number of known bugs at large
on production systems :(.  It seems the circular replication bug is
particularly bad, I'm surprised more people aren't having issues with it.

> I would consider at least the patch for ITS#8413 as mandatory as well.

Hmm, I haven't run into that one yet myself, but thanks for the pointer.

> From: Ondřej Kuzník
> Sent: Monday, March 27, 2017 8:10 AM
> 
> I've had a look whether I could reproduce the issue somehow and there is
> a potential crasher if the accesslog entry contained "reqmod:
> eduPersonAffiliation:+". Can you confirm whether you have entries like
> this in your logdb?

There aren't currently any entries like that in the log, although the last
crash was a week or two ago. I will check again right after the next crash.
However, here is one of the log entries that appeared to correlate with what
was being processed in the backtrace of one of the crashes:

dn: reqStart=20170214120013.000001Z,cn=accesslog

objectClass: auditModify

structuralObjectClass: auditModify

reqStart: 20170214120013.000001Z

reqEnd: 20170214120013.000002Z

reqType: modify

reqSession: 37859

reqAuthzID: cn=idmgmt,ou=user,ou=service,dc=csupomona,dc=edu

reqDN: uid=vntruong,ou=user,dc=csupomona,dc=edu

reqResult: 0

reqMod: eduPersonAffiliation:=

reqMod: eduPersonPrimaryAffiliation:=

reqMod: csupomonaEduPersonAffiliation:=

reqMod: entryCSN:= 20170214120013.628665Z#000000#000#000000

reqMod: modifiersName:= cn=idmgmt,ou=user,ou=service,dc=csupomona,dc=edu

reqMod: modifyTimestamp:= 20170214120013Z

reqEntryUUID: bd5ba51c-7a1b-4bdb-8bf3-fe90552f5909

entryCSN: 20170214120013.628665Z#000000#000#000000

entryUUID: 4737c48c-e46e-45a4-ba4b-2eb61540b27b

creatorsName: cn=accesslog

createTimestamp: 20170214120013Z

modifiersName: cn=accesslog

modifyTimestamp: 20170214120013Z


And it does not appear to have that signature.

Thanks.

--On Thursday, March 30, 2017 8:20 PM +0000 henson@acm.org wrote:

>> From: Quanah Gibson-Mount
>> Sent: Monday, March 27, 2017 8:09 AM
>>
>> There's too many outstanding issues to release 2.4.45, including this
>> one.
>
> Well, are those new issues that don't exist in 2.4.44, or simply existing
> bugs already in the wild that haven't been fixed? If the former, then
> certainly you don't want to release fresh bugs 8-/, but if the latter, it
> seems to me that pushing a release that improves the state of production
> by fixing some known bugs, even if other existing known bugs still remain
> to be quashed, is preferable than leaving a larger number of known bugs
> at large on production systems :(.  It seems the circular replication bug
> is particularly bad, I'm surprised more people aren't having issues with
> it.

2.4.43 introduced new bugs and those were continued in 2.4.44.  I think 
it's important that the regressions are addressed before 2.4.45 is 
released. In addition, this very bug report indicates a regression between 
2.4.44 and what's currently scheduled for 2.4.45, which is further 
reasoning not to release 2.4.45 at this time.

--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

> From: Quanah Gibson-Mount
> Sent: Thursday, March 30, 2017 3:14 PM
>
> released. In addition, this very bug report indicates a regression between
> 2.4.44 and what's currently scheduled for 2.4.45, which is further
> reasoning not to release 2.4.45 at this time.

Hmm, I started seeing this problem when I upgraded from 2.4.43 to 2.4.44+ITS
8432 patch. I've never run stock 2.4.44; so I can't say whether this issue
would occur without the ITS 8432 patch...

--On Friday, March 31, 2017 1:05 AM +0000 henson@acm.org wrote:

>> From: Quanah Gibson-Mount
>> Sent: Thursday, March 30, 2017 3:14 PM
>>
>> released. In addition, this very bug report indicates a regression
>> between 2.4.44 and what's currently scheduled for 2.4.45, which is
>> further reasoning not to release 2.4.45 at this time.
>
> Hmm, I started seeing this problem when I upgraded from 2.4.43 to
> 2.4.44+ITS 8432 patch. I've never run stock 2.4.44; so I can't say
> whether this issue would occur without the ITS 8432 patch...

Either way, it's a significant regression that needs fixing before I'd be 
comforable with a 2.4.45 release.  Interestingly, I never saw this problem 
@ Zimbra, even though it is constantly modifying data and has 
2.4.44+ITS8432. ;)  So I think there's something specific to your use case 
(the hardest kinds to track down!).

--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

> From: Quanah Gibson-Mount
> Sent: Thursday, March 30, 2017 5:16 PM
>
> 2.4.44+ITS8432. ;)  So I think there's something specific to your use case
> (the hardest kinds to track down!).

* Specific to my use case - check

* Not reproducible - check

* Only occurs sporadically at randomly dispersed intervals - check

Perhaps we've hit a heisenbug ;). I was kind of hoping it would get fixed by
accident by other updates in the next release :).

On Thu, Mar 30, 2017 at 12:30:09PM -0700, Paul B. Henson wrote:
>> From: Ondřej Kuzník
>> Sent: Monday, March 27, 2017 8:10 AM
>> 
>> I've had a look whether I could reproduce the issue somehow and there is
>> a potential crasher if the accesslog entry contained "reqmod:
>> eduPersonAffiliation:+". Can you confirm whether you have entries like
>> this in your logdb?
> 
> There aren't currently any entries like that in the log, although the last
> crash was a week or two ago. I will check again right after the next crash.
> However, here is one of the log entries that appeared to correlate with what
> was being processed in the backtrace of one of the crashes:
> 
> [...]
> 
> And it does not appear to have that signature.

Hi Paul,
"modify: replace" entries should result in modify_replace_values being
called instead, where modify_add_values is called when the mod is
LDAP_MOD_ADD (set up from reqMod values starting "attr:+").

Not sure if there is a way to reset the LDAPResult *msg in
syncrepl_message_to_op to retrieve the entry from there but that would
be the place I'd look when you get a another crash.

Regards,

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

So slapd crashed two days in a row, yesterday morning and this morning
:(. Same backtrace. What's worse whereas unlike today and every time
before it was relatively harmless, something *weird* happened yesterday.
I did my usual of just restarting slapd on the master and letting the
load balancer fail back over to it and having it catch back up on
replication from the changes made to the secondary, which has always
been fine before other than maybe having to tune up whatever change was
in the middle of being committed when the primary actually died.

But somehow yesterday things that happened like a week and a half ago
just mysteriously disappeared 8-/. 10 groups that were created on 3/22
were just *gone* without a trace. I had to go dig up ldif backups from
then and manually readd them back to the directory to clean stuff up. It
looks like there's some group membership corruption too, I'm going to
look at it in more detail tomorrow, it should be identical to our AD
environment so I can compare and clean up against that. It doesn't look
like any users fell into limbo, I diff'd the user DN's from the backup
the day before against the one after and they're all still there modulo
5 that were deleted intentionally.

I haven't been overly concerned about this issue, while it's been
annoying, it hasn't had much of a production impact. But it just turned
ugly :(, assuming this was caused by the crash, and I can't see how it
wouldn't have been...

<sigh>. Is there any logging or something I could turn on that would
help for the next occurance but wouldn't be an excessive impact in terms
of load or disk utilization on a production box?

Thanks...

henson@acm.org wrote:
> So slapd crashed two days in a row, yesterday morning and this morning
> :(. Same backtrace.

You should probably switch to a non-optimized build. The backtrace you posted 
initially had most parameters optimized out, which also makes things harder to 
trace.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

On Mon, Apr 03, 2017 at 02:44:48PM +0100, Howard Chu wrote:

> You should probably switch to a non-optimized build. The backtrace you
> posted initially had most parameters optimized out, which also makes
> things harder to trace.

Ok, I installed binaries built with -O0 on the master server that's been
failing; I'll post a new backtrace the next time it crashes, thanks.

On Mon, Apr 03, 2017 at 02:44:48PM +0100, Howard Chu wrote:

> You should probably switch to a non-optimized build. The backtrace you
> posted initially had most parameters optimized out, which also makes
> things harder to trace.

Well, as luck (8-/) would have it, it crashed both yesterday and today
on the non-optimized build. Here are the full backtraces from both
cores, I hope they provide some clue as to what's going on. Let me know
if you want me to do any more post mortem debugging on the cores. Thanks
much.


Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004b8dbb in modify_add_values (e=0x7fac037fca30, mod=0x7fabfc000a90, permissive=0, 
    text=0x7fac037fd000, textbuf=0x7fac037fca80 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/mods.c:61
61		if ( !BER_BVISNULL( &mod->sm_values[mod->sm_numvals] )) {
[Current thread is 1 (Thread 0x7fac037fe700 (LWP 1292))]
(gdb) bt full
#0  0x00000000004b8dbb in modify_add_values (e=0x7fac037fca30, mod=0x7fabfc000a90, permissive=0, 
    text=0x7fac037fd000, textbuf=0x7fac037fca80 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/mods.c:61
        rc = -67095872
        op = 0x5866e0 "add"
        a = 0x7fac037fc8a0
        pmod = {sm_desc = 0x16d53c0, sm_values = 0x0, sm_nvalues = 0x0, sm_numvals = 0, sm_op = 0, 
          sm_flags = 0, sm_type = {bv_len = 34, bv_val = 0x7fabfc000a78 "uid=bran\220\n"}}
        __PRETTY_FUNCTION__ = "modify_add_values"
#1  0x00007fadf301c774 in mdb_modify_internal (op=0x7fac037fd730, tid=0x1a993c0, modlist=0x7fabfc000a50, 
    e=0x7fac037fca30, text=0x7fac037fd000, 
    textbuf=0x7fac037fca80 "modify/delete: eduPersonAffiliation: no such attribute", textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/back-mdb/modify.c:137
        rc = 32684
        err = 0
        mod = 0x7fabfc000a90
        ml = 0x7fabfc000a90
        save_attrs = 0x7fabfc0032c0
        ap = 0x7fac00000000
        glue_attr_delete = 0
        got_delete = 0
        __PRETTY_FUNCTION__ = "mdb_modify_internal"
#2  0x00007fadf301e5f2 in mdb_modify (op=0x7fac037fd730, rs=0x7fac037fcfe0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/back-mdb/modify.c:624
        mdb = 0x7fadf7dbb010
        e = 0x7fabfc003270
        manageDSAit = 2
00\000\340\317\177\003\254\177\000\000\060\327\177\003\254\177\000\000 \313\177\003\254\177\000\000\000\000\000\000\260\000\000\000 \313\177\003\254\177\000\000\324\024\214\367\000\000\000\000(&\000\374\253\177\000\000\060\327\177\003\254\177\000\000p-n\001\000\000\000\000P/n\001\000\000\000\000\320%\000\374\253\177\000\000\370%\000\374\253\177\000\000`\322\177\003\254\177\000\000"...
        textlen = 256
        txn = 0x1a993c0
        opinfo = {moi_oe = {oe_next = {sle_next = 0x7fac037fcf90}, oe_key = 0x7fadf7dbb010}, 
          moi_txn = 0x1a993c0, moi_ref = 1, moi_flag = 0 '\000'}
        moi = 0x7fac037fc9e0
        dummy = {e_id = 171976, e_name = {bv_len = 34, 
            bv_val = 0x7fabfc003240 "uid=brandonl,ou=user,dc=cpp,dc=edu"}, e_nname = {bv_len = 34, 
            bv_val = 0x7fabfc003d20 "uid=brandonl,ou=user,dc=cpp,dc=edu"}, e_attrs = 0x7fac2442b588, 
          e_ocflags = 256, e_bv = {bv_len = 0, bv_val = 0x0}, e_private = 0x7fabfc003270}
        preread_ctrl = 0x0
        postread_ctrl = 0x0
        ctrls = {0x0, 0x0, 0x7fabfc0009d0, 0x50, 0x0, 0x7fac037fc800}
        num_ctrls = 0
        numads = 57
#3  0x00000000004dbc55 in overlay_op_walk (op=0x7fac037fd730, rs=0x7fac037fcfe0, which=op_modify, 
    oi=0x16e2b90, on=0x0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:677
        func = 0x7fadf323ddd8 <bi+88>
        rc = 32768
#4  0x00000000004dbea0 in over_op_func (op=0x7fac037fd730, rs=0x7fac037fcfe0, which=op_modify)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:730
        oi = 0x16e2b90
        on = 0x16e3d20
        be = 0x16d4b60
        db = {bd_info = 0x7fadf323dd80 <bi>, bd_self = 0x16d4b60, 
ts 14 times>, "\001", be_flags = 563464, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, 
            sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, 
            sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 64}, be_suffix = 0x16d3bf0, 
          be_nsuffix = 0x16d4270, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, 
            bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x16d4490 "cn=ldaproot,dc=cpp,dc=edu"}, 
          be_rootndn = {bv_len = 25, bv_val = 0x16d44e0 "cn=ldaproot,dc=cpp,dc=edu"}, be_rootpw = {
            bv_len = 14, bv_val = 0x16a7a30 "XXXXXXXXXXX"}, be_max_deref_depth = 15, be_def_limit = {
            lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, 
            lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x16d3b20, be_acl = 0x16d5eb0, 
          be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, 
          be_update_refs = 0x0, be_pending_csn_list = 0x19966a0, be_pcl_mutex = {__data = {__lock = 0, 
              __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
                __prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, 
          be_syncinfo = 0x16e3970, be_pb = 0x0, be_cf_ocs = 0x7fadf323dac0 <mdbocs>, 
          be_private = 0x7fadf7dbb010, be_next = {stqe_next = 0x0}}
        cb = {sc_next = 0x7fac037fcfb0, sc_response = 0x4daa2c <over_back_response>, sc_cleanup = 0x0, 
          sc_writewait = 0x0, sc_private = 0x16e2b90}
        sc = 0x0
        rc = 32768
        __PRETTY_FUNCTION__ = "over_op_func"
#5  0x00000000004dc003 in over_op_modify (op=0x7fac037fd730, rs=0x7fac037fcfe0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:769
No locals.
#6  0x00000000004c84b4 in syncrepl_message_to_op (si=0x16e4630, op=0x7fac037fd730, msg=0x7fabfc4a1560)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:2394
        oes = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x4c7516 <syncrepl_message_to_op>}, 
          oe_si = 0x16e4630}
        ber = 0x7fabfc4a76a0
        modlist = 0x7fabfc498e20
        ls = 0x7dfc40 <accesslog_sc>
        rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, 
          sr_text = 0x7fac037fca80 "modify/delete: eduPersonAffiliation: no such attribute", sr_ref = 0x0, 
          sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, 
              r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {
              r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0}
        cb = {sc_next = 0x7fabfc0025a0, sc_response = 0x4cfda6 <null_callback>, sc_cleanup = 0x0, 
          sc_writewait = 0x0, sc_private = 0x0}
        text = 0xa <error: Cannot access memory at address 0xa>
        txtbuf = "\220\320\177\003\254\177\000\000\000\321\177\003\254\177\000\000\300\320\177\003\254\177\000\000\030\000\000\000\060\000\000\000p\321\177\003\254\177\000\000\260\320\177\003\254\177\000\000\000\000\000\000\003\000\000\000 \000\000\374\253\177\000\000\377\377\377\377\377\377\377\377\000\000\000\017\000\000\000\000\257\350\377+\254\177\000\000\000\000\000\000\000\000\000\000`\321\177\003\254\177\000\000\000\000\000\000\000\000\000\000\257\350\377+\254\177", '\000' <repeats 11 times>, "\347\177\003\254\177\000\000P\347M", '\000' <repeats 13 times>, "`\322\177\003\254\177\000\000OOq\374\253\177\000\000@Oq\374\253\177\000\000`\325e\001\000\000\000\000OOq\374\253\177\000\000HOq\374\253\177\000\000\000\000\000\000\000\000\000\000"...
        textlen = 256
        bdn = {bv_len = 34, bv_val = 0x7fabfc492c3c "uid=brandonl,ou=user,dc=cpp,dc=edu"}
        dn = {bv_len = 34, bv_val = 0x7fabfc000d30 "\220\022"}
        ndn = {bv_len = 34, bv_val = 0x7fabfc000db0 "p\023"}
        bv = {bv_len = 0, bv_val = 0x0}
        bv2 = {bv_len = 0, bv_val = 0x0}
        bvals = 0x7fabfc41f360
        rdn = {bv_len = 0, bv_val = 0x0}
        sup = {bv_len = 0, bv_val = 0x0}
        prdn = {bv_len = 0, bv_val = 0x0}
        nrdn = {bv_len = 0, bv_val = 0x0}
        psup = {bv_len = 0, bv_val = 0x0}
        nsup = {bv_len = 0, bv_val = 0x0}
        rc = 0
        deleteOldRdn = 0
        freeReqDn = 1
        do_graduate = 1
#7  0x00000000004c31cd in do_syncrep2 (op=0x7fac037fd730, si=0x16e4630)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1013
        match = 0
        syncUUID = {{bv_len = 16, bv_val = 0x7fabfc4c8cd7 "7\245|\231:\352AM\246\305\066\037i&\201\232"}, {
            bv_len = 0, bv_val = 0x3000000020 <error: Cannot access memory at address 0x3000000020>}}
        cookie = {bv_len = 15, bv_val = 0x7fabfc4c8ce9 "rid=003,sid=003"}
        rctrls = 0x7fabfc49f8f0
        rctrlp = 0x7fabfc492de0
        bdn = {bv_len = 44, bv_val = 0x7fabfc492bea "reqStart=20170415103641.000016Z,cn=accesslog"}
        si_tag = 140376648999836
        entry = 0x7fabfc49f8f8
        punlock = -1
        syncstate = 1
        retdata = 0x570e74
        retoid = 0x7fadf6ac95c0 ""
        syncUUIDs = 0x0
        len = 15
        berbuf = {
          buffer = "\002\000\001", '\000' <repeats 29 times>, "ЌL\374\253\177\000\000\370\214L\374\253\177\000\000\370\214L\374\253\177", '\000' <repeats 26 times>, "\020\000\000\000\254\177\000\000\060\327\177\003\254\177\000\000\200\323\177\003\254\177\000\000\260Э\367\255\177\000\000\030\n\000\374\001\000\000\000\000\061~\000\000\000\000\000\200\323\177\003\254\177\000\000\260Э\367\255\177\000\000H\327\177\003\254\177\000\000 \v\020\024\254\177\000\000\000\324\177\003\254\177\000\000\377J\260\367\255\177\000\000\300\323\177\003\001\000\000\000,\324\177\003\254\177\000\000\060\327\177\003\001\000\000\000\020\t\020"..., ialign = 65538, 
          lalign = 65538, falign = 9.18382988e-41, dalign = 3.2380074297143616e-319, 
          palign = 0x10002 <error: Cannot access memory at address 0x10002>}
        ber = 0x7fac037fd2e0
        msg = 0x7fabfc4a1560
        syncCookie = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 3, octet_str = {bv_len = 15, 
            bv_val = 0x7fabfc714f40 "rid=003,sid=003"}, sid = 3, sc_next = {stqe_next = 0x0}}
        syncCookie_req = {ctxcsn = 0x7fabfc40cfe0, sids = 0x7fabfc4b99f0, numcsns = 4, rid = 3, octet_str = {
            bv_len = 183, 
            bv_val = 0x7fabfc41b880 "rid=003,sid=000,csn=20170415103641.410645Z#000000#000#000000;20151030223206.483980Z#000000#001#000000;20160115030222.053144Z#000000#002#000000;20170406033805.626505Z#000000#003#000000"}, sid = 0, sc_next = {stqe_next = 0x0}}
        rc = 0
        err = 0
        modlist = 0x0
        m = 1426881987
        tout_p = 0x7fac037fd210
        tout = {tv_sec = 0, tv_usec = 0}
        refreshDeletes = 0
        empty = "empty"
        __PRETTY_FUNCTION__ = "do_syncrep2"
#8  0x00000000004c54c8 in do_syncrepl (ctx=0x7fac037fdbc0, arg=0x16e4b10)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1564
        rtask = 0x16e4b10
        si = 0x16e4630
        conn = {c_struct_state = SLAP_C_UNINITIALIZED, c_conn_state = SLAP_C_INVALID, c_conn_idx = -1, 
          c_sd = 0, c_close_reason = 0x0, c_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
              __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_sb = 0x0, c_starttime = 0, 
          c_activitytime = 0, c_connid = 18446744073709551615, c_peer_domain = {bv_len = 0, 
            bv_val = 0x568650 ""}, c_peer_name = {bv_len = 0, bv_val = 0x568650 ""}, 
          c_listener = 0x570900 <dummy_list>, c_sasl_bind_mech = {bv_len = 0, bv_val = 0x0}, c_sasl_dn = {
            bv_len = 0, bv_val = 0x0}, c_sasl_authz_dn = {bv_len = 0, bv_val = 0x0}, c_authz_backend = 0x0, 
          c_authz_cookie = 0x0, c_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {
              bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, 
            sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, c_protocol = 0, c_ops = {
            stqh_first = 0x0, stqh_last = 0x0}, c_pending_ops = {stqh_first = 0x0, stqh_last = 0x0}, 
          c_write1_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write1_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_write2_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write2_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_currentber = 0x0, c_writers = 0, c_writing = 0 '\000', c_sasl_bind_in_progress = 0 '\000', 
          c_writewaiter = 0 '\000', c_is_udp = 0 '\000', c_is_tls = 0 '\000', c_needs_tls_accept = 0 '\000', 
          c_sasl_layers = 0 '\000', c_sasl_done = 0 '\000', c_sasl_authctx = 0x0, c_sasl_sockctx = 0x0, 
          c_sasl_extra = 0x0, c_sasl_bindop = 0x0, c_pagedresults_state = {ps_be = 0x0, ps_size = 0, 
            ps_count = 0, ps_cookie = 0, ps_cookieval = {bv_len = 0, bv_val = 0x0}}, c_n_ops_received = 0, 
          c_n_ops_executing = 0, c_n_ops_pending = 0, c_n_ops_completed = 0, c_n_get = 0, c_n_read = 0, 
          c_n_write = 0, c_extensions = 0x0, c_clientfunc = 0x0, c_clientarg = 0x0, 
          c_send_ldap_result = 0x45801c <slap_send_ldap_result>, 
          c_send_search_entry = 0x458cf0 <slap_send_search_entry>, 
          c_send_search_reference = 0x45b0d8 <slap_send_search_reference>, 
          c_send_ldap_extended = 0x45886d <slap_send_ldap_extended>, 
          c_send_ldap_intermediate = 0x458ade <slap_send_ldap_intermediate>}
        opbuf = {ob_op = {o_hdr = 0x7fac037fd8a0, o_tag = 102, o_time = 1492252601, o_tincr = 74, 
            o_bd = 0x7fac037fcc90, o_req_dn = {bv_len = 34, 
              bv_val = 0x7fabfc492e70 "uid=brandonl,ou=user,dc=cpp,dc=edu"}, o_req_ndn = {bv_len = 34, 
              bv_val = 0x7fabfc492ea0 "uid=brandonl,ou=user,dc=cpp,dc=edu"}, o_request = {oq_add = {
                rs_modlist = 0x7fabfc000a50, rs_e = 0x1}, oq_bind = {rb_method = -67106224, rb_cred = {
                  bv_len = 1, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_mech = {
                  bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x7fabfc000a50}, oq_modify = {
                rs_mods = {rs_modlist = 0x7fabfc000a50, rs_no_opattrs = 1 '\001'}, rs_increment = 0}, 
              oq_modrdn = {rs_mods = {rs_modlist = 0x7fabfc000a50, rs_no_opattrs = 1 '\001'}, 
                rs_deleteoldrdn = 0, rs_newrdn = {bv_len = 0, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, 
                  bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0}, oq_search = {rs_scope = -67106224, 
                rs_deref = 32683, rs_slimit = 1, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, 
                rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {
                rs_msgid = -67106224}, oq_cancel = {rs_msgid = -67106224}, oq_extended = {rs_reqoid = {
                  bv_len = 140376643996240, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {
                    bv_len = 140376643996240, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                  rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {
                  bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, 
            o_groups = 0x0, o_do_not_cache = 0 '\000', o_is_auth_check = 0 '\000', 
            o_dont_replicate = 0 '\000', o_acl_priv = ACL_NONE, o_nocaching = 0 '\000', 
            o_delete_glue_parent = 0 '\000', o_no_schema_check = 1 '\001', o_no_subordinate_glue = 0 '\000', 
            o_ctrlflag = '\000' <repeats 14 times>, "\002", '\000' <repeats 16 times>, 
            o_controls = 0x7fac037fd9f0, o_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, 
              sai_dn = {bv_len = 25, bv_val = 0x16d4490 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ndn = {
                bv_len = 25, bv_val = 0x16d44e0 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ssf = 0, 
              sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, 
            o_callback = 0x7fabfc0025d0, o_ctrls = 0x0, o_csn = {bv_len = 40, 
              bv_val = 0x7fabfc492ed0 "20170415103641.599920Z#000000#000#000000"}, o_private = 0x0, 
            o_extra = {slh_first = 0x7fac037fc9e0}, o_next = {stqe_next = 0x0}}, ob_hdr = {oh_opid = 0, 
            oh_connid = 3, oh_conn = 0x7fac037fd470, oh_msgid = 0, oh_protocol = 0, 
            oh_tid = 140376769816320, oh_threadctx = 0x7fac037fdbc0, oh_tmpmemctx = 0x7fabfc0009d0, 
            oh_tmpmfuncs = 0x7dfb80 <slap_sl_mfuncs>, oh_counters = 0x8156c0 <slap_counters>, 
            oh_log_prefix = "conn=-1 op=0", '\000' <repeats 243 times>, oh_extensions = 0x0}, ob_controls = {
            0x0 <repeats 17 times>, 0x7fac037fd260, 0x0 <repeats 14 times>}}
        op = 0x7fac037fd730
        rc = 0
        dostop = 0
        s = 13
        i = -141804259
        defer = 1
        fail = 0
        freeinfo = 0
        be = 0x16d4b60
#9  0x0000000000440411 in connection_read_thread (ctx=0x7fac037fdbc0, argv=0xd)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/connection.c:1296
        rc = 0
        cri = {op = 0x0, func = 0x4c4ec6 <do_syncrepl>, arg = 0x16e4b10, ctx = 0x7fac037fdbc0, nullop = 0}
        s = 13
#10 0x00007fadf7adba4b in ldap_int_thread_pool_wrapper (xpool=0x1667330)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/libraries/libldap_r/tpool.c:696
        pool = 0x1667330
        task = 0x7fac0c339570
        work_list = 0x16673c8
        ctx = {ltu_id = 140376769816320, ltu_key = {{ltk_key = 0x43f808 <conn_counter_init>, 
              ltk_data = 0x7fabfc0008c0, ltk_free = 0x43f65a <conn_counter_destroy>}, {
              ltk_key = 0x4ba0e3 <slap_sl_mem_init>, ltk_data = 0x7fabfc0009d0, 
              ltk_free = 0x4b9f08 <slap_sl_mem_destroy>}, {ltk_key = 0x1996710, ltk_data = 0x7fabfc100a20, 
              ltk_free = 0x7fadf303113d <mdb_reader_free>}, {ltk_key = 0x7fadf3025bce <search_stack>, 
              ltk_data = 0x7fac00ffc010, ltk_free = 0x7fadf3025bab <search_stack_free>}, {
              ltk_key = 0x7fadf30224aa <scope_chunk_get>, ltk_data = 0x7fabfc1028b0, 
              ltk_free = 0x7fadf3022462 <scope_chunk_free>}, {ltk_key = 0x45c638 <slap_op_free>, 
              ltk_data = 0x7fabf456a930, ltk_free = 0x45c58b <slap_op_q_destroy>}, {ltk_key = 0x178fe70, 
              ltk_data = 0x7fabfc49f9a0, ltk_free = 0x7fadf303113d <mdb_reader_free>}, {ltk_key = 0x0, 
              ltk_data = 0x7fabf44f5db0, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, 
              ltk_free = 0x0} <repeats 24 times>}}
        kctx = 0x0
        i = 32
        keyslot = 948
        hash = 3735031732
        __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#11 0x00007fadf7124494 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00007fadf6817edd in clone () from /lib64/libc.so.6
No symbol table info available.


Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00000000004b8dbb in modify_add_values (e=0x7fa0b59f0a30, mod=0x7fa0a0000980, permissive=0, 
    text=0x7fa0b59f1000, textbuf=0x7fa0b59f0a80 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/mods.c:61
61		if ( !BER_BVISNULL( &mod->sm_values[mod->sm_numvals] )) {
[Current thread is 1 (Thread 0x7fa0b59f2700 (LWP 10464))]
(gdb) bt full
#0  0x00000000004b8dbb in modify_add_values (e=0x7fa0b59f0a30, mod=0x7fa0a0000980, permissive=0, 
    text=0x7fa0b59f1000, textbuf=0x7fa0b59f0a80 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/mods.c:61
        rc = -1610599912
        op = 0x5866e0 "add"
        a = 0x7fa0b59f08a0
        pmod = {sm_desc = 0x1e9a3c0, sm_values = 0x0, sm_nvalues = 0x0, sm_numvals = 0, sm_op = 0, 
          sm_flags = 0, sm_type = {bv_len = 34, bv_val = 0x7fa0a0000968 "uid=aaal\200\t"}}
        __PRETTY_FUNCTION__ = "modify_add_values"
#1  0x00007fa277718774 in mdb_modify_internal (op=0x7fa0b59f1730, tid=0x225e3c0, modlist=0x7fa0a0000940, 
    e=0x7fa0b59f0a30, text=0x7fa0b59f1000, 
    textbuf=0x7fa0b59f0a80 "modify/delete: eduPersonAffiliation: no such attribute", textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/back-mdb/modify.c:137
        rc = 32672
        err = 0
        mod = 0x7fa0a0000980
        ml = 0x7fa0a0000980
        save_attrs = 0x7fa0a0003218
        ap = 0x7fa000000000
        glue_attr_delete = 0
        got_delete = 0
        __PRETTY_FUNCTION__ = "mdb_modify_internal"
#2  0x00007fa27771a5f2 in mdb_modify (op=0x7fa0b59f1730, rs=0x7fa0b59f0fe0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/back-mdb/modify.c:624
        mdb = 0x7fa27c4b7010
        e = 0x7fa0a00031c8
        manageDSAit = 2
\000\340\017\237\265\240\177\000\000\060\027\237\265\240\177\000\000 \v\237\265\240\177\000\000\000\000\000\000\260\000\000\000 \v\237\265\240\177\000\000\324\324\373{\000\000\000\000\030%\000\240\240\177\000\000\060\027\237\265\240\177\000\000p}\352\001\000\000\000\000P\177\352\001\000\000\000\000\300$\000\240\240\177\000\000\350$\000\240\240\177\000\000`\022\237\265\240\177\000\000"...
        textlen = 256
        txn = 0x225e3c0
        opinfo = {moi_oe = {oe_next = {sle_next = 0x7fa0b59f0f90}, oe_key = 0x7fa27c4b7010}, 
          moi_txn = 0x225e3c0, moi_ref = 1, moi_flag = 0 '\000'}
        moi = 0x7fa0b59f09e0
        dummy = {e_id = 200434, e_name = {bv_len = 34, 
            bv_val = 0x7fa0a0003198 "uid=aaalmora,ou=user,dc=cpp,dc=edu"}, e_nname = {bv_len = 34, 
            bv_val = 0x7fa0a0003ce0 "uid=aaalmora,ou=user,dc=cpp,dc=edu"}, e_attrs = 0x7fa098ba4938, 
          e_ocflags = 256, e_bv = {bv_len = 0, bv_val = 0x0}, e_private = 0x7fa0a00031c8}
        preread_ctrl = 0x0
        postread_ctrl = 0x0
        ctrls = {0x0, 0x0, 0x7fa0a00008c0, 0x50, 0x0, 0x7fa0b59f0800}
        num_ctrls = 0
        numads = 57
#3  0x00000000004dbc55 in overlay_op_walk (op=0x7fa0b59f1730, rs=0x7fa0b59f0fe0, which=op_modify, 
    oi=0x1ea7b90, on=0x0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:677
        func = 0x7fa277939dd8 <bi+88>
        rc = 32768
#4  0x00000000004dbea0 in over_op_func (op=0x7fa0b59f1730, rs=0x7fa0b59f0fe0, which=op_modify)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:730
        oi = 0x1ea7b90
        on = 0x1ea8d20
        be = 0x1e99b60
        db = {bd_info = 0x7fa277939d80 <bi>, bd_self = 0x1e99b60, 
ts 14 times>, "\001", be_flags = 563464, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, 
            sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, 
            sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 64}, be_suffix = 0x1e98bf0, 
          be_nsuffix = 0x1e99270, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, 
            bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x1e99490 "cn=ldaproot,dc=cpp,dc=edu"}, 
          be_rootndn = {bv_len = 25, bv_val = 0x1e994e0 "cn=ldaproot,dc=cpp,dc=edu"}, be_rootpw = {
            bv_len = 14, bv_val = 0x1e6ca30 "XXXXXXXXXXXX"}, be_max_deref_depth = 15, be_def_limit = {
            lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, 
            lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x1e98b20, be_acl = 0x1e9aeb0, 
          be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, 
          be_update_refs = 0x0, be_pending_csn_list = 0x215b6a0, be_pcl_mutex = {__data = {__lock = 0, 
              __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
                __prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, 
          be_syncinfo = 0x1ea8970, be_pb = 0x0, be_cf_ocs = 0x7fa277939ac0 <mdbocs>, 
          be_private = 0x7fa27c4b7010, be_next = {stqe_next = 0x0}}
        cb = {sc_next = 0x7fa0b59f0fb0, sc_response = 0x4daa2c <over_back_response>, sc_cleanup = 0x0, 
          sc_writewait = 0x0, sc_private = 0x1ea7b90}
        sc = 0x0
        rc = 32768
        __PRETTY_FUNCTION__ = "over_op_func"
#5  0x00000000004dc003 in over_op_modify (op=0x7fa0b59f1730, rs=0x7fa0b59f0fe0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/backover.c:769
No locals.
#6  0x00000000004c84b4 in syncrepl_message_to_op (si=0x1ea8970, op=0x7fa0b59f1730, msg=0x7fa0a012df60)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:2394
        oes = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x4c7516 <syncrepl_message_to_op>}, 
          oe_si = 0x1ea8970}
        ber = 0x7fa0a029eb70
        modlist = 0x7fa0a0127460
        ls = 0x7dfc40 <accesslog_sc>
        rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, 
          sr_text = 0x7fa0b59f0a80 "modify/delete: eduPersonAffiliation: no such attribute", sr_ref = 0x0, 
          sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, 
              r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {
              r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0}
        cb = {sc_next = 0x7fa0a0002490, sc_response = 0x4cfda6 <null_callback>, sc_cleanup = 0x0, 
          sc_writewait = 0x0, sc_private = 0x0}
        text = 0xa <error: Cannot access memory at address 0xa>
        txtbuf = "\220\020\237\265\240\177\000\000\000\021\237\265\240\177\000\000\300\020\237\265\240\177\000\000\030\000\000\000\060\000\000\000p\021\237\265\240\177\000\000\260\020\237\265\240\177\000\000\000\000\000\000\003\000\000\000 \000\000\240\240\177\000\000\377\377\377\377\377\377\377\377\000\000\000\017\000\000\000\000?\034\237\266\240\177\000\000\000\000\000\000\000\000\000\000`\021\237\265\240\177\000\000\000\000\000\000\000\000\000\000?\034\237\266\240\177", '\000' <repeats 11 times>, "'\237\265\240\177\000\000P\347M", '\000' <repeats 13 times>, "`\022\237\265\240\177\000\000\177\233\023\240\240\177\000\000p\233\023\240\240\177\000\000`%\342\001\000\000\000\000\177\233\023\240\240\177\000\000x\233\023\240\240\177\000\000\000\000\000\000\000\000\000\000"...
        textlen = 256
        bdn = {bv_len = 34, bv_val = 0x7fa0a042265b "uid=aaalmora,ou=user,dc=cpp,dc=edu"}
        dn = {bv_len = 34, bv_val = 0x7fa0a0000c20 "\310\021"}
        ndn = {bv_len = 34, bv_val = 0x7fa0a0000ca0 "\250\022"}
        bv = {bv_len = 0, bv_val = 0x0}
        bv2 = {bv_len = 0, bv_val = 0x0}
        bvals = 0x7fa0a0106580
        rdn = {bv_len = 0, bv_val = 0x0}
        sup = {bv_len = 0, bv_val = 0x0}
        prdn = {bv_len = 0, bv_val = 0x0}
        nrdn = {bv_len = 0, bv_val = 0x0}
        psup = {bv_len = 0, bv_val = 0x0}
        nsup = {bv_len = 0, bv_val = 0x0}
        rc = 0
        deleteOldRdn = 0
        freeReqDn = 1
        do_graduate = 1
#7  0x00000000004c31cd in do_syncrep2 (op=0x7fa0b59f1730, si=0x1ea8970)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1013
        match = 0
        syncUUID = {{bv_len = 16, bv_val = 0x7fa0a0144147 "4w\232m2\230F>\273\200\032\350\227o\252i"}, {
            bv_len = 0, bv_val = 0x3000000020 <error: Cannot access memory at address 0x3000000020>}}
        cookie = {bv_len = 15, bv_val = 0x7fa0a0144159 "rid=001,sid=001"}
        rctrls = 0x7fa0a0111800
        rctrlp = 0x7fa0a0157460
        bdn = {bv_len = 44, bv_val = 0x7fa0a0422609 "reqStart=20170416103057.000005Z,cn=accesslog"}
        si_tag = 140327924125852
        entry = 0x7fa0a0111808
        punlock = -1
        syncstate = 1
        retdata = 0x570e74
        retoid = 0x7fa27b1c55c0 ""
        syncUUIDs = 0x0
        len = 15
        berbuf = {
          buffer = "\002\000\001", '\000' <repeats 29 times>, "@A\024\240\240\177\000\000hA\024\240\240\177\000\000hA\024\240\240\177", '\000' <repeats 26 times>, "\017\000\000\000\240\177\000\000\060\027\237\265\240\177\000\000\200\023\237\265\240\177\000\000\260\220\035|\242\177\000\000\b\t\000\240\001\000\000\000\000\061~\000\000\000\000\000\200\023\237\265\240\177\000\000\260\220\035|\242\177\000\000H\027\237\265\240\177\000\000 \v\020\230\240\177\000\000\000\024\237\265\240\177\000\000\377\n |\242\177\000\000\300\023\237\265\001\000\000\000,\024\237\265\240\177\000\000\060\027\237\265\001\000\000\000"..., ialign = 65538, lalign = 65538, 
          falign = 9.18382988e-41, dalign = 3.2380074297143616e-319, 
          palign = 0x10002 <error: Cannot access memory at address 0x10002>}
        ber = 0x7fa0b59f12e0
        msg = 0x7fa0a012df60
        syncCookie = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 1, octet_str = {bv_len = 15, 
            bv_val = 0x7fa0a0139b70 "rid=001,sid=001"}, sid = 1, sc_next = {stqe_next = 0x0}}
        syncCookie_req = {ctxcsn = 0x7fa0a04230d0, sids = 0x7fa0a012bb00, numcsns = 4, rid = 1, octet_str = {
            bv_len = 183, 
            bv_val = 0x7fa0a015e3f0 "rid=001,sid=000,csn=20170416081903.399068Z#000000#000#000000;20170416051743.054642Z#000000#001#000000;20160115030222.053144Z#000000#002#000000;20170416043935.843860Z#000000#003#000000"}, sid = 0, sc_next = {stqe_next = 0x0}}
        rc = 0
        err = 0
        modlist = 0x0
        m = 1426881987
        tout_p = 0x7fa0b59f1210
        tout = {tv_sec = 0, tv_usec = 0}
        refreshDeletes = 0
        empty = "empty"
        __PRETTY_FUNCTION__ = "do_syncrep2"
#8  0x00000000004c54c8 in do_syncrepl (ctx=0x7fa0b59f1bc0, arg=0x1ea3e70)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1564
        rtask = 0x1ea3e70
        si = 0x1ea8970
        conn = {c_struct_state = SLAP_C_UNINITIALIZED, c_conn_state = SLAP_C_INVALID, c_conn_idx = -1, 
          c_sd = 0, c_close_reason = 0x0, c_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
              __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_sb = 0x0, c_starttime = 0, 
          c_activitytime = 0, c_connid = 18446744073709551615, c_peer_domain = {bv_len = 0, 
            bv_val = 0x568650 ""}, c_peer_name = {bv_len = 0, bv_val = 0x568650 ""}, 
          c_listener = 0x570900 <dummy_list>, c_sasl_bind_mech = {bv_len = 0, bv_val = 0x0}, c_sasl_dn = {
            bv_len = 0, bv_val = 0x0}, c_sasl_authz_dn = {bv_len = 0, bv_val = 0x0}, c_authz_backend = 0x0, 
          c_authz_cookie = 0x0, c_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {
              bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, 
            sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, c_protocol = 0, c_ops = {
            stqh_first = 0x0, stqh_last = 0x0}, c_pending_ops = {stqh_first = 0x0, stqh_last = 0x0}, 
          c_write1_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write1_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_write2_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write2_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_currentber = 0x0, c_writers = 0, c_writing = 0 '\000', c_sasl_bind_in_progress = 0 '\000', 
          c_writewaiter = 0 '\000', c_is_udp = 0 '\000', c_is_tls = 0 '\000', c_needs_tls_accept = 0 '\000', 
          c_sasl_layers = 0 '\000', c_sasl_done = 0 '\000', c_sasl_authctx = 0x0, c_sasl_sockctx = 0x0, 
          c_sasl_extra = 0x0, c_sasl_bindop = 0x0, c_pagedresults_state = {ps_be = 0x0, ps_size = 0, 
            ps_count = 0, ps_cookie = 0, ps_cookieval = {bv_len = 0, bv_val = 0x0}}, c_n_ops_received = 0, 
          c_n_ops_executing = 0, c_n_ops_pending = 0, c_n_ops_completed = 0, c_n_get = 0, c_n_read = 0, 
          c_n_write = 0, c_extensions = 0x0, c_clientfunc = 0x0, c_clientarg = 0x0, 
          c_send_ldap_result = 0x45801c <slap_send_ldap_result>, 
          c_send_search_entry = 0x458cf0 <slap_send_search_entry>, 
          c_send_search_reference = 0x45b0d8 <slap_send_search_reference>, 
          c_send_ldap_extended = 0x45886d <slap_send_ldap_extended>, 
          c_send_ldap_intermediate = 0x458ade <slap_send_ldap_intermediate>}
        opbuf = {ob_op = {o_hdr = 0x7fa0b59f18a0, o_tag = 102, o_time = 1492338657, o_tincr = 65, 
            o_bd = 0x7fa0b59f0c90, o_req_dn = {bv_len = 34, 
              bv_val = 0x7fa0a0160390 "uid=aaalmora,ou=user,dc=cpp,dc=edu"}, o_req_ndn = {bv_len = 34, 
              bv_val = 0x7fa0a0127430 "uid=aaalmora,ou=user,dc=cpp,dc=edu"}, o_request = {oq_add = {
                rs_modlist = 0x7fa0a0000940, rs_e = 0x1}, oq_bind = {rb_method = -1610610368, rb_cred = {
                  bv_len = 1, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_mech = {
                  bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x7fa0a0000940}, oq_modify = {
                rs_mods = {rs_modlist = 0x7fa0a0000940, rs_no_opattrs = 1 '\001'}, rs_increment = 0}, 
              oq_modrdn = {rs_mods = {rs_modlist = 0x7fa0a0000940, rs_no_opattrs = 1 '\001'}, 
                rs_deleteoldrdn = 0, rs_newrdn = {bv_len = 0, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, 
                  bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0}, oq_search = {rs_scope = -1610610368, 
                rs_deref = 32672, rs_slimit = 1, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, 
                rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {
                rs_msgid = -1610610368}, oq_cancel = {rs_msgid = -1610610368}, oq_extended = {rs_reqoid = {
                  bv_len = 140327855851840, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {
                    bv_len = 140327855851840, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                  rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {
                  bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, 
            o_groups = 0x0, o_do_not_cache = 0 '\000', o_is_auth_check = 0 '\000', 
            o_dont_replicate = 0 '\000', o_acl_priv = ACL_NONE, o_nocaching = 0 '\000', 
            o_delete_glue_parent = 0 '\000', o_no_schema_check = 1 '\001', o_no_subordinate_glue = 0 '\000', 
            o_ctrlflag = '\000' <repeats 14 times>, "\002", '\000' <repeats 16 times>, 
            o_controls = 0x7fa0b59f19f0, o_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, 
              sai_dn = {bv_len = 25, bv_val = 0x1e99490 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ndn = {
                bv_len = 25, bv_val = 0x1e994e0 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ssf = 0, 
              sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, 
            o_callback = 0x7fa0a00024c0, o_ctrls = 0x0, o_csn = {bv_len = 40, 
              bv_val = 0x7fa0a0109a60 "20170416103057.530384Z#000000#000#000000"}, o_private = 0x0, 
            o_extra = {slh_first = 0x7fa0b59f09e0}, o_next = {stqe_next = 0x0}}, ob_hdr = {oh_opid = 0, 
            oh_connid = 1, oh_conn = 0x7fa0b59f1470, oh_msgid = 0, oh_protocol = 0, 
            oh_tid = 140328218601216, oh_threadctx = 0x7fa0b59f1bc0, oh_tmpmemctx = 0x7fa0a00008c0, 
            oh_tmpmfuncs = 0x7dfb80 <slap_sl_mfuncs>, oh_counters = 0x8156c0 <slap_counters>, 
            oh_log_prefix = "conn=-1 op=0", '\000' <repeats 243 times>, oh_extensions = 0x0}, ob_controls = {
            0x0 <repeats 17 times>, 0x7fa0b59f1260, 0x0 <repeats 14 times>}}
        op = 0x7fa0b59f1730
        rc = 0
        dostop = 0
        s = 18
        i = 2080111901
        defer = 1
        fail = 0
        freeinfo = 0
        be = 0x1e99b60
#9  0x0000000000440411 in connection_read_thread (ctx=0x7fa0b59f1bc0, argv=0x12)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/connection.c:1296
        rc = 0
        cri = {op = 0x0, func = 0x4c4ec6 <do_syncrepl>, arg = 0x1ea3e70, ctx = 0x7fa0b59f1bc0, nullop = 0}
        s = 18
#10 0x00007fa27c1d7a4b in ldap_int_thread_pool_wrapper (xpool=0x1e2c330)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/libraries/libldap_r/tpool.c:696
        pool = 0x1e2c330
        task = 0x7fa0b0000e40
        work_list = 0x1e2c3c8
        ctx = {ltu_id = 140328218601216, ltu_key = {{ltk_key = 0x4ba0e3 <slap_sl_mem_init>, 
              ltk_data = 0x7fa0a00008c0, ltk_free = 0x4b9f08 <slap_sl_mem_destroy>}, {ltk_key = 0x215b710, 
              ltk_data = 0x7fa0a0ba3eb0, ltk_free = 0x7fa27772d13d <mdb_reader_free>}, {
              ltk_key = 0x7fa277721bce <search_stack>, ltk_data = 0x7fa096fff010, 
              ltk_free = 0x7fa277721bab <search_stack_free>}, {ltk_key = 0x7fa27771e4aa <scope_chunk_get>, 
              ltk_data = 0x7fa0a0ba5d40, ltk_free = 0x7fa27771e462 <scope_chunk_free>}, {
              ltk_key = 0x43f808 <conn_counter_init>, ltk_data = 0x7fa0a012d4f0, 
              ltk_free = 0x43f65a <conn_counter_destroy>}, {ltk_key = 0x45c638 <slap_op_free>, 
              ltk_data = 0x7fa0a825ab60, ltk_free = 0x45c58b <slap_op_q_destroy>}, {ltk_key = 0x1f54e70, 
              ltk_data = 0x7fa0a01c3fa0, ltk_free = 0x7fa27772d13d <mdb_reader_free>}, {ltk_key = 0x0, 
              ltk_data = 0x7fa09c04d880, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, 
              ltk_free = 0x0} <repeats 24 times>}}
        kctx = 0x0
        i = 32
        keyslot = 634
        hash = 3535879802
        __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#11 0x00007fa27b820494 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#12 0x00007fa27af13edd in clone () from /lib64/libc.so.6
No symbol table info available.

On Sun, Apr 16, 2017 at 06:20:06PM +0000, henson@acm.org wrote:
> On Mon, Apr 03, 2017 at 02:44:48PM +0100, Howard Chu wrote:
> 
>> You should probably switch to a non-optimized build. The backtrace you
>> posted initially had most parameters optimized out, which also makes
>> things harder to trace.
> 
> Well, as luck (8-/) would have it, it crashed both yesterday and today
> on the non-optimized build. Here are the full backtraces from both
> cores, I hope they provide some clue as to what's going on. Let me know
> if you want me to do any more post mortem debugging on the cores. Thanks
> much.
> 
> 
> Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> (gdb) bt full
> #6  0x00000000004c84b4 in syncrepl_message_to_op (si=0x16e4630, op=0x7fac037fd730, msg=0x7fabfc4a1560)
>     at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:2394
> #7  0x00000000004c31cd in do_syncrep2 (op=0x7fac037fd730, si=0x16e4630)
>     at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1013
>         bdn = {bv_len = 44, bv_val = 0x7fabfc492bea "reqStart=20170415103641.000016Z,cn=accesslog"}
> 
> 
> Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> (gdb) bt full
> #6  0x00000000004c84b4 in syncrepl_message_to_op (si=0x1ea8970, op=0x7fa0b59f1730, msg=0x7fa0a012df60)
>     at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:2394
> #7  0x00000000004c31cd in do_syncrep2 (op=0x7fa0b59f1730, si=0x1ea8970)
>     at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.44-r1/work/openldap-2.4.44/servers/slapd/syncrepl.c:1013
>         bdn = {bv_len = 44, bv_val = 0x7fa0a0422609 "reqStart=20170416103057.000005Z,cn=accesslog"}

Hi Paul,
could you post the two accesslog entries (or at least an outline)
referenced in the backtrace?
- reqStart=20170415103641.000016Z,cn=accesslog
- reqStart=20170416103057.000005Z,cn=accesslog

I assume you are still running the same code in production, correct?

Thanks,
Ondrej

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

On Wed, Apr 19, 2017 at 05:37:35PM +0100, Ondřej Kuzník wrote:

> could you post the two accesslog entries (or at least an outline)
> referenced in the backtrace?

That's weird. I can't find these exact entries. Based on the username, I
found ones that are very close to the timestamp, but ones that match
these exact timestamps do not exist. Last time Quanah asked me to look
up the matching accesslog entry from the crash I believe I was able to
find it.

> - reqStart=20170415103641.000016Z,cn=accesslog

dn: reqStart=20170415103641.000068Z,cn=accesslog
objectClass: auditModify
structuralObjectClass: auditModify
reqStart: 20170415103641.000068Z
reqEnd: 20170415103641.000069Z
reqType: modify
reqSession: 792228
reqAuthzID: cn=idmgmt,ou=user,ou=service,dc=cpp,dc=edu
reqDN: uid=brandonl,ou=user,dc=cpp,dc=edu
reqResult: 0
reqMod: eduPersonAffiliation:=
reqMod: eduPersonPrimaryAffiliation:=
reqMod: cppEduPersonAffiliation:=
reqMod: entryCSN:= 20170415103641.599920Z#000000#000#000000
reqMod: modifiersName:= cn=idmgmt,ou=user,ou=service,dc=cpp,dc=edu
reqMod: modifyTimestamp:= 20170415103641Z
reqEntryUUID: 8b33981d-25ee-4f7b-b439-41c273e94b61
entryUUID: b132db96-9ebe-4ee5-9fd2-5257b0c352d2
creatorsName: cn=accesslog
createTimestamp: 20170415103641Z
entryCSN: 20170415103641.599920Z#000000#000#000000
modifiersName: cn=accesslog
modifyTimestamp: 20170415103641Z

> - reqStart=20170416103057.000005Z,cn=accesslog

dn: reqStart=20170416103057.000045Z,cn=accesslog
objectClass: auditModify
structuralObjectClass: auditModify
reqStart: 20170416103057.000045Z
reqEnd: 20170416103057.000046Z
reqType: modify
reqSession: 54887
reqAuthzID: cn=idmgmt,ou=user,ou=service,dc=cpp,dc=edu
reqDN: uid=aaalmora,ou=user,dc=cpp,dc=edu
reqResult: 0
reqMod: eduPersonAffiliation:=
reqMod: eduPersonPrimaryAffiliation:=
reqMod: cppEduPersonAffiliation:=
reqMod: entryCSN:= 20170416103057.530384Z#000000#000#000000
reqMod: modifiersName:= cn=idmgmt,ou=user,ou=service,dc=cpp,dc=edu
reqMod: modifyTimestamp:= 20170416103057Z
reqEntryUUID: 153aa3e7-cc27-48ce-b6f2-e14215348941
entryCSN: 20170416103057.530384Z#000000#000#000000
entryUUID: b0c18fef-134e-4c8e-b073-fcdff58d3d93
creatorsName: cn=accesslog
createTimestamp: 20170416103057Z
modifiersName: cn=accesslog
modifyTimestamp: 20170416103057Z

> I assume you are still running the same code in production, correct?

Yes. It actually crashed 4 days in a row :(, the two days I provided
backtraces for and then the following two days. Just for something to do
I slapcat'd the db and accesslog and reloaded it after the fourth day in
case there was some weird mdb corruption or something. It hasn't crashed
since, but it crashes randomly so that could be meaningless <sigh>. I
wouldn't think reloaded the db would have affected finding the matching
accesslog entries but that is something different than last time too.
And of course now this is the non-optimized build.

Thanks...

On Wed, Apr 19, 2017 at 01:03:15PM -0700, Paul B. Henson wrote:
> On Wed, Apr 19, 2017 at 05:37:35PM +0100, Ondřej Kuzník wrote:
>> could you post the two accesslog entries (or at least an outline)
>> referenced in the backtrace?
> 
> That's weird. I can't find these exact entries. Based on the username, I
> found ones that are very close to the timestamp, but ones that match
> these exact timestamps do not exist. Last time Quanah asked me to look
> up the matching accesslog entry from the crash I believe I was able to
> find it.

Hi Paul,
those come from master with ServerID 0? Maybe you can find them in
another server's accesslog?

>> - reqStart=20170415103641.000016Z,cn=accesslog
> 
> dn: reqStart=20170415103641.000068Z,cn=accesslog
> reqDN: uid=brandonl,ou=user,dc=cpp,dc=edu
> reqMod: entryCSN:= 20170415103641.599920Z#000000#000#000000
> entryCSN: 20170415103641.599920Z#000000#000#000000
> 
>> - reqStart=20170416103057.000005Z,cn=accesslog
> 
> dn: reqStart=20170416103057.000045Z,cn=accesslog
> reqDN: uid=aaalmora,ou=user,dc=cpp,dc=edu
> reqMod: entryCSN:= 20170416103057.530384Z#000000#000#000000
> entryCSN: 20170416103057.530384Z#000000#000#000000
> 
>> I assume you are still running the same code in production, correct?
> 
> Yes. It actually crashed 4 days in a row :(, the two days I provided
> backtraces for and then the following two days. Just for something to do
> I slapcat'd the db and accesslog and reloaded it after the fourth day in
> case there was some weird mdb corruption or something. It hasn't crashed
> since, but it crashes randomly so that could be meaningless <sigh>. I
> wouldn't think reloaded the db would have affected finding the matching
> accesslog entries but that is something different than last time too.
> And of course now this is the non-optimized build.

Would you be able to try the patch I provided earlier on one of your
servers?

Thanks,

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

--On Thursday, April 20, 2017 9:34 AM +0000 okuznik@symas.com wrote:

> On Wed, Apr 19, 2017 at 01:03:15PM -0700, Paul B. Henson wrote:
>> On Wed, Apr 19, 2017 at 05:37:35PM +0100, Ond=C5=99ej Kuzn=C3=ADk wrote=
> :
>>> could you post the two accesslog entries (or at least an outline)
>>> referenced in the backtrace?
>> =20
>> That's weird. I can't find these exact entries. Based on the username, =
> I
>> found ones that are very close to the timestamp, but ones that match
>> these exact timestamps do not exist. Last time Quanah asked me to look
>> up the matching accesslog entry from the crash I believe I was able to
>> find it.
>
> Hi Paul,
> those come from master with ServerID 0? Maybe you can find them in
> another server's accesslog?

Hi Paul,

You stated previously that you are in 4-way MMR.  It's not valid to have a 
serverID of 0 in an MMR environment (See 
<http://www.openldap.org/its/index.cgi/?findid=8635>).

--Quanah



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

> From: Ondřej Kuzník
> Sent: Thursday, April 20, 2017 1:34 AM
>
> those come from master with ServerID 0? Maybe you can find them in
> another server's accesslog?

Hmm, no, I don't see them in any of the three other accesslogs either.

> Would you be able to try the patch I provided earlier on one of your
> servers?

This one?

ftp://ftp.openldap.org/incoming/Ondrej-Kuznik-20170327-ITS8609-ignore-invalid-accesslog-ops.patch

Sure, I can drop this in on the master server that has been crashing and see what happens.

Thanks…

> From: Quanah Gibson-Mount
> Sent: Thursday, April 20, 2017 8:07 AM
> 
> You stated previously that you are in 4-way MMR.  It's not valid to have a
> serverID of 0 in an MMR environment (See
> <http://www.openldap.org/its/index.cgi/?findid=8635>).

Well now, that's quite a fresh ITS 8-/. I'm guessing quite a few people have
serverID's of 0 in their replicated environments, as I don't know that I've
ever seen that information before. What are the ramifications of having a
server with an ID of 0 in a replicated environment? What is the procedure
for remediating the issue? Is it as simple as shutting down that server,
updating the configuration to have a serverID of 4 and restarting it? Or
does the database need to be stripped of all CSN's which have an ID of 0 via
slapcat, grep -v, and then slapadd with the -w option on the master and then
reloading from ldif on all the replicas?

--On Thursday, April 20, 2017 9:02 PM +0000 henson@acm.org wrote:

>> From: Quanah Gibson-Mount
>> Sent: Thursday, April 20, 2017 8:07 AM
>>
>> You stated previously that you are in 4-way MMR.  It's not valid to have
>> a serverID of 0 in an MMR environment (See
>> <http://www.openldap.org/its/index.cgi/?findid=8635>).
>
> Well now, that's quite a fresh ITS 8-/. I'm guessing quite a few people
> have serverID's of 0 in their replicated environments, as I don't know
> that I've ever seen that information before. What are the ramifications
> of having a server with an ID of 0 in a replicated environment? What is
> the procedure for remediating the issue? Is it as simple as shutting down
> that server, updating the configuration to have a serverID of 4 and
> restarting it? Or does the database need to be stripped of all CSN's
> which have an ID of 0 via slapcat, grep -v, and then slapadd with the -w
> option on the master and then reloading from ldif on all the replicas?

It's fine for there to be legacy entryCSNs and a contextCSN for serverID of 
0.  However, it is not fine for any master in an MMR setup to have a 
specific serverID of 0.  If you're running with serverID's 0-3, I'd simply 
try an ldapmodify on the server with 0 to set its serverID to 4, and then 
do a modification against it, so it generates a new contextCSN that gets 
pushed out to all nodes.

--Quanah


--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

quanah@symas.com wrote:
> --On Thursday, April 20, 2017 9:02 PM +0000 henson@acm.org wrote:
>
>>> From: Quanah Gibson-Mount
>>> Sent: Thursday, April 20, 2017 8:07 AM
>>>
>>> You stated previously that you are in 4-way MMR.  It's not valid to have
>>> a serverID of 0 in an MMR environment (See
>>> <http://www.openldap.org/its/index.cgi/?findid=8635>).
>>
>> Well now, that's quite a fresh ITS 8-/. I'm guessing quite a few people
>> have serverID's of 0 in their replicated environments, as I don't know
>> that I've ever seen that information before. What are the ramifications
>> of having a server with an ID of 0 in a replicated environment? What is
>> the procedure for remediating the issue? Is it as simple as shutting down
>> that server, updating the configuration to have a serverID of 4 and
>> restarting it? Or does the database need to be stripped of all CSN's
>> which have an ID of 0 via slapcat, grep -v, and then slapadd with the -w
>> option on the master and then reloading from ldif on all the replicas?
>
> It's fine for there to be legacy entryCSNs and a contextCSN for serverID of
> 0.  However, it is not fine for any master in an MMR setup to have a
> specific serverID of 0.  If you're running with serverID's 0-3, I'd simply
> try an ldapmodify on the server with 0 to set its serverID to 4, and then
> do a modification against it, so it generates a new contextCSN that gets
> pushed out to all nodes.

Performance-wise, it would be better to delete the SID 0 contextCSN value too.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

--On Friday, April 21, 2017 2:26 AM +0100 Howard Chu <hyc@symas.com> wrote:

>> It's fine for there to be legacy entryCSNs and a contextCSN for serverID
>> of 0.  However, it is not fine for any master in an MMR setup to have a
>> specific serverID of 0.  If you're running with serverID's 0-3, I'd
>> simply try an ldapmodify on the server with 0 to set its serverID to 4,
>> and then do a modification against it, so it generates a new contextCSN
>> that gets pushed out to all nodes.
>
> Performance-wise, it would be better to delete the SID 0 contextCSN value
> too.

Yeah, definitely not optimal, but less of a worry at the moment. ;)

--Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

> From: Ondřej Kuzník
> Sent: Thursday, April 20, 2017 1:34 AM
> 
> Would you be able to try the patch I provided earlier on one of your
> servers?

Sorry for the long delay on this. I had built an updated binary with the patch, but was waiting for the server to crash to drop it in. As luck would have it, it went almost a month without crashing 8-/. It crashed yesterday though, so it is now running with the patch (Ondrej-Kuznik-20170327-ITS8609-ignore-invalid-accesslog-ops.patch), although it might take some time to see if it makes any difference. Or it might crash again tomorrow. <sigh>.

Thanks much…

> From: Quanah Gibson-Mount
> Sent: Thursday, April 20, 2017 1:26 PM
> 
> It's fine for there to be legacy entryCSNs and a contextCSN for serverID
of
> 0.  However, it is not fine for any master in an MMR setup to have a
> specific serverID of 0.  If you're running with serverID's 0-3, I'd simply
> try an ldapmodify on the server with 0 to set its serverID to 4, and then
> do a modification against it, so it generates a new contextCSN that gets
> pushed out to all nodes.

I went ahead and updated the serverID from 0 to 4 when I dropped in the
patched binary, it seems to have gone smoothly. I'm planning to migrate the
memberOf configuration over the summer to the dynamic list implementation,
which is going to require a dump to ldif and reload, I'll go ahead and strip
out the legacy serverid 0 contextCSN's then.

Thanks.

I've been running 2.4.44 with the test patch since May and haven't seen
a crash since, for what that's worth. We upgraded to 2.4.45 last week,
without that patch. I'm going to see if the crash shows up again, and if
it does, apply the patch to 2.4.45 and see if it goes away again.

Thanks...

changed notes

changed notes

On Fri, Sep 15, 2017 at 07:54:22PM -0700, Paul B. Henson wrote:
> I've been running 2.4.44 with the test patch since May and haven't seen
> a crash since, for what that's worth. We upgraded to 2.4.45 last week,
> without that patch. I'm going to see if the crash shows up again, and if
> it does, apply the patch to 2.4.45 and see if it goes away again.

Well, since upgrading, I've now seen two crashes; I don't have it
compiled with debugging right now so the trace is limited:

#0  0x0000000000483eb9 in modify_add_values ()
#1  0x00007f7b80d45bc3 in mdb_modify_internal () from
/usr/lib64/openldap/openldap/back_mdb.so
#2  0x00007f7b80d46f55 in mdb_modify () from
/usr/lib64/openldap/openldap/back_mdb.so
#3  0x000000000049a6d2 in overlay_op_walk ()
#4  0x000000000049a82a in ?? ()
#5  0x000000000048bbef in ?? ()
#6  0x000000000048f8cf in ?? ()
#7  0x0000000000493ae3 in ?? ()
#8  0x00007f7b858327f2 in ?? () from /usr/lib64/libldap_r-2.4.so.2
#9  0x00007f7b84bb03e4 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f7b848f23ed in clone () from /lib64/libc.so.6

but it looks like the same issue to me. I'm going to go ahead and
recompile 2.4.45 with the ignore-invalid-accesslog-ops.patch and run
that for a while and see what happens.

Thanks...

On Fri, Oct 06, 2017 at 06:45:41PM -0700, Paul B. Henson wrote:

> but it looks like the same issue to me. I'm going to go ahead and
> recompile 2.4.45 with the ignore-invalid-accesslog-ops.patch and run
> that for a while and see what happens.

Sadness :(, not one but two of my boxes crashed today, both running the
patch, with what appears to be the same issue:


Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000483eb9 in modify_add_values (e=e@entry=0x7f75994f9440, mod=mod@entry=0x7f758c0016c8, 
    permissive=0, text=text@entry=0x7f75994f99a0, 
    textbuf=textbuf@entry=0x7f75994f94c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=textlen@entry=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/mods.c:61
61		if ( !BER_BVISNULL( &mod->sm_values[mod->sm_numvals] )) {
[Current thread is 1 (Thread 0x7f75994fb700 (LWP 44707))]
(gdb) bt full
#0  0x0000000000483eb9 in modify_add_values (e=e@entry=0x7f75994f9440, mod=mod@entry=0x7f758c0016c8, 
    permissive=0, text=text@entry=0x7f75994f99a0, 
    textbuf=textbuf@entry=0x7f75994f94c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=textlen@entry=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/mods.c:61
        rc = <optimized out>
        op = 0x4c1265 "add"
        a = <optimized out>
        pmod = {sm_desc = <optimized out>, sm_values = 0x0, sm_nvalues = 0x0, sm_numvals = <optimized out>, 
          sm_op = <optimized out>, sm_flags = <optimized out>, sm_type = {bv_len = <optimized out>, 
            bv_val = <optimized out>}}
        __PRETTY_FUNCTION__ = "modify_add_values"
#1  0x00007f779dbd9bc3 in mdb_modify_internal (op=op@entry=0x7f75994fa600, tid=tid@entry=0x16bbae0, 
    modlist=0x7f758c001688, e=e@entry=0x7f75994f9440, text=text@entry=0x7f75994f99a0, 
    textbuf=textbuf@entry=0x7f75994f94c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/back-mdb/modify.c:137
        rc = <optimized out>
        err = <optimized out>
        mod = 0x7f758c0016c8
        ml = 0x7f758c0016c8
        save_attrs = 0x7f758c002708
        ap = 0x7f77a2997010
        glue_attr_delete = <optimized out>
        got_delete = 0
        __PRETTY_FUNCTION__ = "mdb_modify_internal"
#2  0x00007f779dbdaf55 in mdb_modify (op=0x7f75994fa600, rs=0x7f75994f9980)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/back-mdb/modify.c:624
        mdb = 0x7f77a2997010
        e = 0x7f758c0026b8
        manageDSAit = <optimized out>
---Type <return> to continue, or q <return> to quit---
        textbuf = "modify/delete: eduPersonAffiliation: no such attribute\000\000\276B2\214u\177\000\000\000\000\001\000\000\000\000\000\301B2\214u\177\000\000\276B2\214u\177\000\000\267B2\214u\177\000\000\220\031\000\214u\177\000\000\000\320\070\355\212˄\256\000\200", '\000' <repeats 15 times>, "\246O\231u\177\000\000pV0\001\000\000\000\000\220\031\000\214u\177\000\000PX0\001\000\000\000\000\000\200\000\000\000\000\000\000\236\313J\000\000\000\000\000\370[0\001\000\000\000\000\270\031\000\214u\177\000\000\000\246O\231u\177\000\000\260"...
        txn = 0x16bbae0
        opinfo = {moi_oe = {oe_next = {sle_next = 0x7f75994f9930}, oe_key = 0x7f77a2997010}, 
          moi_txn = 0x16bbae0, moi_ref = 1, moi_flag = 0 '\000'}
        moi = 0x7f75994f9420
        dummy = {e_id = 84455, e_name = {bv_len = 34, 
            bv_val = 0x7f758c002688 "uid=jredmond,ou=user,dc=cpp,dc=edu"}, e_nname = {bv_len = 34, 
            bv_val = 0x7f758c0031f0 "uid=jredmond,ou=user,dc=cpp,dc=edu"}, e_attrs = 0x7f75c42d5ed8, 
          e_ocflags = 256, e_bv = {bv_len = 0, bv_val = 0x0}, e_private = 0x7f758c0026b8}
        preread_ctrl = 0x0
        postread_ctrl = 0x0
        ctrls = {0x0, 0x2, 0xffffffffffffffff, 0x0, 0x0, 0x7f75994f9400}
        num_ctrls = 0
        numads = 61
#3  0x000000000049a6c2 in overlay_op_walk (op=op@entry=0x7f75994fa600, rs=0x7f75994f9980, which=op_modify, 
    oi=0x1305490, on=<optimized out>)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/backover.c:677
        func = <optimized out>
        rc = <optimized out>
#4  0x000000000049a81a in over_op_func (op=0x7f75994fa600, rs=<optimized out>, which=<optimized out>)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/backover.c:730
        oi = <optimized out>
        on = <optimized out>
        be = 0x12f6f70
        db = {bd_info = 0x7f779ddf3c60 <bi>, bd_self = 0x12f6f70, 
          be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\000\000\001", '\000' <repeats 14 times>, "\001", be_flags = 563464, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, 
---Type <return> to continue, or q <return> to quit---
            sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, 
            sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 64}, be_suffix = 0x12f6000, 
          be_nsuffix = 0x12f6680, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, 
            bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0x12f68a0 "cn=ldaproot,dc=cpp,dc=edu"}, 
          be_rootndn = {bv_len = 25, bv_val = 0x12f68f0 "cn=ldaproot,dc=cpp,dc=edu"}, be_rootpw = {
            bv_len = 14, bv_val = 0x12c72b0 ""}, be_max_deref_depth = 15, be_def_limit = {
            lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, 
            lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x12f5f30, be_acl = 0x12f82c0, 
          be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, 
          be_update_refs = 0x0, be_pending_csn_list = 0x15b8d90, be_pcl_mutex = {__data = {__lock = 0, 
              __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
                __prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, 
          be_syncinfo = 0x13060d0, be_pb = 0x0, be_cf_ocs = 0x7f779ddf3820 <mdbocs>, 
          be_private = 0x7f77a2997010, be_next = {stqe_next = 0x0}}
        cb = {sc_next = 0x7f75994f9950, sc_response = 0x499a10 <over_back_response>, sc_cleanup = 0x0, 
          sc_private = 0x1305490, sc_writewait = 0x0}
        sc = <optimized out>
        rc = 32768
        __PRETTY_FUNCTION__ = "over_op_func"
#5  0x000000000048bbdf in syncrepl_message_to_op (si=si@entry=0x1306800, op=op@entry=0x7f75994fa600, 
    msg=0x7f758c112430)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:2423
        oes = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x48b010 <syncrepl_message_to_op>}, 
          oe_si = 0x1306800}
        ber = 0x7f758c1162e0
        modlist = 0x7f758c129dc0
        ls = 0x72e2a0 <accesslog_sc>
        rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, 
          sr_text = 0x7f75994f94c0 "modify/delete: eduPersonAffiliation: no such attribute", sr_ref = 0x0, 
          sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, 
              r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {
              r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0}
---Type <return> to continue, or q <return> to quit---
        cb = {sc_next = 0x7f758c001960, sc_response = 0x48ab80 <null_callback>, sc_cleanup = 0x0, 
          sc_private = 0x0, sc_writewait = 0x0}
        text = 0x29 <error: Cannot access memory at address 0x29>
        txtbuf = "@\237O\231u\177\000\000\261\215J\242w\177\000\000@\237O\231u\177\000\000\020\234O\231u\177\000\000\020'M\000\000\000\000\000\026\220J\242w\177\000\000@\237O\231u\177\000\000\020'M\000\000\000\000\000\220\232O\231u\177\000\000`\374J\242w\177\000\000\020'M\000\000\000\000\000@\235J\242w\177\000\000\020\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377\200\232O\017\000\000\000\000\f\\1\214u\177\000\000 \234O\231u\177\000\000\b\\1\214u\177\000\000\001\000\000\000\000\000\000\000\017\\1\214u\177\000\000\340\234O\231u\177\000\000@\231(\001\000\000\000\000\000h0\001\000\000\000\000\213\307I\000\000\000\000\000\020", '\000' <repeats 23 times>...
        bdn = {bv_len = 34, bv_val = 0x7f758c10fc4c "uid=jredmond,ou=user,dc=cpp,dc=edu"}
        dn = {bv_len = 34, bv_val = 0x7f758c001968 ""}
        ndn = {bv_len = 34, bv_val = 0x7f758c0019e8 "\""}
        bv = {bv_len = 0, bv_val = 0x0}
        bv2 = {bv_len = 0, bv_val = 0x0}
        bvals = 0x7f758c1144a0
        rdn = {bv_len = 0, bv_val = 0x0}
        sup = {bv_len = 0, bv_val = 0x0}
        prdn = {bv_len = 0, bv_val = 0x0}
        nrdn = {bv_len = 0, bv_val = 0x0}
        psup = {bv_len = 0, bv_val = 0x0}
        nsup = {bv_len = 0, bv_val = 0x0}
        rc = 0
        deleteOldRdn = 0
        freeReqDn = 1
        do_graduate = 1
#6  0x000000000048f8bf in do_syncrep2 (op=op@entry=0x7f75994fa600, si=si@entry=0x1306800)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:1014
        match = <optimized out>
        syncUUID = {{bv_len = 16, 
            bv_val = 0x7f758c11b257 "\006_\367\067\304\032H\273\273F \355\022\271\b\f"}, {
            bv_len = 140143060033504, 
---Type <return> to continue, or q <return> to quit---
            bv_val = 0x7f7500000001 <error: Cannot access memory at address 0x7f7500000001>}}
        cookie = {bv_len = 15, bv_val = 0x7f758c11b269 "rid=002,sid=001"}
        rctrls = 0x7f758c10fb80
        rctrlp = <optimized out>
        bdn = {bv_len = 44, bv_val = 0x7f758c10fbfa "reqStart=20171101104541.000063Z,cn=accesslog"}
        si_tag = <optimized out>
        entry = <optimized out>
        punlock = -1
        syncstate = 1
        retdata = 0x0
        retoid = 0x0
        syncUUIDs = 0x7f758c11f0f0
        len = 15
        berbuf = {
          buffer = "\002\000\001", '\000' <repeats 29 times>, "P\262\021\214u\177\000\000x\262\021\214u\177\000\000x\262\021\214u\177", '\000' <repeats 26 times>, "\244q\021\234u\177\000\000m\252\365\240w\177\000\000\340\237O\231u\177\000\000b\232\356\240w\177\000\000\001\000\000\000\000\000\000\000ԏ\357\240w\177\000\000\364:\n\277\325\362^\203\250\336\vʶy\"\277=w3\340\204(\302\375P\262\021\214u\177\000\000\060\000\000\000\000\000\000\000\200\373\020\214u\177\000\000\060\000\000\000\000\000\000\000ТO\231u\177\000\000\000\242O\231u\177\000\000СO\231"..., ialign = 65538, lalign = 65538, falign = 9.18382988e-41, dalign = 3.2380074297143616e-319, 
          palign = 0x10002 <error: Cannot access memory at address 0x10002>}
        ber = 0x7f75994f9f40
        msg = 0x7f758c112430
        syncCookie = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 2, octet_str = {bv_len = 15, 
            bv_val = 0x7f758c315c00 "rid=002,sid=001"}, sid = 1, sc_next = {stqe_next = 0x0}}
        syncCookie_req = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 2, octet_str = {bv_len = 15, 
            bv_val = 0x7f758c105fc0 "rid=002,sid=001"}, sid = 1, sc_next = {stqe_next = 0x0}}
        rc = 0
        err = 0
        modlist = 0x0
        m = 11
        tout_p = 0x7f75994f9c00
        tout = {tv_sec = 0, tv_usec = 0}
---Type <return> to continue, or q <return> to quit---
        refreshDeletes = 0
        empty = "empty"
#7  0x0000000000493ad3 in do_syncrepl (ctx=ctx@entry=0x7f75994fab90, arg=arg@entry=0x1306c20)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:1565
        rtask = 0x1306c20
        si = 0x1306800
        conn = {c_struct_state = SLAP_C_UNINITIALIZED, c_conn_state = SLAP_C_INVALID, c_conn_idx = -1, 
          c_sd = 0, c_close_reason = 0x0, c_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
              __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_sb = 0x0, c_starttime = 0, 
          c_activitytime = 0, c_connid = 18446744073709551615, c_peer_domain = {bv_len = 0, 
            bv_val = 0x4c7052 ""}, c_peer_name = {bv_len = 0, bv_val = 0x4c7052 ""}, 
          c_listener = 0x4c9ee0 <dummy_list>, c_sasl_bind_mech = {bv_len = 0, bv_val = 0x0}, c_sasl_dn = {
            bv_len = 0, bv_val = 0x0}, c_sasl_authz_dn = {bv_len = 0, bv_val = 0x0}, c_authz_backend = 0x0, 
          c_authz_cookie = 0x0, c_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {
              bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, 
            sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, c_protocol = 0, c_ops = {
            stqh_first = 0x0, stqh_last = 0x0}, c_pending_ops = {stqh_first = 0x0, stqh_last = 0x0}, 
          c_write1_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write1_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_write2_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write2_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_currentber = 0x0, c_writers = 0, c_writing = 0 '\000', c_sasl_bind_in_progress = 0 '\000', 
          c_writewaiter = 0 '\000', c_is_udp = 0 '\000', c_is_tls = 0 '\000', c_needs_tls_accept = 0 '\000', 
          c_sasl_layers = 0 '\000', c_sasl_done = 0 '\000', c_sasl_authctx = 0x0, c_sasl_sockctx = 0x0, 
          c_sasl_extra = 0x0, c_sasl_bindop = 0x0, c_pagedresults_state = {ps_be = 0x0, ps_size = 0, 
---Type <return> to continue, or q <return> to quit---
            ps_count = 0, ps_cookie = 0, ps_cookieval = {bv_len = 0, bv_val = 0x0}}, c_n_ops_received = 0, 
          c_n_ops_executing = 0, c_n_ops_pending = 0, c_n_ops_completed = 0, c_n_get = 0, c_n_read = 0, 
          c_n_write = 0, c_extensions = 0x0, c_clientfunc = 0x0, c_clientarg = 0x0, 
          c_send_ldap_result = 0x440a60 <slap_send_ldap_result>, 
          c_send_search_entry = 0x4414c0 <slap_send_search_entry>, 
          c_send_search_reference = 0x442a00 <slap_send_search_reference>, 
          c_send_ldap_extended = 0x441190 <slap_send_ldap_extended>, 
          c_send_ldap_intermediate = 0x441330 <slap_send_ldap_intermediate>}
        opbuf = {ob_op = {o_hdr = 0x7f75994fa770, o_tag = 102, o_time = 1509533141, o_tincr = 61, 
            o_bd = 0x7f75994f9690, o_req_dn = {bv_len = 34, 
              bv_val = 0x7f758c11f0f0 "uid=jredmond,ou=user,dc=cpp,dc=edu"}, o_req_ndn = {bv_len = 34, 
              bv_val = 0x7f758c11d4c0 "uid=jredmond,ou=user,dc=cpp,dc=edu"}, o_request = {oq_add = {
                rs_modlist = 0x7f758c001688, rs_e = 0x1}, oq_bind = {rb_method = -1946151288, rb_cred = {
                  bv_len = 1, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_mech = {
                  bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x7f758c001688}, oq_modify = {
                rs_mods = {rs_modlist = 0x7f758c001688, rs_no_opattrs = 1 '\001'}, rs_increment = 0}, 
              oq_modrdn = {rs_mods = {rs_modlist = 0x7f758c001688, rs_no_opattrs = 1 '\001'}, 
                rs_deleteoldrdn = 0, rs_newrdn = {bv_len = 0, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, 
                  bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0}, oq_search = {rs_scope = -1946151288, 
                rs_deref = 32629, rs_slimit = 1, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, 
                rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {
                rs_msgid = -1946151288}, oq_cancel = {rs_msgid = -1946151288}, oq_extended = {rs_reqoid = {
                  bv_len = 140142836717192, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {
                    bv_len = 140142836717192, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                  rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {
                  bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, 
            o_groups = 0x0, o_do_not_cache = 0 '\000', o_is_auth_check = 0 '\000', 
            o_dont_replicate = 0 '\000', o_acl_priv = ACL_NONE, o_nocaching = 0 '\000', 
            o_delete_glue_parent = 0 '\000', o_no_schema_check = 1 '\001', o_no_subordinate_glue = 0 '\000', 
            o_ctrlflag = '\000' <repeats 14 times>, "\002", '\000' <repeats 16 times>, 
            o_controls = 0x7f75994fa8c0, o_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, 
              sai_dn = {bv_len = 25, bv_val = 0x12f68a0 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ndn = {
---Type <return> to continue, or q <return> to quit---
                bv_len = 25, bv_val = 0x12f68f0 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ssf = 0, 
              sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, 
            o_callback = 0x7f758c001990, o_ctrls = 0x0, o_csn = {bv_len = 40, 
              bv_val = 0x7f758c3242a0 "20171101104541.389271Z#000000#004#000000"}, o_private = 0x0, 
            o_extra = {slh_first = 0x7f75994f9420}, o_next = {stqe_next = 0x0}}, ob_hdr = {oh_opid = 0, 
            oh_connid = 2, oh_conn = 0x7f75994fa340, oh_msgid = 0, oh_protocol = 0, 
            oh_tid = 140143060039424, oh_threadctx = 0x7f75994fab90, oh_tmpmemctx = 0x7f758c001150, 
            oh_tmpmfuncs = 0x72e140 <slap_sl_mfuncs>, oh_counters = 0x731640 <slap_counters>, 
            oh_log_prefix = "conn=-1 op=0", '\000' <repeats 243 times>, oh_extensions = 0x0}, ob_controls = {
            0x0 <repeats 17 times>, 0x7f75994f9ce0, 0x0 <repeats 14 times>}}
        op = 0x7f75994fa600
        rc = 0
        dostop = 0
        s = 60
        i = <optimized out>
        defer = 1
        fail = 0
        freeinfo = 0
        be = 0x12f6f70
#8  0x0000000000430bdf in connection_read_thread (ctx=0x7f75994fab90, argv=0x3c)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/connection.c:1296
        rc = <optimized out>
        cri = {op = 0x0, func = 0x4937f0 <do_syncrepl>, arg = 0x1306c20, ctx = 0x7f75994fab90, 
          nullop = <optimized out>}
        s = 60
#9  0x00007f77a26c67f2 in ldap_int_thread_pool_wrapper (xpool=0x1293710)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/libraries/libldap_r/tpool.c:696
        pool = 0x1293710
        task = 0x7f75d8000b80
        work_list = <optimized out>
        ctx = {ltu_id = 140143060039424, ltu_key = {{ltk_key = 0x42e370 <conn_counter_init>, 
---Type <return> to continue, or q <return> to quit---
              ltk_data = 0x7f758c001040, ltk_free = 0x42e450 <conn_counter_destroy>}, {
              ltk_key = 0x484d20 <slap_sl_mem_init>, ltk_data = 0x7f758c001150, 
              ltk_free = 0x484bf0 <slap_sl_mem_destroy>}, {ltk_key = 0x15b8e00, ltk_data = 0x7f758c1011a0, 
              ltk_free = 0x7f779dbe7730 <mdb_reader_free>}, {ltk_key = 0x7f779dbdd5f0 <search_stack>, 
              ltk_data = 0x7f7586fff010, ltk_free = 0x7f779dbdd700 <search_stack_free>}, {
              ltk_key = 0x7f779dbdd550 <scope_chunk_get>, ltk_data = 0x7f75989fa010, 
              ltk_free = 0x7f779dbdd6d0 <scope_chunk_free>}, {ltk_key = 0x443960 <slap_op_free>, 
              ltk_data = 0x7f75cc315ce0, ltk_free = 0x443930 <slap_op_q_destroy>}, {ltk_key = 0x13b2430, 
              ltk_data = 0x7f758c119050, ltk_free = 0x7f779dbe7730 <mdb_reader_free>}, {ltk_key = 0x0, 
              ltk_data = 0x7f75c8130aa0, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, 
              ltk_free = 0x0} <repeats 24 times>}}
        kctx = <optimized out>
        keyslot = <optimized out>
        hash = <optimized out>
        __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#10 0x00007f77a1a443e4 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007f77a17863ed in clone () from /lib64/libc.so.6
No symbol table info available.


Core was generated by `/usr/lib64/openldap/slapd -u ldap -g ldap -h ldaps:// ldap://'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000483eb9 in modify_add_values (e=e@entry=0x7f306e4f6440, mod=mod@entry=0x7f3060000ad0, 
    permissive=0, text=text@entry=0x7f306e4f69a0, 
    textbuf=textbuf@entry=0x7f306e4f64c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=textlen@entry=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/mods.c:61
61		if ( !BER_BVISNULL( &mod->sm_values[mod->sm_numvals] )) {
[Current thread is 1 (Thread 0x7f306e4f8700 (LWP 85701))]
(gdb) bt full
#0  0x0000000000483eb9 in modify_add_values (e=e@entry=0x7f306e4f6440, mod=mod@entry=0x7f3060000ad0, 
    permissive=0, text=text@entry=0x7f306e4f69a0, 
    textbuf=textbuf@entry=0x7f306e4f64c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=textlen@entry=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/mods.c:61
        rc = <optimized out>
        op = 0x4c1265 "add"
        a = <optimized out>
        pmod = {sm_desc = <optimized out>, sm_values = 0x0, sm_nvalues = 0x0, sm_numvals = <optimized out>, 
          sm_op = <optimized out>, sm_flags = <optimized out>, sm_type = {bv_len = <optimized out>, 
            bv_val = <optimized out>}}
        __PRETTY_FUNCTION__ = "modify_add_values"
#1  0x00007f3251ec4bc3 in mdb_modify_internal (op=op@entry=0x7f306e4f7600, tid=tid@entry=0xffc950, 
    modlist=0x7f3060000a90, e=e@entry=0x7f306e4f6440, text=text@entry=0x7f306e4f69a0, 
    textbuf=textbuf@entry=0x7f306e4f64c0 "modify/delete: eduPersonAffiliation: no such attribute", 
    textlen=256)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/back-mdb/modify.c:137
        rc = <optimized out>
        err = <optimized out>
        mod = 0x7f3060000ad0
        ml = 0x7f3060000ad0
        save_attrs = 0x7f3060001b50
        ap = 0x7f3256c82010
        glue_attr_delete = <optimized out>
        got_delete = 0
        __PRETTY_FUNCTION__ = "mdb_modify_internal"
#2  0x00007f3251ec5f55 in mdb_modify (op=0x7f306e4f7600, rs=0x7f306e4f6980)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/back-mdb/modify.c:624
        mdb = 0x7f3256c82010
        e = 0x7f3060001b00
        manageDSAit = <optimized out>
---Type <return> to continue, or q <return> to quit---
        textbuf = "modify/delete: eduPersonAffiliation: no such attribute\000\000\276\247\020`0\177\000\000\000\000\001\000\000\000\000\000\301\247\020`0\177\000\000\276\247\020`0\177\000\000\267\247\020`0\177\000\000\230\r\000`0\177\000\000\000_\246mQo\262.\000\200", '\000' <repeats 15 times>, "vOn0\177\000\000Pf\304\000\000\000\000\000\230\r\000`0\177\000\000\060h\304\000\000\000\000\000\000\200\000\000\000\000\000\000\236\313J\000\000\000\000\000\330k\304\000\000\000\000\000\300\r\000`0\177\000\000\000vOn0\177\000\000"...
        txn = 0xffc950
        opinfo = {moi_oe = {oe_next = {sle_next = 0x7f306e4f6930}, oe_key = 0x7f3256c82010}, 
          moi_txn = 0xffc950, moi_ref = 1, moi_flag = 0 '\000'}
        moi = 0x7f306e4f6420
        dummy = {e_id = 94173, e_name = {bv_len = 31, 
            bv_val = 0x7f3060001ad8 "uid=jplau,ou=user,dc=cpp,dc=edu"}, e_nname = {bv_len = 31, 
            bv_val = 0x7f30600026a0 "uid=jplau,ou=user,dc=cpp,dc=edu"}, e_attrs = 0x7f3078126340, 
          e_ocflags = 256, e_bv = {bv_len = 0, bv_val = 0x0}, e_private = 0x7f3060001b00}
        preread_ctrl = 0x0
        postread_ctrl = 0x0
        ctrls = {0x0, 0x2, 0xffffffffffffffff, 0x0, 0x0, 0x7f306e4f6400}
        num_ctrls = 0
        numads = 61
#3  0x000000000049a6c2 in overlay_op_walk (op=op@entry=0x7f306e4f7600, rs=0x7f306e4f6980, which=op_modify, 
    oi=0xc46470, on=<optimized out>)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/backover.c:677
        func = <optimized out>
        rc = <optimized out>
#4  0x000000000049a81a in over_op_func (op=0x7f306e4f7600, rs=<optimized out>, which=<optimized out>)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/backover.c:730
        oi = <optimized out>
        on = <optimized out>
        be = 0xc37f50
        db = {bd_info = 0x7f32520dec60 <bi>, bd_self = 0xc37f50, 
          be_ctrls = "\000\001\001\001\000\001\000\000\001\000\000\001\001\000\001\000\000\001", '\000' <repeats 14 times>, "\001", be_flags = 563464, be_restrictops = 0, be_requires = 0, be_ssf_set = {sss_ssf = 0, 
---Type <return> to continue, or q <return> to quit---
            sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, 
            sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 64}, be_suffix = 0xc36fe0, 
          be_nsuffix = 0xc37660, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, 
            bv_val = 0x0}, be_rootdn = {bv_len = 25, bv_val = 0xc37880 "cn=ldaproot,dc=cpp,dc=edu"}, 
          be_rootndn = {bv_len = 25, bv_val = 0xc378d0 "cn=ldaproot,dc=cpp,dc=edu"}, be_rootpw = {
            bv_len = 14, bv_val = 0xc08290 ""}, be_max_deref_depth = 15, be_def_limit = {
            lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = -1, lms_s_hard = 0, lms_s_unchecked = -1, 
            lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0xc36f10, be_acl = 0xc392a0, 
          be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, 
          be_update_refs = 0x0, be_pending_csn_list = 0xdf8120, be_pcl_mutex = {__data = {__lock = 0, 
              __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {
                __prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, 
          be_syncinfo = 0xc470b0, be_pb = 0x0, be_cf_ocs = 0x7f32520de820 <mdbocs>, 
          be_private = 0x7f3256c82010, be_next = {stqe_next = 0x0}}
        cb = {sc_next = 0x7f306e4f6950, sc_response = 0x499a10 <over_back_response>, sc_cleanup = 0x0, 
          sc_private = 0xc46470, sc_writewait = 0x0}
        sc = <optimized out>
        rc = 32768
        __PRETTY_FUNCTION__ = "over_op_func"
#5  0x000000000048bbdf in syncrepl_message_to_op (si=si@entry=0xc477e0, op=op@entry=0x7f306e4f7600, 
    msg=0x7f3060111950)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:2423
        oes = {oe = {oe_next = {sle_next = 0x0}, oe_key = 0x48b010 <syncrepl_message_to_op>}, 
          oe_si = 0xc477e0}
        ber = 0x7f3060115190
        modlist = 0x7f3060108c00
        ls = 0x72e2a0 <accesslog_sc>
        rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, 
          sr_text = 0x7f306e4f64c0 "modify/delete: eduPersonAffiliation: no such attribute", sr_ref = 0x0, 
          sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, 
              r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {
              r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0}
---Type <return> to continue, or q <return> to quit---
        cb = {sc_next = 0x7f3060000d68, sc_response = 0x48ab80 <null_callback>, sc_cleanup = 0x0, 
          sc_private = 0x0, sc_writewait = 0x0}
        text = 0x29 <error: Cannot access memory at address 0x29>
        txtbuf = "@oOn0\177\000\000\261=yV2\177\000\000@oOn0\177\000\000\020lOn0\177\000\000\020'M\000\000\000\000\000\026@yV2\177\000\000@oOn0\177\000\000\020'M\000\000\000\000\000\220jOn0\177\000\000`\254yV2\177\000\000\020'M\000\000\000\000\000@MyV2\177\000\000\020\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377\200jO\017\000\000\000\000\314\370\020`0\177\000\000 lOn0\177\000\000\310\370\020`0\177\000\000\001\000\000\000\000\000\000\000\317\370\020`0\177\000\000\340lOn0\177\000\000\060\251\274\000\000\000\000\000\340w\304\000\000\000\000\000\213\307I\000\000\000\000\000\020", '\000' <repeats 23 times>...
        bdn = {bv_len = 31, bv_val = 0x7f30601157bc "uid=jplau,ou=user,dc=cpp,dc=edu"}
        dn = {bv_len = 31, bv_val = 0x7f3060000d68 ""}
        ndn = {bv_len = 31, bv_val = 0x7f3060000de8 "h\032"}
        bv = {bv_len = 0, bv_val = 0x0}
        bv2 = {bv_len = 0, bv_val = 0x0}
        bvals = 0x7f3060103740
        rdn = {bv_len = 0, bv_val = 0x0}
        sup = {bv_len = 0, bv_val = 0x0}
        prdn = {bv_len = 0, bv_val = 0x0}
        nrdn = {bv_len = 0, bv_val = 0x0}
        psup = {bv_len = 0, bv_val = 0x0}
        nsup = {bv_len = 0, bv_val = 0x0}
        rc = 0
        deleteOldRdn = 0
        freeReqDn = 1
        do_graduate = 1
#6  0x000000000048f8bf in do_syncrep2 (op=op@entry=0x7f306e4f7600, si=si@entry=0xc477e0)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:1014
        match = <optimized out>
        syncUUID = {{bv_len = 16, 
            bv_val = 0x7f306010e577 ")\263\217MC\265F~\211\377\vB\020", <incomplete sequence \350\240>}, {
            bv_len = 139845985857504, 
            bv_val = 0x7f3000000001 <error: Cannot access memory at address 0x7f3000000001>}}
---Type <return> to continue, or q <return> to quit---
        cookie = {bv_len = 15, bv_val = 0x7f306010e589 "rid=002,sid=001"}
        rctrls = 0x7f306010ae50
        rctrlp = <optimized out>
        bdn = {bv_len = 44, bv_val = 0x7f306011576a "reqStart=20171101104541.000020Z,cn=accesslog"}
        si_tag = <optimized out>
        entry = <optimized out>
        punlock = -1
        syncstate = 1
        retdata = 0x0
        retoid = 0x0
        syncUUIDs = 0x7f30601184b0
        len = 15
        berbuf = {
          buffer = "\002\000\001", '\000' <repeats 29 times>, "p\345\020`0\177\000\000\230\345\020`0\177\000\000\230\345\020`0\177", '\000' <repeats 26 times>, "$\243\021t0\177\000\000mZ$U2\177\000\000\340oOn0\177\000\000bJ\035U2\177\000\000\001\000\000\000\000\000\000\000\324?\036U2\177\000\000\215\005\217\022\023\366\017\311\035,D\303~M0\\\311\326\350\341e\204Xa0q\020`0\177\000\000\060\000\000\000\000\000\000\000 \246\020`0\177\000\000\060\000\000\000\000\000\000\000\320rOn0\177\000\000\000rOn0\177\000\000"..., ialign = 65538, 
          lalign = 65538, falign = 9.18382988e-41, dalign = 3.2380074297143616e-319, 
          palign = 0x10002 <error: Cannot access memory at address 0x10002>}
        ber = 0x7f306e4f6f40
        msg = 0x7f3060111950
        syncCookie = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 2, octet_str = {bv_len = 15, 
            bv_val = 0x7f306010f8c0 "rid=002,sid=001"}, sid = 1, sc_next = {stqe_next = 0x0}}
        syncCookie_req = {ctxcsn = 0x0, sids = 0x0, numcsns = 0, rid = 2, octet_str = {bv_len = 15, 
            bv_val = 0x7f30601044f0 "rid=002,sid=001"}, sid = 1, sc_next = {stqe_next = 0x0}}
        rc = 0
        err = 0
        modlist = 0x0
        m = 0
        tout_p = 0x7f306e4f6c00
        tout = {tv_sec = 0, tv_usec = 0}
        refreshDeletes = 0
---Type <return> to continue, or q <return> to quit---
        empty = "empty"
#7  0x0000000000493ad3 in do_syncrepl (ctx=ctx@entry=0x7f306e4f7b90, arg=arg@entry=0xc47c00)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/syncrepl.c:1565
        rtask = 0xc47c00
        si = 0xc477e0
        conn = {c_struct_state = SLAP_C_UNINITIALIZED, c_conn_state = SLAP_C_INVALID, c_conn_idx = -1, 
          c_sd = 0, c_close_reason = 0x0, c_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
              __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_sb = 0x0, c_starttime = 0, 
          c_activitytime = 0, c_connid = 18446744073709551615, c_peer_domain = {bv_len = 0, 
            bv_val = 0x4c7052 ""}, c_peer_name = {bv_len = 0, bv_val = 0x4c7052 ""}, 
          c_listener = 0x4c9ee0 <dummy_list>, c_sasl_bind_mech = {bv_len = 0, bv_val = 0x0}, c_sasl_dn = {
            bv_len = 0, bv_val = 0x0}, c_sasl_authz_dn = {bv_len = 0, bv_val = 0x0}, c_authz_backend = 0x0, 
          c_authz_cookie = 0x0, c_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, sai_dn = {
              bv_len = 0, bv_val = 0x0}, sai_ndn = {bv_len = 0, bv_val = 0x0}, sai_ssf = 0, 
            sai_transport_ssf = 0, sai_tls_ssf = 0, sai_sasl_ssf = 0}, c_protocol = 0, c_ops = {
            stqh_first = 0x0, stqh_last = 0x0}, c_pending_ops = {stqh_first = 0x0, stqh_last = 0x0}, 
          c_write1_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write1_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_write2_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, 
              __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' <repeats 39 times>, __align = 0}, c_write2_cv = {__data = {__lock = 0, 
              __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, 
              __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' <repeats 47 times>, __align = 0}, 
          c_currentber = 0x0, c_writers = 0, c_writing = 0 '\000', c_sasl_bind_in_progress = 0 '\000', 
          c_writewaiter = 0 '\000', c_is_udp = 0 '\000', c_is_tls = 0 '\000', c_needs_tls_accept = 0 '\000', 
          c_sasl_layers = 0 '\000', c_sasl_done = 0 '\000', c_sasl_authctx = 0x0, c_sasl_sockctx = 0x0, 
          c_sasl_extra = 0x0, c_sasl_bindop = 0x0, c_pagedresults_state = {ps_be = 0x0, ps_size = 0, 
            ps_count = 0, ps_cookie = 0, ps_cookieval = {bv_len = 0, bv_val = 0x0}}, c_n_ops_received = 0, 
---Type <return> to continue, or q <return> to quit---
          c_n_ops_executing = 0, c_n_ops_pending = 0, c_n_ops_completed = 0, c_n_get = 0, c_n_read = 0, 
          c_n_write = 0, c_extensions = 0x0, c_clientfunc = 0x0, c_clientarg = 0x0, 
          c_send_ldap_result = 0x440a60 <slap_send_ldap_result>, 
          c_send_search_entry = 0x4414c0 <slap_send_search_entry>, 
          c_send_search_reference = 0x442a00 <slap_send_search_reference>, 
          c_send_ldap_extended = 0x441190 <slap_send_ldap_extended>, 
          c_send_ldap_intermediate = 0x441330 <slap_send_ldap_intermediate>}
        opbuf = {ob_op = {o_hdr = 0x7f306e4f7770, o_tag = 102, o_time = 1509533141, o_tincr = 30, 
            o_bd = 0x7f306e4f6690, o_req_dn = {bv_len = 31, 
              bv_val = 0x7f3060109190 "uid=jplau,ou=user,dc=cpp,dc=edu"}, o_req_ndn = {bv_len = 31, 
              bv_val = 0x7f306010d940 "uid=jplau,ou=user,dc=cpp,dc=edu"}, o_request = {oq_add = {
                rs_modlist = 0x7f3060000a90, rs_e = 0x1}, oq_bind = {rb_method = 1610615440, rb_cred = {
                  bv_len = 1, bv_val = 0x0}, rb_edn = {bv_len = 0, bv_val = 0x0}, rb_ssf = 0, rb_mech = {
                  bv_len = 0, bv_val = 0x0}}, oq_compare = {rs_ava = 0x7f3060000a90}, oq_modify = {
                rs_mods = {rs_modlist = 0x7f3060000a90, rs_no_opattrs = 1 '\001'}, rs_increment = 0}, 
              oq_modrdn = {rs_mods = {rs_modlist = 0x7f3060000a90, rs_no_opattrs = 1 '\001'}, 
                rs_deleteoldrdn = 0, rs_newrdn = {bv_len = 0, bv_val = 0x0}, rs_nnewrdn = {bv_len = 0, 
                  bv_val = 0x0}, rs_newSup = 0x0, rs_nnewSup = 0x0}, oq_search = {rs_scope = 1610615440, 
                rs_deref = 32560, rs_slimit = 1, rs_tlimit = 0, rs_limit = 0x0, rs_attrsonly = 0, 
                rs_attrs = 0x0, rs_filter = 0x0, rs_filterstr = {bv_len = 0, bv_val = 0x0}}, oq_abandon = {
                rs_msgid = 1610615440}, oq_cancel = {rs_msgid = 1610615440}, oq_extended = {rs_reqoid = {
                  bv_len = 139845745773200, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                rs_flags = 0, rs_reqdata = 0x0}, oq_pwdexop = {rs_extended = {rs_reqoid = {
                    bv_len = 139845745773200, bv_val = 0x1 <error: Cannot access memory at address 0x1>}, 
                  rs_flags = 0, rs_reqdata = 0x0}, rs_old = {bv_len = 0, bv_val = 0x0}, rs_new = {
                  bv_len = 0, bv_val = 0x0}, rs_mods = 0x0, rs_modtail = 0x0}}, o_abandon = 0, o_cancel = 0, 
            o_groups = 0x0, o_do_not_cache = 0 '\000', o_is_auth_check = 0 '\000', 
            o_dont_replicate = 0 '\000', o_acl_priv = ACL_NONE, o_nocaching = 0 '\000', 
            o_delete_glue_parent = 0 '\000', o_no_schema_check = 1 '\001', o_no_subordinate_glue = 0 '\000', 
            o_ctrlflag = '\000' <repeats 14 times>, "\002", '\000' <repeats 16 times>, 
            o_controls = 0x7f306e4f78c0, o_authz = {sai_method = 0, sai_mech = {bv_len = 0, bv_val = 0x0}, 
              sai_dn = {bv_len = 25, bv_val = 0xc37880 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ndn = {bv_len = 25, 
                bv_val = 0xc378d0 "cn=ldaproot,dc=cpp,dc=edu"}, sai_ssf = 0, sai_transport_ssf = 0, 
---Type <return> to continue, or q <return> to quit---
              sai_tls_ssf = 0, sai_sasl_ssf = 0}, o_ber = 0x0, o_res_ber = 0x0, o_callback = 0x7f3060000d98, 
            o_ctrls = 0x0, o_csn = {bv_len = 40, 
              bv_val = 0x7f306010a7a0 "20171101104537.144920Z#000000#004#000000"}, o_private = 0x0, 
            o_extra = {slh_first = 0x7f306e4f6420}, o_next = {stqe_next = 0x0}}, ob_hdr = {oh_opid = 0, 
            oh_connid = 2, oh_conn = 0x7f306e4f7340, oh_msgid = 0, oh_protocol = 0, 
            oh_tid = 139845985863424, oh_threadctx = 0x7f306e4f7b90, oh_tmpmemctx = 0x7f30600008c0, 
            oh_tmpmfuncs = 0x72e140 <slap_sl_mfuncs>, oh_counters = 0x731640 <slap_counters>, 
            oh_log_prefix = "conn=-1 op=0", '\000' <repeats 243 times>, oh_extensions = 0x0}, ob_controls = {
            0x0 <repeats 17 times>, 0x7f306e4f6ce0, 0x0 <repeats 14 times>}}
        op = 0x7f306e4f7600
        rc = 0
        dostop = 0
        s = 19
        i = <optimized out>
        defer = 1
        fail = 0
        freeinfo = 0
        be = 0xc37f50
#8  0x0000000000430bdf in connection_read_thread (ctx=0x7f306e4f7b90, argv=0x13)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/servers/slapd/connection.c:1296
        rc = <optimized out>
        cri = {op = 0x0, func = 0x4937f0 <do_syncrepl>, arg = 0xc47c00, ctx = 0x7f306e4f7b90, 
          nullop = <optimized out>}
        s = 19
#9  0x00007f32569b17f2 in ldap_int_thread_pool_wrapper (xpool=0xbd4700)
    at /var/lib/portage/tmp/portage/net-nds/openldap-2.4.45-r2/work/openldap-2.4.45/libraries/libldap_r/tpool.c:696
        pool = 0xbd4700
        task = 0x7f308c000ce0
        work_list = <optimized out>
        ctx = {ltu_id = 139845985863424, ltu_key = {{ltk_key = 0x484d20 <slap_sl_mem_init>, 
              ltk_data = 0x7f30600008c0, ltk_free = 0x484bf0 <slap_sl_mem_destroy>}, {ltk_key = 0xef9ca0, 
---Type <return> to continue, or q <return> to quit---
              ltk_data = 0x7f30601018f0, ltk_free = 0x7f3251ed2730 <mdb_reader_free>}, {
              ltk_key = 0x7f3251ec85f0 <search_stack>, ltk_data = 0x7f306c9f6010, 
              ltk_free = 0x7f3251ec8700 <search_stack_free>}, {ltk_key = 0x7f3251ec8550 <scope_chunk_get>, 
              ltk_data = 0x7f306d9f7010, ltk_free = 0x7f3251ec86d0 <scope_chunk_free>}, {
              ltk_key = 0x42e370 <conn_counter_init>, ltk_data = 0x7f30601091f0, 
              ltk_free = 0x42e450 <conn_counter_destroy>}, {ltk_key = 0x443960 <slap_op_free>, 
              ltk_data = 0x7f30681b0200, ltk_free = 0x443930 <slap_op_q_destroy>}, {ltk_key = 0xcf3410, 
              ltk_data = 0x7f306010bef0, ltk_free = 0x7f3251ed2730 <mdb_reader_free>}, {ltk_key = 0x0, 
              ltk_data = 0x7f308424c830, ltk_free = 0x0}, {ltk_key = 0x0, ltk_data = 0x0, 
              ltk_free = 0x0} <repeats 24 times>}}
        kctx = <optimized out>
        keyslot = <optimized out>
        hash = <optimized out>
        __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
#10 0x00007f3255d2f3e4 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007f3255a713ed in clone () from /lib64/libc.so.6
No symbol table info available.

On Wed, Nov 01, 2017 at 08:22:15PM +0000, henson@acm.org wrote:
> On Fri, Oct 06, 2017 at 06:45:41PM -0700, Paul B. Henson wrote:
>> but it looks like the same issue to me. I'm going to go ahead and
>> recompile 2.4.45 with the ignore-invalid-accesslog-ops.patch and run
>> that for a while and see what happens.
> 
> Sadness :(, not one but two of my boxes crashed today, both running the
> patch, with what appears to be the same issue:

Hi Paul,
I've updated the patch slightly to make sure invalid operations get
logged and rejected so some action can be taken. Based on the backtrace,
I can't see how an add modification could have ended up without values
even with the previous version, however.

A patch against master is available here:
https://github.com/mistotebe/openldap/tree/its8609

And you can get one that applies on RE24 here:
ftp://ftp.openldap.org/incoming/ndrej-Kuznik-20170327-ITS8609-ignore-invalid-accesslog-ops.patch

If you still see crashes with the above, I might need you to explore the
core file and accesslog a bit further.

Regards,

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP

On Thu, Apr 18, 2019 at 12:46:57PM +0200, Ondřej Kuzník wrote:

> If you still see crashes with the above, I might need you to explore the
> core file and accesslog a bit further.

Hmm, actually, I haven't seen a crash for quite a while; I don't recall
exactly when the last one was but it's been ages. Right now I'm running
2.4.46 with the patch for ITS#8927 applied, but without the last
ITS#8609 patch you had given me.

I think the last crash was on 2.4.45 with the ITS#8927 patch. Maybe
something in 2.4.46 randomly fixed it?

--On Thursday, April 18, 2019 8:09 PM +0000 henson@acm.org wrote:

> On Thu, Apr 18, 2019 at 12:46:57PM +0200, Ond??ej Kuzn??k wrote:
>
>> If you still see crashes with the above, I might need you to explore the
>> core file and accesslog a bit further.
>
> Hmm, actually, I haven't seen a crash for quite a while; I don't recall
> exactly when the last one was but it's been ages. Right now I'm running
> 2.4.46 with the patch for ITS#8927 applied, but without the last
> ITS#8609 patch you had given me.
>
> I think the last crash was on 2.4.45 with the ITS#8927 patch. Maybe
> something in 2.4.46 randomly fixed it?

Hi Paul,

Thanks for the follow up.  I'm going to go ahead and close this ITS.  If 
you do happen to see a crash again in the future, please follow up and 
we'll address it at that time.

Regards,
Quanah

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

Apparently fixed in 2.4.46 by other fixes.

changed notes
changed state Open to Closed