michael@stroeder.com wrote:
> Full_Name: Michael Ströder
> Version: 2.4.39 with patches
> OS:
> URL:
> Submission from: (NULL) (212.227.35.93)
>
>
> slapd occasionally seg faults.
> We can't reproduce it with a certain configuration.
>
> This is a custom Debian Wheezy package based on OpenLDAP 2.4.39
> linked against OpenSSL under a separate prefix.
>
> slapo-rwm is patched according to ITS#7723 but I can't tell whether
> it's related to ITS#7723 or not. Hence the separate ITS.

What behavior do you get by reverting the #7723 patch?
>
> master commit for the reference counting patch used:
> http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commit;h=5f524c4465d38c2c81e472cae9702f2a51888e8f
>
> Here's the stack trace (obfuscated filter and search base):
>
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) bt full
> #0  0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #1  0x00000000004f776b in rewrite_var_cmp (c1=0x7f8abaffb680, c2=0x7f8a884fba80)
> at var.c:43
>          v1 = 0x7f8abaffb680
>          v2 = 0x7f8a884fba80
>          __PRETTY_FUNCTION__ = "rewrite_var_cmp"
> #2  0x0000000000500920 in avl_find (root=0x7f8a892823b0, data=0x7f8abaffb680,
> fcmp=0x4f76a8 <rewrite_var_cmp>) at avl.c:545
>          cmp = 0
> #3  0x00000000004f797e in rewrite_var_find (tree=0x7f8a892823b0,
> name=0x7f8c9ff4b319 "searchFilter") at var.c:115
>          var = {lv_name = 0x7f8c9ff4b319 "searchFilter", lv_flags = -1534643252,
> lv_value = {bv_len = 140233568059984, bv_val = 0x7f8a89523fa0 ""}}
>          __PRETTY_FUNCTION__ = "rewrite_var_find"
> #4  0x00000000004f5f51 in rewrite_session_var_set_f (info=0x1b4f770,
> cookie=0x7f8c9fee2510, name=0x7f8c9ff4b319 "searchFilter",
>      value=0x7f8aac1d93a0
> "(&(member=uid=xxxxx,cn=yyyyy,ou=zzzzz)(objectClass=posixGroup)(cn=*))",
> flags=15) at session.c:222
>          session = 0x7f8a89523f90
>          var = 0x1ffb
>          __PRETTY_FUNCTION__ = "rewrite_session_var_set_f"
> #5  0x00007f8c9ff41836 in rwm_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0)
> at rwm.c:955
>          on = 0x1b4f530
>          rwmap = 0x1b4f710
>          rc = 32652
>          dc = {rwmap = 0x7f8abaffb960, conn = 0x0, ctx = 0xffffffffffffffa8
> <Address 0xffffffffffffffa8 out of bounds>, rs = 0x2e90e50}
>          fstr = {bv_len = 0, bv_val = 0x0}
>          f = 0x0
>          an = 0x0
>          text = 0x0
>          roc = 0x7f8aac1d95e8
> #6  0x00000000004cf13b in overlay_op_walk (op=0x7f8aadb03a70, rs=0x7f8abaffcac0,
> which=op_search, oi=0x1b4d270, on=0x1b4f530) at backover.c:661
>          func = 0x1b4f588
>          rc = 32768
> #7  0x00000000004cf3ef in over_op_func (op=0x7f8aadb03a70, rs=0x7f8abaffcac0,
> which=op_search) at backover.c:723
>          oi = 0x1b4d270
>          on = 0x1b4f530
>          be = 0x772340
>          db = {bd_info = 0x1b4f530, bd_self = 0x772340, be_ctrls = "\000", '\001'
> <repeats 16 times>, '\000' <repeats 15 times>, be_flags = 131840, be_restrictops
> = 0, be_requires = 19, be_ssf_set = {
>              sss_ssf = 128, sss_transport = 0, sss_tls = 0, sss_sasl = 0,
> sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0,
> sss_update_sasl = 0, sss_simple_bind = 0},
>            be_suffix = 0x1acae40, be_nsuffix = 0x1acae90, be_schemadn = {bv_len =
> 12, bv_val = 0x1ba32d0 "cn=Subschema"}, be_schemandn = {bv_len = 12, bv_val =
> 0x1ba2c80 "cn=subschema"}, be_rootdn = {
>              bv_len = 0, bv_val = 0x0}, be_rootndn = {bv_len = 0, bv_val = 0x0},
> be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit =
> {lms_t_soft = 3600, lms_t_hard = 0,
>              lms_s_soft = 500, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr =
> 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x1b54bd0,
> be_dfltaccess = ACL_READ,
>            be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0},
> be_update_refs = 0x0, be_pending_csn_list = 0x0, be_pcl_mutex = {__data =
> {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
>                __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
> __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x0, be_pb =
> 0x0, be_cf_ocs = 0x768e88, be_private = 0x0,
>            be_next = {stqe_next = 0x1acaee0}}
>          cb = {sc_next = 0x0, sc_response = 0x4ce266 <over_back_response>,
> sc_cleanup = 0, sc_private = 0x1b4d270}
>          sc = 0x7f8aac1d9578
>          rc = 32768
>          __PRETTY_FUNCTION__ = "over_op_func"
> #8  0x00000000004cf4cf in over_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0)
> at backover.c:750
> No locals.
> #9  0x0000000000440982 in do_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) at
> search.c:247
>          base = {bv_len = 8, bv_val = 0x7f8a8801d589 "ou=zzzzz"}
>          siz = 7
>          off = 0
>          i = 7
> #10 0x000000000043d2da in connection_operation (ctx=0x7f8abaffcba0,
> arg_v=0x7f8aadb03a70) at connection.c:1155
>          rc = 80
>          cancel = 32650
>          op = 0x7f8aadb03a70
>          rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0,
> sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un =
> {sru_search = {r_entry = 0x0, r_attr_flags = 0,
>                r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref
> = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata
> = 0x0}}, sr_flags = 0}
>          tag = 99
>          opidx = SLAP_OP_SEARCH
>          conn = 0x7f8c9fee2510
>          memctx = 0x7f8aac028250
>          memctx_null = 0x0
>          memsiz = 1048576
>          __PRETTY_FUNCTION__ = "connection_operation"
> #11 0x00007f8ca4871c81 in ldap_int_thread_pool_wrapper (xpool=0x1aa7f70) at
> tpool.c:688
>          pool = 0x1aa7f70
>          task = 0x7f8aa57bb7a0
>          work_list = 0x1aa8008
>          ctx = {ltu_id = 140233819543296, ltu_key = {{ltk_key = 0x43ce31,
> ltk_data = 0x7f8aac028140, ltk_free = 0x43cc83 <conn_counter_destroy>}, {ltk_key
> = 0x4af7af, ltk_data = 0x7f8aac028250,
>                ltk_free = 0x4af5d4 <slap_sl_mem_destroy>}, {ltk_key = 0x1c58fc0,
> ltk_data = 0x7f8aac024fb0, ltk_free = 0x7f8ca15c4122 <mdb_reader_free>},
> {ltk_key = 0x7f8ca15b6b94,
>                ltk_data = 0x7f8a9f0f7010, ltk_free = 0x7f8ca15b6b4c
> <scope_chunk_free>}, {ltk_key = 0x457c69, ltk_data = 0x7f8a88063040, ltk_free =
> 0x457bbc <slap_op_q_destroy>}, {
>                ltk_key = 0x7f8ca15b9a39, ltk_data = 0x7f8a94be7010, ltk_free =
> 0x7f8ca15b9a16 <search_stack_free>}, {ltk_key = 0x1c67c80, ltk_data =
> 0x7f8ab46aadb0,
>                ltk_free = 0x7f8ca15c4122 <mdb_reader_free>}, {ltk_key = 0x0,
> ltk_data = 0x0, ltk_free = 0} <repeats 25 times>}}
>          kctx = 0x0
> ---Type <return> to continue, or q <return> to quit---
>          i = 32
>          keyslot = 377
>          hash = 3746453881
>          __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper"
> #12 0x00007f8ca2feab50 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> No symbol table info available.
> #13 0x00007f8ca2d350ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
> No symbol table info available.
> #14 0x0000000000000000 in ?? ()
> No symbol table info available.
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

hyc@symas.com wrote:
>> slapd occasionally seg faults.
>> We can't reproduce it with a certain configuration.
>>
>> This is a custom Debian Wheezy package based on OpenLDAP 2.4.39
>> linked against OpenSSL under a separate prefix.
>>
>> slapo-rwm is patched according to ITS#7723 but I can't tell whether
>> it's related to ITS#7723 or not. Hence the separate ITS.
> 
> What behavior do you get by reverting the #7723 patch?

The patch was applied because we had these seg faults before.
I'd rather say nothing changed in our case with the patch.
Crashes happen once or twice a week or so.
Also it was impossible to crash the non-patched installation with a test
script acting like described in ITS#7723.

Fixing this would be highly appreciated but I can't provide a simple config
reproducing it.

The following rwm directives are in the frontend part:

overlay rwm
rwm-rewriteEngine on
rwm-drop-unrequested-attrs no
# uid=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching (uid=foo)
rwm-rewriteMap slapd uid2dn "ldap:///ou=xxxxx?entryDN?sub?"
rwm-rewriteContext bindDN
rwm-rewriteRule "^(uid=[^,]+),ou=xxxxx$" "${uid2dn($1)}" ":@I"
# serverFqdn=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching
(serverFqdn=foo)
rwm-rewriteMap slapd fqdn2dn "ldap:///ou=xxxxx?entryDN?sub?"
rwm-rewriteContext bindDN
rwm-rewriteRule "^(serverFqdn=[^,]+),ou=xxxxx$" "${fqdn2dn($1)}" ":@I"

In a former configuration version these directives were in the backend
ou=xxxxx part. Because of the seg faults I moved it which made things slightly
better but hard to tell. In another configuration variant I even experienced
seg faults with *slapcat*.

This is a two-layer replication topology with several MMR providers and
read-only consumers which use SASL/EXTERNAL with client certs for
authentication and authz-regexp mapping to authz-DNs. If things are wrong
during consumer initialization sometimes even the providers crashes.

Ciao, Michael.

Michael Ströder wrote:
> hyc@symas.com wrote:
>>> slapd occasionally seg faults.
>>> We can't reproduce it with a certain configuration.
>>>
>>> This is a custom Debian Wheezy package based on OpenLDAP 2.4.39
>>> linked against OpenSSL under a separate prefix.
>>>
>>> slapo-rwm is patched according to ITS#7723 but I can't tell whether
>>> it's related to ITS#7723 or not. Hence the separate ITS.
>>
>> What behavior do you get by reverting the #7723 patch?
>
> The patch was applied because we had these seg faults before.
> I'd rather say nothing changed in our case with the patch.
> Crashes happen once or twice a week or so.
> Also it was impossible to crash the non-patched installation with a test
> script acting like described in ITS#7723.
>
> Fixing this would be highly appreciated but I can't provide a simple config
> reproducing it.

Can you run slapd with ElectricFence and post stack traces and diagnostics 
from any crashes there?
>
> The following rwm directives are in the frontend part:
>
> overlay rwm
> rwm-rewriteEngine on
> rwm-drop-unrequested-attrs no
> # uid=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching (uid=foo)
> rwm-rewriteMap slapd uid2dn "ldap:///ou=xxxxx?entryDN?sub?"
> rwm-rewriteContext bindDN
> rwm-rewriteRule "^(uid=[^,]+),ou=xxxxx$" "${uid2dn($1)}" ":@I"
> # serverFqdn=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching
> (serverFqdn=foo)
> rwm-rewriteMap slapd fqdn2dn "ldap:///ou=xxxxx?entryDN?sub?"
> rwm-rewriteContext bindDN
> rwm-rewriteRule "^(serverFqdn=[^,]+),ou=xxxxx$" "${fqdn2dn($1)}" ":@I"
>
> In a former configuration version these directives were in the backend
> ou=xxxxx part. Because of the seg faults I moved it which made things slightly
> better but hard to tell. In another configuration variant I even experienced
> seg faults with *slapcat*.
>
> This is a two-layer replication topology with several MMR providers and
> read-only consumers which use SASL/EXTERNAL with client certs for
> authentication and authz-regexp mapping to authz-DNs. If things are wrong
> during consumer initialization sometimes even the providers crashes.
>
> Ciao, Michael.
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

On Tue, 15 Apr 2014 04:35:26 -0700 Howard Chu <hyc@symas.com> wrote
> Can you run slapd with ElectricFence and post stack traces and diagnostics 
> from any crashes there?

Hmm, I will look into it.
But I'm a bit cautious running such a component in production.
Would running with MALLOC_CHECK_ be also reasonable helpful?

BTW: The configuration uses back-mdb.

Another strange thing to note:
The archived syslog files revealed that the bind-DN rewriting is currently
*not* used by the LDAP clients at all. So I really wonder where it even calls
into slapo-rwm during normal operation.

Ciao, Michael.

Michael Ströder wrote:
> On Tue, 15 Apr 2014 04:35:26 -0700 Howard Chu <hyc@symas.com> wrote
>> Can you run slapd with ElectricFence and post stack traces and diagnostics
>> from any crashes there?
>
> Hmm, I will look into it.
> But I'm a bit cautious running such a component in production.
> Would running with MALLOC_CHECK_ be also reasonable helpful?

Possibly using MALLOC_CHECK_=3 could give us a clue, yes.
>
> BTW: The configuration uses back-mdb.
>
> Another strange thing to note:
> The archived syslog files revealed that the bind-DN rewriting is currently
> *not* used by the LDAP clients at all. So I really wonder where it even calls
> into slapo-rwm during normal operation.

Good question.
>
> Ciao, Michael.
>
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

One more thing to note:
We see some coincidence between the seg faults and situations where the load on
the machine goes up and number of pending threads in back-monitor had values in
the range of 16 to 22 (threads is 16 for four cores).