Full_Name: Michael Str�der Version: 2.4.39 with patches OS: URL: Submission from: (NULL) (212.227.35.93) slapd occasionally seg faults. We can't reproduce it with a certain configuration. This is a custom Debian Wheezy package based on OpenLDAP 2.4.39 linked against OpenSSL under a separate prefix. slapo-rwm is patched according to ITS#7723 but I can't tell whether it's related to ITS#7723 or not. Hence the separate ITS. master commit for the reference counting patch used: http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commit;h=5f524c4465d38c2c81e472cae9702f2a51888e8f Here's the stack trace (obfuscated filter and search base): Program terminated with signal 11, Segmentation fault. #0 0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt full #0 0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00000000004f776b in rewrite_var_cmp (c1=0x7f8abaffb680, c2=0x7f8a884fba80) at var.c:43 v1 = 0x7f8abaffb680 v2 = 0x7f8a884fba80 __PRETTY_FUNCTION__ = "rewrite_var_cmp" #2 0x0000000000500920 in avl_find (root=0x7f8a892823b0, data=0x7f8abaffb680, fcmp=0x4f76a8 <rewrite_var_cmp>) at avl.c:545 cmp = 0 #3 0x00000000004f797e in rewrite_var_find (tree=0x7f8a892823b0, name=0x7f8c9ff4b319 "searchFilter") at var.c:115 var = {lv_name = 0x7f8c9ff4b319 "searchFilter", lv_flags = -1534643252, lv_value = {bv_len = 140233568059984, bv_val = 0x7f8a89523fa0 ""}} __PRETTY_FUNCTION__ = "rewrite_var_find" #4 0x00000000004f5f51 in rewrite_session_var_set_f (info=0x1b4f770, cookie=0x7f8c9fee2510, name=0x7f8c9ff4b319 "searchFilter", value=0x7f8aac1d93a0 "(&(member=uid=xxxxx,cn=yyyyy,ou=zzzzz)(objectClass=posixGroup)(cn=*))", flags=15) at session.c:222 session = 0x7f8a89523f90 var = 0x1ffb __PRETTY_FUNCTION__ = "rewrite_session_var_set_f" #5 0x00007f8c9ff41836 in rwm_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) at rwm.c:955 on = 0x1b4f530 rwmap = 0x1b4f710 rc = 32652 dc = {rwmap = 0x7f8abaffb960, conn = 0x0, ctx = 0xffffffffffffffa8 <Address 0xffffffffffffffa8 out of bounds>, rs = 0x2e90e50} fstr = {bv_len = 0, bv_val = 0x0} f = 0x0 an = 0x0 text = 0x0 roc = 0x7f8aac1d95e8 #6 0x00000000004cf13b in overlay_op_walk (op=0x7f8aadb03a70, rs=0x7f8abaffcac0, which=op_search, oi=0x1b4d270, on=0x1b4f530) at backover.c:661 func = 0x1b4f588 rc = 32768 #7 0x00000000004cf3ef in over_op_func (op=0x7f8aadb03a70, rs=0x7f8abaffcac0, which=op_search) at backover.c:723 oi = 0x1b4d270 on = 0x1b4f530 be = 0x772340 db = {bd_info = 0x1b4f530, bd_self = 0x772340, be_ctrls = "\000", '\001' <repeats 16 times>, '\000' <repeats 15 times>, be_flags = 131840, be_restrictops = 0, be_requires = 19, be_ssf_set = { sss_ssf = 128, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x1acae40, be_nsuffix = 0x1acae90, be_schemadn = {bv_len = 12, bv_val = 0x1ba32d0 "cn=Subschema"}, be_schemandn = {bv_len = 12, bv_val = 0x1ba2c80 "cn=subschema"}, be_rootdn = { bv_len = 0, bv_val = 0x0}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit = {lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = 500, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x1b54bd0, be_dfltaccess = ACL_READ, be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pending_csn_list = 0x0, be_pcl_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x0, be_pb = 0x0, be_cf_ocs = 0x768e88, be_private = 0x0, be_next = {stqe_next = 0x1acaee0}} cb = {sc_next = 0x0, sc_response = 0x4ce266 <over_back_response>, sc_cleanup = 0, sc_private = 0x1b4d270} sc = 0x7f8aac1d9578 rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #8 0x00000000004cf4cf in over_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) at backover.c:750 No locals. #9 0x0000000000440982 in do_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) at search.c:247 base = {bv_len = 8, bv_val = 0x7f8a8801d589 "ou=zzzzz"} siz = 7 off = 0 i = 7 #10 0x000000000043d2da in connection_operation (ctx=0x7f8abaffcba0, arg_v=0x7f8aadb03a70) at connection.c:1155 rc = 80 cancel = 32650 op = 0x7f8aadb03a70 rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x7f8c9fee2510 memctx = 0x7f8aac028250 memctx_null = 0x0 memsiz = 1048576 __PRETTY_FUNCTION__ = "connection_operation" #11 0x00007f8ca4871c81 in ldap_int_thread_pool_wrapper (xpool=0x1aa7f70) at tpool.c:688 pool = 0x1aa7f70 task = 0x7f8aa57bb7a0 work_list = 0x1aa8008 ctx = {ltu_id = 140233819543296, ltu_key = {{ltk_key = 0x43ce31, ltk_data = 0x7f8aac028140, ltk_free = 0x43cc83 <conn_counter_destroy>}, {ltk_key = 0x4af7af, ltk_data = 0x7f8aac028250, ltk_free = 0x4af5d4 <slap_sl_mem_destroy>}, {ltk_key = 0x1c58fc0, ltk_data = 0x7f8aac024fb0, ltk_free = 0x7f8ca15c4122 <mdb_reader_free>}, {ltk_key = 0x7f8ca15b6b94, ltk_data = 0x7f8a9f0f7010, ltk_free = 0x7f8ca15b6b4c <scope_chunk_free>}, {ltk_key = 0x457c69, ltk_data = 0x7f8a88063040, ltk_free = 0x457bbc <slap_op_q_destroy>}, { ltk_key = 0x7f8ca15b9a39, ltk_data = 0x7f8a94be7010, ltk_free = 0x7f8ca15b9a16 <search_stack_free>}, {ltk_key = 0x1c67c80, ltk_data = 0x7f8ab46aadb0, ltk_free = 0x7f8ca15c4122 <mdb_reader_free>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 25 times>}} kctx = 0x0 ---Type <return> to continue, or q <return> to quit--- i = 32 keyslot = 377 hash = 3746453881 __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #12 0x00007f8ca2feab50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 No symbol table info available. #13 0x00007f8ca2d350ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #14 0x0000000000000000 in ?? () No symbol table info available.
michael@stroeder.com wrote: > Full_Name: Michael Ströder > Version: 2.4.39 with patches > OS: > URL: > Submission from: (NULL) (212.227.35.93) > > > slapd occasionally seg faults. > We can't reproduce it with a certain configuration. > > This is a custom Debian Wheezy package based on OpenLDAP 2.4.39 > linked against OpenSSL under a separate prefix. > > slapo-rwm is patched according to ITS#7723 but I can't tell whether > it's related to ITS#7723 or not. Hence the separate ITS. What behavior do you get by reverting the #7723 patch? > > master commit for the reference counting patch used: > http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=commit;h=5f524c4465d38c2c81e472cae9702f2a51888e8f > > Here's the stack trace (obfuscated filter and search base): > > Program terminated with signal 11, Segmentation fault. > #0 0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > (gdb) bt full > #0 0x00007f8ca2d8a29d in ?? () from /lib/x86_64-linux-gnu/libc.so.6 > No symbol table info available. > #1 0x00000000004f776b in rewrite_var_cmp (c1=0x7f8abaffb680, c2=0x7f8a884fba80) > at var.c:43 > v1 = 0x7f8abaffb680 > v2 = 0x7f8a884fba80 > __PRETTY_FUNCTION__ = "rewrite_var_cmp" > #2 0x0000000000500920 in avl_find (root=0x7f8a892823b0, data=0x7f8abaffb680, > fcmp=0x4f76a8 <rewrite_var_cmp>) at avl.c:545 > cmp = 0 > #3 0x00000000004f797e in rewrite_var_find (tree=0x7f8a892823b0, > name=0x7f8c9ff4b319 "searchFilter") at var.c:115 > var = {lv_name = 0x7f8c9ff4b319 "searchFilter", lv_flags = -1534643252, > lv_value = {bv_len = 140233568059984, bv_val = 0x7f8a89523fa0 ""}} > __PRETTY_FUNCTION__ = "rewrite_var_find" > #4 0x00000000004f5f51 in rewrite_session_var_set_f (info=0x1b4f770, > cookie=0x7f8c9fee2510, name=0x7f8c9ff4b319 "searchFilter", > value=0x7f8aac1d93a0 > "(&(member=uid=xxxxx,cn=yyyyy,ou=zzzzz)(objectClass=posixGroup)(cn=*))", > flags=15) at session.c:222 > session = 0x7f8a89523f90 > var = 0x1ffb > __PRETTY_FUNCTION__ = "rewrite_session_var_set_f" > #5 0x00007f8c9ff41836 in rwm_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) > at rwm.c:955 > on = 0x1b4f530 > rwmap = 0x1b4f710 > rc = 32652 > dc = {rwmap = 0x7f8abaffb960, conn = 0x0, ctx = 0xffffffffffffffa8 > <Address 0xffffffffffffffa8 out of bounds>, rs = 0x2e90e50} > fstr = {bv_len = 0, bv_val = 0x0} > f = 0x0 > an = 0x0 > text = 0x0 > roc = 0x7f8aac1d95e8 > #6 0x00000000004cf13b in overlay_op_walk (op=0x7f8aadb03a70, rs=0x7f8abaffcac0, > which=op_search, oi=0x1b4d270, on=0x1b4f530) at backover.c:661 > func = 0x1b4f588 > rc = 32768 > #7 0x00000000004cf3ef in over_op_func (op=0x7f8aadb03a70, rs=0x7f8abaffcac0, > which=op_search) at backover.c:723 > oi = 0x1b4d270 > on = 0x1b4f530 > be = 0x772340 > db = {bd_info = 0x1b4f530, bd_self = 0x772340, be_ctrls = "\000", '\001' > <repeats 16 times>, '\000' <repeats 15 times>, be_flags = 131840, be_restrictops > = 0, be_requires = 19, be_ssf_set = { > sss_ssf = 128, sss_transport = 0, sss_tls = 0, sss_sasl = 0, > sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, > sss_update_sasl = 0, sss_simple_bind = 0}, > be_suffix = 0x1acae40, be_nsuffix = 0x1acae90, be_schemadn = {bv_len = > 12, bv_val = 0x1ba32d0 "cn=Subschema"}, be_schemandn = {bv_len = 12, bv_val = > 0x1ba2c80 "cn=subschema"}, be_rootdn = { > bv_len = 0, bv_val = 0x0}, be_rootndn = {bv_len = 0, bv_val = 0x0}, > be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 0, be_def_limit = > {lms_t_soft = 3600, lms_t_hard = 0, > lms_s_soft = 500, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = > 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x1b54bd0, > be_dfltaccess = ACL_READ, > be_extra_anlist = 0x0, be_update_ndn = {bv_len = 0, bv_val = 0x0}, > be_update_refs = 0x0, be_pending_csn_list = 0x0, be_pcl_mutex = {__data = > {__lock = 0, __count = 0, __owner = 0, __nusers = 0, > __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, > __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x0, be_pb = > 0x0, be_cf_ocs = 0x768e88, be_private = 0x0, > be_next = {stqe_next = 0x1acaee0}} > cb = {sc_next = 0x0, sc_response = 0x4ce266 <over_back_response>, > sc_cleanup = 0, sc_private = 0x1b4d270} > sc = 0x7f8aac1d9578 > rc = 32768 > __PRETTY_FUNCTION__ = "over_op_func" > #8 0x00000000004cf4cf in over_op_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) > at backover.c:750 > No locals. > #9 0x0000000000440982 in do_search (op=0x7f8aadb03a70, rs=0x7f8abaffcac0) at > search.c:247 > base = {bv_len = 8, bv_val = 0x7f8a8801d589 "ou=zzzzz"} > siz = 7 > off = 0 > i = 7 > #10 0x000000000043d2da in connection_operation (ctx=0x7f8abaffcba0, > arg_v=0x7f8aadb03a70) at connection.c:1155 > rc = 80 > cancel = 32650 > op = 0x7f8aadb03a70 > rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 0, > sr_matched = 0x0, sr_text = 0x0, sr_ref = 0x0, sr_ctrls = 0x0, sr_un = > {sru_search = {r_entry = 0x0, r_attr_flags = 0, > r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref > = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata > = 0x0}}, sr_flags = 0} > tag = 99 > opidx = SLAP_OP_SEARCH > conn = 0x7f8c9fee2510 > memctx = 0x7f8aac028250 > memctx_null = 0x0 > memsiz = 1048576 > __PRETTY_FUNCTION__ = "connection_operation" > #11 0x00007f8ca4871c81 in ldap_int_thread_pool_wrapper (xpool=0x1aa7f70) at > tpool.c:688 > pool = 0x1aa7f70 > task = 0x7f8aa57bb7a0 > work_list = 0x1aa8008 > ctx = {ltu_id = 140233819543296, ltu_key = {{ltk_key = 0x43ce31, > ltk_data = 0x7f8aac028140, ltk_free = 0x43cc83 <conn_counter_destroy>}, {ltk_key > = 0x4af7af, ltk_data = 0x7f8aac028250, > ltk_free = 0x4af5d4 <slap_sl_mem_destroy>}, {ltk_key = 0x1c58fc0, > ltk_data = 0x7f8aac024fb0, ltk_free = 0x7f8ca15c4122 <mdb_reader_free>}, > {ltk_key = 0x7f8ca15b6b94, > ltk_data = 0x7f8a9f0f7010, ltk_free = 0x7f8ca15b6b4c > <scope_chunk_free>}, {ltk_key = 0x457c69, ltk_data = 0x7f8a88063040, ltk_free = > 0x457bbc <slap_op_q_destroy>}, { > ltk_key = 0x7f8ca15b9a39, ltk_data = 0x7f8a94be7010, ltk_free = > 0x7f8ca15b9a16 <search_stack_free>}, {ltk_key = 0x1c67c80, ltk_data = > 0x7f8ab46aadb0, > ltk_free = 0x7f8ca15c4122 <mdb_reader_free>}, {ltk_key = 0x0, > ltk_data = 0x0, ltk_free = 0} <repeats 25 times>}} > kctx = 0x0 > ---Type <return> to continue, or q <return> to quit--- > i = 32 > keyslot = 377 > hash = 3746453881 > __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" > #12 0x00007f8ca2feab50 in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > No symbol table info available. > #13 0x00007f8ca2d350ed in clone () from /lib/x86_64-linux-gnu/libc.so.6 > No symbol table info available. > #14 0x0000000000000000 in ?? () > No symbol table info available. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
hyc@symas.com wrote: >> slapd occasionally seg faults. >> We can't reproduce it with a certain configuration. >> >> This is a custom Debian Wheezy package based on OpenLDAP 2.4.39 >> linked against OpenSSL under a separate prefix. >> >> slapo-rwm is patched according to ITS#7723 but I can't tell whether >> it's related to ITS#7723 or not. Hence the separate ITS. > > What behavior do you get by reverting the #7723 patch? The patch was applied because we had these seg faults before. I'd rather say nothing changed in our case with the patch. Crashes happen once or twice a week or so. Also it was impossible to crash the non-patched installation with a test script acting like described in ITS#7723. Fixing this would be highly appreciated but I can't provide a simple config reproducing it. The following rwm directives are in the frontend part: overlay rwm rwm-rewriteEngine on rwm-drop-unrequested-attrs no # uid=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching (uid=foo) rwm-rewriteMap slapd uid2dn "ldap:///ou=xxxxx?entryDN?sub?" rwm-rewriteContext bindDN rwm-rewriteRule "^(uid=[^,]+),ou=xxxxx$" "${uid2dn($1)}" ":@I" # serverFqdn=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching (serverFqdn=foo) rwm-rewriteMap slapd fqdn2dn "ldap:///ou=xxxxx?entryDN?sub?" rwm-rewriteContext bindDN rwm-rewriteRule "^(serverFqdn=[^,]+),ou=xxxxx$" "${fqdn2dn($1)}" ":@I" In a former configuration version these directives were in the backend ou=xxxxx part. Because of the seg faults I moved it which made things slightly better but hard to tell. In another configuration variant I even experienced seg faults with *slapcat*. This is a two-layer replication topology with several MMR providers and read-only consumers which use SASL/EXTERNAL with client certs for authentication and authz-regexp mapping to authz-DNs. If things are wrong during consumer initialization sometimes even the providers crashes. Ciao, Michael.
Michael Ströder wrote: > hyc@symas.com wrote: >>> slapd occasionally seg faults. >>> We can't reproduce it with a certain configuration. >>> >>> This is a custom Debian Wheezy package based on OpenLDAP 2.4.39 >>> linked against OpenSSL under a separate prefix. >>> >>> slapo-rwm is patched according to ITS#7723 but I can't tell whether >>> it's related to ITS#7723 or not. Hence the separate ITS. >> >> What behavior do you get by reverting the #7723 patch? > > The patch was applied because we had these seg faults before. > I'd rather say nothing changed in our case with the patch. > Crashes happen once or twice a week or so. > Also it was impossible to crash the non-patched installation with a test > script acting like described in ITS#7723. > > Fixing this would be highly appreciated but I can't provide a simple config > reproducing it. Can you run slapd with ElectricFence and post stack traces and diagnostics from any crashes there? > > The following rwm directives are in the frontend part: > > overlay rwm > rwm-rewriteEngine on > rwm-drop-unrequested-attrs no > # uid=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching (uid=foo) > rwm-rewriteMap slapd uid2dn "ldap:///ou=xxxxx?entryDN?sub?" > rwm-rewriteContext bindDN > rwm-rewriteRule "^(uid=[^,]+),ou=xxxxx$" "${uid2dn($1)}" ":@I" > # serverFqdn=foo,ou=xxxxx -> entryDN of entry within ou=xxxxx matching > (serverFqdn=foo) > rwm-rewriteMap slapd fqdn2dn "ldap:///ou=xxxxx?entryDN?sub?" > rwm-rewriteContext bindDN > rwm-rewriteRule "^(serverFqdn=[^,]+),ou=xxxxx$" "${fqdn2dn($1)}" ":@I" > > In a former configuration version these directives were in the backend > ou=xxxxx part. Because of the seg faults I moved it which made things slightly > better but hard to tell. In another configuration variant I even experienced > seg faults with *slapcat*. > > This is a two-layer replication topology with several MMR providers and > read-only consumers which use SASL/EXTERNAL with client certs for > authentication and authz-regexp mapping to authz-DNs. If things are wrong > during consumer initialization sometimes even the providers crashes. > > Ciao, Michael. > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
On Tue, 15 Apr 2014 04:35:26 -0700 Howard Chu <hyc@symas.com> wrote > Can you run slapd with ElectricFence and post stack traces and diagnostics > from any crashes there? Hmm, I will look into it. But I'm a bit cautious running such a component in production. Would running with MALLOC_CHECK_ be also reasonable helpful? BTW: The configuration uses back-mdb. Another strange thing to note: The archived syslog files revealed that the bind-DN rewriting is currently *not* used by the LDAP clients at all. So I really wonder where it even calls into slapo-rwm during normal operation. Ciao, Michael.
Michael Ströder wrote: > On Tue, 15 Apr 2014 04:35:26 -0700 Howard Chu <hyc@symas.com> wrote >> Can you run slapd with ElectricFence and post stack traces and diagnostics >> from any crashes there? > > Hmm, I will look into it. > But I'm a bit cautious running such a component in production. > Would running with MALLOC_CHECK_ be also reasonable helpful? Possibly using MALLOC_CHECK_=3 could give us a clue, yes. > > BTW: The configuration uses back-mdb. > > Another strange thing to note: > The archived syslog files revealed that the bind-DN rewriting is currently > *not* used by the LDAP clients at all. So I really wonder where it even calls > into slapo-rwm during normal operation. Good question. > > Ciao, Michael. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
One more thing to note: We see some coincidence between the seg faults and situations where the load on the machine goes up and number of pending threads in back-monitor had values in the range of 16 to 22 (threads is 16 for four cores).