[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#3227)



Also, note that if you submit a large number of simultaneous connections,
those that exceed the number of available threads are queued and remain
pending.  I guess the sigsegv is a bug, and it would be nice to be able
to track it down.  I haven't been able to generate it on my system, so it
might be something related to your setupo, or at least something that
depends on the rest of the environmet.   However, in your case, if you
think your production system may be undergoing a high load, you might try
to increase the number of available threads.

p.

> Pierangelo Masarati wrote:
>> According to the link you sent, each instance of your application
>> is trying to send 512 simultanoeus requests to slapd:
>>
>>         perl load-test.pl --server=server-name --num-forks=512
>>
>> since slapd cannot handle more than 1024 file descriptors (as far
>> as I know, because of an intrinsic limitation in glibc's select)
>> you're likely to be exausting system resources.  The core dump
>> you're showing is meaningless to me, because it shows the error
>> occurring in an obscure and generic internal of glibc rather than
>> in some specific part of slapd, starting from generic low level I/O
>> routines of libldap.  Can you reproduce the problem with a more
>> limited load?
>
> Yes, with "--num-forks=32" run on each of two machines, the server still
> crashes with the same problem.  It performed fine with "--num-forks=16"
> and "--num-forks=24".
>
> Please consider that the # of file descriptors is at least doubled,
> because the LDAP backend is being used for each request to rewrite from
> "o=WFU,c=US" to "ou=Users,dc=wfu,dc=edu".
>
> With "lsof" monitoring, the pattern seems to be
>    1. normal # conns
>    2. quickly increasing # conns
>    3. hanging until one or both processes killed
>    4. unresponsive until # connections goes down
>    5. normal # conns
>    6. a lockup
>    7. crash
>
> During testing, I may have found a better gdb backtrace, too!  Check out
> the "__assert_fail" statement.  Thank you very much!
>
> #0  0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> No symbol table info available.
> #1  0xb7262a29 in raise () from /lib/tls/libc.so.6
> No symbol table info available.
> #2  0xb7264255 in abort () from /lib/tls/libc.so.6
> No symbol table info available.
> #3  0xb725c559 in __assert_fail () from /lib/tls/libc.so.6
> No symbol table info available.
> #4  0x0806ad32 in slap_op_free (op=0x8b6405d8) at operation.c:66
>          slap_empty_bv_dup = {bv_len = 2433723312, bv_val = 0xb7380b7c
> "xÚ"}
> #5  0x0807391e in do_abandon (op=0x8233d40, rs=0x910fa870) at
> abandon.c:107
>          id = 1971
>          o = (Operation *) 0x8b6405d8
>          i = 7
> #6  0x0805a591 in connection_operation (ctx=0x910fa900, arg_v=0x8233d40)
> at connection.c:1047
>          rc = -1025
>          op = (Operation *) 0x8233d40
>          rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err =
> 0, sr_matched = 0x0, sr_text = 0x0,
>    sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_sasl = {r_sasldata = 0x0},
> sru_extended = {r_rspoid = 0x0,
>        r_rspdata = 0x0}, sru_search = {r_entry = 0x0, r_attrs = 0x0,
> r_nentries = 0, r_v2ref = 0x0}}, sr_flags = 0}
>          tag = 80
>          oldtag = 80
>          conn = (Connection *) 0x96a9d088
>          memctx = (void *) 0x82d7930
>          memctx_null = (void *) 0x0
>          memsiz = 1048576
> #7  0x080de3b6 in ldap_int_thread_pool_wrapper (xpool=0x8154fc0) at
> tpool.c:467
>          pool = (struct ldap_int_thread_pool_s *) 0x8154fc0
>          ctx = (ldap_int_thread_ctx_t *) 0x90874260
>          ltc_key = {{ltk_key = 0x8097a48, ltk_data = 0x82d7930, ltk_free
> = 0x8097a18 <sl_mem_destroy>}, {
>      ltk_key = 0x81ad600, ltk_data = 0x132, ltk_free = 0x80bc6d0
> <bdb_locker_id_free>}, {ltk_key = 0x80af37d,
>      ltk_data = 0x88dfd008, ltk_free = 0x80af365 <search_stack_free>},
> {ltk_key = 0x0, ltk_data = 0x0,
>      ltk_free = 0} <repeats 29 times>}
>          tid = 2433723312
>          i = 507
>          keyslot = 507
>          hash = 507
> #8  0xb7377dac in start_thread () from /lib/tls/libpthread.so.0
> No symbol table info available.
> #9  0xb7316a8a in clone () from /lib/tls/libc.so.6
> No symbol table info available.
> (gdb) bt full
> #0  0xb75ebc32 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
> No symbol table info available.
> #1  0xb7262a29 in raise () from /lib/tls/libc.so.6
> No symbol table info available.
> #2  0xb7264255 in abort () from /lib/tls/libc.so.6
> No symbol table info available.
> #3  0xb725c559 in __assert_fail () from /lib/tls/libc.so.6
> No symbol table info available.
> #4  0x0806ad32 in slap_op_free (op=0x8b6405d8) at operation.c:66
>          slap_empty_bv_dup = {bv_len = 2433723312, bv_val = 0xb7380b7c
> "xÚ"}
> #5  0x0807391e in do_abandon (op=0x8233d40, rs=0x910fa870) at
> abandon.c:107
>          id = 1971
>          o = (Operation *) 0x8b6405d8
>          i = 7
> #6  0x0805a591 in connection_operation (ctx=0x910fa900, arg_v=0x8233d40)
> at connection.c:1047
>          rc = -1025
>          op = (Operation *) 0x8233d40
>          rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err =
> 0, sr_matched = 0x0, sr_text = 0x0,
>    sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_sasl = {r_sasldata = 0x0},
> sru_extended = {r_rspoid = 0x0,
>        r_rspdata = 0x0}, sru_search = {r_entry = 0x0, r_attrs = 0x0,
> r_nentries = 0, r_v2ref = 0x0}}, sr_flags = 0}
>          tag = 80
>          oldtag = 80
>          conn = (Connection *) 0x96a9d088
>          memctx = (void *) 0x82d7930
>          memctx_null = (void *) 0x0
>          memsiz = 1048576
> #7  0x080de3b6 in ldap_int_thread_pool_wrapper (xpool=0x8154fc0) at
> tpool.c:467
>          pool = (struct ldap_int_thread_pool_s *) 0x8154fc0
>          ctx = (ldap_int_thread_ctx_t *) 0x90874260
>          ltc_key = {{ltk_key = 0x8097a48, ltk_data = 0x82d7930, ltk_free
> = 0x8097a18 <sl_mem_destroy>}, {
>      ltk_key = 0x81ad600, ltk_data = 0x132, ltk_free = 0x80bc6d0
> <bdb_locker_id_free>}, {ltk_key = 0x80af37d,
>      ltk_data = 0x88dfd008, ltk_free = 0x80af365 <search_stack_free>},
> {ltk_key = 0x0, ltk_data = 0x0,
>      ltk_free = 0} <repeats 29 times>}
>          tid = 2433723312
>          i = 507
>          keyslot = 507
>          hash = 507
> #8  0xb7377dac in start_thread () from /lib/tls/libpthread.so.0
> No symbol table info available.
> #9  0xb7316a8a in clone () from /lib/tls/libc.so.6
> No symbol table info available.
>
>
> --
>             John Borwick
>         Systems Administrator
>        Wake Forest University | web  http://www.wfu.edu/~borwicjh
>        Winston-Salem, NC, USA | GPG key ID               56D60872
>


-- 
Pierangelo Masarati
mailto:pierangelo.masarati@sys-net.it


    SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497