OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Incoming/6878
Full headers

From: tjgates@castlebranch.com
Subject: ppcache segfault with tavl_delete
Compose comment
Download message
State:
0 replies:
9 followups: 1 2 3 4 5 6 7 8 9

Major security issue: yes  no

Notes:

Notification:


Date: Thu, 24 Mar 2011 16:28:18 +0000
From: tjgates@castlebranch.com
To: openldap-its@OpenLDAP.org
Subject: ppcache segfault with tavl_delete
Full_Name: Tyler Gates
Version: 2.4.21
OS: Ubuntu 10.04.2 LTS/CentOS 5.4
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (24.106.216.17)


I have a caching proxy server (2.4.21) in front of two multi-masters. After
about 1 to 3 days it will segfault only when caching is enabled. Log files are
normally pretty useless but are normally in the operation of removing stale
queries: 

Mar 24 09:55:36 directory-proxy2 slapd[5363]: REMOVING TEMP ATTR :
TEMPLATE=ba2c2b64-ea65-102f-96bd-2f354737277c
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STALE QUERY REMOVED, SIZE=0
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STORED QUERIES = 22
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STALE QUERY REMOVED, CACHE =1
entries
Mar 24 09:55:36 directory-proxy2 slapd[5363]: Lock CR index = 0xb8848c00

I've attached to the running process using gdb and got the following:
**************************************************************************
Program received signal SIGPIPE, Broken pipe.

Program received signal SIGPIPE, Broken pipe.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb61b4b70 (LWP 5365)]
tavl_delete (root=0x0, data=0xb890bac0, fcmp=0xb6b64910
<pcache_query_cmp>) at
/build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c:200
200     /build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c: No such file or
directory.
        in /build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c
(gdb)
Continuing.
[Thread 0xb1effb70 (LWP 7113) exited]
[Thread 0xb2d9cb70 (LWP 10996) exited]
[Thread 0xb3a9fb70 (LWP 9738) exited]
[Thread 0xb47a2b70 (LWP 8550) exited]
[Thread 0xb65b5b70 (LWP 5364) exited]
[Thread 0xb61b4b70 (LWP 5365) exited]
[Thread 0xb5cb2b70 (LWP 5673) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) bt
No stack.
**********************************************************************

As you can see I cannot get a back trace, but I still have the gdb shell running
so I run any other commands if needed.
I've run version 2.4.18 on a CentOS machine with the same setup and I get
segfaults only when pcache is enabled although I've never run it under gdb.

Followup 1

Download message
Date: Thu, 24 Mar 2011 10:34:25 -0700
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: tjgates@castlebranch.com, openldap-its@openldap.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
--On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com wrote:

> Full_Name: Tyler Gates
> Version: 2.4.21
> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (24.106.216.17)
>
>
> I have a caching proxy server (2.4.21) in front of two multi-masters.
> After about 1 to 3 days it will segfault only when caching is enabled.
> Log files are normally pretty useless but are normally in the operation
> of removing stale queries:

Bug reports with 2.4.21 are not being pursued.  Please use a current 
release (2.4.24) and verify whether or not the issue still exists there.

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 2

Download message
Cc: "openldap-its@openldap.org" <openldap-its@openldap.org>
From: Tyler Gates <tjgates@castlebranch.com>
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
Date: Sun, 27 Mar 2011 14:58:38 -0400
To: Quanah Gibson-Mount <quanah@zimbra.com>
I've installed version 2.4.24 and been running it for a few days now. I'll
report back in a few days on it's outcome.


On Mar 24, 2011, at 1:34 PM, Quanah Gibson-Mount <quanah@zimbra.com>
wrote:

> --On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com wrote:
> 
>> Full_Name: Tyler Gates
>> Version: 2.4.21
>> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (24.106.216.17)
>> 
>> 
>> I have a caching proxy server (2.4.21) in front of two multi-masters.
>> After about 1 to 3 days it will segfault only when caching is enabled.
>> Log files are normally pretty useless but are normally in the operation
>> of removing stale queries:
> 
> Bug reports with 2.4.21 are not being pursued.  Please use a current
release (2.4.24) and verify whether or not the issue still exists there.
> 
> --Quanah
> 
> 
> --
> 
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration



Followup 3

Download message
Date: Wed, 30 Mar 2011 11:11:04 -0400
From: Tyler Gates <tjgates@castlebranch.com>
To: Quanah Gibson-Mount <quanah@zimbra.com>
CC: openldap-its@openldap.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
Just crashed:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb07f3b70 (LWP 24359)]
0xb77a7430 in __kernel_vsyscall ()
(gdb)
(gdb) bt
#0  0xb77a7430 in __kernel_vsyscall ()
#1  0xb727a651 in *__GI_raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb727da82 in *__GI_abort () at abort.c:92
#3  0xb72b149d in __libc_message (do_abort=2, fmt=0xb7385f98 "*** glibc
detected *** %s: %s: 0x%s ***\n") at
../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb72bb591 in malloc_printerr (action=<value optimized out>, str=0x6
<Address 0x6 out of bounds>, ptr=0xb8d93a40) at malloc.c:6266
#5  0xb72bcde8 in _int_free (av=<value optimized out>, p=<value
optimized out>) at malloc.c:4794
#6  0xb72bfecd in *__GI___libc_free (mem=0xb8d93a40) at malloc.c:3738
#7  0xb7750c20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
#8  0xb7750caf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
#9  0xb7750cf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
#10 0xb7808f3d in attr_clean (a=0xb688ff64) at
/tmp/buildd/openldap-2.4.24/servers/slapd/attr.c:146
#11 0xb7808fdb in attrs_free (a=0xb688ff64) at
/tmp/buildd/openldap-2.4.24/servers/slapd/attr.c:196
#12 0xb6e2db0f in hdb_cache_modify (bdb=0xb8c104b0, e=0xb8c57b5c,
newAttrs=0xb688ed64, txn=0xb8d93ea8, lock=0xb07f11f0) at cache.c:1226
#13 0xb6e1901c in hdb_modify (op=0xb07f152c, rs=0xb07f1368) at modify.c:669
#14 0xb6e47258 in merge_entry (op=<value optimized out>, e=0xb8c58264,
dup=0, query_uuid=0xb8c9b130)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:874
#15 0xb6e4effc in cache_entries (op=<value optimized out>,
query_uuid=<value optimized out>)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:2291
#16 0xb6e4f3ed in pcache_op_cleanup (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:2396
#17 0xb78102ee in slap_cleanup_play (op=<value optimized out>,
rs=<value
optimized out>) at /tmp/buildd/openldap-2.4.24/servers/slapd/result.c:539
#18 0xb7810fa9 in send_ldap_response (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/result.c:724
#19 0xb781209c in slap_send_ldap_result (op=0xb8c829d8, rs=0xb07f2ffc)
at /tmp/buildd/openldap-2.4.24/servers/slapd/result.c:851
#20 0xb6e6a987 in ldap_back_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/back-ldap/search.c:574
#21 0xb787802b in overlay_op_walk (op=0xb8c829d8, rs=0xb07f2ffc,
which=op_search, oi=0xb8c10fe0, on=0xb8c10260)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/backover.c:669
#22 0xb7878d58 in over_op_func (op=<value optimized out>, rs=<value
optimized out>, which=op_search)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/backover.c:721
#23 0xb7800b54 in fe_op_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/search.c:372
#24 0xb78014e7 in do_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/search.c:217
#25 0xb77fd994 in connection_operation (ctx=0xb07f31dc,
arg_v=0xb8c829d8) at
/tmp/buildd/openldap-2.4.24/servers/slapd/connection.c:1113
#26 0xb77fe492 in connection_read_thread (ctx=0xb07f31dc, argv=0x28) at
/tmp/buildd/openldap-2.4.24/servers/slapd/connection.c:1249
#27 0xb77634f4 in ?? () from /usr/lib/libldap_r-2.4.so.2
#28 0xb73af96e in start_thread (arg=0xb07f3b70) at pthread_create.c:300
#29 0xb731da4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb)


she gdb shell is still open, let me know if I can provide any further
information.

On 03/24/2011 01:34 PM, Quanah Gibson-Mount wrote:
> --On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com
> wrote:
>
>> Full_Name: Tyler Gates
>> Version: 2.4.21
>> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (24.106.216.17)
>>
>>
>> I have a caching proxy server (2.4.21) in front of two multi-masters.
>> After about 1 to 3 days it will segfault only when caching is enabled.
>> Log files are normally pretty useless but are normally in the operation
>> of removing stale queries:
>
> Bug reports with 2.4.21 are not being pursued.  Please use a current
> release (2.4.24) and verify whether or not the issue still exists there.
>
> --Quanah
>
>
> -- 
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration

-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or diss

Message of length 5211 truncated


Followup 4

Download message
Date: Sun, 3 Apr 2011 17:17:40 -0400 (EDT)
From: Tyler Gates <tjgates@castlebranch.com>
To: Quanah Gibson-Mount <quanah@zimbra.com>
Cc: openldap-its@openldap.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
------=_Part_481677_474224743.1301865460773
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Just checking in.. Is there anything I can do to help? If it helps, running
slapd in debug mode wouldn't crash on me after over 2 1/2 weeks of running.
Maybe a race condition among all the different threads? 


-- 

Tyler Gates 
Systems Administrator 
Castle Branch Inc. 
910-815-3880 ext 7230 
tjgates@castlebranch.com 

This e-mail message, including any attachments, may contain private, 
confidential, and privileged information for the restricted use of the 
intended recipient(s). If you are not the intended recipient(s), you 
may NOT use, disclose, copy, or disseminate this information. Please 
notify the sender by return e-mail of this misdirected correspondence 
and destroy all copies of the original message including all attachments. 
Your cooperation is appreciated. 

------=_Part_481677_474224743.1301865460773
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html><head><style type='text/css'>p { margin: 0;
}</style></head><body><div style='font-family: Arial;
font-size: 12pt; color: #000000'>Just checking in.. Is there anything I can
do to help? If it helps, running slapd in debug mode wouldn't crash on me after
over 2 1/2 weeks of running. Maybe a race condition among all the different
threads?<br><span><br><br>-- <br><br>Tyler
Gates<br>Systems Administrator<br>Castle Branch
Inc.<br>910-815-3880 ext
7230<br>tjgates@castlebranch.com<br><br>This e-mail message,
including any attachments, may contain private,<br>confidential, and
privileged information for the restricted use of the<br>intended
recipient(s). If you are not the intended recipient(s), you<br>may NOT
use, disclose, copy, or disseminate this information. Please<br>notify the
sender by return e-mail of this misdirected correspondence<br>and destroy
all copies of the original message including all attachments.<br>Your
cooperation is appreciated.<br></span></div></body></
 html>
------=_Part_481677_474224743.1301865460773--



Followup 5

Download message
Date: Wed, 08 Jun 2011 08:40:26 -0400
From: Tyler Gates <tjgates@castlebranch.com>
To: openldap-its@OpenLDAP.org
CC: Howard Chu <hyc@symas.com>
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
After applying the patches for ITS #6848 and ITS #6898 I still get crashes.
Here's the latest:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xae1edb70 (LWP 6121)]
0xb76e2430 in __kernel_vsyscall ()
(gdb)
(gdb)
(gdb) bt
#0  0xb76e2430 in __kernel_vsyscall ()
#1  0xb71b4651 in *__GI_raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb71b7a82 in *__GI_abort () at abort.c:92
#3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
detected *** %s: %s: 0x%s ***\n") at
../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb71f5591 in malloc_printerr (action=<value optimized out>, str=0x6
<Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
#5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
optimized out>) at malloc.c:4794
#6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
#7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
#8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
#9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
#10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
/tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
#11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
/tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
#12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
#13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at modify.c:662
#14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
query_uuid=<value optimized out>)
    at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
#15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
/tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
#16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
#17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
#18 0xb7257a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb) continue
Continuing.
[Thread 0xaddecb70 (LWP 6122) exited]
[Thread 0xae1edb70 (LWP 6121) exited]
[Thread 0xae7f0b70 (LWP 6120) exited]
[Thread 0xaedf3b70 (LWP 6119) exited]
[Thread 0xaf2f5b70 (LWP 6118) exited]
[Thread 0xaf7f7b70 (LWP 6117) exited]
[Thread 0xafbf8b70 (LWP 6116) exited]
[Thread 0xafff9b70 (LWP 6115) exited]
[Thread 0xb04fbb70 (LWP 6114) exited]
[Thread 0xb08fcb70 (LWP 6113) exited]
[Thread 0xb582ab70 (LWP 5868) exited]
[Thread 0xb5320b70 (LWP 5869) exited]
[Thread 0xb4e1eb70 (LWP 5870) exited]
[Thread 0xb4a1db70 (LWP 5871) exited]
[Thread 0xb461cb70 (LWP 5872) exited]
[Thread 0xb421bb70 (LWP 5873) exited]
[Thread 0xb5c2bb70 (LWP 5867) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
(gdb)


-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or disseminate this information. Please
notify the sender by return e-mail of this misdirected correspondence
and destroy all copies of the original message including all attachments.
Your cooperation is appreciated.



Followup 6

Download message
Date: Wed, 08 Jun 2011 11:26:17 -0700
From: Howard Chu <hyc@symas.com>
To: Tyler Gates <tjgates@castlebranch.com>
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
Tyler Gates wrote:
> After applying the patches for ITS #6848 and ITS #6898 I still get crashes.
> Here's the latest:

Looks like a double-free. Can you reproduce this consistently? If so, what are 
the steps? can you run slapd under valgrind and reproduce this?
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0xae1edb70 (LWP 6121)]
> 0xb76e2430 in __kernel_vsyscall ()
> (gdb)
> (gdb)
> (gdb) bt
> #0  0xb76e2430 in __kernel_vsyscall ()
> #1  0xb71b4651 in *__GI_raise (sig=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
> detected *** %s: %s: 0x%s ***\n") at
> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> #4  0xb71f5591 in malloc_printerr (action=<value optimized out>,
str=0x6
> <Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
> #5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
> optimized out>) at malloc.c:4794
> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
> #8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
> #9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at modify.c:662
> #14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
> query_uuid=<value optimized out>)
>      at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
> #17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
> #18 0xb7257a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
> (gdb) continue
> Continuing.
> [Thread 0xaddecb70 (LWP 6122) exited]
> [Thread 0xae1edb70 (LWP 6121) exited]
> [Thread 0xae7f0b70 (LWP 6120) exited]
> [Thread 0xaedf3b70 (LWP 6119) exited]
> [Thread 0xaf2f5b70 (LWP 6118) exited]
> [Thread 0xaf7f7b70 (LWP 6117) exited]
> [Thread 0xafbf8b70 (LWP 6116) exited]
> [Thread 0xafff9b70 (LWP 6115) exited]
> [Thread 0xb04fbb70 (LWP 6114) exited]
> [Thread 0xb08fcb70 (LWP 6113) exited]
> [Thread 0xb582ab70 (LWP 5868) exited]
> [Thread 0xb5320b70 (LWP 5869) exited]
> [Thread 0xb4e1eb70 (LWP 5870) exited]
> [Thread 0xb4a1db70 (LWP 5871) exited]
> [Thread 0xb461cb70 (LWP 5872) exited]
> [Thread 0xb421bb70 (LWP 5873) exited]
> [Thread 0xb5c2bb70 (LWP 5867) exited]
>
> Program terminated with signal SIGABRT, Aborted.
> The program no longer exists.
> (gdb)
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 7

Download message
Date: Wed, 08 Jun 2011 14:28:02 -0700
From: Howard Chu <hyc@symas.com>
To: Tyler Gates <tjgates@castlebranch.com>
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
Howard Chu wrote:
> Tyler Gates wrote:
>> After applying the patches for ITS #6848 and ITS #6898 I still get
crashes.
>> Here's the latest:

??? neither #6848 nor #6898 are relevant to pcache.

I think you should try the RE24 branch at this point, as a number of other 
pcache-related fixes have been merged there.
>
> Looks like a double-free. Can you reproduce this consistently? If so, what
are
> the steps? can you run slapd under valgrind and reproduce this?
>>
>> Program received signal SIGABRT, Aborted.
>> [Switching to Thread 0xae1edb70 (LWP 6121)]
>> 0xb76e2430 in __kernel_vsyscall ()
>> (gdb)
>> (gdb)
>> (gdb) bt
>> #0  0xb76e2430 in __kernel_vsyscall ()
>> #1  0xb71b4651 in *__GI_raise (sig=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
>> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
>> detected *** %s: %s: 0x%s ***\n") at
>> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
>> #4  0xb71f5591 in malloc_printerr (action=<value optimized out>,
str=0x6
>> <Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
>> #5  0xb71f6de8 in _int_free (av=<value optimized out>,
p=<value
>> optimized out>) at malloc.c:4794
>> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
>> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
>> #8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
>> #9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
>> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
>> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
>> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
>> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
>> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at
modify.c:662
>> #14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
>> query_uuid=<value optimized out>)
>>       at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
>> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
>> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
>> #17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
>> #18 0xb7257a4e in clone () at
../sysdeps/unix/sysv/linux/i386/clone.S:130
>> (gdb) continue
>> Continuing.
>> [Thread 0xaddecb70 (LWP 6122) exited]
>> [Thread 0xae1edb70 (LWP 6121) exited]
>> [Thread 0xae7f0b70 (LWP 6120) exited]
>> [Thread 0xaedf3b70 (LWP 6119) exited]
>> [Thread 0xaf2f5b70 (LWP 6118) exited]
>> [Thread 0xaf7f7b70 (LWP 6117) exited]
>> [Thread 0xafbf8b70 (LWP 6116) exited]
>> [Thread 0xafff9b70 (LWP 6115) exited]
>> [Thread 0xb04fbb70 (LWP 6114) exited]
>> [Thread 0xb08fcb70 (LWP 6113) exited]
>> [Thread 0xb582ab70 (LWP 5868) exited]
>> [Thread 0xb5320b70 (LWP 5869) exited]
>> [Thread 0xb4e1eb70 (LWP 5870) exited]
>> [Thread 0xb4a1db70 (LWP 5871) exited]
>> [Thread 0xb461cb70 (LWP 5872) exited]
>> [Thread 0xb421bb70 (LWP 5873) exited]
>> [Thread 0xb5c2bb70 (LWP 5867) exited]
>>
>> Program terminated with signal SIGABRT, Aborted.
>> The program no longer exists.
>> (gdb)
>>
>>
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 8

Download message
Date: Wed, 8 Jun 2011 18:27:28 -0400 (EDT)
From: Tyler Gates <tjgates@castlebranch.com>
To: Howard Chu <hyc@symas.com>
Cc: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
----- "Howard Chu" <hyc@symas.com> wrote:

> Howard Chu wrote:
> > Tyler Gates wrote:
> >> After applying the patches for ITS #6848 and ITS #6898 I still get
> crashes.
> >> Here's the latest:
> 
> ??? neither #6848 nor #6898 are relevant to pcache.

I figured probably not but thought it may be important to know in terms as this
is not just a straight 2.4.25 build...

> 
> I think you should try the RE24 branch at this point, as a number of
> other 
> pcache-related fixes have been merged there.

Are you referring to OPENLDAP_REL_ENG_2_4 in the git tree?
I'm not sure if I'll have the time (work) to tackle that but I'll see what I can
do.

> >
> > Looks like a double-free. Can you reproduce this consistently? If
> so, what are
> > the steps? can you run slapd under valgrind and reproduce this?

I can reproduce it consistently in terms that it crashes reliably after 2 to 4
days with 1 to 2 Linux hosts connected via pam-ldap account sessions. Other than
that I've not been able to find a way to make it crash on demand. I do know it
won't crash if it is run in the foreground with -d -1.

I'm running it now under valgrind, any options you are looking for in
particular? I'll report back the next time it crashes and/or I can try RE24.


> >>
> >> Program received signal SIGABRT, Aborted.
> >> [Switching to Thread 0xae1edb70 (LWP 6121)]
> >> 0xb76e2430 in __kernel_vsyscall ()
> >> (gdb)
> >> (gdb)
> >> (gdb) bt
> >> #0  0xb76e2430 in __kernel_vsyscall ()
> >> #1  0xb71b4651 in *__GI_raise (sig=6) at
> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
> >> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "***
> glibc
> >> detected *** %s: %s: 0x%s ***\n") at
> >> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> >> #4  0xb71f5591 in malloc_printerr (action=<value optimized
out>,
> str=0x6
> >> <Address 0x6 out of bounds>, ptr=0xb9584e80) at
malloc.c:6266
> >> #5  0xb71f6de8 in _int_free (av=<value optimized out>,
p=<value
> >> optimized out>) at malloc.c:4794
> >> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at
> malloc.c:3738
> >> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
> >> #8  0xb768acaf in ber_bvarray_free_x () from
> /usr/lib/liblber-2.4.so.2
> >> #9  0xb768acf5 in ber_bvarray_free () from
> /usr/lib/liblber-2.4.so.2
> >> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
> >> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
> >> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
> >> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at
> cache.c:1238
> >> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at
> modify.c:662
> >> #14 0xb6d52e81 in remove_query_data (op=<value optimized
out>,
> >> query_uuid=<value optimized out>)
> >>       at
> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
> >> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc,
> arg=0xb955e018) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
> >> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
> >> #17 0xb72e996e in start_thread (arg=0xae1edb70) at
> pthread_create.c:300
> >> #18 0xb7257a4e in clone () at
> ../sysdeps/unix/sysv/linux/i386/clone.S:130
> >> (gdb) continue
> >> Continuing.
> >> [Thread 0xaddecb70 (LWP 6122) exited]
> >> [Thread 0xae1edb70 (LWP 6121) exited]
> >> [Thread 0xae7f0b70 (LWP 6120) exited]
> >> [Thread 0xaedf3b70 (LWP 6119) exited]
> >> [Thread 0xaf2f5b70 (LWP 6118) exited]
> >> [Thread 0xaf7f7b70 (LWP 6117) exited]
> >> [Thread 0xafbf8b70 (LWP 6116) exited]
> >> [Thread 0xafff9b70 (LWP 6115) exited]
> >> [Thread 0xb04fbb70 (LWP 6114) exited]
> >> [Thread 0xb08fcb70 (LWP 6113) exited]
> >> [Thread 0xb582ab70 (LWP 5868) exited]
> >> [Thread 0xb5320b70 (LWP 5869) exited]
> >> [Thread 0xb4e1eb70 (LWP 5870) exited]
> >> [Thread 0xb4a1db70 (LWP 5871) exited]
> >> [Thread 0xb461cb70 (LWP 5872) exited]
> >> [Thread 0xb421bb70 (LWP 5873) exited]
> >> [Thread 0xb5c2bb70 (LWP 5867) exited]
> >>
> >> Program terminated with signal SIGABRT, Aborted.
> >> The program no longer exists.
> >> (gdb)
> >>
> >>
> >
> >
> 
> 
> -- 
>    -- Howard Chu

Message of length 5189 truncated


Followup 9

Download message
Date: Thu, 16 Jun 2011 12:27:18 -0400
From: Tyler Gates <tjgates@castlebranch.com>
To: openldap-its@OpenLDAP.org
CC: Howard Chu <hyc@symas.com>
Subject: Re: (ITS#6878) ppcache segfault with tavl_delete
This is a multi-part message in MIME format.
--------------050208080705010102070004
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

I can't get this to crash under valgrind, just like it wont crash when
run in debug mode.

There were a few nasty messages however so I've attached the output from
it thus far.


-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or disseminate this information. Please
notify the sender by return e-mail of this misdirected correspondence
and destroy all copies of the original message including all attachments.
Your cooperation is appreciated.


--------------050208080705010102070004
Content-Type: text/plain;
 name="slapd.valgrind.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="slapd.valgrind.txt"

==25268== Memcheck, a memory error detector
==25268== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==25268== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
copyright info
==25268== Command: /usr/sbin/slapd -h ldap:///\ ldapi:///\ ldaps:/// -g openldap
-u openldap -F /etc/ldap/slapd.d
==25268== Parent PID: 25106
==25268== 
==25268== 
==25268== HEAP SUMMARY:
==25268==     in use at exit: 682,293 bytes in 14,370 blocks
==25268==   total heap usage: 47,421 allocs, 33,051 frees, 8,849,431 bytes
allocated
==25268== 
==25268== LEAK SUMMARY:
==25268==    definitely lost: 199 bytes in 6 blocks
==25268==    indirectly lost: 13,619 bytes in 450 blocks
==25268==      possibly lost: 32,944 bytes in 10 blocks
==25268==    still reachable: 635,531 bytes in 13,904 blocks
==25268==         suppressed: 0 bytes in 0 blocks
==25268== Rerun with --leak-check=full to see details of leaked memory
==25268== 
==25268== For counts of detected and suppressed errors, rerun with: -v
==25268== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 134 from 14)
==25273== Thread 4:
==25273== Conditional jump or move depends on uninitialised value(s)
==25273==    at 0x49A249F: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49A3310: __log_put (in /usr/lib/libdb-4.8.so)
==25273==    by 0x494E6F7: __db_addrem_log (in /usr/lib/libdb-4.8.so)
==25273==    by 0x495CEB6: __db_pitem (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48AEAB3: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48B0447: __bam_iitem (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48A9889: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x4955D0D: __dbc_iput (in /usr/lib/libdb-4.8.so)
==25273==    by 0x495817C: __dbc_put (in /usr/lib/libdb-4.8.so)
==25273==    by 0x495EBE7: __dbc_put_pp (in /usr/lib/libdb-4.8.so)
==25273==    by 0x5493825: hdb_idl_insert_key (idl.c:876)
==25273==    by 0x548A5E8: hdb_key_change (key.c:93)
==25273== 
==25273== Syscall param pwrite64(buf) points to uninitialised byte(s)
==25273==    at 0x4C1AC0B: pwrite64 (pwrite64.c:54)
==25273==    by 0x49B9AB7: __os_io (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49A11DA: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49A1E7E: __log_flush_int (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49A376C: __log_put (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49C8AEA: __txn_regop_log (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49C5784: __txn_commit (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49C5850: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x54820B7: hdb_modify (modify.c:670)
==25273==    by 0x54B0257: merge_entry (pcache.c:874)
==25273==    by 0x54B7FFB: cache_entries (pcache.c:2291)
==25273==    by 0x54B83EC: pcache_op_cleanup (pcache.c:2396)
==25273==  Address 0x6e254b4 is not stack'd, malloc'd or (recently) free'd
==25273== 
==25273== Thread 3:
==25273== Conditional jump or move depends on uninitialised value(s)
==25273==    at 0x49A249F: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49A3310: __log_put (in /usr/lib/libdb-4.8.so)
==25273==    by 0x494E6F7: __db_addrem_log (in /usr/lib/libdb-4.8.so)
==25273==    by 0x495CBE0: __db_ditem (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48AB7EE: __bam_ditem (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48AA39D: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x48AACFD: ??? (in /usr/lib/libdb-4.8.so)
==25273==    by 0x49546C6: __dbc_close (in /usr/lib/libdb-4.8.so)
==25273==    by 0x495FC62: __dbc_close_pp (in /usr/lib/libdb-4.8.so)
==25273==    by 0x5492ED7: hdb_idl_delete_key (idl.c:1034)
==25273==    by 0x548A533: hdb_key_change (key.c:97)
==25273==    by 0x5489AD4: indexer (index.c:214)
==25273== 
==25273== Warning: invalid file descriptor -1 in syscall write()
==25273== Thread 4:
==25273== Invalid read of size 4
==25273==    at 0x486C562: ??? (in /usr/lib/libldap_r-2.4.so.2.7.0)
==25273==    by 0x486AEBE: ??? (in /usr/lib/libldap_r-2.4.so.2.7.0)
==25273==  

Message of length 47126 truncated

Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org