Issue 6878 - slapo-ppcache segfault with tavl_delete
Summary: slapo-ppcache segfault with tavl_delete
Status: VERIFIED SUSPENDED
Alias: None
Product: OpenLDAP
Classification: Unclassified
Component: overlays (show other issues)
Version: 2.4.21
Hardware: All All
: --- normal
Target Milestone: 2.6.0
Assignee: OpenLDAP project
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-03-24 16:28 UTC by tgates81@gmail.com
Modified: 2021-06-24 15:54 UTC (History)
0 users

See Also:


Attachments
slapd.valgrind.txt (44.91 KB, text/plain)
2011-06-16 16:27 UTC, tgates81@gmail.com
Details

Note You need to log in before you can comment on or make changes to this issue.
Description tgates81@gmail.com 2011-03-24 16:28:18 UTC
Full_Name: Tyler Gates
Version: 2.4.21
OS: Ubuntu 10.04.2 LTS/CentOS 5.4
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (24.106.216.17)


I have a caching proxy server (2.4.21) in front of two multi-masters. After
about 1 to 3 days it will segfault only when caching is enabled. Log files are
normally pretty useless but are normally in the operation of removing stale
queries: 

Mar 24 09:55:36 directory-proxy2 slapd[5363]: REMOVING TEMP ATTR :
TEMPLATE=ba2c2b64-ea65-102f-96bd-2f354737277c
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STALE QUERY REMOVED, SIZE=0
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STORED QUERIES = 22
Mar 24 09:55:36 directory-proxy2 slapd[5363]: STALE QUERY REMOVED, CACHE =1
entries
Mar 24 09:55:36 directory-proxy2 slapd[5363]: Lock CR index = 0xb8848c00

I've attached to the running process using gdb and got the following:
**************************************************************************
Program received signal SIGPIPE, Broken pipe.

Program received signal SIGPIPE, Broken pipe.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb61b4b70 (LWP 5365)]
tavl_delete (root=0x0, data=0xb890bac0, fcmp=0xb6b64910 <pcache_query_cmp>) at
/build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c:200
200     /build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c: No such file or
directory.
        in /build/buildd/openldap-2.4.21/libraries/liblutil/tavl.c
(gdb)
Continuing.
[Thread 0xb1effb70 (LWP 7113) exited]
[Thread 0xb2d9cb70 (LWP 10996) exited]
[Thread 0xb3a9fb70 (LWP 9738) exited]
[Thread 0xb47a2b70 (LWP 8550) exited]
[Thread 0xb65b5b70 (LWP 5364) exited]
[Thread 0xb61b4b70 (LWP 5365) exited]
[Thread 0xb5cb2b70 (LWP 5673) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb) bt
No stack.
**********************************************************************

As you can see I cannot get a back trace, but I still have the gdb shell running
so I run any other commands if needed.
I've run version 2.4.18 on a CentOS machine with the same setup and I get
segfaults only when pcache is enabled although I've never run it under gdb.
Comment 1 Quanah Gibson-Mount 2011-03-24 17:34:25 UTC
--On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com wrote:

> Full_Name: Tyler Gates
> Version: 2.4.21
> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (24.106.216.17)
>
>
> I have a caching proxy server (2.4.21) in front of two multi-masters.
> After about 1 to 3 days it will segfault only when caching is enabled.
> Log files are normally pretty useless but are normally in the operation
> of removing stale queries:

Bug reports with 2.4.21 are not being pursued.  Please use a current 
release (2.4.24) and verify whether or not the issue still exists there.

--Quanah


--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Comment 2 tgates81@gmail.com 2011-03-27 18:58:38 UTC
I've installed version 2.4.24 and been running it for a few days now. I'll report back in a few days on it's outcome.


On Mar 24, 2011, at 1:34 PM, Quanah Gibson-Mount <quanah@zimbra.com> wrote:

> --On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com wrote:
> 
>> Full_Name: Tyler Gates
>> Version: 2.4.21
>> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (24.106.216.17)
>> 
>> 
>> I have a caching proxy server (2.4.21) in front of two multi-masters.
>> After about 1 to 3 days it will segfault only when caching is enabled.
>> Log files are normally pretty useless but are normally in the operation
>> of removing stale queries:
> 
> Bug reports with 2.4.21 are not being pursued.  Please use a current release (2.4.24) and verify whether or not the issue still exists there.
> 
> --Quanah
> 
> 
> --
> 
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration

Comment 3 tgates81@gmail.com 2011-03-30 15:11:04 UTC
Just crashed:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb07f3b70 (LWP 24359)]
0xb77a7430 in __kernel_vsyscall ()
(gdb)
(gdb) bt
#0  0xb77a7430 in __kernel_vsyscall ()
#1  0xb727a651 in *__GI_raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb727da82 in *__GI_abort () at abort.c:92
#3  0xb72b149d in __libc_message (do_abort=2, fmt=0xb7385f98 "*** glibc
detected *** %s: %s: 0x%s ***\n") at
../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb72bb591 in malloc_printerr (action=<value optimized out>, str=0x6
<Address 0x6 out of bounds>, ptr=0xb8d93a40) at malloc.c:6266
#5  0xb72bcde8 in _int_free (av=<value optimized out>, p=<value
optimized out>) at malloc.c:4794
#6  0xb72bfecd in *__GI___libc_free (mem=0xb8d93a40) at malloc.c:3738
#7  0xb7750c20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
#8  0xb7750caf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
#9  0xb7750cf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
#10 0xb7808f3d in attr_clean (a=0xb688ff64) at
/tmp/buildd/openldap-2.4.24/servers/slapd/attr.c:146
#11 0xb7808fdb in attrs_free (a=0xb688ff64) at
/tmp/buildd/openldap-2.4.24/servers/slapd/attr.c:196
#12 0xb6e2db0f in hdb_cache_modify (bdb=0xb8c104b0, e=0xb8c57b5c,
newAttrs=0xb688ed64, txn=0xb8d93ea8, lock=0xb07f11f0) at cache.c:1226
#13 0xb6e1901c in hdb_modify (op=0xb07f152c, rs=0xb07f1368) at modify.c:669
#14 0xb6e47258 in merge_entry (op=<value optimized out>, e=0xb8c58264,
dup=0, query_uuid=0xb8c9b130)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:874
#15 0xb6e4effc in cache_entries (op=<value optimized out>,
query_uuid=<value optimized out>)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:2291
#16 0xb6e4f3ed in pcache_op_cleanup (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/overlays/pcache.c:2396
#17 0xb78102ee in slap_cleanup_play (op=<value optimized out>, rs=<value
optimized out>) at /tmp/buildd/openldap-2.4.24/servers/slapd/result.c:539
#18 0xb7810fa9 in send_ldap_response (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/result.c:724
#19 0xb781209c in slap_send_ldap_result (op=0xb8c829d8, rs=0xb07f2ffc)
at /tmp/buildd/openldap-2.4.24/servers/slapd/result.c:851
#20 0xb6e6a987 in ldap_back_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/back-ldap/search.c:574
#21 0xb787802b in overlay_op_walk (op=0xb8c829d8, rs=0xb07f2ffc,
which=op_search, oi=0xb8c10fe0, on=0xb8c10260)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/backover.c:669
#22 0xb7878d58 in over_op_func (op=<value optimized out>, rs=<value
optimized out>, which=op_search)
    at /tmp/buildd/openldap-2.4.24/servers/slapd/backover.c:721
#23 0xb7800b54 in fe_op_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/search.c:372
#24 0xb78014e7 in do_search (op=0xb8c829d8, rs=0xb07f2ffc) at
/tmp/buildd/openldap-2.4.24/servers/slapd/search.c:217
#25 0xb77fd994 in connection_operation (ctx=0xb07f31dc,
arg_v=0xb8c829d8) at
/tmp/buildd/openldap-2.4.24/servers/slapd/connection.c:1113
#26 0xb77fe492 in connection_read_thread (ctx=0xb07f31dc, argv=0x28) at
/tmp/buildd/openldap-2.4.24/servers/slapd/connection.c:1249
#27 0xb77634f4 in ?? () from /usr/lib/libldap_r-2.4.so.2
#28 0xb73af96e in start_thread (arg=0xb07f3b70) at pthread_create.c:300
#29 0xb731da4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb)


she gdb shell is still open, let me know if I can provide any further
information.

On 03/24/2011 01:34 PM, Quanah Gibson-Mount wrote:
> --On Thursday, March 24, 2011 4:28 PM +0000 tjgates@castlebranch.com
> wrote:
>
>> Full_Name: Tyler Gates
>> Version: 2.4.21
>> OS: Ubuntu 10.04.2 LTS/CentOS 5.4
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (24.106.216.17)
>>
>>
>> I have a caching proxy server (2.4.21) in front of two multi-masters.
>> After about 1 to 3 days it will segfault only when caching is enabled.
>> Log files are normally pretty useless but are normally in the operation
>> of removing stale queries:
>
> Bug reports with 2.4.21 are not being pursued.  Please use a current
> release (2.4.24) and verify whether or not the issue still exists there.
>
> --Quanah
>
>
> -- 
>
> Quanah Gibson-Mount
> Sr. Member of Technical Staff
> Zimbra, Inc
> A Division of VMware, Inc.
> --------------------
> Zimbra ::  the leader in open source messaging and collaboration

-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or disseminate this information. Please
notify the sender by return e-mail of this misdirected correspondence
and destroy all copies of the original message including all attachments.
Your cooperation is appreciated.

Comment 4 tgates81@gmail.com 2011-04-03 21:17:40 UTC
Just checking in.. Is there anything I can do to help? If it helps, running slapd in debug mode wouldn't crash on me after over 2 1/2 weeks of running. Maybe a race condition among all the different threads? 


-- 

Tyler Gates 
Systems Administrator 
Castle Branch Inc. 
910-815-3880 ext 7230 
tjgates@castlebranch.com 

This e-mail message, including any attachments, may contain private, 
confidential, and privileged information for the restricted use of the 
intended recipient(s). If you are not the intended recipient(s), you 
may NOT use, disclose, copy, or disseminate this information. Please 
notify the sender by return e-mail of this misdirected correspondence 
and destroy all copies of the original message including all attachments. 
Your cooperation is appreciated. 
Comment 5 tgates81@gmail.com 2011-06-08 12:40:26 UTC
After applying the patches for ITS #6848 and ITS #6898 I still get crashes.
Here's the latest:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xae1edb70 (LWP 6121)]
0xb76e2430 in __kernel_vsyscall ()
(gdb)
(gdb)
(gdb) bt
#0  0xb76e2430 in __kernel_vsyscall ()
#1  0xb71b4651 in *__GI_raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2  0xb71b7a82 in *__GI_abort () at abort.c:92
#3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
detected *** %s: %s: 0x%s ***\n") at
../sysdeps/unix/sysv/linux/libc_fatal.c:189
#4  0xb71f5591 in malloc_printerr (action=<value optimized out>, str=0x6
<Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
#5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
optimized out>) at malloc.c:4794
#6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
#7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
#8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
#9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
#10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
/tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
#11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
/tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
#12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
#13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at modify.c:662
#14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
query_uuid=<value optimized out>)
    at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
#15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
/tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
#16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
#17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
#18 0xb7257a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
(gdb) continue
Continuing.
[Thread 0xaddecb70 (LWP 6122) exited]
[Thread 0xae1edb70 (LWP 6121) exited]
[Thread 0xae7f0b70 (LWP 6120) exited]
[Thread 0xaedf3b70 (LWP 6119) exited]
[Thread 0xaf2f5b70 (LWP 6118) exited]
[Thread 0xaf7f7b70 (LWP 6117) exited]
[Thread 0xafbf8b70 (LWP 6116) exited]
[Thread 0xafff9b70 (LWP 6115) exited]
[Thread 0xb04fbb70 (LWP 6114) exited]
[Thread 0xb08fcb70 (LWP 6113) exited]
[Thread 0xb582ab70 (LWP 5868) exited]
[Thread 0xb5320b70 (LWP 5869) exited]
[Thread 0xb4e1eb70 (LWP 5870) exited]
[Thread 0xb4a1db70 (LWP 5871) exited]
[Thread 0xb461cb70 (LWP 5872) exited]
[Thread 0xb421bb70 (LWP 5873) exited]
[Thread 0xb5c2bb70 (LWP 5867) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.
(gdb)


-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or disseminate this information. Please
notify the sender by return e-mail of this misdirected correspondence
and destroy all copies of the original message including all attachments.
Your cooperation is appreciated.

Comment 6 Howard Chu 2011-06-08 18:26:17 UTC
Tyler Gates wrote:
> After applying the patches for ITS #6848 and ITS #6898 I still get crashes.
> Here's the latest:

Looks like a double-free. Can you reproduce this consistently? If so, what are 
the steps? can you run slapd under valgrind and reproduce this?
>
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0xae1edb70 (LWP 6121)]
> 0xb76e2430 in __kernel_vsyscall ()
> (gdb)
> (gdb)
> (gdb) bt
> #0  0xb76e2430 in __kernel_vsyscall ()
> #1  0xb71b4651 in *__GI_raise (sig=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
> detected *** %s: %s: 0x%s ***\n") at
> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> #4  0xb71f5591 in malloc_printerr (action=<value optimized out>, str=0x6
> <Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
> #5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
> optimized out>) at malloc.c:4794
> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
> #8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
> #9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at modify.c:662
> #14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
> query_uuid=<value optimized out>)
>      at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
> #17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
> #18 0xb7257a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
> (gdb) continue
> Continuing.
> [Thread 0xaddecb70 (LWP 6122) exited]
> [Thread 0xae1edb70 (LWP 6121) exited]
> [Thread 0xae7f0b70 (LWP 6120) exited]
> [Thread 0xaedf3b70 (LWP 6119) exited]
> [Thread 0xaf2f5b70 (LWP 6118) exited]
> [Thread 0xaf7f7b70 (LWP 6117) exited]
> [Thread 0xafbf8b70 (LWP 6116) exited]
> [Thread 0xafff9b70 (LWP 6115) exited]
> [Thread 0xb04fbb70 (LWP 6114) exited]
> [Thread 0xb08fcb70 (LWP 6113) exited]
> [Thread 0xb582ab70 (LWP 5868) exited]
> [Thread 0xb5320b70 (LWP 5869) exited]
> [Thread 0xb4e1eb70 (LWP 5870) exited]
> [Thread 0xb4a1db70 (LWP 5871) exited]
> [Thread 0xb461cb70 (LWP 5872) exited]
> [Thread 0xb421bb70 (LWP 5873) exited]
> [Thread 0xb5c2bb70 (LWP 5867) exited]
>
> Program terminated with signal SIGABRT, Aborted.
> The program no longer exists.
> (gdb)
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 7 Howard Chu 2011-06-08 21:28:02 UTC
Howard Chu wrote:
> Tyler Gates wrote:
>> After applying the patches for ITS #6848 and ITS #6898 I still get crashes.
>> Here's the latest:

??? neither #6848 nor #6898 are relevant to pcache.

I think you should try the RE24 branch at this point, as a number of other 
pcache-related fixes have been merged there.
>
> Looks like a double-free. Can you reproduce this consistently? If so, what are
> the steps? can you run slapd under valgrind and reproduce this?
>>
>> Program received signal SIGABRT, Aborted.
>> [Switching to Thread 0xae1edb70 (LWP 6121)]
>> 0xb76e2430 in __kernel_vsyscall ()
>> (gdb)
>> (gdb)
>> (gdb) bt
>> #0  0xb76e2430 in __kernel_vsyscall ()
>> #1  0xb71b4651 in *__GI_raise (sig=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
>> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
>> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "*** glibc
>> detected *** %s: %s: 0x%s ***\n") at
>> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
>> #4  0xb71f5591 in malloc_printerr (action=<value optimized out>, str=0x6
>> <Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
>> #5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
>> optimized out>) at malloc.c:4794
>> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at malloc.c:3738
>> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
>> #8  0xb768acaf in ber_bvarray_free_x () from /usr/lib/liblber-2.4.so.2
>> #9  0xb768acf5 in ber_bvarray_free () from /usr/lib/liblber-2.4.so.2
>> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
>> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
>> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
>> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at cache.c:1238
>> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at modify.c:662
>> #14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
>> query_uuid=<value optimized out>)
>>       at /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
>> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc, arg=0xb955e018) at
>> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
>> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
>> #17 0xb72e996e in start_thread (arg=0xae1edb70) at pthread_create.c:300
>> #18 0xb7257a4e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
>> (gdb) continue
>> Continuing.
>> [Thread 0xaddecb70 (LWP 6122) exited]
>> [Thread 0xae1edb70 (LWP 6121) exited]
>> [Thread 0xae7f0b70 (LWP 6120) exited]
>> [Thread 0xaedf3b70 (LWP 6119) exited]
>> [Thread 0xaf2f5b70 (LWP 6118) exited]
>> [Thread 0xaf7f7b70 (LWP 6117) exited]
>> [Thread 0xafbf8b70 (LWP 6116) exited]
>> [Thread 0xafff9b70 (LWP 6115) exited]
>> [Thread 0xb04fbb70 (LWP 6114) exited]
>> [Thread 0xb08fcb70 (LWP 6113) exited]
>> [Thread 0xb582ab70 (LWP 5868) exited]
>> [Thread 0xb5320b70 (LWP 5869) exited]
>> [Thread 0xb4e1eb70 (LWP 5870) exited]
>> [Thread 0xb4a1db70 (LWP 5871) exited]
>> [Thread 0xb461cb70 (LWP 5872) exited]
>> [Thread 0xb421bb70 (LWP 5873) exited]
>> [Thread 0xb5c2bb70 (LWP 5867) exited]
>>
>> Program terminated with signal SIGABRT, Aborted.
>> The program no longer exists.
>> (gdb)
>>
>>
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 8 tgates81@gmail.com 2011-06-08 22:27:28 UTC
----- "Howard Chu" <hyc@symas.com> wrote:

> Howard Chu wrote:
> > Tyler Gates wrote:
> >> After applying the patches for ITS #6848 and ITS #6898 I still get
> crashes.
> >> Here's the latest:
> 
> ??? neither #6848 nor #6898 are relevant to pcache.

I figured probably not but thought it may be important to know in terms as this is not just a straight 2.4.25 build...

> 
> I think you should try the RE24 branch at this point, as a number of
> other 
> pcache-related fixes have been merged there.

Are you referring to OPENLDAP_REL_ENG_2_4 in the git tree?
I'm not sure if I'll have the time (work) to tackle that but I'll see what I can do.

> >
> > Looks like a double-free. Can you reproduce this consistently? If
> so, what are
> > the steps? can you run slapd under valgrind and reproduce this?

I can reproduce it consistently in terms that it crashes reliably after 2 to 4 days with 1 to 2 Linux hosts connected via pam-ldap account sessions. Other than that I've not been able to find a way to make it crash on demand. I do know it won't crash if it is run in the foreground with -d -1.

I'm running it now under valgrind, any options you are looking for in particular? I'll report back the next time it crashes and/or I can try RE24.


> >>
> >> Program received signal SIGABRT, Aborted.
> >> [Switching to Thread 0xae1edb70 (LWP 6121)]
> >> 0xb76e2430 in __kernel_vsyscall ()
> >> (gdb)
> >> (gdb)
> >> (gdb) bt
> >> #0  0xb76e2430 in __kernel_vsyscall ()
> >> #1  0xb71b4651 in *__GI_raise (sig=6) at
> >> ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> >> #2  0xb71b7a82 in *__GI_abort () at abort.c:92
> >> #3  0xb71eb49d in __libc_message (do_abort=2, fmt=0xb72bff98 "***
> glibc
> >> detected *** %s: %s: 0x%s ***\n") at
> >> ../sysdeps/unix/sysv/linux/libc_fatal.c:189
> >> #4  0xb71f5591 in malloc_printerr (action=<value optimized out>,
> str=0x6
> >> <Address 0x6 out of bounds>, ptr=0xb9584e80) at malloc.c:6266
> >> #5  0xb71f6de8 in _int_free (av=<value optimized out>, p=<value
> >> optimized out>) at malloc.c:4794
> >> #6  0xb71f9ecd in *__GI___libc_free (mem=0xb9584e80) at
> malloc.c:3738
> >> #7  0xb768ac20 in ber_memfree_x () from /usr/lib/liblber-2.4.so.2
> >> #8  0xb768acaf in ber_bvarray_free_x () from
> /usr/lib/liblber-2.4.so.2
> >> #9  0xb768acf5 in ber_bvarray_free () from
> /usr/lib/liblber-2.4.so.2
> >> #10 0xb7743e5d in attr_clean (a=0xb6dcc06c) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:148
> >> #11 0xb7743efb in attrs_free (a=0xb6dcc06c) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/attr.c:198
> >> #12 0xb6d84b7f in hdb_cache_modify (bdb=0xb9536e10, e=0xb94f1dac,
> >> newAttrs=0xb6dd6884, txn=0xb2176970, lock=0xae1ec930) at
> cache.c:1238
> >> #13 0xb6d7008c in hdb_modify (op=0xae1eccfc, rs=0xae1ecad0) at
> modify.c:662
> >> #14 0xb6d52e81 in remove_query_data (op=<value optimized out>,
> >> query_uuid=<value optimized out>)
> >>       at
> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:1829
> >> #15 0xb6d53d0b in consistency_check (ctx=0xae1ed1dc,
> arg=0xb955e018) at
> >> /tmp/buildd/openldap-2.4.25/servers/slapd/overlays/pcache.c:3520
> >> #16 0xb769d8b4 in ?? () from /usr/lib/libldap_r-2.4.so.2
> >> #17 0xb72e996e in start_thread (arg=0xae1edb70) at
> pthread_create.c:300
> >> #18 0xb7257a4e in clone () at
> ../sysdeps/unix/sysv/linux/i386/clone.S:130
> >> (gdb) continue
> >> Continuing.
> >> [Thread 0xaddecb70 (LWP 6122) exited]
> >> [Thread 0xae1edb70 (LWP 6121) exited]
> >> [Thread 0xae7f0b70 (LWP 6120) exited]
> >> [Thread 0xaedf3b70 (LWP 6119) exited]
> >> [Thread 0xaf2f5b70 (LWP 6118) exited]
> >> [Thread 0xaf7f7b70 (LWP 6117) exited]
> >> [Thread 0xafbf8b70 (LWP 6116) exited]
> >> [Thread 0xafff9b70 (LWP 6115) exited]
> >> [Thread 0xb04fbb70 (LWP 6114) exited]
> >> [Thread 0xb08fcb70 (LWP 6113) exited]
> >> [Thread 0xb582ab70 (LWP 5868) exited]
> >> [Thread 0xb5320b70 (LWP 5869) exited]
> >> [Thread 0xb4e1eb70 (LWP 5870) exited]
> >> [Thread 0xb4a1db70 (LWP 5871) exited]
> >> [Thread 0xb461cb70 (LWP 5872) exited]
> >> [Thread 0xb421bb70 (LWP 5873) exited]
> >> [Thread 0xb5c2bb70 (LWP 5867) exited]
> >>
> >> Program terminated with signal SIGABRT, Aborted.
> >> The program no longer exists.
> >> (gdb)
> >>
> >>
> >
> >
> 
> 
> -- 
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com
>    Director, Highland Sun     http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 9 tgates81@gmail.com 2011-06-16 16:27:18 UTC
I can't get this to crash under valgrind, just like it wont crash when
run in debug mode.

There were a few nasty messages however so I've attached the output from
it thus far.


-- 

Tyler Gates
Systems Administrator
Castle Branch Inc.
910-815-3880 ext 7230
tjgates@castlebranch.com

This e-mail message, including any attachments, may contain private,
confidential, and privileged information for the restricted use of the
intended recipient(s). If you are not the intended recipient(s), you
may NOT use, disclose, copy, or disseminate this information. Please
notify the sender by return e-mail of this misdirected correspondence
and destroy all copies of the original message including all attachments.
Your cooperation is appreciated.

Comment 10 Quanah Gibson-Mount 2017-04-08 00:06:39 UTC
moved from Incoming to Software Bugs
Comment 11 Quanah Gibson-Mount 2021-06-24 15:54:26 UTC
Likely resolved at this point.