Issue 6361 - Failed assertion in slapd when running on OpenWRT/glibc
Summary: Failed assertion in slapd when running on OpenWRT/glibc
Status: VERIFIED FIXED
Alias: None
Product: OpenLDAP
Classification: Unclassified
Component: slapd (show other issues)
Version: 2.4.16
Hardware: All All
: --- normal
Target Milestone: ---
Assignee: OpenLDAP project
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-11-04 02:13 UTC by mike@flyn.org
Modified: 2014-08-01 21:04 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description mike@flyn.org 2009-11-04 02:13:24 UTC
Full_Name: W. Michael Petullo
Version: 2.4.16
OS: OpenWRT/Linux/glibc
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (76.6.126.162)


I am running slapd on OpenWRT/MIPS32/Linux/glibc. I have found that after a
period of running, slapd terminates with:

slapd: daemon.c:983: slapd_set_read: Assertion
((((&slap_daemon.sd_actives)->__fds_bits)[(((s)) / (8 * sizeof (__fd_mask)))] &
((__fd_mask) 1 << (((s)) % (8 * sizeof (___fd_mask))))) != 0) failed.

$ ldd /usr/sbin/slapd 
	libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2 (0x2aad8000)
	liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2 (0x2ab20000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x2ab3c000)
	libdb-4.7.so => /usr/lib/libdb-4.7.so (0x2ab50000)
	libresolv.so.2 => /lib/libresolv.so.2 (0x2ac92000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aca6000)
	libc.so.6 => /lib/libc.so.6 (0x2ad3e000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x2ae98000)
	/lib/ld.so.1 (0x2aaa8000)

$ slapd -V
@(#) $OpenLDAP: slapd 2.4.16 (Nov  3 2009 08:04:35) $
Comment 1 Howard Chu 2009-11-04 03:40:44 UTC
mike@flyn.org wrote:
> Full_Name: W. Michael Petullo
> Version: 2.4.16
> OS: OpenWRT/Linux/glibc
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (76.6.126.162)
> 
> 
> I am running slapd on OpenWRT/MIPS32/Linux/glibc. I have found that after a
> period of running, slapd terminates with:

Please try to reproduce this with a current release (2.4.19).
> 
> slapd: daemon.c:983: slapd_set_read: Assertion
> ((((&slap_daemon.sd_actives)->__fds_bits)[(((s)) / (8 * sizeof (__fd_mask)))] &
> ((__fd_mask) 1 << (((s)) % (8 * sizeof (___fd_mask))))) != 0) failed.
> 
> $ ldd /usr/sbin/slapd 
> 	libldap_r-2.4.so.2 => /usr/lib/libldap_r-2.4.so.2 (0x2aad8000)
> 	liblber-2.4.so.2 => /usr/lib/liblber-2.4.so.2 (0x2ab20000)
> 	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x2ab3c000)
> 	libdb-4.7.so => /usr/lib/libdb-4.7.so (0x2ab50000)
> 	libresolv.so.2 => /lib/libresolv.so.2 (0x2ac92000)
> 	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2aca6000)
> 	libc.so.6 => /lib/libc.so.6 (0x2ad3e000)
> 	libpthread.so.0 => /lib/libpthread.so.0 (0x2ae98000)
> 	/lib/ld.so.1 (0x2aaa8000)
> 
> $ slapd -V
> @(#) $OpenLDAP: slapd 2.4.16 (Nov  3 2009 08:04:35) $
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 2 mike@flyn.org 2009-11-05 03:02:57 UTC
Yes, I have just been able to recreate this in 2.4.19. The line in
daemon.c at which the assertion occurs in this version is 994.

I have also seen:

ldap_write: want=22 error=Broken pipe
...
slapd: connection.c:786: connection_closing: Assertion 'c->c_struct_state
== SLAP_C_USED' failed

One things to note is that, in addition to being on MIPS32, my platform
has only 32MB of memory. I also have a swap file that is 64MB.

Comment 3 Howard Chu 2009-11-13 22:40:18 UTC
mike@flyn.org wrote:
> Yes, I have just been able to recreate this in 2.4.19. The line in
> daemon.c at which the assertion occurs in this version is 994.
> 
> I have also seen:
> 
> ldap_write: want=22 error=Broken pipe
> ...
> slapd: connection.c:786: connection_closing: Assertion 'c->c_struct_state
> == SLAP_C_USED' failed

Please provide stack traces for each of these assertions. In this case it
sounds like connection_closing() got called multiple times for the same
connection, after it had already been fully closed. A trace would help confirm
that.

> One things to note is that, in addition to being on MIPS32, my platform
> has only 32MB of memory. I also have a swap file that is 64MB.

In such an environment you won't have enough memory for more than 1 or 2
server threads. Have you already configured this?

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 4 Howard Chu 2009-11-13 22:49:14 UTC
changed notes
changed state Open to Test
moved from Incoming to Software Bugs
Comment 5 mike@flyn.org 2009-11-14 17:12:58 UTC
> Please provide stack traces for each of these assertions. In this case it
> sounds like connection_closing() got called multiple times for the same
> connection, after it had already been fully closed. A trace would help
> confirm
> that.

This is a little odd:

(gdb) run -d 255
Starting program: /usr/sbin/slapd -d 255
must compile with LDAP_DEBUG for debugging

*** "CRASH" ***

Program exited normally.
(gdb) ba
No stack.

I have been running a script, while ldapsearch -x; do sleep 10; done, that
eventually crashes slapd. But gdb provides little feedback, as noted
above.

>> One things to note is that, in addition to being on MIPS32, my platform
>> has only 32MB of memory. I also have a swap file that is 64MB.
>
> In such an environment you won't have enough memory for more than 1 or 2
> server threads. Have you already configured this?

I have just reconfigured slapd to use one thread and removed all indexes
from the configuration. So far, I have not seen a crash. I am no using two
clients to query slapd and have removed that sleep statements so that the
server is continuously queried. I've been running this for about one hour
and will report back if there are any negative results.

Comment 6 mike@flyn.org 2009-11-15 23:55:04 UTC
Followup 4 states that slapd did not initially crash when using "threads
1" and no indexes. In this case, slapd did eventually crash as before.

Comment 7 Howard Chu 2009-11-16 02:08:30 UTC
mike@flyn.org wrote:
> Followup 4 states that slapd did not initially crash when using "threads
> 1" and no indexes. In this case, slapd did eventually crash as before.

I have removed the two asserts you previously reported; please try the current
code in CVS HEAD.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Comment 8 Quanah Gibson-Mount 2009-11-17 17:08:14 UTC
changed notes
changed state Test to Release
Comment 9 mike@flyn.org 2009-11-19 00:34:13 UTC
CVS HEAD, including the two changes related to assertions (the change log references ITS#6361), may fix this issue. I have been running the new slapd for over 24 hours with no problems. This is much longer than slapd would run for me before the new code.
Comment 10 Howard Chu 2009-11-30 05:34:49 UTC
changed notes
changed state Release to Closed
Comment 11 OpenLDAP project 2014-08-01 21:04:26 UTC
asserts replaced in HEAD
asserts replaced in RE24