OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Development/6300
Full headers

From: bduncan@apple.com
Subject: Add kqueue support to slapd
Compose comment
Download message
State:
0 replies:
12 followups: 1 2 3 4 5 6 7 8 9 10 11 12

Major security issue: yes  no

Notes:

Notification:


Date: Tue, 22 Sep 2009 19:05:32 +0000
From: bduncan@apple.com
To: openldap-its@OpenLDAP.org
Subject: Add kqueue support to slapd
Full_Name: Bryan Duncan
Version: 2.4.16
OS: Mac OS X 10.6
URL: ftp://ftp.openldap.org/incoming/bryan-duncan-kqueue-090922.patch
Submission from: (NULL) (17.224.21.109)


Added support for using kqueue in slapd (for systems that support kqueue(2).

Followup 1

Download message
From: Bryan Duncan <bduncan@apple.com>
Subject: Re: (ITS#6300) Add kqueue support to slapd
Date: Tue, 22 Sep 2009 12:22:58 -0700
Cc: bduncan@apple.com
To: openldap-its@OpenLDAP.org
Subject should have been: "PATCH - Added kqueue support to slapd" to  
make clear this ITS tracks a patch that has been submitted.

-- Bryan



Followup 2

Download message
Date: Tue, 22 Sep 2009 13:02:16 -0700
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: bduncan@apple.com, openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd
--On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com wrote:

> Subject should have been: "PATCH - Added kqueue support to slapd" to
> make clear this ITS tracks a patch that has been submitted.

What OS(es) has this kqueue support been tested with?  For example, the 
last time I looked at kqueue on OSX 10.5, it was very marginal, and missing 
most of the important pieces, which meant it wouldn't work for OpenLDAP or 
Postfix, etc.

--Quanah



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 3

Download message
Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Tue, 22 Sep 2009 14:40:54 -0700
To: Quanah Gibson-Mount <quanah@zimbra.com>, openldap-its@openldap.org
On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:

> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com  
> wrote:
>
>> Subject should have been: "PATCH - Added kqueue support to slapd" to
>> make clear this ITS tracks a patch that has been submitted.
>
> What OS(es) has this kqueue support been tested with?  For example,  
> the last time I looked at kqueue on OSX 10.5, it was very marginal,  
> and missing most of the important pieces, which meant it wouldn't  
> work for OpenLDAP or Postfix, etc.

It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX- 
specific about the kqueue usage in the patch.  As far as kqueue  
support in OS X 10.5, from what I see, most everything is there.   
Certainly all of the kqueue features in this patch are supported in OS  
X 10.5.

-- Bryan Duncan



Followup 4

Download message
Date: Wed, 11 Nov 2009 09:04:35 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Bryan Duncan <bduncan@apple.com>, openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd
--On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan 
<bduncan@apple.com> wrote:


> It was tested on OS X 10.6.  Should work on all BSDs; nothing
> OSX-specific about the kqueue usage in the patch.  As far as kqueue
> support in OS X 10.5, from what I see, most everything is there.
> Certainly all of the kqueue features in this patch are supported in OS X
> 10.5.

Hi Bryan,

The URL to the patch you submitted with this ITS doesn't work.  Can you 
please put the patch somewhere accessible and let us know where it is?

Thanks,
Quanah



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 5

Download message
Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Wed, 11 Nov 2009 09:36:51 -0800
Cc: openldap-its@openldap.org
To: Quanah Gibson-Mount <quanah@zimbra.com>
On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:

> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan =
<bduncan@apple.com> wrote:
>=20
>=20
>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>> OSX-specific about the kqueue usage in the patch.  As far as kqueue
>> support in OS X 10.5, from what I see, most everything is there.
>> Certainly all of the kqueue features in this patch are supported in =
OS X
>> 10.5.
>=20
> Hi Bryan,
>=20
> The URL to the patch you submitted with this ITS doesn't work.  Can =
you please put the patch somewhere accessible and let us know where it =
is?
>=20
> Thanks,
> Quanah

Hi Quanah,

My apologies for the bad URL.

The new URL is: =
http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch
If you use a web browser to get it, there are extra steps: select the =
file "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl =
can pull it down directly with no extra steps.

-- Bryan=



Followup 6

Download message
Date: Wed, 11 Nov 2009 09:45:59 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Bryan Duncan <bduncan@apple.com>
cc: openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd
--On Wednesday, November 11, 2009 9:36 AM -0800 Bryan Duncan 
<bduncan@apple.com> wrote:

>
> On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:
>
>> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan
>> <bduncan@apple.com> wrote:
>>
>>
>>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>>> OSX-specific about the kqueue usage in the patch.  As far as kqueue
>>> support in OS X 10.5, from what I see, most everything is there.
>>> Certainly all of the kqueue features in this patch are supported in
OS X
>>> 10.5.
>>
>> Hi Bryan,
>>
>> The URL to the patch you submitted with this ITS doesn't work.  Can you
>> please put the patch somewhere accessible and let us know where it is?
>>
>> Thanks,
>> Quanah
>
> Hi Quanah,
>
> My apologies for the bad URL.
>
> The new URL is:
> http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch If you
> use a web browser to get it, there are extra steps: select the file
> "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl can
> pull it down directly with no extra steps.

Hi Bryan,

Thanks for the quick response.  Hopefully we can get this incorporated for 
OpenLDAP 2.5 series, and I'm interested in playing with it against 2.4 to 
see how things change when used with Mac.

Thanks!

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 7

Download message
Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Wed, 11 Nov 2009 11:29:42 -0800
Cc: openldap-its@openldap.org
To: Quanah Gibson-Mount <quanah@zimbra.com>
On Nov 11, 2009, at 9:45 AM, Quanah Gibson-Mount wrote:

> --On Wednesday, November 11, 2009 9:36 AM -0800 Bryan Duncan =
<bduncan@apple.com> wrote:
>=20
>>=20
>> On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:
>>=20
>>> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan
>>> <bduncan@apple.com> wrote:
>>>=20
>>>=20
>>>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>>>> OSX-specific about the kqueue usage in the patch.  As far as
kqueue
>>>> support in OS X 10.5, from what I see, most everything is
there.
>>>> Certainly all of the kqueue features in this patch are
supported in =
OS X
>>>> 10.5.
>>>=20
>>> Hi Bryan,
>>>=20
>>> The URL to the patch you submitted with this ITS doesn't work.  Can
=
you
>>> please put the patch somewhere accessible and let us know where it
=
is?
>>>=20
>>> Thanks,
>>> Quanah
>>=20
>> Hi Quanah,
>>=20
>> My apologies for the bad URL.
>>=20
>> The new URL is:
>> http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch If =
you
>> use a web browser to get it, there are extra steps: select the file
>> "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl
=
can
>> pull it down directly with no extra steps.
>=20
> Hi Bryan,
>=20
> Thanks for the quick response.  Hopefully we can get this incorporated =
for OpenLDAP 2.5 series, and I'm interested in playing with it against =
2.4 to see how things change when used with Mac.
>=20
> Thanks!
>=20
> --Quanah


Awesome!  I look forward to seeing it OpenLDAP.  If there's anything I =
can do to help, please let me know.

Thanks.=20
-- Bryan



Followup 8

Download message
Date: Sun, 22 Nov 2009 18:43:58 -0800
From: Howard Chu <hyc@symas.com>
To: bduncan@apple.com
CC: openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd
bduncan@apple.com wrote:
> On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:
>
>> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com
>> wrote:
>>
>>> Subject should have been: "PATCH - Added kqueue support to slapd"
to
>>> make clear this ITS tracks a patch that has been submitted.
>>
>> What OS(es) has this kqueue support been tested with?  For example,
>> the last time I looked at kqueue on OSX 10.5, it was very marginal,
>> and missing most of the important pieces, which meant it wouldn't
>> work for OpenLDAP or Postfix, etc.
>
> It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX-
> specific about the kqueue usage in the patch.  As far as kqueue
> support in OS X 10.5, from what I see, most everything is there.
> Certainly all of the kqueue features in this patch are supported in OS
> X 10.5.

To expand on this: there are a lot of comments around the web that kqueue was 
fairly broken at least up to 10.5. E.g. this:

http://lists.schmorp.de/pipermail/libev/2008q4/000523.html

Since we also use a pipe for the wake_sds[] in our event loop, if kqueue only 
supports sockets, then it cannot be used reliably in slapd.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 9

Download message
Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Thu, 3 Dec 2009 12:41:10 -0800
Cc: openldap-its@openldap.org
To: Howard Chu <hyc@symas.com>
On Nov 22, 2009, at 6:43 PM, Howard Chu wrote:

> bduncan@apple.com wrote:
>> On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:
>>=20
>>> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com
>>> wrote:
>>>=20
>>>> Subject should have been: "PATCH - Added kqueue support to
slapd" =
to
>>>> make clear this ITS tracks a patch that has been submitted.
>>>=20
>>> What OS(es) has this kqueue support been tested with?  For example,
>>> the last time I looked at kqueue on OSX 10.5, it was very marginal,
>>> and missing most of the important pieces, which meant it wouldn't
>>> work for OpenLDAP or Postfix, etc.
>>=20
>> It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX-
>> specific about the kqueue usage in the patch.  As far as kqueue
>> support in OS X 10.5, from what I see, most everything is there.
>> Certainly all of the kqueue features in this patch are supported in =
OS
>> X 10.5.
>=20
> To expand on this: there are a lot of comments around the web that =
kqueue was fairly broken at least up to 10.5. E.g. this:
>=20
> http://lists.schmorp.de/pipermail/libev/2008q4/000523.html

Perhaps I missed something in that thread, but the only specific =
complaint was regarding kqueue and stdin.  And it's true that OS X's =
kqueue didn't support EVFILT_{READ|WRITE} on *devices* (stdin, tty's, =
/dev/* ) until 10.6.  I don't believe OpenLDAP actually needs that =
feature, though.

>=20
> Since we also use a pipe for the wake_sds[] in our event loop, if =
kqueue only supports sockets, then it cannot be used reliably in slapd.

The OS X kqueue has *always* supported EVFILT_{READ|WRITE} on sockets, =
pipes, fifos, files.  Same for kqueue in the BSDs.  (FWIW, I believe the =
BSDs have always supported devices in kqueue.)

I think everything OpenLDAP needs from kqueue has been, and is, =
supported by OS X and the BSDs.  If there is a specific kqueue filter or =
fd type that OpenLDAP needs and isn't supported, please let me know.

Although I don't think it's necessary, you could add a test to the =
configure script to ensure kqueue supports the types of fd's OpenLDAP =
needs by doing something like this:

    int fd; /* or int fds[2]; */
    /* do something to create an fd here:  open(2), pipe(2), socket(2), =
etc. */
    struct kevent ev;
    EV_SET(&ev, fd, EVFILT_READ, EV_ADD, 0, 0, 0);
    int kq =3D kqueue();
    if (kq < 0) {
        ac_have_kqueue =3D false;
    } else {
        struct timespec to =3D {0,1};  /* anything so that kevent will =
return */
        int ret =3D kevent(kq, &ev, 1, &ev, 1, &to);
        close(kq);
        ac_have_kqueue =3D (ret >=3D 0);
    }

If the fd isn't supported 'ret' should be -1 and errno should be =
ENOTSUP.

-- Bryan Duncan



Followup 10

Download message
Date: Mon, 11 Sep 2017 15:45:09 -0700
From: Quanah Gibson-Mount <quanah@fast-mail.org>
To: bduncan@apple.com
cc: openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd
Hi Bryan,

Unfortunately this patch is no longer available with Apple's switch to 
iCloud.  Would you be able to upload it to the OpenLDAP FTP server?

Thanks,
Quanah



Followup 11

Download message
Date: Wed, 13 Sep 2017 13:06:18 -0700
From: Quanah Gibson-Mount <quanah@symas.com>
To: openldap-its@OpenLDAP.org, bduncan@apple.com
Subject: Re: (ITS#6300) Add kqueue support to slapd
Hi Bryan,

I managed to track down a copy of your patch, and uploaded it to the 
OpenLDAP FTP server as bryan-duncan-its6300.patch.  I will see about 
getting this updated for current OpenLDAP, for possible inclusion in 
OpenLDAP 2.5.

Regards,
Quanah

--On Monday, September 11, 2017 11:45 PM +0000 quanah@fast-mail.org wrote:

> Hi Bryan,
>
> Unfortunately this patch is no longer available with Apple's switch to
> iCloud.  Would you be able to upload it to the OpenLDAP FTP server?
>
> Thanks,
> Quanah
>
>
>



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>




Followup 12

Download message
Date: Tue, 24 Apr 2018 14:45:06 -0700
From: Quanah Gibson-Mount <quanah@symas.com>
To: openldap-its@openldap.org
Subject: Re: (ITS#6300) Issues with kqueue after fork
--==========C0CB1C4A269A2DB8FD87==========
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Date: Monday, October 23, 2017 12:43 AM -0700
From: Xin Li
To: Quanah Gibson-Mount

Hi Quanah,

I finally got some time to twiddle with OpenLDAP (again).  It looks like
the EBADF is expected, because fork(2) says:

           =E2=80=A2   The child process has its own copy of the parent's
               descriptors, except for descriptors returned by
               kqueue(2), which are not inherited from the parent
               process.

And slapd does have a fork() after the initial kqueue(), which rendered
it invalid.

The kqueue() is part of SLAP_SOCK_INIT(), which is part of
slapd_daemon_init().  Then, after that, fork() happen in lutil_detach(),
and doing this hack would (incorrectly) make slapd to start:

diff --git a/libraries/liblutil/detach.c b/libraries/liblutil/detach.c
index c6464c038..f47d36548 100644
--- a/libraries/liblutil/detach.c
+++ b/libraries/liblutil/detach.c
@@ -73,7 +73,7 @@ lutil_detach( int debug, int do_close )
 #ifdef HAVE_THR
                        pid =3D fork1();
 #else
-                       pid =3D fork();
+                       pid =3D rfork(RFPROC);
 #endif
                        switch ( pid )
                        {

It is incorrect because the code in slapd/main.c seems to expect the
child to write a "1" to it before exiting with EXIT_SUCCESS() and
obviously if the two processes shares the same file table, the parent
would consider the start was failed because read() would fail.

I think the right fix would be to move the lutil_detach() to before
slapd_daemon_init(), see attachment, but it seems that some code after
the initial daemon initialization is still trying to output to stderr, etc.

What do you think?

Cheers,

--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>

--==========C0CB1C4A269A2DB8FD87==========
Content-Type: message/rfc822;
 name="Re: kqueue in OpenLDAP for FreeBSD (fwd)"

Date: Tue, 24 Apr 2018 14:28:57 -0700
From: Quanah Gibson-Mount <quanah@symas.com>
Reply-To: Quanah Gibson-Mount <quanah@symas.com>
To: Howard Chu <hyc@symas.com>
Subject: Re: kqueue in OpenLDAP for FreeBSD (fwd)
Message-ID: <49431EE1A7D095C608D778A4@[192.168.1.10]>
X-Mailer: Mulberry/4.0.9a1 (Win32)
MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="==========EF6328C82A1CE901E152=========="

--==========EF6328C82A1CE901E152==========
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

kqueue issues with slapd

--Quanah

------------ Forwarded Message ------------
Date: Monday, October 23, 2017 12:43 AM -0700
From: Xin Li <delphij@delphij.net>
To: Quanah Gibson-Mount <quanah@symas.com>, Xin LI
<delphij@gmail.com>
Cc: d@delphij.net, Pietro Cerutti <gahr@gahr.ch>, Xin LI=20
<delphij@freebsd.org>
Subject: Re: kqueue in OpenLDAP for FreeBSD

Hi, Quanah,

I finally got some time to twiddle with OpenLDAP (again).  It looks like
the EBADF is expected, because fork(2) says:

           =E2=80=A2   The child process has its own copy of the parent's
               descriptors, except for descriptors returned by
               kqueue(2), which are not inherited from the parent
               process.

And slapd does have a fork() after the initial kqueue(), which rendered
it invalid.

The kqueue() is part of SLAP_SOCK_INIT(), which is part of
slapd_daemon_init().  Then, after that, fork() happen in lutil_detach(),
and doing this hack would (incorrectly) make slapd to start:

diff --git a/libraries/liblutil/detach.c b/libraries/liblutil/detach.c
index c6464c038..f47d36548 100644
--- a/libraries/liblutil/detach.c
+++ b/libraries/liblutil/detach.c
@@ -73,7 +73,7 @@ lutil_detach( int debug, int do_close )
 #ifdef HAVE_THR
                        pid =3D fork1();
 #else
-                       pid =3D fork();
+                       pid =3D rfork(RFPROC);
 #endif
                        switch ( pid )
                        {

It is incorrect because the code in slapd/main.c seems to expect the
child to write a "1" to it before exiting with EXIT_SUCCESS() and
obviously if the two processes shares the same file table, the parent
would consider the start was failed because read() would fail.

I think the right fix would be to move the lutil_detach() to before
slapd_daemon_init(), see attachment, but it seems that some code after
the initial daemon initialization is still trying to output to stderr, etc.

What do you think?

Cheers,

On 10/20/17 09:00, Quanah Gibson-Mount wrote:
> --On Friday, October 20, 2017 8:15 AM -0700 Quanah Gibson-Mount
> <quanah@symas.com> wrote:
>
>> Thanks, I'll play around with the build some.=C2=A0 I'm limiting slapd
=
to as
>>

Message of length 21542 truncated

Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org