Viewing Development/6300
Full headers

Major security issue: yes  no

Notes:

Notification:

Date: Tue, 22 Sep 2009 19:05:32 +0000
From: bduncan@apple.com
To: openldap-its@OpenLDAP.org
Subject: Add kqueue support to slapd

Full_Name: Bryan Duncan
Version: 2.4.16
OS: Mac OS X 10.6
URL: ftp://ftp.openldap.org/incoming/bryan-duncan-kqueue-090922.patch
Submission from: (NULL) (17.224.21.109)


Added support for using kqueue in slapd (for systems that support kqueue(2).

Followup 1

From: Bryan Duncan <bduncan@apple.com>
Subject: Re: (ITS#6300) Add kqueue support to slapd
Date: Tue, 22 Sep 2009 12:22:58 -0700
Cc: bduncan@apple.com
To: openldap-its@OpenLDAP.org

Subject should have been: "PATCH - Added kqueue support to slapd" to  
make clear this ITS tracks a patch that has been submitted.

-- Bryan

Followup 2

Date: Tue, 22 Sep 2009 13:02:16 -0700
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: bduncan@apple.com, openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd

--On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com wrote:

> Subject should have been: "PATCH - Added kqueue support to slapd" to
> make clear this ITS tracks a patch that has been submitted.

What OS(es) has this kqueue support been tested with?  For example, the 
last time I looked at kqueue on OSX 10.5, it was very marginal, and missing 
most of the important pieces, which meant it wouldn't work for OpenLDAP or 
Postfix, etc.

--Quanah



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Followup 3

Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Tue, 22 Sep 2009 14:40:54 -0700
To: Quanah Gibson-Mount <quanah@zimbra.com>, openldap-its@openldap.org

On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:

> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com  
> wrote:
>
>> Subject should have been: "PATCH - Added kqueue support to slapd" to
>> make clear this ITS tracks a patch that has been submitted.
>
> What OS(es) has this kqueue support been tested with?  For example,  
> the last time I looked at kqueue on OSX 10.5, it was very marginal,  
> and missing most of the important pieces, which meant it wouldn't  
> work for OpenLDAP or Postfix, etc.

It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX- 
specific about the kqueue usage in the patch.  As far as kqueue  
support in OS X 10.5, from what I see, most everything is there.   
Certainly all of the kqueue features in this patch are supported in OS  
X 10.5.

-- Bryan Duncan

Followup 4

Date: Wed, 11 Nov 2009 09:04:35 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Bryan Duncan <bduncan@apple.com>, openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd

--On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan 
<bduncan@apple.com> wrote:


> It was tested on OS X 10.6.  Should work on all BSDs; nothing
> OSX-specific about the kqueue usage in the patch.  As far as kqueue
> support in OS X 10.5, from what I see, most everything is there.
> Certainly all of the kqueue features in this patch are supported in OS X
> 10.5.

Hi Bryan,

The URL to the patch you submitted with this ITS doesn't work.  Can you 
please put the patch somewhere accessible and let us know where it is?

Thanks,
Quanah



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Followup 5

Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Wed, 11 Nov 2009 09:36:51 -0800
Cc: openldap-its@openldap.org
To: Quanah Gibson-Mount <quanah@zimbra.com>

On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:

> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan =
<bduncan@apple.com> wrote:
>=20
>=20
>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>> OSX-specific about the kqueue usage in the patch.  As far as kqueue
>> support in OS X 10.5, from what I see, most everything is there.
>> Certainly all of the kqueue features in this patch are supported in =
OS X
>> 10.5.
>=20
> Hi Bryan,
>=20
> The URL to the patch you submitted with this ITS doesn't work.  Can =
you please put the patch somewhere accessible and let us know where it =
is?
>=20
> Thanks,
> Quanah

Hi Quanah,

My apologies for the bad URL.

The new URL is: =
http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch
If you use a web browser to get it, there are extra steps: select the =
file "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl =
can pull it down directly with no extra steps.

-- Bryan=

Followup 6

Date: Wed, 11 Nov 2009 09:45:59 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: Bryan Duncan <bduncan@apple.com>
cc: openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd

--On Wednesday, November 11, 2009 9:36 AM -0800 Bryan Duncan 
<bduncan@apple.com> wrote:

>
> On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:
>
>> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan
>> <bduncan@apple.com> wrote:
>>
>>
>>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>>> OSX-specific about the kqueue usage in the patch.  As far as kqueue
>>> support in OS X 10.5, from what I see, most everything is there.
>>> Certainly all of the kqueue features in this patch are supported in
OS X
>>> 10.5.
>>
>> Hi Bryan,
>>
>> The URL to the patch you submitted with this ITS doesn't work.  Can you
>> please put the patch somewhere accessible and let us know where it is?
>>
>> Thanks,
>> Quanah
>
> Hi Quanah,
>
> My apologies for the bad URL.
>
> The new URL is:
> http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch If you
> use a web browser to get it, there are extra steps: select the file
> "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl can
> pull it down directly with no extra steps.

Hi Bryan,

Thanks for the quick response.  Hopefully we can get this incorporated for 
OpenLDAP 2.5 series, and I'm interested in playing with it against 2.4 to 
see how things change when used with Mac.

Thanks!

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Followup 7

Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Wed, 11 Nov 2009 11:29:42 -0800
Cc: openldap-its@openldap.org
To: Quanah Gibson-Mount <quanah@zimbra.com>

On Nov 11, 2009, at 9:45 AM, Quanah Gibson-Mount wrote:

> --On Wednesday, November 11, 2009 9:36 AM -0800 Bryan Duncan =
<bduncan@apple.com> wrote:
>=20
>>=20
>> On Nov 11, 2009, at 9:04 AM, Quanah Gibson-Mount wrote:
>>=20
>>> --On Tuesday, September 22, 2009 2:40 PM -0700 Bryan Duncan
>>> <bduncan@apple.com> wrote:
>>>=20
>>>=20
>>>> It was tested on OS X 10.6.  Should work on all BSDs; nothing
>>>> OSX-specific about the kqueue usage in the patch.  As far as
kqueue
>>>> support in OS X 10.5, from what I see, most everything is
there.
>>>> Certainly all of the kqueue features in this patch are
supported in =
OS X
>>>> 10.5.
>>>=20
>>> Hi Bryan,
>>>=20
>>> The URL to the patch you submitted with this ITS doesn't work.  Can
=
you
>>> please put the patch somewhere accessible and let us know where it
=
is?
>>>=20
>>> Thanks,
>>> Quanah
>>=20
>> Hi Quanah,
>>=20
>> My apologies for the bad URL.
>>=20
>> The new URL is:
>> http://public.me.com/bryan.duncan/bryan-duncan.kqueue.090922.patch If =
you
>> use a web browser to get it, there are extra steps: select the file
>> "bryan-duncan.kqueue.090922.patch" and click download.  ftp & curl
=
can
>> pull it down directly with no extra steps.
>=20
> Hi Bryan,
>=20
> Thanks for the quick response.  Hopefully we can get this incorporated =
for OpenLDAP 2.5 series, and I'm interested in playing with it against =
2.4 to see how things change when used with Mac.
>=20
> Thanks!
>=20
> --Quanah


Awesome!  I look forward to seeing it OpenLDAP.  If there's anything I =
can do to help, please let me know.

Thanks.=20
-- Bryan

Followup 8

Date: Sun, 22 Nov 2009 18:43:58 -0800
From: Howard Chu <hyc@symas.com>
To: bduncan@apple.com
CC: openldap-its@openldap.org
Subject: Re: (ITS#6300) Add kqueue support to slapd

bduncan@apple.com wrote:
> On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:
>
>> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com
>> wrote:
>>
>>> Subject should have been: "PATCH - Added kqueue support to slapd"
to
>>> make clear this ITS tracks a patch that has been submitted.
>>
>> What OS(es) has this kqueue support been tested with?  For example,
>> the last time I looked at kqueue on OSX 10.5, it was very marginal,
>> and missing most of the important pieces, which meant it wouldn't
>> work for OpenLDAP or Postfix, etc.
>
> It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX-
> specific about the kqueue usage in the patch.  As far as kqueue
> support in OS X 10.5, from what I see, most everything is there.
> Certainly all of the kqueue features in this patch are supported in OS
> X 10.5.

To expand on this: there are a lot of comments around the web that kqueue was 
fairly broken at least up to 10.5. E.g. this:

http://lists.schmorp.de/pipermail/libev/2008q4/000523.html

Since we also use a pipe for the wake_sds[] in our event loop, if kqueue only 
supports sockets, then it cannot be used reliably in slapd.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

Followup 9

Subject: Re: (ITS#6300) Add kqueue support to slapd
From: Bryan Duncan <bduncan@apple.com>
Date: Thu, 3 Dec 2009 12:41:10 -0800
Cc: openldap-its@openldap.org
To: Howard Chu <hyc@symas.com>

On Nov 22, 2009, at 6:43 PM, Howard Chu wrote:

> bduncan@apple.com wrote:
>> On Sep 22, 2009, at 1:02 PM, Quanah Gibson-Mount wrote:
>>=20
>>> --On Tuesday, September 22, 2009 7:23 PM +0000 bduncan@apple.com
>>> wrote:
>>>=20
>>>> Subject should have been: "PATCH - Added kqueue support to
slapd" =
to
>>>> make clear this ITS tracks a patch that has been submitted.
>>>=20
>>> What OS(es) has this kqueue support been tested with?  For example,
>>> the last time I looked at kqueue on OSX 10.5, it was very marginal,
>>> and missing most of the important pieces, which meant it wouldn't
>>> work for OpenLDAP or Postfix, etc.
>>=20
>> It was tested on OS X 10.6.  Should work on all BSDs; nothing OSX-
>> specific about the kqueue usage in the patch.  As far as kqueue
>> support in OS X 10.5, from what I see, most everything is there.
>> Certainly all of the kqueue features in this patch are supported in =
OS
>> X 10.5.
>=20
> To expand on this: there are a lot of comments around the web that =
kqueue was fairly broken at least up to 10.5. E.g. this:
>=20
> http://lists.schmorp.de/pipermail/libev/2008q4/000523.html

Perhaps I missed something in that thread, but the only specific =
complaint was regarding kqueue and stdin.  And it's true that OS X's =
kqueue didn't support EVFILT_{READ|WRITE} on *devices* (stdin, tty's, =
/dev/* ) until 10.6.  I don't believe OpenLDAP actually needs that =
feature, though.

>=20
> Since we also use a pipe for the wake_sds[] in our event loop, if =
kqueue only supports sockets, then it cannot be used reliably in slapd.

The OS X kqueue has *always* supported EVFILT_{READ|WRITE} on sockets, =
pipes, fifos, files.  Same for kqueue in the BSDs.  (FWIW, I believe the =
BSDs have always supported devices in kqueue.)

I think everything OpenLDAP needs from kqueue has been, and is, =
supported by OS X and the BSDs.  If there is a specific kqueue filter or =
fd type that OpenLDAP needs and isn't supported, please let me know.

Although I don't think it's necessary, you could add a test to the =
configure script to ensure kqueue supports the types of fd's OpenLDAP =
needs by doing something like this:

    int fd; /* or int fds[2]; */
    /* do something to create an fd here:  open(2), pipe(2), socket(2), =
etc. */
    struct kevent ev;
    EV_SET(&ev, fd, EVFILT_READ, EV_ADD, 0, 0, 0);
    int kq =3D kqueue();
    if (kq < 0) {
        ac_have_kqueue =3D false;
    } else {
        struct timespec to =3D {0,1};  /* anything so that kevent will =
return */
        int ret =3D kevent(kq, &ev, 1, &ev, 1, &to);
        close(kq);
        ac_have_kqueue =3D (ret >=3D 0);
    }

If the fd isn't supported 'ret' should be -1 and errno should be =
ENOTSUP.

-- Bryan Duncan