Re: (ITS#3665) Multi-Listener Thread Support

[Emmanuel Lecharny <elecharny@gmail.com>]

 On 7/31/10 3:28 AM, Howard Chu wrote:
> Revisiting an old thread...
>
> http://www.openldap.org/lists/openldap-devel/200504/msg00066.html
>
> I'm definitely seeing our current listener running out of steam on 
> servers with more than 12 or so cores, and some work in this direction 
> will definitely help. First it would be a good idea to classify all of 
> the tasks that the current listener manages, before deciding how to 
> divide them among multiple threads.
>
> The listener is responsible for many events right now:
>   signal/shutdown processing
>   idle timeout checks
>   write timeout checks
>   runqueue scheduling
>   listener socket events
>   threadpool pauses
>   client socket read events
>   client socket write events
>
> Splitting the client socket handling across multiple threads will 
> bring the greatest improvement in scalability. Just need to check our 
> thinking and make sure the remaining division of labor still makes sense.
>
> There are two cases that annoy me in our current design - why don't we 
> just dedicate a thread to each listener socket, and let it block on 
> accept() ? That would eliminate a bit of churn in the current select() 
> workload.
>
> Likewise, why don't we just let writer threads block in write(), 
> instead of having them ask the listener to listen for writability on 
> their socket? Or, if we're using non-blocking sockets, why don't we 
> let the writer threads block in their own select call, instead of 
> relying on the central thread to do the select and re-dispatch?
If your writer threads block until the socket is writeable, how many 
threads will you need to have if many client never read the data ? You 
might quickly not have any remaining writing threads available... The 
select() is supposed to tell you when a socket is ready to accept more 
write, then it's time to select a writing thread to push data into this 
socket. You just need a pool of writing thread, and when a writing 
thread has pushed data into the socket, then it becomes available for 
the next write.
> The first, obvious answer is this: when threads are blocked in system 
> calls like accept(), we can't simply wake them up again for shutdown 
> events or other situations. I believe the obvious fix here is to use 
> select() in each thread, waiting for both the target fd and the 
> wake_sds fd which is written to whenever a signal is caught. Off the 
> top of my head I'm not sure, when several threads are selecting on the 
> same fd, if they all receive a Readable event or if only one of them 
> will. Anyone know?
Why would you have more than one select() ? Wouldn't it be better to 
have one thread processing the select() and dispatching the operation to 
a pool of threads ?

All in all, what costs CPU consumption in a server is most certainly the 
processing of incoming and outgoing requests, not the processing of the 
select() operation, no ?

Im' maybe a bit tainted by Java, but it's really based on the same 
mechanisms under the hood...

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com