Re: (ITS#3665) Multi-Listener Thread Support

[Howard Chu <hyc@symas.com>]

Revisiting an old thread...

http://www.openldap.org/lists/openldap-devel/200504/msg00066.html

I'm definitely seeing our current listener running out of steam on servers 
with more than 12 or so cores, and some work in this direction will definitely 
help. First it would be a good idea to classify all of the tasks that the 
current listener manages, before deciding how to divide them among multiple 
threads.

The listener is responsible for many events right now:
  signal/shutdown processing
  idle timeout checks
  write timeout checks
  runqueue scheduling
  listener socket events
  threadpool pauses
  client socket read events
  client socket write events

Splitting the client socket handling across multiple threads will bring the 
greatest improvement in scalability. Just need to check our thinking and make 
sure the remaining division of labor still makes sense.

There are two cases that annoy me in our current design - why don't we just 
dedicate a thread to each listener socket, and let it block on accept() ? That 
would eliminate a bit of churn in the current select() workload.

Likewise, why don't we just let writer threads block in write(), instead of 
having them ask the listener to listen for writability on their socket? Or, if 
we're using non-blocking sockets, why don't we let the writer threads block in 
their own select call, instead of relying on the central thread to do the 
select and re-dispatch?

The first, obvious answer is this: when threads are blocked in system calls 
like accept(), we can't simply wake them up again for shutdown events or other 
situations. I believe the obvious fix here is to use select() in each thread, 
waiting for both the target fd and the wake_sds fd which is written to 
whenever a signal is caught. Off the top of my head I'm not sure, when several 
threads are selecting on the same fd, if they all receive a Readable event or 
if only one of them will. Anyone know?

I don't think the remaining tasks really involve much overhead. So ideally, we 
can handle the idle/write timeout and runqueue scheduling in a single thread, 
which can also be responsible for the main signal/shutdown processing.

The listener sockets can each have their own dedicated thread, selecting on 
their listener socket and the signal fd.

A small number of threads can handle the bulk of the client socket events. We 
would continue to use a power-of-two descriptor table, and a power-of-two set 
of threads here. The division of labor would simply be
    thread# = FD % number of threads
(There's no need to split the current connection table into multiple arrays, 
we just divvy things up so that a given thread only accesses its own slots in 
the array.)

Comments?
--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/