Re: (ITS#3835) Lightweight Listener Thread

[slim@us.ibm.com]

I confirm that your fixes to the patch are valid.



hyc@symas.com 
Sent by: owner-openldap-bugs@OpenLDAP.org
07/05/2005 09:50 PM

To
openldap-its@OpenLDAP.org
cc

Subject
Re: (ITS#3835) Lightweight Listener Thread






Looks interesting. I have not tested the patch yet, but a quick read 
thru shows a couple problems:

The definition of connection_op_queue() is ifdef'd, but it is invoked 
unconditionally in connection_op_activate().

In slapd_suspend() there is a check "#ifdef SLAP_EVENTS_ARE_INDEXED" but 
this macro is always defined; it is the value (1 or 0) that is 
significant. Judging from the comments I believe the correct test is 
actually "#if !SLAP_EVENTS_ARE_INDEXED" here.

slim@us.ibm.com wrote:
> Full_Name: sang seok lim
> Version: Current HEAD
> OS: SUSE Linux 9
> URL: ftp://ftp.openldap.org/incoming/lightweight_listener.diff
> Submission from: (NULL) (129.34.20.23)
>
>
> This patch enables multi-threaded processing of incoming connection
> establishment and LDAP request reading/parsing so as to make the 
listener thread
> of slapd more lightweight. Lightening a listener thread will make slapd 
more
> responsive and robust.
>
> The current slapd consists of a single listener thread and a thread 
pool. The
> listener thread is in charge of handling incoming connections, 
reading/parsing
> LDAP requests, waking up blocked operation, etc. The first two 
operations are
> CPU-cycles consuming routines and are executed against over all 
triggered events
> one by one, which limits the parallelism of connection management in 
slapd. For
> example, in this architecture, if the listener thread is stalled or 
occupied by
> an abnormal connection, it will hinder processing of other normal 
connections at
> once.
>
> As a remedy, this patch enables a listener thread to hand-off the 
processing of
> each triggered event to the worker threads instead of having the 
listener thread
> process all triggered events by itself.
>
> At the code level, slapd_handle_listener() and connection_read() are
> multi-threaded. On the reception of a new connection, the listener does 
only
> select() on a listening socket and then all necessary work will be done 
by a
> separate worker thread. Likewise, on the reception of new LDAP requests, 
it only
> select()s the corresponding event and then submits the event to the 
worker
> thread pool.
>
> Performance numbers with this patch are as follows,
>
> System-under-test:
> H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory
> O/S: SUSE Linux 9
> 10K entries (cached in an entry cache)
> # of concurrent connections : 100
>
> Throughput
> W/O patch: 18,450 op/sec
> W patch: 18,845 op/sec
>
> Throughput with 4,000 idle TCP connections
> W/O patch:  8,748 op/sec
> W patch: 11,383 op/sec
>
> Throughput with 2 second delay in one out of a hundred incoming 
connections: ex)
> 2 seconds delay in reverse DNS look-up
> W/O patch: 505 op/sec
> W patch: 10,659 op/sec
> W patch+SLAP_PERSISTENT_ACCEPT_THREAD(see below): 806 op/sec
>
> The experiment confirms that with the patch the slapd becomes more 
responsive
> and robust to an unexpected delay in a listener thread and a large 
number of
> idle connections.
>
> In addition, if SLAP_PERSISTENT_ACCEPT_THREAD is turned on in the patch, 
the
> listener will create a single, persistent, and standalone thread which 
executes
> slapd_handle_listener() in a loop. It achieves a very high connection 
accept()
> rate. But its use is confined to a thread-supported platform only and it 
is not
> multi-threaded so that it lacks parallelism in establishing connections. 
For
> now, its multi-threading extension is under consideration.
>
> accept() rate comparison
> W SLAP_PERSISTENT_ACCEPT_THREAD: 5,230/sec
> W/O SLAP_PERSISTENT_ACCEPT_THREAD: 510/sec
>
> This patch has been devised as the second approach to the connection 
management
> in slapd (you can see the first approach, multi-listener threads in 
ITS#3665). I
> really appreciate any comments to the patch.
>
> Sang-Seok Lim
> IBM T.J Watson Research,
> P.O. Box 218, Yorktown Heights, NY 10598
> slim@us.ibm.com
>
>
>
>
> 


-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support