(ITS#3835) Lightweight Listener Thread

[slim@us.ibm.com]

Full_Name: sang seok lim
Version: Current HEAD
OS: SUSE Linux 9
URL: ftp://ftp.openldap.org/incoming/lightweight_listener.diff
Submission from: (NULL) (129.34.20.23)


This patch enables multi-threaded processing of incoming connection
establishment and LDAP request reading/parsing so as to make the listener thread
of slapd more lightweight. Lightening a listener thread will make slapd more
responsive and robust.

The current slapd consists of a single listener thread and a thread pool. The
listener thread is in charge of handling incoming connections, reading/parsing
LDAP requests, waking up blocked operation, etc. The first two operations are
CPU-cycles consuming routines and are executed against over all triggered events
one by one, which limits the parallelism of connection management in slapd. For
example, in this architecture, if the listener thread is stalled or occupied by
an abnormal connection, it will hinder processing of other normal connections at
once.

As a remedy, this patch enables a listener thread to hand-off the processing of
each triggered event to the worker threads instead of having the listener thread
process all triggered events by itself.

At the code level, slapd_handle_listener() and connection_read() are
multi-threaded. On the reception of a new connection, the listener does only
select() on a listening socket and then all necessary work will be done by a
separate worker thread. Likewise, on the reception of new LDAP requests, it only
select()s the corresponding event and then submits the event to the worker
thread pool.

Performance numbers with this patch are as follows,

System-under-test:
H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory
O/S: SUSE Linux 9
10K entries (cached in an entry cache)
# of concurrent connections : 100

Throughput
W/O patch: 18,450 op/sec
W patch: 18,845 op/sec

Throughput with 4,000 idle TCP connections
W/O patch:  8,748 op/sec
W patch: 11,383 op/sec

Throughput with 2 second delay in one out of a hundred incoming connections: ex)
2 seconds delay in reverse DNS look-up
W/O patch: 505 op/sec
W patch: 10,659 op/sec
W patch+SLAP_PERSISTENT_ACCEPT_THREAD(see below): 806 op/sec

The experiment confirms that with the patch the slapd becomes more responsive
and robust to an unexpected delay in a listener thread and a large number of
idle connections.

In addition, if SLAP_PERSISTENT_ACCEPT_THREAD is turned on in the patch, the
listener will create a single, persistent, and standalone thread which executes
slapd_handle_listener() in a loop. It achieves a very high connection accept()
rate. But its use is confined to a thread-supported platform only and it is not
multi-threaded so that it lacks parallelism in establishing connections. For
now, its multi-threading extension is under consideration.

accept() rate comparison
W SLAP_PERSISTENT_ACCEPT_THREAD: 5,230/sec
W/O SLAP_PERSISTENT_ACCEPT_THREAD: 510/sec

This patch has been devised as the second approach to the connection management
in slapd (you can see the first approach, multi-listener threads in ITS#3665). I
really appreciate any comments to the patch.

Sang-Seok Lim
IBM T.J Watson Research,
P.O. Box 218, Yorktown Heights, NY 10598
slim@us.ibm.com