(ITS#3975) semaphore-based load conditioning in slapd

[slim@us.ibm.com]

Full_Name: sang seok lim
Version: HEAD
OS: SUSE Linux 9
URL: ftp://ftp.openldap.org/incoming/sangseok-1025.diff
Submission from: (NULL) (129.34.20.23)


This patch enables a semaphore-based resource access control in slapd so as to
mitigate the listener thread overheads, mainly from select(). The resource is
defined as available worker threads in a thread pool at given time. Mitigating
the overheads improves the throughput of slapd especially when it handles a
number of concurrent connections.

The current slapd consists of a single listener thread and a thread pool
managing multiple worker threads. The listener thread monitors incoming
operations by using select() and submits them to the pool for further
processing.  Then worker threads  process the submitted operations. In the
current architecture, even when there is no available worker thread, the
listener keeps select()ing, processing the incoming requests, and submitting
them to the pool. These submitted requests will not be processed until any
worker thread becomes available. The select() operation itself is not cheap,
when a number of concurrent connections is handled in slapd.

This patch utilizes a lazy semaphore, the variant of a semaphore, in order to
eliminate the excessive selecting() overheads of the listener thread. With the
semaphore, before submitting an operation, a thread checks if there is any
available worker thread by calling ldap_lazy_sem_wait(). If there is no
available worker thread, ldap_lazy_sem_wait() will be blocked, putting the
thread into sleep. After completing the operation, a worker thread calls
ldap_lazy_sem_post() which increases the count of the semaphore and wakes up
blocked threads if there is any. In brief, with the semaphore, if there is no
available worker thread in the pool, the listener thread will be put into sleep
until the predefined (lazyness in newly defined semaphore of this patch) number
of threads becomes available. 

The performance numbers of the patched slapd are as follows,

System-under-test:
H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory
O/S: SUSE Linux 9
10K entries (all cached in an entry cache)
10K entry random search

ftp://ftp.openldap.org/incoming/sangseok-perf.PNG

The bars and the lines represent search throughput and select() rate
respectively. The search performance of slapd was improved up to 15% in the
range of 128 ~ 10,000 connections due to the significant decrease in select()
rate. The experiment confirms that the patch relieves the select() overheads
when there is a large number of concurrent connections to slapd.

This patch consists of three parts:
(1) Semaphore-based load conditioning (within SLAP_SEM_LOAD_CONTROL)
(2) Lightweight listener (within SLAP_LIGHTWEIGHT_LISTENER)
Please check ITS#3835 to see what it is.
http://www.openldap.org/its/index.cgi/Development?id=3835
* This patch obsoletes the previously submitted patch of ITS#3835.
(3) Multiple connection array support (within SLAP_MULTI_CONN_ARRAY)
- Execution of connection_read() is protected by a global mutex
(connections_mutex). So even though the execution is parallelized by
multi-threaded processing in ITS#3835, it is still serialized by the mutex which
incurs 1-3% CPU idleness. By using this patch, it minimizes the chance of lock
conflicting so as to eliminate the CPU idle and increase throughput. It is
supplemental to part (2).

*Each part can be applied independently from other parts. Recommended
combinations, however, are (1)+(2)+(3) for responsiveness and throughput, (1)
only for  throughput, or (2)+(3) only for responsiveness improvement.

I really appreciate any comments to the patch.

Sang-Seok Lim
IBM T.J Watson Research,
P.O. Box 218, Yorktown Heights, NY 10598
slim@us.ibm.com