Full_Name: sang seok lim Version: Current HEAD OS: SUSE Linux 9 URL: ftp://ftp.openldap.org/incoming/lightweight_listener.diff Submission from: (NULL) (129.34.20.23) This patch enables multi-threaded processing of incoming connection establishment and LDAP request reading/parsing so as to make the listener thread of slapd more lightweight. Lightening a listener thread will make slapd more responsive and robust. The current slapd consists of a single listener thread and a thread pool. The listener thread is in charge of handling incoming connections, reading/parsing LDAP requests, waking up blocked operation, etc. The first two operations are CPU-cycles consuming routines and are executed against over all triggered events one by one, which limits the parallelism of connection management in slapd. For example, in this architecture, if the listener thread is stalled or occupied by an abnormal connection, it will hinder processing of other normal connections at once. As a remedy, this patch enables a listener thread to hand-off the processing of each triggered event to the worker threads instead of having the listener thread process all triggered events by itself. At the code level, slapd_handle_listener() and connection_read() are multi-threaded. On the reception of a new connection, the listener does only select() on a listening socket and then all necessary work will be done by a separate worker thread. Likewise, on the reception of new LDAP requests, it only select()s the corresponding event and then submits the event to the worker thread pool. Performance numbers with this patch are as follows, System-under-test: H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory O/S: SUSE Linux 9 10K entries (cached in an entry cache) # of concurrent connections : 100 Throughput W/O patch: 18,450 op/sec W patch: 18,845 op/sec Throughput with 4,000 idle TCP connections W/O patch: 8,748 op/sec W patch: 11,383 op/sec Throughput with 2 second delay in one out of a hundred incoming connections: ex) 2 seconds delay in reverse DNS look-up W/O patch: 505 op/sec W patch: 10,659 op/sec W patch+SLAP_PERSISTENT_ACCEPT_THREAD(see below): 806 op/sec The experiment confirms that with the patch the slapd becomes more responsive and robust to an unexpected delay in a listener thread and a large number of idle connections. In addition, if SLAP_PERSISTENT_ACCEPT_THREAD is turned on in the patch, the listener will create a single, persistent, and standalone thread which executes slapd_handle_listener() in a loop. It achieves a very high connection accept() rate. But its use is confined to a thread-supported platform only and it is not multi-threaded so that it lacks parallelism in establishing connections. For now, its multi-threading extension is under consideration. accept() rate comparison W SLAP_PERSISTENT_ACCEPT_THREAD: 5,230/sec W/O SLAP_PERSISTENT_ACCEPT_THREAD: 510/sec This patch has been devised as the second approach to the connection management in slapd (you can see the first approach, multi-listener threads in ITS#3665). I really appreciate any comments to the patch. Sang-Seok Lim IBM T.J Watson Research, P.O. Box 218, Yorktown Heights, NY 10598 slim@us.ibm.com
Looks interesting. I have not tested the patch yet, but a quick read thru shows a couple problems: The definition of connection_op_queue() is ifdef'd, but it is invoked unconditionally in connection_op_activate(). In slapd_suspend() there is a check "#ifdef SLAP_EVENTS_ARE_INDEXED" but this macro is always defined; it is the value (1 or 0) that is significant. Judging from the comments I believe the correct test is actually "#if !SLAP_EVENTS_ARE_INDEXED" here. slim@us.ibm.com wrote: > Full_Name: sang seok lim > Version: Current HEAD > OS: SUSE Linux 9 > URL: ftp://ftp.openldap.org/incoming/lightweight_listener.diff > Submission from: (NULL) (129.34.20.23) > > > This patch enables multi-threaded processing of incoming connection > establishment and LDAP request reading/parsing so as to make the listener thread > of slapd more lightweight. Lightening a listener thread will make slapd more > responsive and robust. > > The current slapd consists of a single listener thread and a thread pool. The > listener thread is in charge of handling incoming connections, reading/parsing > LDAP requests, waking up blocked operation, etc. The first two operations are > CPU-cycles consuming routines and are executed against over all triggered events > one by one, which limits the parallelism of connection management in slapd. For > example, in this architecture, if the listener thread is stalled or occupied by > an abnormal connection, it will hinder processing of other normal connections at > once. > > As a remedy, this patch enables a listener thread to hand-off the processing of > each triggered event to the worker threads instead of having the listener thread > process all triggered events by itself. > > At the code level, slapd_handle_listener() and connection_read() are > multi-threaded. On the reception of a new connection, the listener does only > select() on a listening socket and then all necessary work will be done by a > separate worker thread. Likewise, on the reception of new LDAP requests, it only > select()s the corresponding event and then submits the event to the worker > thread pool. > > Performance numbers with this patch are as follows, > > System-under-test: > H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory > O/S: SUSE Linux 9 > 10K entries (cached in an entry cache) > # of concurrent connections : 100 > > Throughput > W/O patch: 18,450 op/sec > W patch: 18,845 op/sec > > Throughput with 4,000 idle TCP connections > W/O patch: 8,748 op/sec > W patch: 11,383 op/sec > > Throughput with 2 second delay in one out of a hundred incoming connections: ex) > 2 seconds delay in reverse DNS look-up > W/O patch: 505 op/sec > W patch: 10,659 op/sec > W patch+SLAP_PERSISTENT_ACCEPT_THREAD(see below): 806 op/sec > > The experiment confirms that with the patch the slapd becomes more responsive > and robust to an unexpected delay in a listener thread and a large number of > idle connections. > > In addition, if SLAP_PERSISTENT_ACCEPT_THREAD is turned on in the patch, the > listener will create a single, persistent, and standalone thread which executes > slapd_handle_listener() in a loop. It achieves a very high connection accept() > rate. But its use is confined to a thread-supported platform only and it is not > multi-threaded so that it lacks parallelism in establishing connections. For > now, its multi-threading extension is under consideration. > > accept() rate comparison > W SLAP_PERSISTENT_ACCEPT_THREAD: 5,230/sec > W/O SLAP_PERSISTENT_ACCEPT_THREAD: 510/sec > > This patch has been devised as the second approach to the connection management > in slapd (you can see the first approach, multi-listener threads in ITS#3665). I > really appreciate any comments to the patch. > > Sang-Seok Lim > IBM T.J Watson Research, > P.O. Box 218, Yorktown Heights, NY 10598 > slim@us.ibm.com > > > > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
Hi Slim, I patched OpenLDAP 2.3.4 with your patch (plus Howard's comments), and ran it on a small DB (10,002 entry database). I then queried that database with 12 clients (all using persistent connections, rather than bind/search/unbind/close). After 7-8 minutes of constant querying, 11 clients disappeared -- Their state remains established, but they are no longer able to send data to the server that results in anything being logged when logging is turned on... It's like they are talking to empty air. The 12th client continues to query without problem. So something appears to be losing context after a period of time. I'm going to test with 12 clients that bind/search/unbind/close as well. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Tuesday, July 05, 2005 11:04 PM -0700 Quanah Gibson-Mount <quanah@symas.com> wrote: > Hi Slim, > > I patched OpenLDAP 2.3.4 with your patch (plus Howard's comments), and > ran it on a small DB (10,002 entry database). > > I then queried that database with 12 clients (all using persistent > connections, rather than bind/search/unbind/close). After 7-8 minutes of > constant querying, 11 clients disappeared -- Their state remains > established, but they are no longer able to send data to the server that > results in anything being logged when logging is turned on... It's like > they are talking to empty air. The 12th client continues to query > without problem. So something appears to be losing context after a > period of time. > > I'm going to test with 12 clients that bind/search/unbind/close as well. The results here were a bit worse. The server and clients stopped communicating after 4 seconds. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
I confirm that your fixes to the patch are valid. hyc@symas.com Sent by: owner-openldap-bugs@OpenLDAP.org 07/05/2005 09:50 PM To openldap-its@OpenLDAP.org cc Subject Re: (ITS#3835) Lightweight Listener Thread Looks interesting. I have not tested the patch yet, but a quick read thru shows a couple problems: The definition of connection_op_queue() is ifdef'd, but it is invoked unconditionally in connection_op_activate(). In slapd_suspend() there is a check "#ifdef SLAP_EVENTS_ARE_INDEXED" but this macro is always defined; it is the value (1 or 0) that is significant. Judging from the comments I believe the correct test is actually "#if !SLAP_EVENTS_ARE_INDEXED" here. slim@us.ibm.com wrote: > Full_Name: sang seok lim > Version: Current HEAD > OS: SUSE Linux 9 > URL: ftp://ftp.openldap.org/incoming/lightweight_listener.diff > Submission from: (NULL) (129.34.20.23) > > > This patch enables multi-threaded processing of incoming connection > establishment and LDAP request reading/parsing so as to make the listener thread > of slapd more lightweight. Lightening a listener thread will make slapd more > responsive and robust. > > The current slapd consists of a single listener thread and a thread pool. The > listener thread is in charge of handling incoming connections, reading/parsing > LDAP requests, waking up blocked operation, etc. The first two operations are > CPU-cycles consuming routines and are executed against over all triggered events > one by one, which limits the parallelism of connection management in slapd. For > example, in this architecture, if the listener thread is stalled or occupied by > an abnormal connection, it will hinder processing of other normal connections at > once. > > As a remedy, this patch enables a listener thread to hand-off the processing of > each triggered event to the worker threads instead of having the listener thread > process all triggered events by itself. > > At the code level, slapd_handle_listener() and connection_read() are > multi-threaded. On the reception of a new connection, the listener does only > select() on a listening socket and then all necessary work will be done by a > separate worker thread. Likewise, on the reception of new LDAP requests, it only > select()s the corresponding event and then submits the event to the worker > thread pool. > > Performance numbers with this patch are as follows, > > System-under-test: > H/W: IBM eServer OpenPower with 2 1.5GHz CPUs (SMT) and 16GB memory > O/S: SUSE Linux 9 > 10K entries (cached in an entry cache) > # of concurrent connections : 100 > > Throughput > W/O patch: 18,450 op/sec > W patch: 18,845 op/sec > > Throughput with 4,000 idle TCP connections > W/O patch: 8,748 op/sec > W patch: 11,383 op/sec > > Throughput with 2 second delay in one out of a hundred incoming connections: ex) > 2 seconds delay in reverse DNS look-up > W/O patch: 505 op/sec > W patch: 10,659 op/sec > W patch+SLAP_PERSISTENT_ACCEPT_THREAD(see below): 806 op/sec > > The experiment confirms that with the patch the slapd becomes more responsive > and robust to an unexpected delay in a listener thread and a large number of > idle connections. > > In addition, if SLAP_PERSISTENT_ACCEPT_THREAD is turned on in the patch, the > listener will create a single, persistent, and standalone thread which executes > slapd_handle_listener() in a loop. It achieves a very high connection accept() > rate. But its use is confined to a thread-supported platform only and it is not > multi-threaded so that it lacks parallelism in establishing connections. For > now, its multi-threading extension is under consideration. > > accept() rate comparison > W SLAP_PERSISTENT_ACCEPT_THREAD: 5,230/sec > W/O SLAP_PERSISTENT_ACCEPT_THREAD: 510/sec > > This patch has been devised as the second approach to the connection management > in slapd (you can see the first approach, multi-listener threads in ITS#3665). I > really appreciate any comments to the patch. > > Sang-Seok Lim > IBM T.J Watson Research, > P.O. Box 218, Yorktown Heights, NY 10598 > slim@us.ibm.com > > > > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
Hi Quanah, I'm investigating the problem. Could you test the patch without the Howard's second fix. My current thought is that slapd_suspend/slapd_resume always have to be mutex (slap_daemon.sd_mutex)-protected in both epoll() and select() cases. Thank you. Sang-Seok quanah@symas.com Sent by: owner-openldap-bugs@OpenLDAP.org 07/06/2005 02:52 AM To openldap-its@OpenLDAP.org cc Subject Re: (ITS#3835) Lightweight Listener Thread --On Tuesday, July 05, 2005 11:04 PM -0700 Quanah Gibson-Mount <quanah@symas.com> wrote: > Hi Slim, > > I patched OpenLDAP 2.3.4 with your patch (plus Howard's comments), and > ran it on a small DB (10,002 entry database). > > I then queried that database with 12 clients (all using persistent > connections, rather than bind/search/unbind/close). After 7-8 minutes of > constant querying, 11 clients disappeared -- Their state remains > established, but they are no longer able to send data to the server that > results in anything being logged when logging is turned on... It's like > they are talking to empty air. The 12th client continues to query > without problem. So something appears to be losing context after a > period of time. > > I'm going to test with 12 clients that bind/search/unbind/close as well. The results here were a bit worse. The server and clients stopped communicating after 4 seconds. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Wednesday, July 06, 2005 11:00 AM -0400 Sang s Lim <slim@us.ibm.com> wrote: > Hi Quanah, > > I'm investigating the problem. > Could you test the patch without the Howard's second fix. > My current thought is that slapd_suspend/slapd_resume always have to > be mutex (slap_daemon.sd_mutex)-protected in both epoll() and select() > cases. > Thank you. > > Sang-Seok Hi Sang-Seok, I'm rebuilding it with the if's returned to ifdefs, and will let you know the results. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Wednesday, July 06, 2005 12:12 PM -0700 Quanah Gibson-Mount <quanah@symas.com> wrote: > > > --On Wednesday, July 06, 2005 11:00 AM -0400 Sang s Lim <slim@us.ibm.com> > wrote: > >> Hi Quanah, >> >> I'm investigating the problem. >> Could you test the patch without the Howard's second fix. >> My current thought is that slapd_suspend/slapd_resume always have to >> be mutex (slap_daemon.sd_mutex)-protected in both epoll() and select() >> cases. >> Thank you. >> >> Sang-Seok > > Hi Sang-Seok, > > I'm rebuilding it with the if's returned to ifdefs, and will let you know > the results. Hi Sang-Seok, I've tested now with the new build, and it works correctly. So I'm guessing the ifdef's are not even needed, since things must always be mutexed. As for the results: Search rate with patch: 868.232 ops/second Searchrate without patch: 910.171 ops/second I.e., its faster without the patch for me on Solaris. I've not yet tested with it being compiled with the second flag set. This is on a 2 CPU SunFire V210 with 2GB of RAM. The database has 10,000 entries. It has an 11k entry cache and a 33k idl cache. The DB has a 384 MB cache for BDB. The attribute being queried (uid) is indexed eq, and it is an exact uid= search that is being performed. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
>I've tested now with the new build, and it works correctly. So I'm >guessing the ifdef's are not even needed, since things must always be >mutexed. No they are not. So with fixes, I uploaded the patch again. ftp://ftp.openldap.org/incoming/sangseok-050707.diff > >As for the results: > >Search rate with patch: 868.232 ops/second >Searchrate without patch: 910.171 ops/second > >I.e., its faster without the patch for me on Solaris. I've not yet tested >with it being compiled with the second flag set. > >This is on a 2 CPU SunFire V210 with 2GB of RAM. The database has 10,000 >entries. It has an 11k entry cache and a 33k idl cache. The DB has a 384 >MB cache for BDB. The attribute being queried (uid) is indexed eq, and it >is an exact uid= search that is being performed. With your H/W and configuration, 910 ops/second seems to be very low, considering all entries were cached in the entry cache and indexed. I have several questions. What benchmark did you use? Are the numbers maximum throughput or CPUs were under-utilized during the test? And could you test the patch with a large number of idle connections (4K ~16K)? It'll be very interesting and helpful to see how the patch works with a large number of idle connections in other platform. Thank you. --Sang-Seok
--On Thursday, July 07, 2005 10:28 AM -0400 Sang s Lim <slim@us.ibm.com> wrote: >> I've tested now with the new build, and it works correctly. So I'm >> guessing the ifdef's are not even needed, since things must always be >> mutexed. > > No they are not. So with fixes, I uploaded the patch again. > ftp://ftp.openldap.org/incoming/sangseok-050707.diff > >> >> As for the results: >> >> Search rate with patch: 868.232 ops/second >> Searchrate without patch: 910.171 ops/second >> >> I.e., its faster without the patch for me on Solaris. I've not yet > tested >> with it being compiled with the second flag set. >> >> This is on a 2 CPU SunFire V210 with 2GB of RAM. The database has >> 10,000 > >> entries. It has an 11k entry cache and a 33k idl cache. The DB has a > 384 >> MB cache for BDB. The attribute being queried (uid) is indexed eq, and > it >> is an exact uid= search that is being performed. > > With your H/W and configuration, 910 ops/second seems to be very low, > considering all entries were cached in the entry cache and indexed. > I have several questions. What benchmark did you use? > Are the numbers maximum throughput or CPUs were under-utilized during the > test? > And could you test the patch with a large number of idle connections (4K > ~16K)? > It'll be very interesting and helpful to see how the patch works > with a large number of idle connections in other platform. > Thank you. Hi Sang, Note that in this case, an operation is: "connect, bind, search, receive result, unbind, disconnect". This search rate is in line with what I expect from a Solaris system. Compartively, throwing GSSAPI in the mix nets me 174 ops/second average. The benchmarking software I'm using is slamd <http://www.slamd.com/>. The CPU's are highly utilized (99.x%) during the searches. I'm not particularly interested in testing with a large number of idle connections. The solution to idle clients is the idletimeout setting in slapd.conf. I'd have to figure out how to set up a large number of idle connections as well. I was hoping this patch was more designed at increasing throughput performance. --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
> I'm not particularly interested in testing with a large number of idle > connections. The solution to idle clients is the idletimeout setting in > slapd.conf. I'd have to figure out how to set up a large number of idle > connections as well. A simple socket program which establishes TCP connections to slapd without sending subsequent LDAP requests will work for the experiment with a number of idle connections. > > I was hoping this patch was more designed at increasing throughput > performance. Even though this patch only aims at responsive and resilient improvement, I think it is an important issue as you can see in the following link. http://www.openldap.org/lists/openldap-software/200505/msg00206.html This patch may relieve the problem posted in the link. BTW, I'm also looking into throughput improvement. So I hope a throughput improvement patch is provided soon. -Sang-Seok
--On Friday, July 08, 2005 8:13 PM -0400 Sang s Lim <slim@us.ibm.com> wrote: >> I'm not particularly interested in testing with a large number of idle >> connections. The solution to idle clients is the idletimeout setting in > >> slapd.conf. I'd have to figure out how to set up a large number of idle > >> connections as well. > > A simple socket program which establishes TCP connections to slapd without > sending subsequent LDAP requests will work for the experiment with a > number of idle connections. > >> >> I was hoping this patch was more designed at increasing throughput >> performance. > > Even though this patch only aims at responsive and resilient improvement, > I think it is an important issue as you can see in the following link. > http://www.openldap.org/lists/openldap-software/200505/msg00206.html > This patch may relieve the problem posted in the link. I can see how it might help relieve the problem in that post, although I'm not sure if I would call the patch more responsive (where I define that as a betterment in getting results). Definately more resilient, in that connections will be accepted (if not acted upon). > BTW, I'm also looking into throughput improvement. So I hope a throughput > improvement patch > is provided soon. Wonderful. :) --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
> I've tested now with the new build, and it works correctly. So I'm > guessing the ifdef's are not even needed, since things must always be > mutexed. > > As for the results: > > Search rate with patch: 868.232 ops/second > Searchrate without patch: 910.171 ops/second > > I.e., its faster without the patch for me on Solaris. I've not yet tested > with it being compiled with the second flag set. To find the cause of the performance degradation, I have thoroughly reanalyze the patch and found there is an expensive operation, WAKE_LISTENER in "slapd_resume". So I eliminated the operation in the patch. The following patch has been tested and showed no performance degradation in my environment as before. ftp://ftp.openldap.org/incoming/sangseok-050711.diff Sang-Seok
--On Monday, July 11, 2005 7:52 PM -0400 Sang s Lim <slim@us.ibm.com> wrote: >> I've tested now with the new build, and it works correctly. So I'm >> guessing the ifdef's are not even needed, since things must always be >> mutexed. >> >> As for the results: >> >> Search rate with patch: 868.232 ops/second >> Searchrate without patch: 910.171 ops/second >> >> I.e., its faster without the patch for me on Solaris. I've not yet > tested >> with it being compiled with the second flag set. > > To find the cause of the performance degradation, I have thoroughly > reanalyze the patch and found there is an expensive operation, > WAKE_LISTENER in "slapd_resume". So I eliminated the operation in the > patch. > The following patch has been tested and showed no performance degradation > in my environment as before. > ftp://ftp.openldap.org/incoming/sangseok-050711.diff > > Sang-Seok Hi Sang-Seok, I'll give this new version a try. :) --Quanah -- Quanah Gibson-Mount Product Engineer Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
changed state Open to Feedback moved from Incoming to Software Enhancements
First off, thanks for your work on this, Sang-Seok. It will be a big help. I've started running the lightweight listener code with OPENLDAP_REL_ENG_2_3 (7 July) on one host in our LDAP cluster. Average CPU consumption for the listener thread has dropped from ~45% of one CPU to ~35% of one CPU (a ~25% drop in CPU consumption). I suspect I'm not seeing bigger gains because I have tcpwrappers and reverse DNS resolution disabled, so the only things being offloaded to other threads are connection_read() and connection initialization. I've also looked at ITS #3665, which I may try soon. If there are any specific things I can try with either ITS, please let me know. john -- John Morrissey _o /\ ---- __o jwm@horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
John Morrissey <jwm@horde.net> wrote on 08/01/2005 03:34:27 PM: > First off, thanks for your work on this, Sang-Seok. It will be a big help. Than you. > I've started running the lightweight listener code with OPENLDAP_REL_ENG_2_3 > (7 July) on one host in our LDAP cluster. Average CPU consumption for the > listener thread has dropped from ~45% of one CPU to ~35% of one CPU (a ~25% > drop in CPU consumption). Could you elaborate on incoming request characteristics of the host. - H/W spec of the host. - the number of concurrent connections. - incoming request types: search, modify, add, etc. - incoming request pattern: how many requests per single bind(). I think the 35% CPU consumption of the thread is too much. In my test environment, even with 10k active concurrent connections, the listener(W the patch) thread CPU consumption was less than roughly 10%. > I suspect I'm not seeing bigger gains because I have tcpwrappers and reverse > DNS resolution disabled, so the only things being offloaded to other threads > are connection_read() and connection initialization. I am also working an another patch which aims at the listener thread load control. We can collaborate together to see if the patch will work. Sang-Seok
On Tue, Aug 02, 2005 at 08:57:37AM -0400, Sang s Lim wrote: > Could you elaborate on incoming request characteristics of the host. > - H/W spec of the host. Sure. They all have dual 2.4GHz Xeon CPUs and 2GB RAM. They run Debian Linux (sarge) with the most recent Debian kernel, 2.4.27. We have another machine in the cluster running a 2.6 kernel, which allows us to use NPTL for threading instead of LinuxThreads. It runs OpenLDAP 2.1.30. The 2.4 -> 2.6 kernel move did not lower CPU consumption significantly, perhaps a few percentage points. > - the number of concurrent connections. Steady-state is about 600 or 700, but we sometimes see peaks of twice that. > - incoming request types: search, modify, add, etc. They're always under heavy search load with relatively few writes; I'm guessing one modify/add/delete every two or three seconds. > - incoming request pattern: how many requests per single bind(). We have two different patterns. The first binds, performs a single search that returns a single entry, then unbinds. The second type persistent and are long-lived. They bind, then issue hundreds or thousands of searches as requests come in to the supported service (i.e., one search for each e-mail message delivered, etc.) These connections can persist for several hours or even days. In our case with the lightweight listener code, I suspect a lot of the CPU consumption is in select(). An strace(1) of it yields heavy select() activity and not much else; just some small read()s, time(), sched_yield(), and kill() calls. john -- John Morrissey _o /\ ---- __o jwm@horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__
> In our case with the lightweight listener code, I suspect a lot of the CPU > consumption is in select(). An strace(1) of it yields heavy select() > activity and not much else; just some small read()s, time(), sched_yield(), > and kill() calls. What I am working on is to use a semaphore for load conditioning between a listener thread and worker threads. It reduces listener overheads (Marjory comes from select()) significantly when there is a large number of concurrent connections(128>). This patch will be submitted soon. Sang-Seok
moved from Software Enhancements to Development
> What I am working on is to use a semaphore for load conditioning between a > listener thread > and worker threads. It reduces listener overheads (Marjory comes from > select()) significantly > when there is a large number of concurrent connections(128>). This patch > will be submitted soon. The new patch for mitigating the select() overheads mentioned above is newly submitted to ITS which makes the lightweight patch of this ITS obsolete. Please check the following link, http://www.OpenLDAP.org/its/index.cgi?findid=3975 Sang Seok
changed notes
changed state Feedback to Closed
moved from Development to Archive.Development
See ITS#3975