Re: Default threads value in OpenLDAP

[Howard Chu <hyc@symas.com>]

Pierangelo Masarati wrote:
> I'd like to mention a different scenario, where proxy databases need to 
> deal with a mix of slow and fast targets.  What we experienced is that 
> concurrency can be heavily penalized by this sort of mix of targets when 
> few threads are available, because inevitably operations affecting slow 
> targets eat up resources that remain idle in ldap_result() while they 
> could be used to deal with fast target related operations, while now 
> they have to remain pending.  In some cases, we had to use up to 128 
> threads (we even experimented with 512) with big waste of system resources.
>
> A solution could be to redesign the proxies so that request and response 
> are handled independently by different threads, using "client" 
> connections that detect activities on persistent connection handlers 
> towards the targets.  Together with a customer, we quickly prototyped 
> something like this (back-aldap, standing for "asynchronous ldap"), 
> which is just a toy right now, but it showed some potential.
>
> In the meanwhile, I'd like slapd to maintain as much efficiency as 
> possible when running with lots of threads.

One of the things I've been considering is to extend the thread pool 
manager to keep track of how often a particular thread cycles through 
the pool. This would allow us to differentiate long-running operation 
threads from the cheaper/faster operations. Then we could automatically 
spawn a new thread to compensate for long-running operations 
monopolizing an existing thread. (The long-running thread can just exit 
when it finally completes.) The net effect would be that we would have a 
target number of configured threads X, and up to 2X if there are X slow 
operations running at once.

--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/