Re: >1024 connections in slapd / select->poll

[Howard Chu <hyc@symas.com>]

Volker.Lendecke@SerNet.DE wrote:

> On Mon, Nov 15, 2004 at 02:47:34PM -0800, Howard Chu wrote:
>  
>
> > lists around. But on a heavily loaded server with thousands of active 
> > sessions, the fact is that your list of interesting descriptors is not 
> > static. Ultimately the server must iterate across all the thousands of 
> > descriptors, because the majority of them are probably active, and the 
> > server must insert and delete descriptors from the list continuously, 
> > because client sessions tend to come and go continually.
> >    
>
> All true, but the argument I think is that epoll has no inherent limits, and is
> optimal for many idle connections.
OK. But most servers where performance limits are being experienced have 
a higher proportion of active connections to idle connections. The most 
widely used protocols - http, smtp, pop, etc. have extremely short 
connection lifetimes.

> With thousands of active connections I think
> a central select/poll/epoll/whatever is broken anyway.
Yes.

> Only a SMP machine with
> a descriptor-passing architecture might help here.
>  
Descriptor-passing per se isn't necessary as long as you have descriptor 
inheritance and shared memory. In that case you only need to provide one 
master process performing select()s on the listener sockets, and a pool 
of child processes inheriting those listeners. The child processes 
rotate through performing accept()s so that the live session sockets are 
only present in the children. You can spawn as many of these child 
processes as necessary to keep the total number of active descriptors in 
each process down to a manageable limit, using shared memory to 
coordinate the rest of the program.

But yes, to your main point, a software design that handles tens of 
thousands of descriptors in a single process is broken. I think the fact 
that epoll/kqueue can scale to tens of thousands of descriptors is 
somewhat academic, since you probably can't do anything useful on those 
connections in a single process. A typical thread subsystem will let you 
use a couple thousand threads, nowhere near enough to service every 
descriptor with a low latency.

(At this point I'm just carrying on this conversation out of theoretical 
interest. I've already decided to look into migrating to libevent 
http://www.monkey.org/~provos/libevent/ to take advantage of these newer 
mechanisms.)

--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support