[PATCH] back-sock

[Brian Candler <B.Candler@pobox.com>]

I have put together a new backend, 'back-sock', based heavily on back-shell. I 
invite people to look, test, and comment; if feedback is positive I'll submit 
it as a patch via ITS. The patch to openldap-2.2.26 is at 
http://psg.com/~brian/software/openldap-backsock.diff.gz

The intention behind this backend is to give something with the flexibility of 
back-perl and the simplicity of back-shell, but with high enough performance 
for real world applications rather than just prototypes.

back-perl is throttled by being single-threaded: each entry into the perl 
interpreter is protected by a mutex call, so that only one request at a time 
can be handled. back-shell is worse as it forks a new child for each request, 
as well as not supporting threading.

back-sock attempts to address this problem by passing requests down a 
unix-domain socket. You can then build a pool of worker processes (e.g. with 
Net::Server in perl) each of which handles one request at a time. A trivial 
sample is included as servers/slapd/back-sock/searchexample.pl

The protocol sent down the socket is the same as back-shell, with the addition 
of a blank line to indicate the end of the request. The worker process reads 
the request, sends its response in back-shell format, then closes the socket.

These worker processes should be persistent and can each hold open a 
persistent database handle, so you should get decent concurrency on 
applications which query a SQL database for example.

Sample performance results on a 2.8GHz Pentium4 running FreeBSD 5.4:
- searchexample.pl as the server backend
- 10 client processes in perl using Net::LDAP each to send 100 requests
- total real time to complete those 1000 requests:

7 seconds       -- back-sock, fresh TCP connection opened for each query
5 seconds       -- back-null, fresh TCP connection opened for each query
4 seconds       -- back-sock, each client holds open a single connection
2 seconds       -- back-null, each client holds open a single connection

This is with the clients and server all running on the same machine.

Unfortunately, I was once able to make openldap crash:

Assertion failed: (!FD_ISSET( s, &slap_daemon.sd_actives)), function 
slapd_daemon_task, file daemon.c, line 1604.
Abort trap (core dumped)

This makes me suspect I'm doing something in a non-thread-safe way. Any advice 
here would be much appreciated.

Missing bits: I haven't written a manpage. Also, I re-used LDAP_DEBUG_SHELL 
rather than defining a new debug flag for this module. See comments marked 
"FIXME" in the patch for other areas I think could be improved.

Regards,

Brian Candler.

--------
P.S. here's the test client program used above

#!/usr/bin/perl -w

use Net::LDAP;

my $uri = "ldap://localhost";;
my $concurrency = 10;
my $numhits = 100;   # per process

sub do_kid();

for (my $i = 0; $i < $concurrency; $i++) {
  my $pid = fork();
  if ($pid == 0) {
    do_kid();
    exit 0;
  }
}

for (my $i = 0; $i < $concurrency; $i++) {
  $pid = wait();
  print STDERR "Child pid $pid terminated\n";
}
exit 0;

sub do_kid() {
  for (my $j =0; $j < $numhits; $j++) {
  my $ldap = Net::LDAP->new($uri) or die "$@";
    $mesg = $ldap->search(
      base=>'dc=example,dc=com',
      filter=>'(objectclass=*)'
    );
    $mesg->code && die $mesg->error;
    #foreach $entry ($mesg->entries) { print STDERR $entry->dn,"\n"; }
  $ldap->unbind;
  }
}