startup costs was Re: filter preprocessing for performance improvement

[Eric Irrgang <erici@motown.cc.utexas.edu>]

I have a big problem whenever I have to restart a server.  Obviously, the 
first search for objectclass=* is going to enumerate the whole directory. 
The problem is that even an anonymous user can cause the server to execute 
this search on the backend even though the ACLs and limits will keep them 
from getting any results.  All it takes is a few poorly configured client 
applications to do some sort of poll and I have connections hanging for 
half an hour until the first objectclass=* search finishes.  I run out of 
threads and every one of them is constantly trying to get CPU time.

What I currently do is to keep a machine from being accessible by taking 
it out of the load-balancer's rotation for the half hour or so that it 
takes for me to do a search for objectclass=*, but I figure there has got 
to be another way.  I have both eq and pres indexes on objectclass.  It's 
just that I have a very big directory.

I'm not trying to speed up the objectclass=* search.  I'm trying to figure 
out how to keep it from impacting the server's responsiveness when it is 
being performed under circumstances where no entries will be returned, 
such as when sizelimits or ACLs (which are evaluated at the frontend after 
the backend has performed the operation, right) will block things.  Any 
suggestions?

One thing I just thought of would be to have a single entry that would 
always be accessible to any searcher and then set 'limits anonymous 
size=1'.  Would that cause the backend operation to be canceled once the 
first entry were returned?  That might save me something.

On Fri, 2 Mar 2007, Howard Chu wrote:

> (objectclass=*) is already a no-op in back-bdb/back-hdb. I made that 
> optimization back in November 2001.
>
> So yes, it's a nice idea, been there, done that.
>
> > I have a problem in that the first time someone performs a search for 
> > 'objectclass=*' after slapd is restarted, the server is really bogged down 
> > for a while.  Once the search has completed once, this is not a problem. I 
> > assume that's due to the IDL cache.  However, I currently have to keep the 
> > server unavailable after restarting slapd for upwards of half an hour while 
> > I do an 'objectclass=*' search the first time.
>
> Changing the behavior of (objectclass=*) isn't going to make any difference. 
> If you specify a filter that is too broad, then slapd has to trawl through 
> every entry in the DB because every entry satisfies the condition.

--
Eric Irrgang - UT Austin ITS Unix Systems - (512)475-9342