index corruption problem (ITS#1349)

[storm@mediaways.net]

Full_Name: Markus Storm
Version: REL_ENG_2 as of 2001-09-23
OS: Linux 2.4
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (195.71.100.155)


Hi all,

we're experiencing a strange case of index corruption:

We're doing searches for a certain uid every few seconds.
After a random period of time (may be a day or a week),
slapd starts responding very slowly to these queries. 
And ONLY these !

We've got subtrees 'small' and 'large' containing entries 
'uid=testuser, ou=small, o=ourbase' and
'uid=testuser, ou=large, o=ourbase'.

Whenever this occurs
ldapsearch -b "ou=small, o=ourbase" "uid=testuser" is still FAST, but
ldapsearch -b "ou=large, o=ourbase" "uid=testuser" is SLOW.


'Large' means there are more than <allids> (aka SLAPD_LDBM_MIN_MAXIDS)
entries within that ou, 'small' means there are less.
We've got several 'small' and several 'large' subtrees, each
of them shows this behaviour.

Some notes:
  
* several clients may simultaneously query server for these entries.
  We use threading.
* searches "uid=xyz", "uid=xyz*" or searches concerning other
  indices are not affected
* slapd starts consuming huge amounts of CPU
* restarting slapd doesn't help, but copying the uid index from
  a standby slave slapd (one that is not actively queried) makes
  the problem disappear


Any idea ?
Someone care to check the allidsthreshold code with regard to threading ?

thanks,
Markus