DNCache in hdb stays over limits

[Ondřej Kuzník <ondrej.kuznik@acision.com>]

Hello,
I am using OpenLDAP with a large database using HDB backend that cannot
fit in the RAM. The bulk of the database is located under a single
suffix like this:
ou=data, dc=example, dc=com

In this example, I used a smaller and simpler database (~20000 entries)
and set DN cache to 1000 entries:
$ ldapsearch -LLL -b "olcDatabase={2}hdb,cn=config" olcDbCacheFree \
 olcDbDNcacheSize olcDbCacheSize

dn: olcDatabase={2}hdb,cn=config
olcDbCacheSize: 1000
olcDbCacheFree: 1
olcDbDNcacheSize: 1000

example entry:
dn: uid=6102328959,ou=data,dc=example,dc=com
objectClass: top
objectClass: account
uid: 6102328959


When querying the database for some of the entries, sometimes the
DN cache gets to a state when it contains all of them (and much more
than the configured maximum) even though no search is being processed.

Here are some observations I made thus far:

1. When performing a search with a base of "ou=data, dc=example, dc=com"
the following happens (no indexes are present on the database):

$ ldapsearch -b "ou=data, dc=example, dc=com" "*"

 i. First everything under "ou=data, dc=example, dc=com" is loaded into
    the DN cache (exhausting available memory and being swapped out if
    needed).

 ii. Then the server starts returning the matching entries, gradually
    freeing the DN cache.
    If a limit on the number of returned entries is set with the search
    (e.g. -z 10), DNs of the entries not returned are not freed until
    they are visited by a search in the future.

$ ldapsearch -LLL -b "cn=Database 2,cn=Databases,cn=Monitor" \
 olmBDBEntryCache olmBDBDNCache    

olmBDBEntryCache: 1000
olmBDBDNCache: 20324

2. When the same search is performed with a base higher ("dc=example,
dc=com") I get the expected behaviour:

$ ldapsearch -b "dc=example, dc=com" "objectClass=account"

The search starts returning the entries instantly and DN cache size
never exceeds 1002 entries (the entry limit "-z" changes nothing).


Is this behaviour expected? Because the slapd-hdb man page is a little
blurry on this. Can something be done on the configuration side to
prevent such behaviour or the only way out is to use BDB backend for the
database? I have tested this with both 2.4.20 and the latest cvs
snapshot.

Regards,
Ondřej Kuzník