Re: slapd index and db errs crashing frequently (ITS#309)

[ngs@elok.com]

Hi Kurt,

Thanks for getting back to me - responses below.

"Kurt D. Zeilenga" wrote:

> >Recently upgraded to OpenLDAP 1.2.7 and Berkeley DB 2.7.7 to get
around memory
> >leaks.
> >Rebuilt the database (about 150,000 entries) from LDIF.
> >Now experience these errors in syslog and slapd restarting every
minute or so.
> >Previous dn.dbb file was 17MB, now it is 299MB.
>
> 1.2.7 generates both eq and sub indices for DNs.  Previous
> versions, due to a programming error, failed to generate
> the sub index.  This error was corrected in 1.2.7.
>   http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/back-
ldbm/init.c.diff?r1=1.3.2.6.2.6&r2=1.3.2.6.2.7
>
> The DN sub index is used to determine which entries to
> check for subtree searches.  Without the sub index, subtree
> search must check all entries.

Okay, I understand the file size now.

>
>
> With 150K entries, a DN sub index could be quite large,
> though ALLIDS blocks should keep it from getting out of hand.
> You can try:
>         1) lowering SLAPD_LDBM_MIN_MAXIDS
>         2) hack the "sub," out of back-ldbm/init.c dn
>         initialization.

ALLIDS has always been a problem for me due to the nature of the data I
work with. The data set is large and
there are generally very simple differentiating attributes, male/female,
age, interests etc. This meant that I
found virtually every useful attribute/value that is indexed exceeding
the ALLIDS threshold and my searches were
taking forever.

Something that would help me to solve this particular problem would be
to understand whether the server
re-orders the LDAP query terms at all in order to optimize the search,
or whether it follows the order of the
terms in the query string.

For example - if the Person value for objectclass has exceeded the
ALLIDs threshold, but the Male value for
gender has not, will a search (&(objectlass=Person)(gender=male)) search
every entry while
(&(gender=male)(objectclass=Person)) will take advantage of the index to
short circuit the search?


>
>
> >Sep 27 15:26:46 work slapd[18596]: ldbm_db_errcall(): ==>
> >/disk2/p2-overflow/slapd/db.new/dn.dbb: page 1347241258 doesn't
exist, create
> >flag not set
>
> Did you run out of disk space (or quota)?

Very unlikely, it's a 4GB disk at 26% capacity, the total size of the db
directory is 500MB. The db is the only
thing on there. I'm beginning to suspect something related to Berkeley
DB 2.7.7, but the earlier version
I worked with was 2.7.3 and they have pulled the 2.7.5 download off the
site.

You wouldn't have the 2.7.5 tar.gz file would you?

Incidentally, in the same situation as the above (slapd 1.2.7 + DB
2.7.7) I tried to use the readonly command in
the slapd.conf file and it didn't seem to have any effect - I could
modify entries at will. Is it meant to work?
Does it apply to the rootdn as well?

Thanks again.

Neil.