[Date Prev][Date Next]
RE: scope not ok errors on very large databases (ITS#3343)
Thanks for the info. I'll try changing the parameters and reloading it next week sometime.
We have 259,423 direct children and about 690,264 total children under the problem node.
From: Howard Chu [mailto:firstname.lastname@example.org]
Sent: Friday, October 01, 2004 4:38 AM
To: Armbrust, Daniel C.
Subject: Re: scope not ok errors on very large databases (ITS#3343)
>Full_Name: Daniel Armbrust
>OS: Fedora Core 2
>Submission from: (NULL) (126.96.36.199)
>I have hit a problem where Openldap seems to stop using its indexes (or just
>does something wrong with them) and instead of quickly do a one level search, it
>ends up scanning the entire database, returning tons of "scope not ok"
>The interesting part is that this only happens on 1 particular node in my
>database. Renaming the node to something else has no affect. It also seems to
>be size related - as when we split up the ldif that we load the database from,
>such that we load this node in pieces, we can load some parts of this node, but
Exactly how large is this "node" ? You say it fails when you do a
one-level search under it - how many immediate children does it have?
There is a known limitation in back-bdb's index design; when any index
slot hits 65536 entries it gets converted from an explicit list of
entries into a "range". If the entries in this slot were not added in
sorted order, then the range may span a large portion of the database.
For example, assuming the slot size was 4, and you had an index slot
with entry IDs
if you added a new entry under this slot, entry ID 99, this index slot
would be converted into a range
which would include quite a large number of entries that really have
nothing to do with that slot.
You can tweak the slot sizes in back-bdb/idl.h BDB_IDL_DB_SIZE and
BDB_IDL_UM_SIZE and recompile. I believe UM_SIZE must always be at least
twice the DB_SIZE. You will also need to dump the database to LDIF
before making this change, and reload from scratch afterward.
Also, loading your database in sorted order will help minimize the
impact of this problem. I.e., make sure that all of the children of a
particular node are loaded contiguously, without other intervening
entries. This only helps when the DIT is relatively flat.
Originally back-hdb did not have this problem, although it does now
because it shares the same search/indexing mechanism.
-- Howard Chu
Chief Architect, Symas Corp. Director, Highland Sun
Symas: Premier OpenSource Development and Support