[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: ldapsearch performance degradation

Howard Chu wrote:
Tim Dyce wrote:
Hi Howard,

Thanks for the help :D

We have been testing in ramdisk as well, to make sure that disk
thrashing is not the root cause.

If your searches are not running long enough to show up for profiling,
increase the number of second level entries until you get something you
can profile.

Ah, there's a bug in your script, it's creating the 2nd-level entries with the
wrong DN so the DB never had more than 250 entries.

Now I've fixed that and run again and can see the behavior you're talking
about. It's actually due to a particular design choice:

Ordinarily at each level of the tree we keep an index tallying all of the
children beneath that point. In back-bdb this index is used for subtree
searches and for onelevel searches. (In back-hdb it's only used for onelevel.)
However, as a write optimization, for the root entry of the DB, we don't
bother to maintain this index, it's simply set to "All entries". (Otherwise in
the back-bdb case, there is far too much write overhead to maintain this index.)

The problem is that "All entries" is actually a range 1 to N where N is the ID
of the last entry in the DB. (And 1 is the ID of the root entry.) As you add
entries, N keeps increasing, but 1 stays constant.

When you do a subtree search, every entryID in the corresponding index slot is
checked. In this case, with a subtree search starting at the root entry, you
will always be iterating through every ID from 1 thru N, even though many of
those IDs have been deleted, and it takes time for BDB to return "no such
object" for all the deleted IDs.

If you do all of your operations under a child entry instead of the database
root entry, the performance will be constant. I've already verified this with
a modified copy of your test. I can post it if you wish.

Just for sake of comparison with your original test setup from

I tweaked the DB_CONFIG parameters to speed up the ldapadds, otherwise it takes too long to run the test. Your cache sizes etc. were all too small...

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/

Attachment: newtest.tar.gz
Description: GNU Zip compressed data