RE: Questions...

["Howard Chu" <hyc@highlandsun.com>]

> -----Original Message-----
> From: owner-openldap-devel@OpenLDAP.org
> [mailto:owner-openldap-devel@OpenLDAP.org]On Behalf Of Quanah Gibson-Mount

> --On Tuesday, January 27, 2004 2:35 PM -0500 Jong
> <jongchoi@OpenLDAP.org> wrote:

> > The difference between ldbm and bdb became significant when
> > indexing came into the picture.
> > So this may be a sign that we should improve the bdb indexing system
> > further.
>
> My data loads always find indexing to be the biggest
> bottleneck with BDB.
> Our systems that index multiple things went up two hours in
> indexing time
> between 2.1 & 2.2 with BDB (from 7 hours to 9 hours).

Well now that you mention it... The index generation logic is the same
between ldbm and bdb. The only difference is that ldbm uses Btrees for
everything by default, and back-bdb uses hashes for the indexes. When I
recompiled back-bdb to use Btrees the load time went down from 18m to 12m:
2.2 hdb (no txn logging)
691.310u 16.220s 12:26.53 94.7% 0+0k 0+0io 76127pf+0w
total 583102
-rw-r--r--   2 hyc      users         208 Jan 27 02:32 DB_CONFIG
-rw-------   1 hyc      users           8 Jan 27 18:30 __db.001
-rw-------   1 hyc      users    29724672 Jan 27 18:42 cn.bdb
-rw-------   1 hyc      users    131985408 Jan 27 18:42 dn2id.bdb
-rw-------   1 hyc      users    431702016 Jan 27 18:43 id2entry.bdb
-rw-------   1 hyc      users     1343488 Jan 27 18:42 objectClass.bdb

Apparently growing a hash table on the fly is very expensive. The test008
seems to run about one second (10-11%) faster now as well.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support