Re: (ITS#3611) Index clustering patch for fast slapadd

[hyc@symas.com]

Howard Chu wrote:

> Why are you explicitly storing the number of IDs in the IDL, when the 
> BDB library already maintains that counter itself?

Never mind. I see that since an ID cluster is written as a single 
element, the BDB counter does not apply.

The idea of grouping writes and buffering/caching the groups is good, 
but this approach still creates a fair amount of transaction log 
overhead. I.e., even if only a few elements of a cluster are changed, 
the entire cluster must eventually be flushed back to the database, and 
since the cluster is fairly large, it will result in large transaction 
log records.

If instead, you retained the current use of individual IDs, and buffered 
a list of deltas such that only individual IDs needed to be flushed back 
to the database, the transaction log overhead would remain small. 
Meanwhile, flushing a large group of writes to an individual index all 
at once would still minimize seeks and thrashing of the BDB cache.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support