Re: back-bdb

[Marijn Meijles <marijn@bitpit.net>]

You wrote:
> >entry caching? back-bdb has transactions, so why not leave the caching (and
> >locking) to bdb?
> 
> I believe entry caching will again prove itself as being
> highly effective.  The performance balance may have shifted
> slightly but this is not due to transactions.  In particular,
I know, but as bdb without transactions is only free-threaded you needed a
locking scheme anyway. now that you have transactions, you have a choice.
(assuming that you maintain the order in which you access the various
tables)

> the conversion from database format to in-memory format should
> be significantly less than that in back-ldbm.  back-ldbm
> uses LDIF in id2entry, back-bdb uses BER.  Optimizing
> BER and memory management will shift the balance further.
yup. with ldif format it's definitely faster, I did some profiling on that.
I'm currently thinking about a format which is (almost) the same on disk
and in memory. Maybe I'll choose BER as well.

> However, it is likely that entry caching will still be
> effective.  We'll see (by performance analysis).
> 
Keep in mind that you can create nasty deadlocks between the backend
cache (entry locking) and the bdb locking. For example:
- don't put the lookup on entries you intent to write lock in the same
  transaction as the one you'll use to write them.
- during the lifespan of a backend write lock, don't use mutexes which
  encapsulate a bdb operation.
- ....

As the balance shifts further, you will reach a point where the extra
overhead of parsing an entry from the bdb becomes less than maintaining
an extra caching/locking layer.

Have you already thought about what kind of cache you want to use?

-- 
Marijn@bitpit.net
---
If someone has a mid-life crisis during a hide-&-seek game, does that
mean he automatically loses because he can't find himself?