[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LMDB stuff

Le 2/3/14 10:14 PM, Howard Chu a Ãcrit :
> Was chatting with Emmanuel Lecharny (who is currently working on
> Mavibot for ApacheDS, an MVCC backend similar to LMDB) and had an
> interesting realization: we can avoid the current issue of long-lived
> reader txns preventing page reclamation.
> The key point is this - currently for any txn, we store in the freeDB
> a list of the pageIDs that were freed in that txn (due to being copied
> or deleted). All we need is to know that any of these pages has been
> copied *twice* since the txn of the outstanding reader. 

> At that point, any such page is so new that the reader would never
> have seen it, and if no other readers are outstanding then the page
> can be safely reclaimed.
> Currently mavibot maintains this revision history in a dedicated Btree
> (much like our freeDB but with more info contained). I'm thinking,
> since we're already going to add a txnID to every page's page header,
> we can simply add a 2nd txnID, recording the txnID of the previous
> change to this page's ancestor. Then, any page where this prevTxnID is
> >= the outstanding reader's txnID can be reclaimed.
> Still thinking about the actual implementation of this, it may make
> more sense to store the prevTxnID in the freeDB than in each page
> header. Ideally we want to be able to grab a chunk of pageIDs
> unambiguously, instead of having to iterate thru each page and read
> its header to determine if it's safe.

You have two use cases :
o when you are adding new revisions, and no new read occurs (this is the
easy use case : you can get rid of any pages being copied twice)
o when a read txn is released (now, you have to go through all the pages
that have been copied before this read txn down to the next read txn,
and remove all pages which have been copied twice, and go up to the next
read or write txn, to do the same work).

Emmanuel LÃcharny