[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: question about use of buffers in lmdb

Luke Kenneth Casson Leighton wrote:
howard, et al, hi,

i notice that LMDB stores data in an "append only" manner (from the
Q&A).  i was thinking of recommending that in python the buffers be
handed around with *direct* and persistent pointers to the data within
the LMDB shm/memory-map.

the reason that i believe this will be successful is because the
circumstances are rather unusual: once data is entered (added) it is
*never* overwritten, updated, changed, or modified.  there simply is
no need [in this application].  so there will only ever be insertion
(once and only once) or controlled and monitored deletion (by age of

under these circumstances is it perfectly reasonable to pass around
the pointers to the buffers returned from value look-ups
*indefinitely*?  if so that would be awesome.

Any time a new value is inserted into an existing page, that page's address will change due to COW. Any time an old page gets sufficiently old that no other txn points to it, it becomes eligible for reuse - thus completely unrelated data could get onto it. So unless you're keeping very close tabs on your application's key+data sizes and know how full a page got, you can't rely on particular addresses staying valid.


  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/