[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: py-lmdb

Volker Lendecke wrote:
On Sun, May 18, 2014 at 12:15:45PM -0700, Howard Chu wrote:
But even tdb only allows one write transaction at a time. I looked
into writing a back-tdb for OpenLDAP back in 2009, before I started
writing LMDB. I know pretty well how tdb works...

Okay, transactioned, safe writes are slow. True. But the
non-transactioned ones have significantly improved in the
very recent past. We get a lot by mutexes (we had to find
out how badly the linux fcntl locks really suck ...), and
also by spreading the load from the freelist to the dead
records in neighboring hash chains. I don't have any
microbenchmarks, but larger-scale benchmark benefit a lot
from those two.

(And of course, fcntl only works for inter-process locking. We needed thread support, which also required mutexes.)

I would like to give lmdb a try in Samba, really. I see that
for 32-bit systems we will probably still need tdb for the
future (pread/pwrite in lmdb anyone in the meantime?  :-)).

That will require app-level buffer cache mgmt and lots of memcpys. Kinda defeats the design of LMDB.

The other blocker when I last took a serious look is that
crashed processes can have harmful effects. Has this changed
in the meantime with automatic cleanup and/or robust
mutexes? I know those might be a bit slower, but I would
love to offer our users the choice at least.

Hallvard has a test branch with robust mutex support, we need to look into merging it...

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/