Re: (ITS#8431) LMDB: Access newly opened database from another transaction

[h.b.furuseth@usit.uio.no]

On 30. mai 2016 18:08, Juerg Bircher wrote:
> If I got it right it is all about protecting against invalid DBIs
> which can happen for example if someone closes database "A" which was
> assigned DBI "5" and later the database "B" is opened and the free
> slot 5 is then assigned to "B". So we could be misled using DBI "5"
> having in mind database "A".
>
> So why don't we extend the MDB_DBI by adding the sequence in the lower
> or higher bits. (...)

Ugh. I didn't expect all my words to miss that badly.  Yes, you caught
one issue.  There are more.  The central point is what I wrote here:

 >> Multi-threaded use of structures, like the ones behind DBIs, must be
 >> synchronized.  But DBIs and mdb_dbi_open() are not mutex-protected,
 >> nor are read-only txns (after the per-thread setup).  So it's up to
 >> the user.  And LMDB mostly can't catch users at less than txn-sized
 >> threading bugs - like the DBI workarounds people keep trying.

I shouldn't really be the one to answer this, Howard might answer
differently if he didn't just fetch a flamethrower.  But anyway:

It's about keeping a clean model of what is going on, which helps
understanding and correctly coding both LMDB itself and user programs.
When you don't work against the model, anyway.

It's about keeping LMDB's internal data structures consistent.  LMDB
must know what the heck is going on when it manages and uses DBIs.
Things like ensuring that two threads do not run mdb_dbi_open() at
the same time, overwriting each others' work and creating a mess.

It's about LMDB being able to know it delivers correct results, at
least if the user doesn't work too hard at thwarting it.

This usually means serializing various parts of the code using locks
or something similar.  But for the sake of speed and simplicity, LMDB
has no relevant locks.  It relies on the user to lock and has some
code to catch some locking errors.  One key word is "tries" - without
locks or other expensive code, it can't always catch them even when it
does try.  So DBI errors are not "this DBI is broken, so drop/refresh
it" but "your program is buggy, so fix it".

And it's about living with the design choices and tradeoffs which
LMDB has made, e.g. not slowing down read-only txns for anything
like locking or extensive error checking.


So anyway, this ITS is about changes to LMDB which will not happen.
Instead it reminded me to add some checks for user errors with DBIs.

-- 
Hallvard