(ITS#8152) mt_flags & MDB_TXN_ERROR is never reset when MDB_MAP_FULL occurs

[dw@botanicus.net]

Full_Name: David Wilson
Version: HEAD
OS: 
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (81.157.85.186)


Since the me_txn0 heap allocation is cached across write transactions, when any
write transaction fails with MDB_MAP_FULL and causes MDB_TXN_ERROR to be set in
me_txn0->mt_flags, it becomes impossible to try another write transaction
without reopening the environment, since that bit is never unset anywhere.

A simple way to trigger this is attempting to put any value larger than the map
size, at which point the environment will always return MDB_BAD_TXN for
subsequent write transactions.

While MAP_FULL and insufficient validation of input by the application are
pretty show-stopper issues already, I doubt wedging the environment state is the
most robust response LMDB could muster in this case.

It seems MDB_TXN_ERROR wants to be unset in either mdb_txn_abort() or
mdb_txn_reset0(). Taking into account the possibility of a parent write
transaction also existing, I'm not sure where and how exactly the bit should be
unset.