[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#8430) Improved handling for large number of databases

On 28/05/16 22:08, Juerg Bircher wrote:
> I understand that it is a tricky thing. But maybe the lazy init is
> still feasible? (...)

Right, lazy init is feasible.  Just keep cross-txn DBI stuff out of
it, I'll take your latest variant to ITS#8431.

Copied from a private mail:
> Maybe a good way in between could be to memcpy txn->mt_env->me_dbflags.
> (...) memcpy uses SSE instructions for x86_64.

Yes. If we make MDB_txn.mt_dbflags an uint16_t*, we can memcpy to it.
That costs <maxdbs> bytes/txn + one more reserved env flag bit in
PERSISTENT_FLAGS: Reserve DB_UNUSED, and keep it always set in
me_dbflags so the memcpy will set it in mt_dbflags.

Or maybe your code can be used without initial memcpy and I'm just
paranoid.  The one problem I can think of is user errors - people
coming up with workarounds like your cross-txn DBI stuff.  LMDB tries
to catch some of that, but maybe late pickup defeats some such checks.

I'll wait for Howard to say something at this point.

BTW, all this reminds me - I should add a test which tries to catch
mdb_dbi_open() in overlapping txns.  Got the code, but forgot about
it while pondering a good error code name.

> However md_flags would need to be moved out of the MDB_dbi struct

Nope.  Sub-DBs with duplicate data have md_flags which do not
correspond to anything saved in env->me_flags.  Your lazy-init code
needs to use the memcpyed flags instead of me_dbflags, that's all.

> We should consider to align on 16 byte boundary for best performance.

It's already aligned.