Re: (ITS#8431) LMDB: Access newly opened database from another transaction

[juerg.bircher@gmail.com]

On 27/05/16 15:11, "Hallvard Breien Furuseth" <h.b.furuseth@usit.uio.no> wrote:

>This works as intended.  Each transaction is atomic, and shall see
>neither DBIs nor data which were committed after the transaction began.
>Cross-txn mdb_dbi_open() is in any case hairy to get right, we screwed
>it up several times before arriving at the now-obvious semantics.
>

It is fine that newly opened database handles shall not be seen by
other transactions as long the transaction runs.. But once the
transaction which opened the handle is committed it should be valid
for any other transaction. This would greatly simplify lazy database
opening of a multithreaded application.

The suggested patch does exactly implement that behaviour. I believe
it does not break anything. Isolation must be guaranteed for the data
but not necessarily for the database handles.

>An application can open and cache its DBIs in a separate startup
>transaction, then commit it and start new transactions.

Yes this would be a simple approach to get around that problem. But I
do not like it as with each opened database the overhead increases
especially if not using my suggested improvements in ITS#8430.

See my measurements with the improved lmdb and the original lmdb.

The test always performs 1000000 iterations on one database while
having 1 database open in the first run and having all databases open
in the second run.

lmdb improved (renew, reset)

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 10] open databases in [0.31975] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [10 of 10] open databases in [0.20350] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 100] open databases in [0.25845] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [100 of 100] open databases in [0.29663] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 1000] open databases in [0.28590] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1000 of 1000] open databases in [0.42897] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 10000] open databases in [0.30004] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [10000 of 10000] open databases in [1.68870] seconds

lmdb improved (begin, commit)

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 10] open databases in [0.36538] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [10 of 10] open databases in [0.35923] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 100] open databases in [0.34858] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [100 of 100] open databases in [0.39294] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 1000] open databases in [0.40222] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1000 of 1000] open databases in [0.54752] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 10000] open databases in [0.78595] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [10000 of 10000] open databases in [2.32414] seconds


lmdb original (renew, reset)

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 10] open databases in [0.18597] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [10 of 10] open databases in [0.21572] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 100] open databases in [0.24173] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [100 of 100] open databases in [0.46497] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 1000] open databases in [0.27127] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1000 of 1000] open databases in [4.18579] seconds

[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [1 of 10000] open databases in [0.30128] seconds
[1000000] iterations (renew, cursor_open, cursor_close, reset) on 1
database with [10000 of 10000] open databases in [49.35048] seconds

lmdb original (begin, commit)


[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 10] open databases in [0.45135] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [10 of 10] open databases in [0.38990] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 100] open databases in [0.47890] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [100 of 100] open databases in [0.69917] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 1000] open databases in [1.84908] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1000 of 1000] open databases in [5.88098] seconds

[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [1 of 10000] open databases in [22.12491] seconds
[1000000] iterations (begin, cursor_open, cursor_close, abort) on 1
database with [10000 of 10000] open databases in [74.53854] seconds

>
>--
> Hallvard

Regards
Juerg