Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)

[hyc@symas.com]

Markus Junginger wrote:
>     Thanks for the report. Unfortunately your suggested fix means the rpage
>     test will always be skipped on read-only cursors, which will break the
>     rpage tracking if the txn is still alive.
> 
> 
> Could you maybe explain a bit about rpages? Why does mdb_cursor_close need to 
> unref pages, but not mdb_cursor_renew? If closing a transaction does not clean 
> up rpages, I would have expected mdb_cursor_renew would have to do. And why 
> does setting MDB_VL32 affect rpages in the first place? To my understanding, 
> MDB_VL32  is about using 64 bits also on 32 bit CPUs, so a differences on page 
> handling seem surprising at first sight.

Think about how LMDB works.

LMDB was originally written for use on 64bit CPUs, with 64bit virtual address 
spaces. It expects to map an entire DB into virtual memory using a single mmap 
call.

Obviously this cannot be done when trying to use a DB larger than 2GB on a 
machine with only a 32bit virtual address space.

When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit 
into a single mmap. It must be mapped in chunks, and the chunks must be 
mapped, unmapped, and remapped on demand. The rpages thus need to be tracked, 
to know which are currently mapped. None of this rpage tracking exists in the 
native non-VL32 build because it's completely unnecessary.

Also note that the MDB_VL32 feature is not officially released. You're free to 
use it if you want, but there is no expectation that it is suitable for 
general use. When you use unreleased features, you're expected to know how to 
support yourself. When you use LMDB at all, you're expected to know how 
virtual addresses work.

Ideally, you should quit using obsolete 32bit CPUs.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/