[Date Prev][Date Next]
On-demand page mapping in LMDB
The vl32 branch on gitorious has some tweaks for LMDB to mmap on individual
page accesses, instead of mmap'ing the entire DB file all at once. This is
primarily to support usage on 32 bit machines, where the address space is too
small to accomodate multiple DBs mapped in competely, all at once.
There's a significant performance hit since almost every page reference
results in an mmap system call. But it also has a much smaller memory footprint.
While the code passes simple tests, it is also broken in one significant
aspect: Currently, advancing a cursor to a new page causes the previous page
to be unmapped. If there are two cursors open on the same DBI in the same
transaction, it's possible that the other cursor will still be pointed to the
old page, and the unmap will break it. There's currently no check to prevent
this, and since read txns don't track all of their open cursors, there is no
way to check (unless we add refcounts to this array, which would be clumsy).
The solution I'm considering is to get rid of the per-txn array of mapped
pages that I added here, and just operate on the cursor stacks directly. In
that case, every cursor will have its own set of mapped read-only pages, and
nothing that happens on one cursor will have any effect on any other cursor.
(For read-only pages. For dirty pages, the usual cursor/page fixups apply as
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/