[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: lmdb VL32 high virtual address usage

Jeremiah Morrill wrote:
This sort of coincides with ITS issue 8347 I just reported.

With ldmb VL32, I’ve noticed that it can eat up a lot of memory space.
Sometimes ~500mb for an 800mb db after reading every record, one by one using
the same readonly transaction/cursor.  I found that modifying MDB_ERPAGE_SIZE
from 16384 to 256 brought virtual memory usage down substantially (~30megs).
I have not noticed any adverse effects from this change or MDB_MAP_FULL.  The
mapped address spaces to seem to get deallocated on the close of the
environment, not the transaction, which sounds like its being cached in the
env->me_rpages for reasons explained in mdb_rpage_get comments.

Is the mem map cache supposed to be this aggressive by design,

Yes. It is designed to use up to 1GB.

 or just because
this area of the code is so new...or simply a usage error on my part?  If it’s
by design, could there be a consideration to make MDB_ERPAGE_SIZE a
#define’able value in the public lmdb.h so one could tweak the behavior?

You're welcome to tweak it in whatever way you see fit. MDB_VL32 is only ever used as a captive library, not linked in from a system package. How you build it is entirely up to you.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/