Re: lmdb VL32 high virtual address usage

[Howard Chu <hyc@symas.com>]

Jeremiah Morrill wrote:
> This sort of coincides with ITS issue 8347 I just reported.
>
> With ldmb VL32, I’ve noticed that it can eat up a lot of memory space.
> Sometimes ~500mb for an 800mb db after reading every record, one by one using
> the same readonly transaction/cursor.  I found that modifying MDB_ERPAGE_SIZE
> from 16384 to 256 brought virtual memory usage down substantially (~30megs).
> I have not noticed any adverse effects from this change or MDB_MAP_FULL.  The
> mapped address spaces to seem to get deallocated on the close of the
> environment, not the transaction, which sounds like its being cached in the
> env->me_rpages for reasons explained in mdb_rpage_get comments.
>
> Is the mem map cache supposed to be this aggressive by design,

Yes. It is designed to use up to 1GB.

 or just because
> this area of the code is so new...or simply a usage error on my part?  If it’s
> by design, could there be a consideration to make MDB_ERPAGE_SIZE a
> #define’able value in the public lmdb.h so one could tweak the behavior?

You're welcome to tweak it in whatever way you see fit. MDB_VL32 is only ever 
used as a captive library, not linked in from a system package. How you build 
it is entirely up to you.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/