[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: lmdb memory usage when writing lots of data with memorymap option

Luc Vlaming wrote:

Currently I am creating support for using LMDB as a new storage backend for
one of our products.
At the moment I am testing import bulk data into lmdb using transactions that
span a single record of 10MB. The total db size afterwards is 5GB. I also
tested with records of 1MB.

I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage
grows very quickly and linear with the amount of data stored into the
database. (memory usage ends up a bit higher than 5GB). when not using
MDB_WRITEMAP, however, memory usage stays very low. Does anyone have a
suggestion what might be wrong and what causes such different behaviour with
and without using the memorymap option?

There is nothing wrong. It is simply writing to the shared memory map.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/