[Date Prev][Date Next] [Chronological] [Thread] [Top]

lmdb memory usage when writing lots of data with memorymap option



Hi,

Currently I am creating support for using LMDB as a new storage backend for one of our products. 
At the moment I am testing import bulk data into lmdb using transactions that span a single record of 10MB. The total db size afterwards is 5GB. I also tested with records of 1MB.

I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage grows very quickly and linear with the amount of data stored into the database. (memory usage ends up a bit higher than 5GB). when not using MDB_WRITEMAP, however, memory usage stays very low. Does anyone have a suggestion what might be wrong and what causes such different behaviour with and without using the memorymap option?

FYI: 
- there are no long lived reads so that should not be the problem.
- I am using the newest git version: ec97f49a6552a8ed599472d665ec7c16463b808c

Regards,
Luc Vlaming