[Date Prev][Date Next] [Chronological] [Thread] [Top]

Antw: Re: lmdb memory usage when writing lots of data with memorymap option

>>> Howard Chu <hyc@symas.com> schrieb am 15.01.2014 um 23:10 in Nachricht
> Luc Vlaming wrote:
>> Hi,
>> Currently I am creating support for using LMDB as a new storage backend for
>> one of our products.
>> At the moment I am testing import bulk data into lmdb using transactions 
> that
>> span a single record of 10MB. The total db size afterwards is 5GB. I also
>> tested with records of 1MB.
>> I noticed a very odd thing: when using the MDB_WRITEMAP option, memory usage
>> grows very quickly and linear with the amount of data stored into the
>> database. (memory usage ends up a bit higher than 5GB). when not using

Maybe for the future make a difference between virtual memory usage and real (resident) memory usage. Especially for Linux this makes a big difference, because a malloc(1GB) actually does not consume any memory until it is actually used.

There's also the "pmap" Utility that can show the detailed difference. For example my small (bdb) slapd has:
# pmap 3668
3668: slapd
00007f601a7f4000   8192K    120K    120K    120K      0K rw-p [anon]
00007f603db4c000  18320K    184K    184K     84K      0K rw-s /var/lib/ldap/__db.003
Total:           808004K  29768K  28657K  27016K  32040K

So of 800MB virtual memory there is only 30MB actually in use...

>> MDB_WRITEMAP, however, memory usage stays very low. Does anyone have a
>> suggestion what might be wrong and what causes such different behaviour with
>> and without using the memorymap option?
> There is nothing wrong. It is simply writing to the shared memory map.

Off-topic: I can remember a statement of the late 80ies where a programmer claimed the 32-bit address space is so large that one does not have to care about garbage collection in virtual address space; just use new addresses. I think even with 64 bit one should always try not to waste address space.


> -- 
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com 
>    Director, Highland Sun     http://highlandsun.com/hyc/ 
>    Chief Architect, OpenLDAP  http://www.openldap.org/project/