[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LMDB on Ramdisk (tmpfs)

Luca Foppiano wrote:
Dear everybody, I'm using one of your component (LMDB) via a java JNDI bindings implementation (https://github.com/deephacks/lmdbjni) and I'm having an issue when I deploy my LMDB file on a tempfs filesystem in RAM. The issue do not occur when the LMDB files are stored on a "normal" filesystem. When the data is in the tempfs ramdisk all the allocated memory ends up being in the Dirty area (it has not been written back to the Filesytem). Here an example using the ramdisk: 

7ce320000000-7cfc20000000 r--s 00000000 00:26 2459
/ramfs/nerd/data/db/db-en/entityEmbeddings/data.mdb
Size:           104857600 kB
Rss:             1255680 kB
Pss:             1255680 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:   1255680 kB    <---
Referenced:      1255680 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd sh mr mw me ms sd

and here an example without:

7ca4fc000000-7cbdfc000000 r--s 00000000 fd:00 11154951
/data/workspace/shared/nerd-data/db/db-en/entityEmbeddings/data.mdb
Size:           104857600 kB
Rss:              838124 kB
Pss:              838124 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    838124 kB       <----
Private_Dirty:         0 kB
Referenced:       764872 kB
Anonymous:             0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd sh mr mw me ms sd
According to my understanding the memory is dirty when 1)there are open transactions, 2) the data has not been written back to the filesystem
Your understanding is incorrect. Dirty pages remain dirty until they are written to stable storage (e.g., disk). A tmpfs/RAMdisk has no stable storage, all of its pages reside only in RAM. That's the point of a RAMdisk.
What I don't understand is why there is a difference between filesystem and ramdisk? Is there any reason? The application (listed above) is not writing on the lmdb, but just reading (using reading transaction). Thank you Luca
Using tmpfs is a waste of RAM. Just use LMDB on a regular filesystem and let the system's pagecache manager take care of memory. 

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/