[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LMDB space amplification

Shu, Xinxin wrote:

I tried 1024 & 2048 bytes value size , but the space amplification
is

~2, so If I want to reduce this space amplification, what's your suggestion?
Have no good ideas at the moment. This is inherent to any DB design that stores metadata co-resident with the data; e.g. same situation exists with BerkeleyDB. If you can compact your data so that it fits in 4080 bytes, or can split it apart (if it also consists of a header and actual data), that would work. 

Cheers,
xinxin

-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com]
Sent: Wednesday, April 22, 2015 3:48 AM
To: Shu, Xinxin; openldap-technical@openldap.org
Subject: Re: LMDB space amplification

Shu, Xinxin wrote:

Hi list,

Recently I have integrated lmdb into ceph , got intial results ,  I
dump all records of lmdb, there are several different size key-value
pairs,

1.  key size : 45 byte, value size = 124 byte 2.  key size : 85 byte,
value size = 1187 byte 3.  key size : 57 byte, value size = 135 byte
4.  key size : 64 byte, value size = 182 byte 5.  key size : 33 byte,
value size = 4096 byte (used for real data, about 80 %)

this db contains about ~ 10000000 records, about 80 percent of records are the last key-value pairs (key size 33 byte, value size 4096 byte), I calculate size of  all key-value pairs, about 33GB , but on-disk size is about 63GB, so space amplification is ~ 2, is this amplification reasonable ? In which situation can lmdb get such large space amplification? How can I reduce this amplification?  If you need any other details , please email me , any tips will be appreciate.


LMDB pages have a 16 byte header. In a system with 4096 byte pages, only
4080 bytes are available for user data, so storing a data value of 4096 bytes will use 2 pages.




--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/