Re: LMDB data growth - overflow pages

[Christian Sell <christian@gsvitec.com>]

Hi,

as far as my scenario goes, I shortly afterward detected serious errors in my
setup that were responsible to the buildup. There is an article on the symas
website
(https://symas.com/understanding-lmdb-database-file-sizes-and-memory-utilization/)
that adresses the question/issue of database growth, I assume you have read
that. Crashes sould of course not happen, but what exactly do you mean by
"crash" anyway?

regards,
Christian

> salesman1 IIS Advertising <sales-iss@outlook.com> hat am 26. Juni 2016 um
> 00:59 geschrieben:
> 
> 
> Re: I HAVE THE SAME PROBLEMS, TOO !
> 
> 
> from your comments
> 
> http://www.openldap.org/lists/openldap-technical/201511/msg00201.html
> 
> 
> 
> I noticed the following using lmdb. Tested version is up-to-date 0.9.18, from
> 2016.
> 
> 
> Database grows much larger than expected in many use cases, and eventually
> crashes, also was
> 
> not a well-behaved database when simple, missing or misbehaved client code
> executes it.
> 
> . 
> 
> Do you think the same ?
> 
> 
> Maybe lbmd is was fine-tuned for other especific needs, or is a toy database,
> or a complicated
> 
> case of application in our scenario. Unfortunatelly is not like Redis yet.
> 
> We can not throw anything special on it without some effort.
> 
> 
> 
> We really want do use lbmd, it's fast, but can't get it to work as expected.
> Lmbd may be incomplete,
> 
> buggy for (the common coder) or ||  (the experienced coder) who tries do use
> it for other needs
> 
> than designed. So it's now a dream but not easily realistic. Needs an expert.
> 
> 
> 
> 
> 1. Database size grows 3x the data we put in, than stops growing - when
> records (key+data) from 410 to 4010 bytes are added.
> 
> -> and consumed much more space than expected for 5000*400 bytes =  2.000.000
> bytes
> 
> 
> 2. Second, when we put key+data > 4096 bytes, database grows much larger and
> crashes a lot of times, with a number
> 
> of page overflows shown on 'mdb_dump ./testdb'
> 
> 
> We made keys ("k:%d", i%5000)  for (i=0; i < 1000000; i++)  // so there are
> 5000 keys, else is overwriting;
> 
> Data is random ascii or fixed memset( 'x',400 )
> 
> 
> Program name: sample2
> 
> We are overwriting old values AND NOT appending, but it grows anyway. Larger
> data than 4096 bytes per key-value
> 
> seems impossible or non-sense at this time. Are we using wrong flags or
> program structure ?
> 
> 
> 
> trials - first run goes fine: #./sample2
> 
> 
> [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l
> Start
> Init done
> written 100000 pairs of len=400 bytes
> -rw-r--r-- 1 root root 2584576 Jun 25 21:43 ./testdb/data.mdb
> -rw-r--r-- 1 root root    8192 Jun 25 21:43 ./testdb/lock.mdb
> [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l
> Start
> Init done
> written 100000 pairs of len=400 bytes
> -rw-r--r-- 1 root root 5173248 Jun 25 21:43 ./testdb/data.mdb
> -rw-r--r-- 1 root root    8192 Jun 25 21:43 ./testdb/lock.mdb
> [root@ip-172-31-13-74 liblmdb]# ./sample2; ls ./testdb/* -l
> Start
> Init done
> written 100000 pairs of len=400 bytes
> -rw-r--r-- 1 root root 7761920 Jun 25 21:43 ./testdb/data.mdb
> -rw-r--r-- 1 root root    8192 Jun 25 21:43 ./testdb/lock.mdb
> [root@ip-172-31-13-74 liblmdb]#
> 
> 
> 
> Looks like it's not replacing old keys and have a page/data size problem,
> 
> 
> Any suggestions ?
> 
> 
> REGARDS !!
> 
> Fabio Martinez - Buenos Aires and São Paulo
> 
> gcc developer
> 
> Brazil
> 
> 
> --
> 
> 
> 
> 
> Hello,
> 
> I am trying to use LMDB to store large (huge) amounts of binary data which,
> for the reason of limiting memory footprint, are split into chunks. Each chunk
> ist stored under a separate key, made up of [collectionId, chunkId], so that I
> can later iterate the chunks using a LMDB cursor. Chunk size is configurable.
> 
> During my tests, I encountered a strange scenario where, after inserting some
> 2000 chunks consisting of 512KB each, the database size had grown to a value
> that was roughly 135 times the calculated size of the data. I ran stat over
> the db and saw that there were > 12000 overflow pages vs. approx. 2000 data
> pages. When I reduced the chunk size to 4060 bytes, the number of overflow
> pages went down to 1000, and the database size went down to the expected
> number (I experimented with different sizes, this was the best result). I did
> not find any documentation that would explain this behavior, or how to deal
> with it. Of course it makes me worry about database bloat and the
> consequences. Can anyone shed light on this?
> 
> thanks,
> Christian
>