Re: lmdb

On Sat, Apr 15, 2017 at 6:16 PM, Howard Chu <hyc@symas.com> wrote:

Patrick Hawley wrote:

I've been puzzled by lmdb's free list use. In some experiments, I've
been seeing that it sometimes grabs pages from the end instead of
using free list pages. This means that the number of pages used
tends to grow until it uses them all. I've tried increasing the map size
(to even quite a large value), and ensuring that write transactions are
not held open long but that doesn't seem to help.

Length of write transactions is irrelevant. Nothing in the LMDB documentation talks about length of write transactions; perhaps you need to reread the docs.

The issue is keeping a lot of read transactions open when writes are in progress.

Thanks for the comment.

I should have added that I'm not using any long-running read transactions;

all read transactions are shortlived. I've observed this effect to occur under

heavy load (many read transactions opened and closed during writes),

and during light load (few read transactions opened and closed during writes).

I did see in the docs that "While any reader exists, writers cannot reuse space in the database file that has become unused in later versions. Due to this, continual use of long-lived read transactions may cause the database to grow without bound" as well as the comments in lmdb.h. But that situation does not seem

to apply to this case where read transactions are closed quickly after opening.

Perhaps it will help to add that the memory use effect is more pronounced if I drop

one the subdatabases; and it is also more pronounced when the values are larger than page size, so requiring overflow pages.

Your reply seems to suggest that there is a problem when many read transactions are open-- whether or not each read transaction is short-lived.
If so, is there a limit to the number of read transaction that can be open

at one time before this problem occurs?

Regardless, do you have any suggestion to help diagnose/mitigate this problem?

I've turned on debugging and also looked at the traces, but haven't yet identified
a pattern that reveals an obvious mitigation strategy-- other than the
suboptimal strategy of dumping the whole database and refilling it from scratch.