LMDB file compaction

[Jens Alfke <jens@mooseyard.com>]

I have some questions/concerns about LMDB file size and the management of free space inside the file. These aren't things I've actually run into yet (I'm still in the investigation phase) but I know LMDB will grow the file up to the maximum map size and not shrink it.

First off, I'm aware that this is an FAQ. I've read the documentation, and I think I understand LMDB's behavior fairly well. I've also read this 2015 post* by Howard Chu:

> This is ultimately the question you must answer - how much space will you actually allow your app to use? Once you have that figure, configure it once and forget about it. It is *STUPID* to ask the OS to allocate space for you and then ask it to deallocate space repeatedly. You wind up fragmenting the disk and killing performance. Either you have enough disk space for the app's needs, or you don't.

This is a fine answer for a server-side use case (which is of course what LMDB was designed for.) But I know from experience that server-side programmers often forget that not everything runs on servers...

It is *not at all* a good answer for most client-side uses, where storage is shared by many different applications from different vendors, the user cannot be required to pre-configure how much space an application will need, and an application generally has no idea how much it'll be asked to store.

It's *especially* non-applicable to mobile applications (the use case I'm investigating.) Phones and tablets are very often storage-constrained, and it's not kosher for an app to be holding onto a lot of free space it's not using. I'm sure everyone reading this has had the experience of trying to free up storage space on their phone by going into various apps and deleting lots of photos / messages / songs / whatever. In the case of an LMDB-based app I would expect that deleting content from the database would return storage space to the operating system. If it didn't, my next step would be to delete the app.

I'm aware that an LMDB database can be compacted by writing a copy. Unfortunately this requires that the disk have free space equal to the current used space of the database, and that's often not the case in the situations where a user most needs to free up space. It would be more efficient to compact the file in place by moving in-use pages at the end of the file into holes, and then truncating the file.

Alternatively, if the filesystem supports sparse files then LMDB could mark free pages as 'holes' in the file; does it do that? But I'm not sure if the filesystems on iOS or Android support this. (I know APFS has sparse file support, but last I checked it did not have a system call to punch a hole in a file.)

—Jens

* https://www.openldap.org/lists/openldap-technical/201511/msg00102.html