Re: Non OpenLDAP use of LMDB

[Howard Chu <hyc@symas.com>]

Harry B wrote:
> Hello,
>
> I am planning to use LMDB to create a resonably large database, few TBs, >
> 500mil keys, on a Fusion IO flash storage.  Memory to storage ratio of the
> available hardware is about 1:10
>
> Assuming the caching of "5 to 10%" of most-frequently-accessed data is good
> enough for my use-case, is this a valid/legitimate use of LMDB ? Or am I using
> the wrong tool for the job?

The information you've provided isn't relevant to answering the question. The 
most important question is, what is your read/write ratio? LMDB is designed 
for read-heavy workloads. We've gotten good performance on it even with a 1:50 
memory to storage ratio, but that's with an 80:20 read:write ratio.

> My other choices are RocksDB (haven't looked at it) or Postgres (using a
> limited subset of features), the latter mainly because we already use it
> across the company.

My tests shows that RocksDB uses about 3x as much memory as LMDB for a given 
workload. Postgres is (at least) an order of magnitude larger/slower. If these 
are your only two alternatives, I'd guess that LMDB will be the best choice.

But asking questions like this is pointless. The only way to get a meaningful 
answer is to prototype your workload and actually profile it for each of your 
potential solutions. Otherwise you're just guessing.
> Any advice is appreciated.
>
> Thanks
> --
> Harry


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/