[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Fwd: Using lmdb as a pure in memory store



Russell Haley wrote:
Thanks so much for your response and thanks for the link. I've been
pouring over that documentation off and on.

The developer currently doing the prototyping has said he ran into an
issue where the use of a very large key value caused the system to
slow down (he didn't give specifics and I didn't have time to ask). He
said it was a sparsely populated record set of about 100,00 rows but
he specifically used a large value for the key as we hope to support
UInt64 for keys (I don't know why, apparently we're indexing
sub-atomic particles?). I saw a setting for  MDB_MAXKEYSIZE which was
0 for computed, but I don't know what that means on an ARM (v6 - 32
bit) system. Would you have any idea what would cause the performance
hit he was referring too?

At a guess, you're slowing down because you're using 64bit values on a 32bit processor and once you exceed the range of a 32bit integer you need twice as many memory accesses to fetch/compare keys. Anyway, if you're using MDB_INTEGERKEY (which you should be, in this case) then you'll never get anywhere near the max keysize.

Sorry if my questions are weird, I'm trying to learn C and embedded
development through osmosis.

Anyway, on a personal note, I was thoroughly happy to get lmdb working
in Lua through

https://github.com/shmul/lightningmdb

and can't wait to try his Mule Round-Robin Database tool

https://github.com/shmul/mule

Awesome stuff.

Russ


On Thu, Sep 22, 2016 at 4:25 PM, Howard Chu <hyc@symas.com> wrote:
Russell Haley wrote:

Hello,

I wasn't fully subscribed when I sent this so I'll send it again.

Any input or reference hints would be great. I love reading manuals. :)


http://lmdb.tech/doc/


Russ


---------- Forwarded message ----------
From: Russell Haley <russ.haley@gmail.com>
Date: Tue, Sep 20, 2016 at 4:52 PM
Subject: Using lmdb as a pure in memory store
To: openldap-technical@openldap.org


Hi there,

We are currently evaluating in memory key value stores for ~100,000 -
200,000 records in an embedded system. I suggested lmdb but it is
being discounted for some reasons I thought I'd validate:

1) It is currently thought that a on disk file is REQUIRED for the
system. Does MDB_NOSYNC turn off the disk caching? Can it be run as a
pure in-memory database? Could the file not just be mapped to a
memdisk?


A file is required. Whether the file is on-disk or not is irrelevant. It can
of course be on a RAMdisk (or, in Linux, a tmpfs), which would then make it
pure in-memory.

MDB_NOSYNC turns off syncing, that's why it is named that.

2) Because these values can come very fast, that the use of a lock
file would cause delay and too much wear on the nand based disk (SSD).


No. That's not how the lock file is used.

I see a no locking option ( MDB_NOLOCK) that would stop a lock file
being written.


This option should only be used if you're implementing your own lock
manager. Hint - no matter what approach you use, any other lock manager you
use will be slower than LMDB's.

Again, another option would be mapping the lock file to
a memdisk to  handle that?


Yes.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/