Re: Fwd: Using lmdb as a pure in memory store

[Russell Haley <russ.haley@gmail.com>]

Thanks so much for your response and thanks for the link. I've been
pouring over that documentation off and on.

The developer currently doing the prototyping has said he ran into an
issue where the use of a very large key value caused the system to
slow down (he didn't give specifics and I didn't have time to ask). He
said it was a sparsely populated record set of about 100,00 rows but
he specifically used a large value for the key as we hope to support
UInt64 for keys (I don't know why, apparently we're indexing
sub-atomic particles?). I saw a setting for  MDB_MAXKEYSIZE which was
0 for computed, but I don't know what that means on an ARM (v6 - 32
bit) system. Would you have any idea what would cause the performance
hit he was referring too?

Sorry if my questions are weird, I'm trying to learn C and embedded
development through osmosis.

Anyway, on a personal note, I was thoroughly happy to get lmdb working
in Lua through

https://github.com/shmul/lightningmdb

and can't wait to try his Mule Round-Robin Database tool

https://github.com/shmul/mule

Awesome stuff.

Russ


On Thu, Sep 22, 2016 at 4:25 PM, Howard Chu <hyc@symas.com> wrote:
> Russell Haley wrote:
>>
>> Hello,
>>
>> I wasn't fully subscribed when I sent this so I'll send it again.
>>
>> Any input or reference hints would be great. I love reading manuals. :)
>
>
> http://lmdb.tech/doc/
>>
>>
>> Russ
>>
>>
>> ---------- Forwarded message ----------
>> From: Russell Haley <russ.haley@gmail.com>
>> Date: Tue, Sep 20, 2016 at 4:52 PM
>> Subject: Using lmdb as a pure in memory store
>> To: openldap-technical@openldap.org
>>
>>
>> Hi there,
>>
>> We are currently evaluating in memory key value stores for ~100,000 -
>> 200,000 records in an embedded system. I suggested lmdb but it is
>> being discounted for some reasons I thought I'd validate:
>>
>> 1) It is currently thought that a on disk file is REQUIRED for the
>> system. Does MDB_NOSYNC turn off the disk caching? Can it be run as a
>> pure in-memory database? Could the file not just be mapped to a
>> memdisk?
>
>
> A file is required. Whether the file is on-disk or not is irrelevant. It can
> of course be on a RAMdisk (or, in Linux, a tmpfs), which would then make it
> pure in-memory.
>
> MDB_NOSYNC turns off syncing, that's why it is named that.
>
>> 2) Because these values can come very fast, that the use of a lock
>> file would cause delay and too much wear on the nand based disk (SSD).
>
>
> No. That's not how the lock file is used.
>
>> I see a no locking option ( MDB_NOLOCK) that would stop a lock file
>> being written.
>
>
> This option should only be used if you're implementing your own lock
> manager. Hint - no matter what approach you use, any other lock manager you
> use will be slower than LMDB's.
>
>> Again, another option would be mapping the lock file to
>> a memdisk to  handle that?
>
>
> Yes.
>
> --
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/