[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LibLmdb non string data



Thanks for the reply. I'll see what I can do to try and stick the identifier byte at a future release. I guess I need to be sure it's not going to adversely effect the sorting order but this is for data so it may not matter that much.  I am not sure I will support the callback functions or multiple get in the initial release either.

Cheers.

On Sat, Dec 3, 2016 at 6:00 AM, Howard Chu <hyc@symas.com> wrote:
Hallvard Breien Furuseth wrote:
On 02. des. 2016 12:26, Tim Uckun wrote:
I am writing a wrapper for liblmdb and one thing I was able to do was to
store simple integers and floats as the value into the database.  This seems
to work pretty well but I am wondering if this is somehow against best
practice.

One bad thing about this is that when you are doing a seek or a cursor
operation the code does not know what kind of data is at the record. I have
some routines to cast them back from the pointers but the user has to know
what to cast them to. It would be very handy if the MDB_val struct had a
field for metadata, even if it just a byte.

LMDB handles binary data fine.  The problem is ordering:

Use a single datatype for each DB, and mdb_set_compare()
mdb_set_dupsort() to order data in that DB properly.  Or
MDB_INTEGERKEY/MDB_INTEGERDUP which do this for unsigned int/size_t.

If that is not feasible and you are mixing different datatypes in a
single database, you need to tag each data item so the compare function
can figure out how to parse and order them all.  Note that a slow
compare function can degrade LMDB's performance.

Note that the compare functions may receive unaligned values.  Though
you can hack alignment 2, 4, or sizeof(size_t) = 4 or 8 by ensuring
*every* key and value in a DB has size = multiple of that alignment.
Keys always have alignment of at least 2.  Hmm, I should add some
code so data will too except when both key and data have uneven size.

Requests to add a user-defined argument to the compare functions have
been rejected.  If you need that, maybe your wrapper can emulate it by
setting thread-specific data before calling LMDB, and then your sort
function can read that data.  That gets cumbersome, though.

If this is all generic, maybe define a DB with meta-info where your
wrapper can look up which datatypes (sizes, compare functions, whatever)
are in which database.

Or just stick your own identifier byte at the beginning of each data value. (Or key, if you're trying to use different type'd keys.)

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/