[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LMDB data corruption/alignment issues on ARM

Martin Lucina wrote:
martin@lucina.net said:
hyc@symas.com said:
My question is: Given that I'm storing C structs directly in LMDB, I need
to get at least word-aligned pointers back from LMDB in order to be able to
access the data safely.

I've not found anything in the documentation or the source about being able
to tweak alignment of key and data values; I *think* that all I need is an
option where LMDB would guarantee a minimum word (4 byte in this case)
alignment of data.

How hard would this be to implement? Would you consider such a feature?

LMDB guarantees 2-byte alignment of keys, and data is generally
stored contiguously with keys. If you want 4-byte alignment, it's
your responsibility to pad your keys to 4-byte boundaries. If you
pad appropriately, your alignment will be preserved.

In this case the keys are size_t (4 bytes on the target) and I'm using
MDB_INTEGERKEY. I'm not sure what you mean by padding the keys to 4-byte
boundaries given that LMDB copies my data into its memory map and I have no
control over where it gets placed.

Ah. What you forgot to mention (and I tracked down in a Reddit comment
somewhere) is that the (key + data) size (and not just the key size) should
be a multiple of the required alignment.

That is what "data is stored contiguously with keys" means.

  The problematic records in
question are using VLAs to store variable length strings; if I pad the
allocated space to the nearest word boundary that seems to fix the issue.


  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/