After power reset, LMDB sub-DBs' key-pairs are missing

[Arto Bendiken <arto@bendiken.net>]

Here's a curious case that I had not encountered with LMDB as yet previously:

1. There was a power reset of a virtual machine with an active LMDB
writer process (standalone use, not OpenLDAP) on an LMDB file
containing three sub-DBs.

2. After rebooting, the previously-populated LMDB file (~7 GB in size)
appears mostly empty, including when examined with mdb_stat or
mdb_dump. Mostly empty meaning that each of three sub-DBs now has only
one K/V entry, instead of 7M+ as they used to. In addition, the main
DB now indicates six entries instead of the expected three (for the
sub-DBs).

3. mdb_copy (with or without -c) does not remedy the situation,
producing a mostly (logically) empty database.

This is with LMDB release 0.9.18 running on Ubuntu 14.04.4 (kernel
3.13.0-79-generic) on an ext4 partition (noatime,nodev,nosuid,noexec)
on Intel SSD storage in SW RAID-1 configuration.

As mentioned, the LMDB file had three sub-DBs, each with 7M+ entries
(as of last backup). No new sub-DBs are created after the database is
initially initialized. After initial creation, these three sub-DBs
only ever get appended to with new key/value pairs, no code ever
deletes or modifies key/value pairs in them. The writer code inserts
new entries one at a time, commits the LMDB transaction, and syncs to
disk.

I've enclosed mdb_stat output from before/after (before being from a
backup, after which numerous more writes had been done). I've also
included mdb_dump output of the main DB and three sub-DBs.

The mdb_dump output for the sub-DBs indicates that they each now
contain only a single entry (instead of 7M+), that entry being in each
case the first key/value pair that was ever inserted into that sub-DB
(ages ago).

The mdb_dump output for the main DB is baffling--instead of the three
expected entries, or the six that mdb_stat indicates after the reboot,
the output includes a multitude of entries--some 2,590. (I've omitted
most of them in the attached, but can provide a copy privately.)

What are my options for recovering an LMDB database in this state, to
the extent possible? Has anyone else experienced a similar scenario?

Thanks,
Arto

-- 
Arto Bendiken | @bendiken | http://ar.to

### BEFORE POWER OUTAGE (from last backup)

$ mdb_stat -ea .
Environment Info
  Map address: (nil)
  Map size: 1099511627776
  Page size: 4096
  Max pages: 268435456
  Number of pages used: 272654
  Last transaction ID: 2210167
  Max readers: 4096
  Number of readers used: 0
Status of Main DB
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 3
Status of sha1:u32
  Tree depth: 4
  Branch pages: 1043
  Leaf pages: 88442
  Overflow pages: 0
  Entries: 7324562
Status of u32:cstr
  Tree depth: 4
  Branch pages: 419
  Leaf pages: 120625
  Overflow pages: 344
  Entries: 7324562
Status of u32:sha1
  Tree depth: 3
  Branch pages: 214
  Leaf pages: 61551
  Overflow pages: 0
  Entries: 7324562

### AFTER POWER OUTAGE

$ ls -l *.mdb
-rw-rw-r-- 1 root root 7448719360 Mar 14 15:25 data.mdb
-rw-rw-r-- 1 root root     262272 Mar 14 16:13 lock.mdb

$ mdb_stat -ea .
Environment Info
  Map address: (nil)
  Map size: 1099511627776
  Page size: 4096
  Max pages: 268435456
  Number of pages used: 1818535
  Last transaction ID: 4132159
  Max readers: 4096
  Number of readers used: 0
Status of Main DB
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 2
  Overflow pages: 0
  Entries: 6
Status of sha1:u32
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 1
Status of u32:cstr
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 1
Status of u32:sha1
  Tree depth: 1
  Branch pages: 0
  Leaf pages: 1
  Overflow pages: 0
  Entries: 1

$ mdb_dump -s sha1:u32 .
VERSION=3
format=bytevalue
database=sha1:u32
type=btree
mapsize=1099511627776
maxreaders=4096
db_pagesize=4096
HEADER=END
 da39a3ee5e6b4b0d3255bfef95601890afd80709
 00000000
DATA=END

$ mdb_dump -s u32:cstr .
VERSION=3
format=bytevalue
database=u32:cstr
type=btree
mapsize=1099511627776
maxreaders=4096
integerkey=1
db_pagesize=4096
HEADER=END
 00000000
 00
DATA=END

$ mdb_dump -s u32:sha1 .
VERSION=3
format=bytevalue
database=u32:sha1
type=btree
mapsize=1099511627776
maxreaders=4096
integerkey=1
db_pagesize=4096
HEADER=END
 00000000
 da39a3ee5e6b4b0d3255bfef95601890afd80709
DATA=END

$ mdb_dump .
VERSION=3
format=bytevalue
type=btree
mapsize=1099511627776
maxreaders=4096
db_pagesize=4096
HEADER=END
 54c93c0000000000
 0a0000000000000071660a000000000069660a000000000066660a000000000054660a00000000000f650a0000000000d4630a00000000008b610a0000000000875f0a0000000000cf4d0a0000000000a2ac090000000000
 55c93c0000000000
 0b00000000000000770f1800000000000d5a17000000000080660a00000000007c660a000000000075660a000000000074660a000000000072660a00000000006d660a00000000006b660a00000000004e3b0a00000000008f0e0a0000000000
 56c93c0000000000

... 2585 entries omitted ...

 0a00000000000000c2d00a0000000000c1d00a0000000000bad00a0000000000b0d00a0000000000afd00a00000000005f5b0a0000000000c9410a00000000006c6a09000000000004590900000000000ec9070000000000
 6dd33c0000000000
 0a00000000000000c3d00a0000000000c0d00a0000000000bfd00a0000000000b9d00a0000000000b8d00a000000000091d00a000000000090d00a000000000082d00a00000000007cd00a00000000000ed9030000000000
 6ed33c0000000000
 0900000000000000c7d00a0000000000c4d00a00000000009fd00a000000000099d00a000000000098d00a000000000081d00a000000000064cf0a0000000000519a090000000000b366000000000000
 6fd33c0000000000
 0a0000000000000042641700000000005874110000000000e0d00a0000000000c9d00a0000000000c5d00a000000000049cf0a0000000000299e0a0000000000916a0a0000000000ae360a0000000000a8130a0000000000
DATA=END