[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Issues arising from creating powerdns backend based on LMDB

Mark Zealey wrote:

4) Creating database with non-sequential keys is very bad (on 4gb
databases, 2* slower than kyoto - about 1h30 and uses more memory).

This was actually a typo - kyoto only takes about 20 minutes to generate
it so 4* slower. However, using a commit every 1m inserts (and because

Try again with commit every 100K inserts.

of a limitation in the perl module also have to close the DB/env and
reopen it) and backing it to a memdisk (which we also have to do with
kyoto), it takes about 10% less time than kyoto. Doing it against normal
disk is still very slow though. Size for a 4gb database was about 10%
more than kyoto, for kyoto outputting a 1.5gb database lmdb did 2.5gb
though. Doesn't matter too much for our purposes however.

Look at mdb_stat -ef on the resulting DB, you'll see that a large amount of pages claimed on disk are actually free pages in the DB. Larger commits leave more old pages behind than smaller commits.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/