[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: back-mdb status

Howard Chu wrote:
A bit of a summary of how the backend is shaping up. I've been testing with a
variety of synthetic LDIFs as well as an actual application database (Zimbra

I noted before that back-mdb's write speeds on disk are quite slow. This is
because a lot of its writes will be to random disk pages, and also the data
writes in a transaction commit are followed by a meta page write, which always
involves a seek to page 0 or page 1 of the DB file. For slapadd -q this effect
can be somewhat hidden because the writes are done with MDB_NOSYNC specified,
so no explicit flushes are performed. In my current tests with synchronous
writes, back-mdb is one half the speed of back-bdb/hdb.
(Even in fully synchronous mode, BDB only writes its transaction logs
synchronously, and those are always sequential writes so there's no seek
overhead to deal with.)

With that said, slapadd -q for a 3.2M entry database on a tmpfs:

back-hdb:	real 75m32.678s  user 84m31.733s   sys 1m0.316s
back-mdb:	real 63m51.048s  user 50m23.125s   sys 13m27.958s

On an XFS partition, the same job took
back-hdb	real 80m34.403s  user 86m2.439s   sys  1m39.662s
back-mdb	real 85m48.598s  user 49m40.606s  sys 14m48.668s

Note that back-hdb runs a trickle-sync thread to flush dirty DB pages in the background, which is why its user time is greater than the real time.

back-mdb actually completed the load in 64m16.19s according to slapadd's progress meter. But back-mdb performs an mdb_env_sync() on close, and that sync took the remaining 21 minutes. (back-hdb pretty much does the same, it does a checkpoint on close, but it skips it in Quick mode. So to be apples-to-apples, back-mdb's final sync should have been omitted as well.)

For back-hdb, BDB was configured with a 32GB environment cache. The resulting
DB directory consumed 14951004KB including data files and environment files.

For back-mdb, MDB was configured with a 32GB mapsize. The resulting DB
directory consumed 18299832KB. The input LDIF was 2.7GB, and there were 29
attributes indexed. Currently MDB is somewhat wasteful with space when dealing
with the sorted-duplicate databases that are used for indexing, there's
definitely room for improvement here.

Also this slapadd was done with tool-threads set to 1, because back-mdb only
allows one writer at a time anyway. There is also obviously room for
improvement here, in terms of a bulk-loading API for the MDB library.

  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/