[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: MDB, read/write mmap



Howard Chu wrote:
> Howard Chu wrote:
>> It seems to me this can only be a viable mode of operation if you're always
>> going to run asynch and don't care much about transaction durability or DB
>> recoverability. Running in this mode offers absolutely zero crash resistance;
>> the entire DB will almost always be irreparably damaged after a system crash.
>>
>> Would you run like that, if it offered you the potential of maybe 10x faster
>> write performance?ï (It could be useful for slapadd -q, certainly.)
> 
> OK, 10x speedup was far too optimistic. Quickly cobbling the changes together,
> it looks more like about a 70% speedup.
> 
> slapadd -q with 5 million entries took 24m16s as originally reported at
> LDAPCon last year.
> 
> With current mdb.master it takes 22m24s:
> 
> real    22m23.984s
> user    26m1.658s
> sys     8m17.415s
> 
> With the writable mmap and no msyncs it took 13m17s.
> 
> real    13m17.225s
> user    22m15.511s
> sys     1m12.533s
> 
> This code is currently available on the map2 branch of my git repo on
> ada.openldap.org. I'll clean it up a bit further then push it to mdb.master
> after some more testing.

The speedup seems to be proportional to the number of indices that are defined
on the database. With Quanah's torture-test LDIF (~6 million entries, 4.9GB),
there's only a small difference between the two when no indices are defined:

Original mdb.master:

real    11m27.385s
user    15m5.489s
sys     6m47.825s

Writemap:

real    10m27.447s
user    14m35.663s
sys     6m10.767s

But with 31 indices defined...

Original:

real    94m35.862s
user    93m31.755s
sys     20m39.693s

Writemap:

real    42m53.499s
user    54m35.509s
sys     7m23.364s

Over a 2:1 speedup.

-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/