Re: (ITS#8943) speed up parallel add/delete

[quanah@symas.com]

--On Wednesday, December 12, 2018 3:54 AM +0000 derek@shannon-data.com 
wrote:

> Full_Name: Derek Zhou
> Version: 2.4 git HEAD
> OS: linux
> URL:
> Submission from: (NULL) (116.236.169.22)
>
>
> the mdb backend does not really support parallelism in rw transactions, so
> add/delete are serialized by a mutex inside the mdb backend as small
> transactions. In my patch, all add/delete are put in a queue and executed
> in a dedicated worker thread. It may seem pointless at first, but by
> doing them all in the same thread, now I can merge concurrent ops from
> different clients into larger transactions, and reduce the number of
> expensive txn_commit calls (fsyncs).

Hi,

Thanks for the report!  I have some questions:

a) Did you test with writemap enabled for mdb?

b) Did you test the effect on other operating systems? (I.e., Windows, 
Solaris, FreeBSD, etc)?

c) Did you test multiple filesystems under Linux(I.e., ext2, ext3, ext4, 
xfs, etc)?

d) What filesystem did you use when testing?  Was journaling enabled or 
disabled?  What was the default flush behavior of the filesystem you used? 
I.e., if you set dbnosync at the mdb level, the underlying filesystem flush 
behavior will then be in effect (with ext4, for example, this defaults to 5 
seconds)

e) Did you test when the database is also writing to an underlying 
accesslog database that is also backed by mdb?

Thanks,
Quanah



--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>