Re: (ITS#7566) ldapadd slower on Linux than BSD

[quanah@zimbra.com]

--On Saturday, April 06, 2013 1:07 PM -0600 - lidutu <lm5@ualberta.ca> 
wrote:

>
>
>
>
>
>
>
> So what is happening on the slapd server while things are paused?
>
> Also, if you use an ldapi:/// socket rather than a tcp/ip socket, do you
> see the same issue?
>
>
> Â 
>
> A have changed to use the ldapi:/// socket and there is no improvement.
>
>
> It seems that one thread of slapd is repeatedly waiting on epoll_wait.
> Strace prints
> epoll_wait(6,
>
> and then pauses for couple of seconds.
>
>
>
> Iotop shows to slapd threads writing to disk. Right after I start
> ldapadd, they write to disk at approximately 60MB/s each. Over time, they
> go down to 2-3 MB/s each.
>
> TIDÂ Â Â  PRIOÂ Â Â  USERÂ Â Â  DISK READÂ Â Â  DISK WRITEÂ Â Â 
> SWAPINÂ Â Â  IOÂ Â Â  COMMAND
> 15264 be/4 ldap        0.00 B/s    2.81 M/s  0.00 % 99.99 %
> slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h
> ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock
> 15255 be/4 ldap        0.00 B/s    3.00 M/s  0.00 % 19.48 %
> slapd -u ldap -g ldap -f /etc/openldap/slapd.conf -h
> ldapi://%2fvar%2frun%2fopenldap%2fslapd.sock

I noticed something similar on ubuntu10 as well.  I upgraded my server to 
ubuntu12, and I also set /proc/sys/vm/dirty_ratio to 90 and 
/proc/sys/vm/dirty_expire_centisecs to 60000, and that helped significantly 
with back-mdb using current OpenLDAP master.  An ldapadd of almost 3.5 
million users takes approximately 4 hours now for my dataset vs 16.  so 4x 
faster.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration