(ITS#7566) ldapadd slower on Linux than BSD

[lm5@ualberta.ca]

Full_Name: Leo Mocofan
Version: 2.4.31
OS: Debian Testing
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (129.128.11.107)


We are currently testing Linux, OpenBSD and FreeBSD to run our LDAP service. Our
tests showed that Linux has great performance numbers in most of the areas but
we found one test that is very slow.

The database has 1 million entries and is 4.5 GB in size. Importing the whole
dababase using slapadd takes minutes on all platforms. When we tried to import
the database using ldapadd we noticed a huge difference: FreeBSD - 42 min,
OpenBSD - 59 min, Linux - 863 min. Changing file systems for backend in Linux
had no effect. Using memory based file system for the backend in Linux the time
for ldapadd dropped to 3 min.

Strace for ldapadd shows:
write(1, "adding new entry \"uid=noname,o"..., 60) = 60
time(NULL)                              = 1365202241
getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0
time(NULL)                              = 1365202241
times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914
getrusage(RUSAGE_SELF, {ru_utime={1, 684105}, ru_stime={0, 212013}, ...}) = 0
time(NULL)                              = 1365202241
times({tms_utime=168, tms_stime=21, tms_cutime=0, tms_cstime=0}) = 1719044914
write(4, "\27\27\27\27\17uJ,\324\324\324\324"..., 1237) = 1237
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 0 (Timeout)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 100) = 1 ([{fd=4, revents=POLLIN}])

Strace for one thread of slapd:
write(12, "\27\27\27\27\17uJ,\324\324\324\324"..., 117) = 117
time([1365202241])                      = 1365202241
sendto(4, "<167>Apr  5 16:50:41 slapd[4513]"..., 79, MSG_NOSIGNAL, NULL, 0) =
79
futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9166, NULL) = 0
futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 0
time(NULL)                              = 1365202242
read(12, "", 5)                         = 0
write(6, "0", 1)                        = 1
futex(0x7f36f8fd6724, FUTEX_WAIT_PRIVATE, 9169, NULL

Strace for another thread of slapd:
futex(0x7f36f8fd66f8, FUTEX_WAKE_PRIVATE, 1) = 1
time(NULL)                              = 1365202241
epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000)
= 1
read(5, "0", 8192)                      = 1
time(NULL)                              = 1365202241
epoll_wait(7, {{EPOLLIN, {u32=4177193872, u64=139874082155408}}}, 1024, 150000)
= 1
epoll_ctl(7, EPOLL_CTL_MOD, 12, {0, {u32=4177193872, u64=139874082155408}}) = 0
futex(0x7f36f8fd6724, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7f36f8fd6720,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
time(NULL)                              = 1365202242
epoll_wait(7, {{EPOLLIN, {u32=4177193844, u64=139874082155380}}}, 1024, 150000)
= 1
read(5, "0", 8192)                      = 1
time(NULL)                              = 1365202242
epoll_wait(7, {}, 1024, 150000)         = 0
time(NULL)                              = 1365202392
epoll_wait(7, 

Ldapadd is stuck in poll system call, while one thread of slapd is waiting on
epoll_wait(7, 
After couple of seconds when apparently nothing happens, then everything starts
working again.

Changing the database backend from to bdb to hdb and mdb doesn't seem to have an
effect. We then tried to force SLAP_EVENT_WAIT to use select instead of
epoll_wait on Linux and ldapadd is still much slower on Linux.

We are runnin Debian testing as of Apr 5, 2013, with glibc-2.13, linux-3.2.41,
libdb-5.1.29, openldap-2.4.31.