(ITS#8438) slapd won't stop (shutdown) on multi-core system under stress

[zvika.ferentz@gmail.com]

Full_Name: zvika ferentz
Version: 2.4.31 (deb7u2)
OS: Debian Wheezy
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (65.51.117.60)


Greetings,

I've noticed that sometimes the "slapd" fails to stop. I can reproduce the
problem only on machines with multiple cores  (more than 2) when there are many
concurrent connections and the server is under a heavy "stress" .

note that this is a debian wheezy x64 so probably it's not the standard vanilla
slapd , however from the logs&source it doesn't look like it's related to the
OS.

In the logs i was able to identify the first "shutdown" log:
    daemon: shutdown requested and initiated
However the next expected shutdown logs are missing ("waiting for ...
operations/tasks to finish" and "slapd stopped") . Hence i believe that it's
related to closing the sockets/listeners (this is my guess based on the source
code, however i never debugged/proved it).
Please note that i'm using virtual environment so i was able to reproduce it
only when i used more than 4 cores.


How to reproduced it:
---------------------
- Open two terminals.
- On terminal #1 i'm just manually running  "slapd restart" commands:
   # /etc/init.d/slapd status ; /etc/init.d/slapd restart
- On terminal #2 i'm running a infinite loops of simple "ldapsearch" (100
concurrent processes running loops of ldapsearch). Terminal #2 is trying to
simulate many concurrent read operations. see "more information" later for the
exact scripts that i used.

Incorrect behavior:
-------------------
The "slapd restart" works a few times, and then the "stop" operation fails.
The stop continues to fail even if i stop all "stress" and terminate all
ldapsearch/connections  (CPU is 99% idle !)

Expected Behavior:
------------------
All slapd stop/restart operations complete successfully 


More Information (optional):D%D
----------------------------
On terminal#2 i used a very simple script to generate a "read only" stress:
# cat > ldaploop.sh << EOF
#!/bin/sh
while true ; do  ldapsearch -x -Z ; done
EOF

# cat > manyloops.sh << "EOF"
#!/bin/sh
for i in `seq 1 100` ; do ( ./ldaploop.sh &) ; done
EOF

As previously mentioned, i ran the "manyloops.sh" to generate 100 running
processes where each one simply runs "ldapsearch" (locally).

thanks in advance,
Zvika Ferentz