Viewing Archive.Incoming/3954
Full headers

Major security issue: yes  no

Notes:
no fix for RE22
Notification:

Date: Fri, 19 Aug 2005 13:21:38 GMT
From: adrian.gschwend@bfh.ch
To: openldap-its@OpenLDAP.org
Subject: slapd stability problems with bdb backend

Full_Name: Adrian Gschwend
Version: 2.2.27
OS: FreeBSD 5.3
URL: ftp://ftp.openldap.org/incoming/adrian-gschwend-050819.gdb-backtrace.log
Submission from: (NULL) (147.87.65.205)


slapd consumes 100% cpu and never quits after add or delete operations. I
described the issues detailed under the following URLs:

http://article.gmane.org/gmane.network.openldap.general/30543
http://article.gmane.org/gmane.network.openldap.general/30574
http://article.gmane.org/gmane.network.openldap.general/30604
http://article.gmane.org/gmane.network.openldap.general/30609
http://article.gmane.org/gmane.network.openldap.general/30624
http://article.gmane.org/gmane.network.openldap.general/30633
http://article.gmane.org/gmane.network.openldap.general/30644

Summary: Our DB contains about 4000 entries (bdb backend), it gets updated via a
meta database which connects to OpenLDAP via the perl-interface. During updates
it often happens that the slapd process hangs, which means it consumes 100% CPU
and does not quit anymore. No more queries can be done. On FreeBSD we have to
kill the process hard (-9) and run db_recover -c afterwards to get a clean DB
before we can restart it.

The problem also occurs on Debian 3.1 with slapd 2.2.23 (all precompiled debian
packages). We redid the database several times (slapcat/slapadd) with various
DB_CONFIG options, see the URLs above for details.

Note that the same add operation that fails works perfectly well once we
restarted slapd so the add itself must be ok. Also it was much harder to
reproduce the problem with a debug version of slapd (CFLAGS+= -g). But it still
does lock, see the gdb thread backtrace attached to this bug

We couldn't do a dbstat -CA yet during a lock, we will provide that as soon as
it locks up again.

If necessary we could provide an ssh (root) login to a test machine with gdb and
a test-environment. If you need that please contact me by email.

Followup 1

Date: Fri, 19 Aug 2005 16:24:31 +0200
From: Adrian Gschwend <adrian.gschwend@bfh.ch>
To: openldap-its@openldap.org
Subject: (ITS#3954) new trace, including db_stat logs

This is a multi-part message in MIME format.
--------------060208010805090107090902
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit


The server just hung again, with a simple add operation. I took the 
chance to do the thread backtrace again and to get db_stat logs as well.

Attached is a tar.gz file that contains the following files:

openldap-its-3954/cn.bdb.stat
openldap-its-3954/dn2id.bdb.stat
openldap-its-3954/entryUUID.bdb.stat
openldap-its-3954/gidNumber.bdb.stat
openldap-its-3954/id2entry.bdb.stat
openldap-its-3954/memberUid.bdb.stat
openldap-its-3954/objectClass.bdb.stat
openldap-its-3954/uid.bdb.stat
openldap-its-3954/uidNumber.bdb.stat
openldap-its-3954/slapd.log
openldap-its-3954/db-backtrace2.log

slapd.log contains the add-entry that locked it, db-backtrace2.log is 
the gdb output.

cu

Adrian
-- 
Adrian Gschwend
System Administrator
Berne University of Applied Sciences
Biel, Switzerland

--------------060208010805090107090902
Content-Type: application/x-gzip;
 name="openldap-its-3954.tar.gz"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="openldap-its-3954.tar.gz"

H4sIADzlBUMAA+2da2/bOBaG5+v4V3BnvrRAmiGpewBj0G2LnQDtbNEL5sNiYdASlXjryF5J
aZP99UtSF0vylY2amQLvCVJbtview9sRqcdOV2uZLROxfrYoi2dO5Lm//DC6UUoDzyP6UZl+
ZIHHmuPqRUYdyhyfc58TylzmOT8Qb/xQtu22KEWuQslXq/LQeSK5WWQH3t9Urnr8Tmy11f9x
dj5P5ueqWQ62h4Wp9vBd1zSNT73eozLmU8qb/lcPnup/L/CDHwgdyf9Ba/pfN8Kh8469/532
//SZ5c/k9Sr+RHJ5tVhlZC1ycSNLmReTpXpZ5qQU86UkxeJ/8oIw3wnZGVnN/yPjcs87s1Wa
XhDH9xzHD88mq+I+i6+rF+kZqUTrczzOHcbVi/1zMimTWZKo55OJfW1erLJ0uVDh3YgyX9xN
6I/DH/MS6/1O9D/9n90FmSlQ/bRnsc457OBZrD27PYu1j19RWd11BbnKV7drmZD5fd2+hXlD
dR4hb1aJJMZerG6zkrxXWeC2IOTZ0Mg/q14dvDypChNGkmTqhMZBQa7lMlEvKfuSL0pZv0qr
UzdF3r16/pK0xshvr16b40XCZVbm9zovka5diyxZyuZI9X/9jBvvQdc7PeLdMUV8m4CdvQEn
GV8kW9EeCNg13j2bgD1TxLUp4psijk2RwBThNkVCU4TatGS4tyWr3PFiKYqi3577WjLS3nlk
4z3a8v7H88sPB72vxZXcHAX7lY7Wo6/E2noIUw+rCSRGq8e20lfXY27qEdjUY77Xe7U0Idu2
bzTExrtvM3wTU8RqKkpTxGoqpqaIY9Ms6d5mud2dbPY2C6PGu82sZiahc2ZTxGRhbpMIVJF9
dbyRN3OZf9yu6b46moTOIpuATRZmoU0Rk4WZzZWGmSzMbMYkM1mY2YxJZrIwsxmTzKROZjMm
2f6Ed7VIfr/VXXZqf5mEx6zGpMktzGpMmoTArMZkfGje2dXR5BZiNSZNbiFWY9LkFmKTctn+
3GIWXh8/Xr48sY7c5BZiM7y5yS3EZnhzk1uIzfDmJiEQm9UPNwmB2AwwbhICoScVCau9YrX0
PM2LKRI41UDqTRaVUfpF1OTvF/nj3eWHVzu6mN6xNJ4LNcZkdkHMylKUQu2n7uKA3tFY/dLm
d3Jcbu9qvbtEcFz3tMASyncENj89MH58Vd5buzBvNKW9VbRUYiHbp+TaxsRHq93emLjtUPCP
K+1d+fVjco4r7b0wWSuduBY+YRScvD4/qrQ3XfeVOmN8X+I3WSGNtibf37qzbjDzrPfDw5k3
xj2NqjFHvKUx3iQecZyPOI1HnDMjps4RR9V4V4YR0/CYt73GSwsj3smon4x4L2HEG25j3HEa
L63XT8a7ETXCraDxtwYj3tgZZeM53gV+/NsXY97tGWMDO95Sb8QNxKiLmLG3I2Pvu75iafRG
3qzye5LmUu0JF0U5oXc8dBNOqXuhBivlDqU/Tv5sAgn7M22b/7fX0tE+AmDD/xlnRL3k+A74
/2MY+D/4P/g/+D/4f8c7+D/4P/h/1zv4P/g/+P+gjuD/4P/g/0M58P9GCfwf/B/8n4D/m9qB
/1tEBf4P/t8qgf+D//elwP/B/8H/wf9hD7Vt/t9L66N8BsCG/6t3CGUBZx74/2MY+D/4P/g/
+D/4f8c7+D/4P/h/1zv4P/g/+P+gjuD/4P/g/0M58P9GCfwf/B/8n4D/m9qB/1tEBf4P/t8q
gf+D//elwP/B/8H/wf9hD7Vt/t9LoI/O/5njav7vUHz//1EM/B/8H/wf/B/8v+Md/B/8H/y/
6x38H/wf/H9QR/B/8H/w/6Ec+H+jBP4P/g/+T8D/Te3A/y2iAv8H/2+VwP/B//tS4P/g/+D/
4P+wh9o2/+9Ov3H+CwAb/k8D/ff/A8YD8P/HMPB/8H/wf/B/8P+Od/B/8H/w/6538H/wf/D/
QR3B/8H/wf+HcuD/jRL4P/g/+D8B/ze1A/+3iAr8H/y/VQL/B//vS4H/g/+D/4P/wx5q2/y/
l6oe//v/blB9/98F/38MA/8H/wf/B/8H/+94B/8H/wf/73oH/wf/B/8f1BH8H/wf/H8oB/7f
KIH/g/+D/xPwf1M78H+LqMD/wf9bJfB/8P++FPg/+D/4P/g/7KG2zf8HWW+ETwBY8X9qvv/v
Mw/8/zEM/B/8H/wf/B/8v+Md/B/8H/y/6x38H/wf/H9QR/B/8H/w/6Ec+H+jBP4P/g/+T8D/
Te3A/y2iAv8H/2+VwP/B//tS4P/g/+D/4P+wh9o2/78d8Zv/lVnxf1/zf595FPz/MQz8H/wf
/B/8H/y/4x38H/wf/L/rHfwf/B/8f1BH8H/wf/D/oRz4f6ME/g/+D/5PwP9N7cD/LaIC/wf/
b5XA/8H/+1Lg/+D/4P/g/7CH2k7+v0kKj//3/5n5/r/D8Pf/H8XA/8H/wf/B/8H/O97B/8H/
wf+73sH/wf/B/wd1BP8H/wf/H8qB/zdK4P/g/+D/BPzf1A783yIq8H/w/1YJ/B/8vy8F/g/+
D/4P/g97qG3z/2Ip1sn5cnU1mo8O/68ROQs81hzrSbLh/4HvEcq8IOCPzP/z1ergBx1EcrPI
Drz/nfL/57dX+mrKvAs3vND8U48FyogZBf8KVbr79wWJV1k29UhqbtA+f/Hi1dsPKqusbsjl
2+nFBSNu5Kn+fGKOiBNGTyc2sqv1lJK/X/7+Ui3jpj/F2fSNyFSmz8+SeDpPr/VDfP2TuoSW
1ysdQfht1OPr6fvLN29fvyJFkU6pvZN3r95/fP2BlOJqGgVE5rl6rZR35dRWipHnL9twL39/
N/vtw+X55Yf3z54nN2/z7Gx1O1VH+uEfettb6GfqlH6NTszr2/M/mT+bi/hTmYtY8lHywOH5
z3zfoc3859z8/Y+AuwHm/2PYh+tcioT45MnmWbGUcr3Irp4+vZj8TPVChYdOlMTBnCwysi7N
ibObWzW4RVnms0W2KMmTp1VK+OW2yH9ZLub6tz71vFids8nPrFWax8mDlHijxALBu0r/WamD
4+Wdqjxzg1D6uryeAbP153K2X2cVi2WjZk7Pn/FzrgWDyc+uFqQh9UKXmqqZmT1LhFqBGSFR
kurgPL7g1FUxeHURV6pBr4vcCO1T5FfxNDgj6vHzVK0s03kqYzc2CvoMVT7QN1rq3vLaflPP
Xv/xVs0jNY1Yr+eEJ9Og20qlLMpYZLFc2vSadKL04b3WXPO10q+/qlJtVdy2KupZfptl+vNi
vYqkUZKYihRyqW/UtS5rd7F25HVCpr4T7Tr/SIiqHwPXk8N+nJWi+ESerEuV2O9ov0+Zy3k7
rpyIi7A3wmP1bylPCMCtFFw3jj3T2rO4vJvpFLW3uk37OW37Oe1QcDy3P4nVCsvUS8SxXP9J
k9bxpalaEV/LZHa/0PTKZs5Gagu2Y87uEjo+aXnIWRK6wrT1bFVUMjNXfyZ1v5a6RrqVDq9n
soos4lHsVDrJXA+Wqo1mGsdZCfp1YFT4aVgJGo0rWarGLmWeieHM3asVNFqRXw3JWuuzjK1i
CutKpiyIg7aSSx2SjUxUywgWuPNKZi5uZoUUeXxtpcRo3ephkNKNUlxrnSzDahlPpPOujKrZ
yRq80pB+GmwGgJGwq5JT66R+OB/orNd2Um6VyGLOuUmB6u3ZIlnOUlmqhv4k78mTuZwad04c
+/SMJHN1qHpGhFQdlYtEZ7kzok411yGfx54TnpFFUujzHCpVFjdJUMmqDOixSLn1areOTOaN
W6UwM3mp8qgc8oRuO7zLaofqIfLiMJXuLm9K

Followup 2

Date: Sat, 27 Aug 2005 22:26:24 -0700
From: Howard Chu <hyc@symas.com>
To: adrian.gschwend@bfh.ch
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#3954) slapd stability problems with bdb backend

adrian.gschwend@bfh.ch wrote:
> Full_Name: Adrian Gschwend
> Version: 2.2.27
> OS: FreeBSD 5.3
> URL: ftp://ftp.openldap.org/incoming/adrian-gschwend-050819.gdb-backtrace.log
> Submission from: (NULL) (147.87.65.205)
>
>
> slapd consumes 100% cpu and never quits after add or delete operations. I
> described the issues detailed under the following URLs:
>   
The backtrace indicates a deadlock in the syncrepl persistent search 
code. This code is known to suffer from a number of deadlock issues in 
OpenLDAP 2.2, the only fix is to use OpenLDAP 2.3.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/