Full_Name: Marvin Mundry Version: 2.4.33 OS: Ubuntu 12.10 URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz Submission from: (NULL) (134.100.2.183) i have slapcat'ed my production environment and slapadded it into an back_mdb database. when deleting members one by one from a large group (48428 Members), after some transactions slapd crashes. when i slapadd my dit to an mdb without having an eq index configured for "member" the removal of members works fine. when doing the same thing (member indexed, slapadding members, starting slapd, removing group members one by one) on a hdb, slapd does not crash. sortvals is set for members as otherwise deletion operations would take a very long time i failed to reproduce the problem without initially slapadding the data from my production environment into the mdb. i have removed all configuration settings not neccessary to reproduce the bug the debug.tar.bz should contain everything that is neccessary (including a description) to reproduce the problem. additionally the file contains a coredump and the full trace output slapd.conf: include /usr/local/etc/openldap/schema/core.schema include /usr/local/etc/openldap/schema/cosine.schema include /usr/local/etc/openldap/schema/inetorgperson.schema include /usr/local/etc/openldap/schema/rfc2307bis.schema pidfile /usr/local/var/run/slapd.pid argsfile /usr/local/var/run/slapd.args modulepath /usr/local/libexec/openldap moduleload back_mdb.la sortvals member database mdb suffix "dc=test,dc=com" rootdn "cn=admin,dc=test,dc=com" rootpw secret directory /usr/local/etc/openldap/test123db/AL index member eq maxsize 1000000000 last lines of slapd -d Trace: 5096b5ae oc_check_allowed type "modifyTimestamp" 5096b5ae sl_malloc 8: ch_malloc #this line repeats 28450 times 5096b5ae => mdb_entry_encode(0x00010170): cn=gr97503,ou=groups,dc=test,dc=com 5096b5ae <= mdb_entry_encode(0x00010170): cn=gr97503,ou=groups,dc=test,dc=com build options: CFLAGS=-O0 -d ./configure --enable-dynamic=yes --enable-syslog=yes --enable-slapd=yes --enable-dynacl=yes --enable-crypt=yes --enable-modules=yes --enable-rewrite=ye s --enable-bdb=mod --enable-hdb=mod --enable-ldap=mod --enable-mdb=mod --enable-meta=mod --enable-monitor=mod --enable-overlays=mod --with-tls=openssl --enable-debug (gdb) bt: #0 0x000002fc000002fa in ?? () #1 0x000002ff000002fd in ?? () #2 0x0000030200000300 in ?? () #3 0x0000030500000303 in ?? () #4 0x0000030800000306 in ?? () #5 0x0000030b00000309 in ?? () #6 0x00007f1400000312 in ?? () #7 0x0000009cffffffbf in ?? () #8 0x00007f1400000011 in ?? () #9 0x00000000000102a3 in ?? () #10 0x00007f140c101880 in ?? () #11 0x00007f140883c200 in ?? () #12 0x0000000000000891 in ?? () #13 0x00007f14155ef060 in ?? () #14 0x00007f14523dd5a6 in ?? () #15 0x00007f1451fa1010 in ?? () #16 0x00000000523dd2a7 in ?? () #17 0x00007f14155ef390 in ?? () #18 0x0000000000000000 in ?? ()
marvin.mundry@uni-hamburg.de wrote: > Full_Name: Marvin Mundry > Version: 2.4.33 > OS: Ubuntu 12.10 > URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz > Submission from: (NULL) (134.100.2.183) > Thanks for the report. The crash has been fixed in git, but your test runs into another (known) issue in MDB. You're working with a very large entry, which libmdb stores in overflow pages. In the current version of libmdb, freespace management for overflow pages is not fully implemented, so every time you update the entry libmdb will always use new pages (instead of reusing old pages). Thus, after a few hundred operations, your 1GB map will be exhausted. It looks like you won't be able to use back-mdb until this feature is fully implemented in libmdb. > i have slapcat'ed my production environment and slapadded it into an back_mdb > database. > > when deleting members one by one from a large group (48428 Members), after some > transactions slapd crashes. > when i slapadd my dit to an mdb without having an eq index configured for > "member" the removal of members works fine. > > when doing the same thing (member indexed, slapadding members, starting slapd, > removing group members one by one) on a hdb, slapd does not crash. > > sortvals is set for members as otherwise deletion operations would take a very > long time > > i failed to reproduce the problem without initially slapadding the data from my > production environment into the mdb. > > i have removed all configuration settings not neccessary to reproduce the bug > > the debug.tar.bz should contain everything that is neccessary (including a > description) to reproduce the problem. additionally the file contains a coredump > and the full trace output > > slapd.conf: > include /usr/local/etc/openldap/schema/core.schema > include /usr/local/etc/openldap/schema/cosine.schema > include /usr/local/etc/openldap/schema/inetorgperson.schema > include /usr/local/etc/openldap/schema/rfc2307bis.schema > pidfile /usr/local/var/run/slapd.pid > argsfile /usr/local/var/run/slapd.args > modulepath /usr/local/libexec/openldap > moduleload back_mdb.la > sortvals member > database mdb > suffix "dc=test,dc=com" > rootdn "cn=admin,dc=test,dc=com" > rootpw secret > directory /usr/local/etc/openldap/test123db/AL > index member eq > maxsize 1000000000 > > > last lines of slapd -d Trace: > 5096b5ae oc_check_allowed type "modifyTimestamp" > 5096b5ae sl_malloc 8: ch_malloc #this line repeats 28450 times > 5096b5ae => mdb_entry_encode(0x00010170): cn=gr97503,ou=groups,dc=test,dc=com > 5096b5ae <= mdb_entry_encode(0x00010170): cn=gr97503,ou=groups,dc=test,dc=com > > build options: > CFLAGS=-O0 -d > ./configure --enable-dynamic=yes --enable-syslog=yes --enable-slapd=yes > --enable-dynacl=yes --enable-crypt=yes --enable-modules=yes --enable-rewrite=ye > s --enable-bdb=mod --enable-hdb=mod --enable-ldap=mod --enable-mdb=mod > --enable-meta=mod --enable-monitor=mod --enable-overlays=mod --with-tls=openssl > > --enable-debug > > (gdb) bt: > #0 0x000002fc000002fa in ?? () > #1 0x000002ff000002fd in ?? () > #2 0x0000030200000300 in ?? () > #3 0x0000030500000303 in ?? () > #4 0x0000030800000306 in ?? () > #5 0x0000030b00000309 in ?? () > #6 0x00007f1400000312 in ?? () > #7 0x0000009cffffffbf in ?? () > #8 0x00007f1400000011 in ?? () > #9 0x00000000000102a3 in ?? () > #10 0x00007f140c101880 in ?? () > #11 0x00007f140883c200 in ?? () > #12 0x0000000000000891 in ?? () > #13 0x00007f14155ef060 in ?? () > #14 0x00007f14523dd5a6 in ?? () > #15 0x00007f1451fa1010 in ?? () > #16 0x00000000523dd2a7 in ?? () > #17 0x00007f14155ef390 in ?? () > #18 0x0000000000000000 in ?? () > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
changed notes changed state Open to Test moved from Incoming to Software Bugs
changed notes changed state Test to Release
hyc@symas.com wrote: > marvin.mundry@uni-hamburg.de wrote: >> Full_Name: Marvin Mundry >> Version: 2.4.33 >> OS: Ubuntu 12.10 >> URL: https://idmswiki.rrz.uni-hamburg.de:8005/debug.tar.bz >> Submission from: (NULL) (134.100.2.183) >> > > Thanks for the report. The crash has been fixed in git, but your test runs > into another (known) issue in MDB. > > You're working with a very large entry, which libmdb stores in overflow pages. > In the current version of libmdb, freespace management for overflow pages is > not fully implemented, so every time you update the entry libmdb will always > use new pages (instead of reusing old pages). Thus, after a few hundred > operations, your 1GB map will be exhausted. > > It looks like you won't be able to use back-mdb until this feature is fully > implemented in libmdb. The overflow page freespace management issue is being addressed in ITS#7455. A patch is currently being tested. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
changed notes changed state Release to Closed
crash fixed in master fixed in RE24