Logged in as guest
Viewing Software Bugs/7455 Full headers
Major security issue: yes no
Notes: fixed in master fixed in RE24 Notification:
Date: Wed, 28 Nov 2012 18:50:06 +0000 From: quanah@openldap.org To: openldap-its@OpenLDAP.org Subject: MDB database grows without bound
Full_Name: Quanah Gibson-Mount Version: 2.4.33 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (74.196.25.250) I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb file grows by about 50MB a day. I.e., the database size on disk *doubles* every day. It is now up to 571MB in size after Here is the DB after a fresh slapadd: zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb 25M data.mdb Here is the DB on the production server: [zimbra@ldap01-zcs db]$ du -c -h data.mdb 571M data.mdb
Date: Wed, 28 Nov 2012 12:16:40 -0800 From: Quanah Gibson-Mount <quanah@zimbra.com> To: openldap-its@openldap.org Subject: Re: (ITS#7455) MDB database grows without bound
--On Wednesday, November 28, 2012 6:50 PM +0000 quanah@OpenLDAP.org wrote: > Here is the DB after a fresh slapadd: > zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb > 25M data.mdb > > Here is the DB on the production server: > [zimbra@ldap01-zcs db]$ du -c -h data.mdb > 571M data.mdb In 1 week: 24,108 MODs 14 Adds 7 Deletes For a client with a large database of several gigs, this will quickly lead to significant problems, even with a large amount of free disk space. If I have a client with a 12GB DB, then they'll hit their DB maxsize of 80GB in just a few days. --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Date: Wed, 28 Nov 2012 13:13:13 -0800 From: Howard Chu <hyc@symas.com> To: quanah@OpenLDAP.org CC: openldap-its@OpenLDAP.org Subject: Re: (ITS#7455) MDB database grows without bound
quanah@OpenLDAP.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.33 > OS: Linux 2.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (74.196.25.250) > > > I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb > file grows by about 50MB a day. I.e., the database size on disk *doubles* every > day. It is now up to 571MB in size after > > > Here is the DB after a fresh slapadd: > zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb > 25M data.mdb > > Here is the DB on the production server: > [zimbra@ldap01-zcs db]$ du -c -h data.mdb > 571M data.mdb Based on the mdb_stat output you pasted, this is simply a case of overflow pages not reusing freelist pages. The significant info here is the freelist info and the number of overflow pages used in the id2e database. Environment Info Map address: (nil) Map size: 85899345920 Page size: 4096 Max pages: 20971520 Number of pages used: 146230 Last transaction ID: 192037 Max readers: 126 Number of readers used: 10 Freelist Status Tree depth: 3 Branch pages: 7 Leaf pages: 668 Overflow pages: 32 Entries: 10671 Free pages: 141005 Status of Main DB Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 38 ... Status of ad2i Tree depth: 2 Branch pages: 1 Leaf pages: 20 Overflow pages: 0 Entries: 976 Status of cn Tree depth: 2 Branch pages: 1 Leaf pages: 231 Overflow pages: 0 Entries: 30224 Status of displayName Tree depth: 2 Branch pages: 1 Leaf pages: 91 Overflow pages: 0 Entries: 13625 Status of dn2i Tree depth: 2 Branch pages: 1 Leaf pages: 114 Overflow pages: 0 Entries: 5935 Status of entryCSN Tree depth: 2 Branch pages: 1 Leaf pages: 37 Overflow pages: 0 Entries: 2967 Status of entryUUID Tree depth: 2 Branch pages: 1 Leaf pages: 26 Overflow pages: 0 Entries: 2967 Status of givenName Tree depth: 2 Branch pages: 1 Leaf pages: 24 Overflow pages: 0 Entries: 3318 Status of id2e Tree depth: 3 Branch pages: 7 Leaf pages: 721 Overflow pages: 1937 Entries: 2968 Status of mail Tree depth: 2 Branch pages: 1 Leaf pages: 141 Overflow pages: 0 Entries: 33250 Status of objectClass Tree depth: 2 Branch pages: 1 Leaf pages: 9 Overflow pages: 0 Entries: 9353 Status of sn Tree depth: 2 Branch pages: 1 Leaf pages: 53 Overflow pages: 0 Entries: 7502 Status of uid Tree depth: 2 Branch pages: 1 Leaf pages: 13 Overflow pages: 0 Entries: 3148 ... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Date: Fri, 30 Nov 2012 05:09:34 -0800 From: Howard Chu <hyc@symas.com> To: openldap-its@openldap.org Subject: Re: (ITS#7455) MDB database grows without bound
This is a multi-part message in MIME format. --------------040901020606020603090609 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Howard Chu wrote: > quanah@OpenLDAP.org wrote: >> Full_Name: Quanah Gibson-Mount >> Version: 2.4.33 >> OS: Linux 2.6 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (74.196.25.250) >> >> >> I have a very small DB (about 25MB from a fresh slapadd). However, the data.mdb >> file grows by about 50MB a day. I.e., the database size on disk *doubles* every >> day. It is now up to 571MB in size after >> >> >> Here is the DB after a fresh slapadd: >> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb >> 25M data.mdb >> >> Here is the DB on the production server: >> [zimbra@ldap01-zcs db]$ du -c -h data.mdb >> 571M data.mdb > > Based on the mdb_stat output you pasted, this is simply a case of overflow > pages not reusing freelist pages. The significant info here is the freelist > info and the number of overflow pages used in the id2e database. Here's the patch we're currently testing for this issue. It appears to work but is maybe not being aggressive enough in reclaiming space. We may want to increase the number of retries a bit more. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ --------------040901020606020603090609 Content-Type: text/plain; charset=UTF-8; name="diff.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="diff.txt" diff --git a/libraries/libmdb/mdb.c b/libraries/libmdb/mdb.c index 251ab6a..117b402 100644 --- a/libraries/libmdb/mdb.c +++ b/libraries/libmdb/mdb.c @@ -1242,6 +1242,7 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) MDB_page *np; pgno_t pgno = P_INVALID; MDB_ID2 mid; + txnid_t oldest = 0, last; int rc; *mp = NULL; @@ -1254,12 +1255,11 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp) if (!txn->mt_env->me_pghead && txn->mt_dbs[FREE_DBI].md_root != P_INVALID) { /* See if there's anything in the free DB */ - int j; MDB_reader *r; MDB_cursor m2; MDB_node *leaf; MDB_val data; - txnid_t *kptr, last; + txnid_t *kptr; mdb_cursor_init(&m2, txn, FREE_DBI, NULL); if (!txn->mt_env->me_pgfirst) { @@ -1282,15 +1282,21 @@ again: last = *(txnid_t *)key.mv_data; } - /* Unusable if referred by a meta page or reader... */ - j = 1; - if (last < txn->mt_txnid-1) { - j = txn->mt_env->me_txns->mti_numreaders; - r = txn->mt_env->me_txns->mti_readers + j; - for (j = -j; j && (last<r[j].mr_txnid || !r[j].mr_pid); j++) ; + { + unsigned int i, nr; + txnid_t mr; + oldest = txn->mt_txnid - 1; + nr = txn->mt_env->me_txns->mti_numreaders; + r = txn->mt_env->me_txns->mti_readers; + for (i=0; i<nr; i++) { + if (!r[i].mr_pid) continue; + mr = r[i].mr_txnid; + if (mr < oldest) + oldest = mr; + } } - if (!j) { + if (oldest > last) { /* It's usable, grab it. */ MDB_oldpages *mop; @@ -1331,29 +1337,108 @@ none: if (txn->mt_env->me_pghead) { MDB_oldpages *mop = txn->mt_env->me_pghead; if (num > 1) { - /* FIXME: For now, always use fresh pages. We - * really ought to search the free list for a - * contiguous range. - */ - ; + MDB_cursor m2; + int retry = 2, readit = 0, n2 = num-1; + unsigned int i, j, k; + + /* If current list is too short, must fetch more and coalesce */ + if (mop->mo_pages[0] < (unsigned)num) + readit = 1; + + mdb_cursor_init(&m2, txn, FREE_DBI, NULL); + do { + if (readit) { + MDB_val key, data; + MDB_oldpages *mop2; + pgno_t *idl; + int exact; + + last = mop->mo_txnid + 1; + + /* We haven't hit the readers list yet? */ + if (!oldest) { + MDB_reader *r; + unsigned int nr; + txnid_t mr; + + oldest = txn->mt_txnid - 1; + nr = txn->mt_env->me_txns->mti_numreaders; + r = txn->mt_env->me_txns->mti_readers; + for (i=0; i<nr; i++) { + if (!r[i].mr_pid) continue; + mr = r[i].mr_txnid; + if (mr < oldest) + oldest = mr; + } + } + + /* There's nothing we can use on the freelist */ + if (oldest - last < 1) + break; + + exact = 0; + key.mv_data = &last; + key.mv_size = sizeof(last); + rc = mdb_cursor_set(&m2, &key, &data, MDB_SET, &exact); + if (rc) + return rc; + idl = (MDB_ID *) data.mv_data; + mop2 = malloc(sizeof(MDB_oldpages) + MDB_IDL_SIZEOF(idl) -
______________ © Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org
