OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Software Bugs/7455
Full headers

From: quanah@openldap.org
Subject: MDB database grows without bound
Compose comment
Download message
State:
0 replies:
3 followups: 1 2 3

Major security issue: yes  no

Notes:

Notification:


Date: Wed, 28 Nov 2012 18:50:06 +0000
From: quanah@openldap.org
To: openldap-its@OpenLDAP.org
Subject: MDB database grows without bound
Full_Name: Quanah Gibson-Mount
Version: 2.4.33
OS: Linux 2.6
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (74.196.25.250)


I have a very small DB (about 25MB from a fresh slapadd).  However, the data.mdb
file grows by about 50MB a day.  I.e., the database size on disk *doubles* every
day.  It is now up to 571MB in size after 


Here is the DB after a fresh slapadd:
zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
25M     data.mdb

Here is the DB on the production server:
[zimbra@ldap01-zcs db]$ du -c -h data.mdb
571M    data.mdb

Followup 1

Download message
Date: Wed, 28 Nov 2012 12:16:40 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: openldap-its@openldap.org
Subject: Re: (ITS#7455) MDB database grows without bound
--On Wednesday, November 28, 2012 6:50 PM +0000 quanah@OpenLDAP.org wrote:

> Here is the DB after a fresh slapadd:
> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
> 25M     data.mdb
>
> Here is the DB on the production server:
> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
> 571M    data.mdb


In 1 week:

24,108 MODs
14 Adds
7 Deletes

For a client with a large database of several gigs, this will quickly lead 
to significant problems, even with a large amount of free disk space.  If I 
have a client with a 12GB DB, then they'll hit their DB maxsize of 80GB in 
just a few days.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration



Followup 2

Download message
Date: Wed, 28 Nov 2012 13:13:13 -0800
From: Howard Chu <hyc@symas.com>
To: quanah@OpenLDAP.org
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#7455) MDB database grows without bound
quanah@OpenLDAP.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.4.33
> OS: Linux 2.6
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (74.196.25.250)
>
>
> I have a very small DB (about 25MB from a fresh slapadd).  However, the
data.mdb
> file grows by about 50MB a day.  I.e., the database size on disk *doubles*
every
> day.  It is now up to 571MB in size after
>
>
> Here is the DB after a fresh slapadd:
> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
> 25M     data.mdb
>
> Here is the DB on the production server:
> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
> 571M    data.mdb

Based on the mdb_stat output you pasted, this is simply a case of overflow 
pages not reusing freelist pages. The significant info here is the freelist 
info and the number of overflow pages used in the id2e database.

   Environment Info
     Map address: (nil)
     Map size: 85899345920
     Page size: 4096
     Max pages: 20971520
     Number of pages used: 146230
     Last transaction ID: 192037
     Max readers: 126
     Number of readers used: 10
   Freelist Status
     Tree depth: 3
     Branch pages: 7
     Leaf pages: 668
     Overflow pages: 32
     Entries: 10671
     Free pages: 141005
   Status of Main DB
     Tree depth: 1
     Branch pages: 0
     Leaf pages: 1
     Overflow pages: 0
     Entries: 38
     ...
   Status of ad2i
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 20
     Overflow pages: 0
     Entries: 976
   Status of cn
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 231
     Overflow pages: 0
     Entries: 30224
   Status of displayName
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 91
     Overflow pages: 0
     Entries: 13625
   Status of dn2i
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 114
     Overflow pages: 0
     Entries: 5935
   Status of entryCSN
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 37
     Overflow pages: 0
     Entries: 2967
   Status of entryUUID
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 26
     Overflow pages: 0
     Entries: 2967
   Status of givenName
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 24
     Overflow pages: 0
     Entries: 3318
   Status of id2e
     Tree depth: 3
     Branch pages: 7
     Leaf pages: 721
     Overflow pages: 1937
     Entries: 2968
   Status of mail
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 141
     Overflow pages: 0
     Entries: 33250
   Status of objectClass
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 9
     Overflow pages: 0
     Entries: 9353
   Status of sn
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 53
     Overflow pages: 0
     Entries: 7502
   Status of uid
     Tree depth: 2
     Branch pages: 1
     Leaf pages: 13
     Overflow pages: 0
     Entries: 3148
     ...


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/



Followup 3

Download message
Date: Fri, 30 Nov 2012 05:09:34 -0800
From: Howard Chu <hyc@symas.com>
To: openldap-its@openldap.org
Subject: Re: (ITS#7455) MDB database grows without bound
This is a multi-part message in MIME format.
--------------040901020606020603090609
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Howard Chu wrote:
> quanah@OpenLDAP.org wrote:
>> Full_Name: Quanah Gibson-Mount
>> Version: 2.4.33
>> OS: Linux 2.6
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (74.196.25.250)
>>
>>
>> I have a very small DB (about 25MB from a fresh slapadd).  However, the
data.mdb
>> file grows by about 50MB a day.  I.e., the database size on disk
*doubles* every
>> day.  It is now up to 571MB in size after
>>
>>
>> Here is the DB after a fresh slapadd:
>> zimbra@zre-ldap002:~/data/ldap/mdb/db$ du -c -h data.mdb
>> 25M     data.mdb
>>
>> Here is the DB on the production server:
>> [zimbra@ldap01-zcs db]$ du -c -h data.mdb
>> 571M    data.mdb
>
> Based on the mdb_stat output you pasted, this is simply a case of overflow
> pages not reusing freelist pages. The significant info here is the freelist
> info and the number of overflow pages used in the id2e database.

Here's the patch we're currently testing for this issue.
It appears to work but is maybe not being aggressive enough in reclaiming 
space. We may want to increase the number of retries a bit more.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

--------------040901020606020603090609
Content-Type: text/plain; charset=UTF-8;
 name="diff.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="diff.txt"

diff --git a/libraries/libmdb/mdb.c b/libraries/libmdb/mdb.c
index 251ab6a..117b402 100644
--- a/libraries/libmdb/mdb.c
+++ b/libraries/libmdb/mdb.c
@@ -1242,6 +1242,7 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp)
 	MDB_page *np;
 	pgno_t pgno = P_INVALID;
 	MDB_ID2 mid;
+	txnid_t oldest = 0, last;
 	int rc;
 
 	*mp = NULL;
@@ -1254,12 +1255,11 @@ mdb_page_alloc(MDB_cursor *mc, int num, MDB_page **mp)
 		if (!txn->mt_env->me_pghead &&
 			txn->mt_dbs[FREE_DBI].md_root != P_INVALID) {
 			/* See if there's anything in the free DB */
-			int j;
 			MDB_reader *r;
 			MDB_cursor m2;
 			MDB_node *leaf;
 			MDB_val data;
-			txnid_t *kptr, last;
+			txnid_t *kptr;
 
 			mdb_cursor_init(&m2, txn, FREE_DBI, NULL);
 			if (!txn->mt_env->me_pgfirst) {
@@ -1282,15 +1282,21 @@ again:
 				last = *(txnid_t *)key.mv_data;
 			}
 
-			/* Unusable if referred by a meta page or reader... */
-			j = 1;
-			if (last < txn->mt_txnid-1) {
-				j = txn->mt_env->me_txns->mti_numreaders;
-				r = txn->mt_env->me_txns->mti_readers + j;
-				for (j = -j; j && (last<r[j].mr_txnid || !r[j].mr_pid); j++) ;
+			{
+				unsigned int i, nr;
+				txnid_t mr;
+				oldest = txn->mt_txnid - 1;
+				nr = txn->mt_env->me_txns->mti_numreaders;
+				r = txn->mt_env->me_txns->mti_readers;
+				for (i=0; i<nr; i++) {
+					if (!r[i].mr_pid) continue;
+					mr = r[i].mr_txnid;
+					if (mr < oldest)
+						oldest = mr;
+				}
 			}
 
-			if (!j) {
+			if (oldest > last) {
 				/* It's usable, grab it.
 				 */
 				MDB_oldpages *mop;
@@ -1331,29 +1337,108 @@ none:
 		if (txn->mt_env->me_pghead) {
 			MDB_oldpages *mop = txn->mt_env->me_pghead;
 			if (num > 1) {
-				/* FIXME: For now, always use fresh pages. We
-				 * really ought to search the free list for a
-				 * contiguous range.
-				 */
-				;
+				MDB_cursor m2;
+				int retry = 2, readit = 0, n2 = num-1;
+				unsigned int i, j, k;
+
+				/* If current list is too short, must fetch more and coalesce */
+				if (mop->mo_pages[0] < (unsigned)num)
+					readit = 1;
+
+				mdb_cursor_init(&m2, txn, FREE_DBI, NULL);
+				do {
+					if (readit) {
+						MDB_val key, data;
+						MDB_oldpages *mop2;
+						pgno_t *idl;
+						int exact;
+
+						last = mop->mo_txnid + 1;
+
+						/* We haven't hit the readers list yet? */
+						if (!oldest) {
+							MDB_reader *r;
+							unsigned int nr;
+							txnid_t mr;
+
+							oldest = txn->mt_txnid - 1;
+							nr = txn->mt_env->me_txns->mti_numreaders;
+							r = txn->mt_env->me_txns->mti_readers;
+							for (i=0; i<nr; i++) {
+								if (!r[i].mr_pid) continue;
+								mr = r[i].mr_txnid;
+								if (mr < oldest)
+									oldest = mr;
+							}
+						}
+
+						/* There's nothing we can use on the freelist */
+						if (oldest - last < 1)
+							break;
+
+						exact = 0;
+						key.mv_data = &last;
+						key.mv_size = sizeof(last);
+						rc = mdb_cursor_set(&m2, &key, &data, MDB_SET, &exact);
+						if (rc)
+							return rc;
+						idl = (MDB_ID *) data.mv_data;
+						mop2 = malloc(sizeof(MDB_oldpages) + MDB_IDL_SIZEOF(idl) -

Message of length 7394 truncated

Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org