OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Archive.Incoming/3565
Full headers

From: daniel.armbrust@mayo.edu
Subject: Programatic Insert Scaleability Problem (#2)
Compose comment
Download message
State:
0 replies:
4 followups: 1 2 3 4

Major security issue: yes  no

Notes:

Notification:


Date: Wed, 23 Feb 2005 14:30:05 GMT
From: daniel.armbrust@mayo.edu
To: openldap-its@OpenLDAP.org
Subject: Programatic Insert Scaleability Problem (#2)
Full_Name: Dan Armbrust
Version: 2.2.23
OS: Fedora Core 3
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (129.176.151.21)


This bug is very similar to this bug
http://www.openldap.org/its/index.cgi?findid=3564  - maybe the same bug. 

But, since that one could be classified as a feature request, and I think this
one is more clearly a bug, I'm filing them separately.

I tried to do the same type of load as I detailed in the above bug report -
except this time, I removed all of the extra indexes that we were adding, and
disabled schema checking (on a hunch) Since the indexing seemed to be the source
of the memory management issue - I had hopes of being able to load the whole
thing without indexes, then slapcat it, and then reload it with slapadd with the
indexes enabled - since I know that works for large datasets.

This time, I got about twice as far in the load process (roughly around 1.2
million dn's - each with about 10 attributes).  Unlike the load process in the
other bug report - this time it did not get slower as it progressed.  It was
running fine up until the point that it kicked out the following error:
LDAP: error code 80 - entry store failed

At this point, the server was using about 4 GB of ram, which I though was rather
excessive.  (Nothing else is being done on the server other than the load - no
searches at all)

I tried to connect to the server with a ldap browser, and slapd segfaulted.
Database size only made it up to 4.3 GB.  

Possibly related:
I have made the following modifications to my instance of 2.2.23 (because of
this bug http://www.openldap.org/its/index.cgi?findid=3343):
In the file 'servers/slapd/back-bdb/idl.h' I modify these two lines:
#define BDB_IDL_DB_SIZE (1<<16) /* 64K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<17) /* 128K IDL in memory*/

so that they read:
#define BDB_IDL_DB_SIZE (1<<18) /* 256K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<19) /* 512K IDL in memory*/

in the file 'include/ldap_pvt_thread.h' on the line that says:
#define LDAP_PVT_THREAD_STACK_SIZE (4*1024*1024)

change it to:
#define LDAP_PVT_THREAD_STACK_SIZE (8*1024*1024)



Followup 1

Download message
Date: Thu, 03 Mar 2005 14:54:53 -0800
From: Howard Chu <hyc@symas.com>
To: daniel.armbrust@mayo.edu
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)
daniel.armbrust@mayo.edu wrote:

>Full_Name: Dan Armbrust
>Version: 2.2.23
>OS: Fedora Core 3
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (129.176.151.21)
>
>
>This bug is very similar to this bug
>http://www.openldap.org/its/index.cgi?findid=3564  - maybe the same bug. 
>
>But, since that one could be classified as a feature request, and I think
this
>one is more clearly a bug, I'm filing them separately.
>
I don't see any difference between the crashes reported in these two 
ITS's. Since this one is more focused though, I'll close the other one.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support



Followup 2

Download message
Date: Mon, 18 Apr 2005 08:48:48 -0700
From: Howard Chu <hyc@symas.com>
To: daniel.armbrust@mayo.edu
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)
I believe this is fixed by ITS#3666.

daniel.armbrust@mayo.edu wrote:

>Full_Name: Dan Armbrust
>Version: 2.2.23
>OS: Fedora Core 3
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (129.176.151.21)
>
>
>This bug is very similar to this bug
>http://www.openldap.org/its/index.cgi?findid=3564  - maybe the same bug. 
>
>But, since that one could be classified as a feature request, and I think
this
>one is more clearly a bug, I'm filing them separately.
>
>I tried to do the same type of load as I detailed in the above bug report -
>except this time, I removed all of the extra indexes that we were adding,
and
>disabled schema checking (on a hunch) Since the indexing seemed to be the
source
>of the memory management issue - I had hopes of being able to load the whole
>thing without indexes, then slapcat it, and then reload it with slapadd with
the
>indexes enabled - since I know that works for large datasets.
>
>This time, I got about twice as far in the load process (roughly around 1.2
>million dn's - each with about 10 attributes).  Unlike the load process in
the
>other bug report - this time it did not get slower as it progressed.  It was
>running fine up until the point that it kicked out the following error:
>LDAP: error code 80 - entry store failed
>
>At this point, the server was using about 4 GB of ram, which I though was
rather
>excessive.  (Nothing else is being done on the server other than the load -
no
>searches at all)
>
>I tried to connect to the server with a ldap browser, and slapd segfaulted.
>Database size only made it up to 4.3 GB.  
>  
>


-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support



Followup 3

Download message
From: "Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu>
To: Howard Chu <hyc@symas.com>
Cc: openldap-its@openldap.org
Subject: RE: (ITS#3565) Programatic Insert Scaleability Problem (#2)
Date: Mon, 18 Apr 2005 12:55:35 -0500
 
Thanks, 

I'll try to test sometime in the near future.

Dan


-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com] 
Sent: Monday, April 18, 2005 10:49 AM
To: Armbrust, Daniel C.
Cc: openldap-its@OpenLDAP.org
Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)

I believe this is fixed by ITS#3666.

daniel.armbrust@mayo.edu wrote:

>Full_Name: Dan Armbrust
>Version: 2.2.23
>OS: Fedora Core 3
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (129.176.151.21)
>
>
>This bug is very similar to this bug
>http://www.openldap.org/its/index.cgi?findid=3564  - maybe the same bug. 
>
>But, since that one could be classified as a feature request, and I think
this
>one is more clearly a bug, I'm filing them separately.
>
>I tried to do the same type of load as I detailed in the above bug report -
>except this time, I removed all of the extra indexes that we were adding,
and
>disabled schema checking (on a hunch) Since the indexing seemed to be the
source
>of the memory management issue - I had hopes of being able to load the whole
>thing without indexes, then slapcat it, and then reload it with slapadd with
the
>indexes enabled - since I know that works for large datasets.
>
>This time, I got about twice as far in the load process (roughly around 1.2
>million dn's - each with about 10 attributes).  Unlike the load process in
the
>other bug report - this time it did not get slower as it progressed.  It was
>running fine up until the point that it kicked out the following error:
>LDAP: error code 80 - entry store failed
>
>At this point, the server was using about 4 GB of ram, which I though was
rather
>excessive.  (Nothing else is being done on the server other than the load -
no
>searches at all)
>
>I tried to connect to the server with a ldap browser, and slapd segfaulted.
>Database size only made it up to 4.3 GB.  
>  
>


-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support



Followup 4

Download message
From: "Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu>
To: Howard Chu <hyc@symas.com>
Cc: openldap-its@openldap.org
Subject: RE: (ITS#3565) Programatic Insert Scaleability Problem (#2)
Date: Fri, 13 May 2005 09:03:35 -0500
I just finished retesting my DB load with 2.2.26 - and it worked!  I didn't
notice any significant memory leaks, and the performance remained more or less
stable through the entire load.  And it didn't crash :) 

Thanks for letting me know about the bug fixes.  This bug can be closed.

For reference, I was able to load (programmatically from java - not slapadd) a
SQL 2.1 GB database with ~2.5 million rows into a 10 GB bdb openldap database,
(with heavy indexing) - in about 16 hours.  Previously, I had only been able to
load about 10% of it.

Thanks for your expertise yet again.

Dan


-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com] 
Sent: Monday, April 18, 2005 10:49 AM
To: Armbrust, Daniel C.
Cc: openldap-its@OpenLDAP.org
Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)

I believe this is fixed by ITS#3666.


Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org