Logged in as guest
Viewing Archive.Incoming/3565 Full headers
Major security issue: yes no
Notes: same as ITS#3666? fixed in 2.2.25 Notification:
Date: Wed, 23 Feb 2005 14:30:05 GMT From: daniel.armbrust@mayo.edu To: openldap-its@OpenLDAP.org Subject: Programatic Insert Scaleability Problem (#2)
Full_Name: Dan Armbrust Version: 2.2.23 OS: Fedora Core 3 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.176.151.21) This bug is very similar to this bug http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. But, since that one could be classified as a feature request, and I think this one is more clearly a bug, I'm filing them separately. I tried to do the same type of load as I detailed in the above bug report - except this time, I removed all of the extra indexes that we were adding, and disabled schema checking (on a hunch) Since the indexing seemed to be the source of the memory management issue - I had hopes of being able to load the whole thing without indexes, then slapcat it, and then reload it with slapadd with the indexes enabled - since I know that works for large datasets. This time, I got about twice as far in the load process (roughly around 1.2 million dn's - each with about 10 attributes). Unlike the load process in the other bug report - this time it did not get slower as it progressed. It was running fine up until the point that it kicked out the following error: LDAP: error code 80 - entry store failed At this point, the server was using about 4 GB of ram, which I though was rather excessive. (Nothing else is being done on the server other than the load - no searches at all) I tried to connect to the server with a ldap browser, and slapd segfaulted. Database size only made it up to 4.3 GB. Possibly related: I have made the following modifications to my instance of 2.2.23 (because of this bug http://www.openldap.org/its/index.cgi?findid=3343): In the file 'servers/slapd/back-bdb/idl.h' I modify these two lines: #define BDB_IDL_DB_SIZE (1<<16) /* 64K IDL on disk*/ #define BDB_IDL_UM_SIZE (1<<17) /* 128K IDL in memory*/ so that they read: #define BDB_IDL_DB_SIZE (1<<18) /* 256K IDL on disk*/ #define BDB_IDL_UM_SIZE (1<<19) /* 512K IDL in memory*/ in the file 'include/ldap_pvt_thread.h' on the line that says: #define LDAP_PVT_THREAD_STACK_SIZE (4*1024*1024) change it to: #define LDAP_PVT_THREAD_STACK_SIZE (8*1024*1024)
Date: Thu, 03 Mar 2005 14:54:53 -0800 From: Howard Chu <hyc@symas.com> To: daniel.armbrust@mayo.edu CC: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)
daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > I don't see any difference between the crashes reported in these two ITS's. Since this one is more focused though, I'll close the other one. -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
Date: Mon, 18 Apr 2005 08:48:48 -0700 From: Howard Chu <hyc@symas.com> To: daniel.armbrust@mayo.edu CC: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2)
I believe this is fixed by ITS#3666. daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > >I tried to do the same type of load as I detailed in the above bug report - >except this time, I removed all of the extra indexes that we were adding, and >disabled schema checking (on a hunch) Since the indexing seemed to be the source >of the memory management issue - I had hopes of being able to load the whole >thing without indexes, then slapcat it, and then reload it with slapadd with the >indexes enabled - since I know that works for large datasets. > >This time, I got about twice as far in the load process (roughly around 1.2 >million dn's - each with about 10 attributes). Unlike the load process in the >other bug report - this time it did not get slower as it progressed. It was >running fine up until the point that it kicked out the following error: >LDAP: error code 80 - entry store failed > >At this point, the server was using about 4 GB of ram, which I though was rather >excessive. (Nothing else is being done on the server other than the load - no >searches at all) > >I tried to connect to the server with a ldap browser, and slapd segfaulted. >Database size only made it up to 4.3 GB. > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
From: "Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu> To: Howard Chu <hyc@symas.com> Cc: openldap-its@openldap.org Subject: RE: (ITS#3565) Programatic Insert Scaleability Problem (#2) Date: Mon, 18 Apr 2005 12:55:35 -0500
Thanks, I'll try to test sometime in the near future. Dan -----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Monday, April 18, 2005 10:49 AM To: Armbrust, Daniel C. Cc: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2) I believe this is fixed by ITS#3666. daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > >I tried to do the same type of load as I detailed in the above bug report - >except this time, I removed all of the extra indexes that we were adding, and >disabled schema checking (on a hunch) Since the indexing seemed to be the source >of the memory management issue - I had hopes of being able to load the whole >thing without indexes, then slapcat it, and then reload it with slapadd with the >indexes enabled - since I know that works for large datasets. > >This time, I got about twice as far in the load process (roughly around 1.2 >million dn's - each with about 10 attributes). Unlike the load process in the >other bug report - this time it did not get slower as it progressed. It was >running fine up until the point that it kicked out the following error: >LDAP: error code 80 - entry store failed > >At this point, the server was using about 4 GB of ram, which I though was rather >excessive. (Nothing else is being done on the server other than the load - no >searches at all) > >I tried to connect to the server with a ldap browser, and slapd segfaulted. >Database size only made it up to 4.3 GB. > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
From: "Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu> To: Howard Chu <hyc@symas.com> Cc: openldap-its@openldap.org Subject: RE: (ITS#3565) Programatic Insert Scaleability Problem (#2) Date: Fri, 13 May 2005 09:03:35 -0500
I just finished retesting my DB load with 2.2.26 - and it worked! I didn't notice any significant memory leaks, and the performance remained more or less stable through the entire load. And it didn't crash :) Thanks for letting me know about the bug fixes. This bug can be closed. For reference, I was able to load (programmatically from java - not slapadd) a SQL 2.1 GB database with ~2.5 million rows into a 10 GB bdb openldap database, (with heavy indexing) - in about 16 hours. Previously, I had only been able to load about 10% of it. Thanks for your expertise yet again. Dan -----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Monday, April 18, 2005 10:49 AM To: Armbrust, Daniel C. Cc: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2) I believe this is fixed by ITS#3666.
______________ © Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org
