Full_Name: Dan Armbrust Version: 2.2.23 OS: Fedora Core 3 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.176.151.21) This bug is very similar to this bug http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. But, since that one could be classified as a feature request, and I think this one is more clearly a bug, I'm filing them separately. I tried to do the same type of load as I detailed in the above bug report - except this time, I removed all of the extra indexes that we were adding, and disabled schema checking (on a hunch) Since the indexing seemed to be the source of the memory management issue - I had hopes of being able to load the whole thing without indexes, then slapcat it, and then reload it with slapadd with the indexes enabled - since I know that works for large datasets. This time, I got about twice as far in the load process (roughly around 1.2 million dn's - each with about 10 attributes). Unlike the load process in the other bug report - this time it did not get slower as it progressed. It was running fine up until the point that it kicked out the following error: LDAP: error code 80 - entry store failed At this point, the server was using about 4 GB of ram, which I though was rather excessive. (Nothing else is being done on the server other than the load - no searches at all) I tried to connect to the server with a ldap browser, and slapd segfaulted. Database size only made it up to 4.3 GB. Possibly related: I have made the following modifications to my instance of 2.2.23 (because of this bug http://www.openldap.org/its/index.cgi?findid=3343): In the file 'servers/slapd/back-bdb/idl.h' I modify these two lines: #define BDB_IDL_DB_SIZE (1<<16) /* 64K IDL on disk*/ #define BDB_IDL_UM_SIZE (1<<17) /* 128K IDL in memory*/ so that they read: #define BDB_IDL_DB_SIZE (1<<18) /* 256K IDL on disk*/ #define BDB_IDL_UM_SIZE (1<<19) /* 512K IDL in memory*/ in the file 'include/ldap_pvt_thread.h' on the line that says: #define LDAP_PVT_THREAD_STACK_SIZE (4*1024*1024) change it to: #define LDAP_PVT_THREAD_STACK_SIZE (8*1024*1024)
daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > I don't see any difference between the crashes reported in these two ITS's. Since this one is more focused though, I'll close the other one. -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
I believe this is fixed by ITS#3666. daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > >I tried to do the same type of load as I detailed in the above bug report - >except this time, I removed all of the extra indexes that we were adding, and >disabled schema checking (on a hunch) Since the indexing seemed to be the source >of the memory management issue - I had hopes of being able to load the whole >thing without indexes, then slapcat it, and then reload it with slapadd with the >indexes enabled - since I know that works for large datasets. > >This time, I got about twice as far in the load process (roughly around 1.2 >million dn's - each with about 10 attributes). Unlike the load process in the >other bug report - this time it did not get slower as it progressed. It was >running fine up until the point that it kicked out the following error: >LDAP: error code 80 - entry store failed > >At this point, the server was using about 4 GB of ram, which I though was rather >excessive. (Nothing else is being done on the server other than the load - no >searches at all) > >I tried to connect to the server with a ldap browser, and slapd segfaulted. >Database size only made it up to 4.3 GB. > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
changed notes changed state Open to Feedback
Thanks, I'll try to test sometime in the near future. Dan -----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Monday, April 18, 2005 10:49 AM To: Armbrust, Daniel C. Cc: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2) I believe this is fixed by ITS#3666. daniel.armbrust@mayo.edu wrote: >Full_Name: Dan Armbrust >Version: 2.2.23 >OS: Fedora Core 3 >URL: ftp://ftp.openldap.org/incoming/ >Submission from: (NULL) (129.176.151.21) > > >This bug is very similar to this bug >http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug. > >But, since that one could be classified as a feature request, and I think this >one is more clearly a bug, I'm filing them separately. > >I tried to do the same type of load as I detailed in the above bug report - >except this time, I removed all of the extra indexes that we were adding, and >disabled schema checking (on a hunch) Since the indexing seemed to be the source >of the memory management issue - I had hopes of being able to load the whole >thing without indexes, then slapcat it, and then reload it with slapadd with the >indexes enabled - since I know that works for large datasets. > >This time, I got about twice as far in the load process (roughly around 1.2 >million dn's - each with about 10 attributes). Unlike the load process in the >other bug report - this time it did not get slower as it progressed. It was >running fine up until the point that it kicked out the following error: >LDAP: error code 80 - entry store failed > >At this point, the server was using about 4 GB of ram, which I though was rather >excessive. (Nothing else is being done on the server other than the load - no >searches at all) > >I tried to connect to the server with a ldap browser, and slapd segfaulted. >Database size only made it up to 4.3 GB. > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
changed notes changed state Feedback to Suspended
I just finished retesting my DB load with 2.2.26 - and it worked! I didn't notice any significant memory leaks, and the performance remained more or less stable through the entire load. And it didn't crash :) Thanks for letting me know about the bug fixes. This bug can be closed. For reference, I was able to load (programmatically from java - not slapadd) a SQL 2.1 GB database with ~2.5 million rows into a 10 GB bdb openldap database, (with heavy indexing) - in about 16 hours. Previously, I had only been able to load about 10% of it. Thanks for your expertise yet again. Dan -----Original Message----- From: Howard Chu [mailto:hyc@symas.com] Sent: Monday, April 18, 2005 10:49 AM To: Armbrust, Daniel C. Cc: openldap-its@OpenLDAP.org Subject: Re: (ITS#3565) Programatic Insert Scaleability Problem (#2) I believe this is fixed by ITS#3666.
changed notes changed state Suspended to Closed
moved from Incoming to Archive.Incoming
same as ITS#3666? fixed in 2.2.25