(ITS#3565) Programatic Insert Scaleability Problem (#2)

[daniel.armbrust@mayo.edu]

Full_Name: Dan Armbrust
Version: 2.2.23
OS: Fedora Core 3
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (129.176.151.21)


This bug is very similar to this bug
http://www.openldap.org/its/index.cgi?findid=3564  - maybe the same bug. 

But, since that one could be classified as a feature request, and I think this
one is more clearly a bug, I'm filing them separately.

I tried to do the same type of load as I detailed in the above bug report -
except this time, I removed all of the extra indexes that we were adding, and
disabled schema checking (on a hunch) Since the indexing seemed to be the source
of the memory management issue - I had hopes of being able to load the whole
thing without indexes, then slapcat it, and then reload it with slapadd with the
indexes enabled - since I know that works for large datasets.

This time, I got about twice as far in the load process (roughly around 1.2
million dn's - each with about 10 attributes).  Unlike the load process in the
other bug report - this time it did not get slower as it progressed.  It was
running fine up until the point that it kicked out the following error:
LDAP: error code 80 - entry store failed

At this point, the server was using about 4 GB of ram, which I though was rather
excessive.  (Nothing else is being done on the server other than the load - no
searches at all)

I tried to connect to the server with a ldap browser, and slapd segfaulted.
Database size only made it up to 4.3 GB.  

Possibly related:
I have made the following modifications to my instance of 2.2.23 (because of
this bug http://www.openldap.org/its/index.cgi?findid=3343):
In the file 'servers/slapd/back-bdb/idl.h' I modify these two lines:
#define BDB_IDL_DB_SIZE (1<<16) /* 64K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<17) /* 128K IDL in memory*/

so that they read:
#define BDB_IDL_DB_SIZE (1<<18) /* 256K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<19) /* 512K IDL in memory*/

in the file 'include/ldap_pvt_thread.h' on the line that says:
#define LDAP_PVT_THREAD_STACK_SIZE (4*1024*1024)

change it to:
#define LDAP_PVT_THREAD_STACK_SIZE (8*1024*1024)