[Date Prev][Date Next]
(ITS#3565) Programatic Insert Scaleability Problem (#2)
Full_Name: Dan Armbrust
OS: Fedora Core 3
Submission from: (NULL) (18.104.22.168)
This bug is very similar to this bug
http://www.openldap.org/its/index.cgi?findid=3564 - maybe the same bug.
But, since that one could be classified as a feature request, and I think this
one is more clearly a bug, I'm filing them separately.
I tried to do the same type of load as I detailed in the above bug report -
except this time, I removed all of the extra indexes that we were adding, and
disabled schema checking (on a hunch) Since the indexing seemed to be the source
of the memory management issue - I had hopes of being able to load the whole
thing without indexes, then slapcat it, and then reload it with slapadd with the
indexes enabled - since I know that works for large datasets.
This time, I got about twice as far in the load process (roughly around 1.2
million dn's - each with about 10 attributes). Unlike the load process in the
other bug report - this time it did not get slower as it progressed. It was
running fine up until the point that it kicked out the following error:
LDAP: error code 80 - entry store failed
At this point, the server was using about 4 GB of ram, which I though was rather
excessive. (Nothing else is being done on the server other than the load - no
searches at all)
I tried to connect to the server with a ldap browser, and slapd segfaulted.
Database size only made it up to 4.3 GB.
I have made the following modifications to my instance of 2.2.23 (because of
this bug http://www.openldap.org/its/index.cgi?findid=3343):
In the file 'servers/slapd/back-bdb/idl.h' I modify these two lines:
#define BDB_IDL_DB_SIZE (1<<16) /* 64K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<17) /* 128K IDL in memory*/
so that they read:
#define BDB_IDL_DB_SIZE (1<<18) /* 256K IDL on disk*/
#define BDB_IDL_UM_SIZE (1<<19) /* 512K IDL in memory*/
in the file 'include/ldap_pvt_thread.h' on the line that says:
#define LDAP_PVT_THREAD_STACK_SIZE (4*1024*1024)
change it to:
#define LDAP_PVT_THREAD_STACK_SIZE (8*1024*1024)