Re: strange performance result

[Andreas <andreas@conectiva.com.br>]

On Tue, Jun 17, 2003 at 09:49:36AM -0400, james liang wrote:
> Many data entries are inserted into openldap in under 100ms.  However, every
> 6 or so entries, an entry will take more than 1 to 2 second to insert.  This
> happens even thought the size of all the records are exact the same. 

I would guess your log file size is too small. Tune at least these two parameters
in your DB_CONFIG file:
set_lg_max      104857600 
set_lg_bsize    26214400 

The lines above set the log file to be 100M max and a buffer size at 100M/4

> 2.  the CPU and file IO are grossly under utilized during the data insertion
> 
> The server is a dual CPU DELL box running Linux (redhat 8).  The CPU
> utilization seems to peak at 20% and the IO utilization is just 15%.  I tried
> to run multiple data insertion processes in parallel to no avail. The
> collective throught put stays flat and the CPU and IO utilization stays the
> same.   There seems to be a common point of contention in the system that is
> severly limiting the degree of parallism.  Another thing I notice is that the
> cpu and io utilization seems to degrade gradually as the list gets larger and
> larger.  

You need to add cache. Check the following parameter in your DB_CONFIG file:
set_cachesize   0 209715200 0 

The line above sets a DB cache of 200M. I don't know how much memory you have, but
check out my results with 1 million entries:
set_cachesize   slapadd time (with indexes)     Obs
1M              1937min                         Two scsi HDs
200M            478min                          1 HD only
800M            194min                          1 HD only

With 800M cache the CPU no longer sits idle here during the bulk load :)

> I tried tweaking dbnosync and cachesize but it didn't help.  I am pretty sure
> the network(100 megabit ethernet) is not the issue.  I tried to run the data
> insertion app directly in the machine that is running openldap and got the
> same result.

You should really use slapadd for this, and there should be no slapd process running.