[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#3851) Berkeley DB Scalability Patch



Quanah,
Do you also have a result without "-q" option ?
- Jong-Hyuk

Quanah Gibson-Mount wrote:

>
>
> --On Wednesday, August 10, 2005 11:55 AM -0400 Jong-Hyuk 
> <jongchoi@watson.ibm.com> wrote:
>
>> Here is a result with various degrees of indexing for 2 million 
>> entries :
>>
>> ftp://ftp.openldap.org/incoming/ScalabilityWithIndexing.png
>>
>> As shown in the graph, for 13 indexing (6 equality, 2 equality &
>> substring, 5 equality, substring & approximate), it took almost a day
>> (only less one hour) without the Berkeley DB scalability patch while it
>> took only six hours with the patch. It would have been decreased further
>> if a separate indexing were performed by using slapindex after the
>> initial slapadd. Although you tested with 21 indexing, the number of
>> substring/approximate indexing are the same at seven in both cases. The
>> effect of additional equality indexing was found negligible through my
>> performance evaluation.
>>
>> As a result, I would say my results contradict yours. With my 
>> performance
>> evaluation configuration, I could not find any tendency that might give
>> me a hint why you had experienced such a problem. One thing I noticed
>> from your last result is that you did not run the experiment to
>> completion. I think it is required to run each run to completion to draw
>> a meaningful result.
>
>
> Hi Jong,
>
> I've run the test to the finish for the unpatched version, and have 
> the patched version running.  The unpatched version will beat the 
> patched version by several days, at the current rate things are going.
>
> For the unpatched load, the results were:
>
> cgi3-linux:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
> [########################################] (100%) 06d08h  2000002 obj
> 277927.430u 2905.190s 152:22:40.95 51.1%        0+0k 0+0io 684572pf+0w
>
>
> Right now, the *patched* load is at:
>
> cgi3-linux:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
> [##########################              ] ( 66%) 06d13h  1331216 obj
>
> and doing a few percent every day.
>
> I think you are not doing a broad enough test to really understand the 
> validity of your patch, and I suggest testing with more than 13 
> indices, and letting it run to completion.
>
> My indices are:
>
> # Indices to maintain
> index   default eq
> index   objectClass
> index   uid
> index   departmentNumber
> index   cn              eq,sub,approx
> index   sn              eq,sub,approx
> index   l
> index   ou
> index   telephonenumber eq,sub
> index   userpassword
> index   givenName       eq,sub,approx
> index   mail
> index   carLicense
> index   employeeType
> index   homephone       eq,sub
> index   mobile  eq,sub
> index   pager   eq,sub
> index   o
> index   roomNumber
> index   preferredLanguage
> index   postalCode
> index   st
>
> and DB_CONFIG is:
>
> # $Id: DB_CONFIG,v 1.1 2004/04/02 19:27:11 quanah Exp $
> set_cachesize 1 713031680 0
> set_lg_regionmax 262144
> set_lg_bsize 2097152
> set_lg_dir /var/log/bdb
>
> Regards,
> Quanah
>
> -- 
> Quanah Gibson-Mount
> Product Engineer
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
>
>
>