Re: (ITS#3851) Berkeley DB Scalability Patch

[jongchoi@watson.ibm.com]

Quanah, thanks for your inputs with experimental data. I believe that 
you need more data points to draw a meaningful comparison, though. 
Especially, I'd like to see 1) what the result for larger DITs, say ones 
having 1 ~ 4 million entries, would be; and 2) the BDB cache sizes and 
actual RSS are needed for each of these experiments. I'm afraid that you 
are looking at only one point in the whole scalability picture.

Actually, I was busy doing baseline scalability evaluation so far. I was 
able to add up to 64 million entries within a reasonable amount of time 
(currently doing the same with 128 million).

- Jong-Hyuk

Quanah Gibson-Mount wrote:

>
>
> --On Friday, July 15, 2005 6:57 PM -0700 Quanah Gibson-Mount 
> <quanah@symas.com> wrote:
>
>> Hi Jong,
>>
>> The second test looks good.
>>
>> Without patch:
>> 630.304 ops/second
>> 182.671 mod/second
>> 447.632 searches/second
>>
>> With patch:
>> 632.156 ops/second
>> 183.216 mod/second
>> 448.941 searches/second
>>
>>
>> So a definite (though slight in my small database case) improvement.
>> Yeah. :)
>
>
> Hi Jong,
>
> I'm curious if you have tested this patch on DB's with a large number 
> of indices.  I put together a 100K entry DB (I unfortunately do not 
> have 12GB RAM machines at my disposal) with 21 indices in it:
>
> index   default eq
> index   objectClass
> index   uid
> index   departmentNumber
> index   cn              eq,sub,approx
> index   sn              eq,sub,approx
> index   l
> index   ou
> index   telephonenumber eq,sub
> index   userpassword
> index   givenName       eq,sub,approx
> index   mail
> index   carLicense
> index   employeeType
> index   homephone       eq,sub
> index   mobile  eq,sub
> index   pager   eq,sub
> index   o
> index   roomNumber
> index   preferredLanguage
> index   postalCode
> index   st
>
>
> I then tested the difference in load times with and without your DB 
> scalability patch.  The effect I saw of having a large set of indices 
> is that the DB scalability patch does not perform as well as the 
> normal BDB process.
>
> Results w/o patch:
> -------------------------------------
> 538.67u 7.76s 9:13.34 98.7%
> 537.08u 7.69s 9:12.05 98.6%
> 535.90u 8.17s 9:10.99 98.7%
>
>
> Results w/ patch:
> -------------------------------------
> 726.48u 13.03s 14:41.07 83.9%
> 727.89u 12.04s 14:37.65 84.3%
> 727.35u 12.06s 14:39.21 84.0%
>
>
> One thing immediately evident is that the CPU usage drops a good 
> 14-15% than without the patch, with a corresponding 1/3 increase in 
> total time to load the database.
>
> Have you done any testing of your patch on large scale DB's with a 
> good number of indices?
>
> Regards,
> Quanah
>
> -- 
> Quanah Gibson-Mount
> Product Engineer
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
>
>
>