[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#3851) Berkeley DB Scalability Patch
--On Thursday, September 01, 2005 10:03 AM -0400 Jong-Hyuk
<jongchoi@watson.ibm.com> wrote:
>
>>
>> Final load time with the BDB scalability pach:
>>
>> cgi3-linux:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
>> [########################################] (100%) 02w05h 2000002 obj
>> \HH
>> 6485.120u 5834.390s 341:15:40.01 1.0% 0+0k 0+0io 981489pf+0w
>>
>>
>> So more than twice as long the 152:22:40.95 time it took to load the
>> database without the patch.
>
>
> I recommend you to confirm your system setup first. Assuming that you are
> experimenting on a reasonably up-to-date platform, more than 6 days for
> adding 2 million entries seems too much even without the BDB scalability
> patch. When I compare your numbers with my results, your configuration is
> 6.6x slower than mine originally (before applying BDB scalability patch).
> Fair comparison could follow after first bring the baseline performance
> in sync.
After moving to a system with 3GB of memory instead of 2GB, I actually see
some performance gain from your patch:
tribes:~> cat 13indices-time-base
sundial1:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
[########################################] (100%) 03d15h 2000002 obj
205538.150u 1335.460s 87:48:16.93 65.4% 0+0k 0+0io 438073pf+0w
tribes:~> cat 13indices-time-patched
sundial1:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
[########################################] (100%) 03d04h 2000002 obj
5123.990u 2264.730s 76:14:54.16 2.6% 0+0k 0+0io 423456pf+0w
Or a savings of 11.5 hours in the 13 indices case.
tribes:~> cat 21indices-time-base
sundial1:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
[########################################] (100%) 04d04h 2000002 obj
232486.230u 1511.880s 100:56:51.30 64.3% 0+0k 0+0io 437784pf+0w
tribes:~> cat 21indices-time-patched
sundial1:/tmp/ldif# time slapadd -q -l 2M-peons.ldif
[########################################] (100%) 03d17h 2000002 obj
5709.420u 2511.500s 89:19:34.61 2.5% 0+0k 0+0io 429203pf+0w
Or a savings of 12 hours in the 21 indices case.
So in both cases, there is a moderate decrease in the amount of time it
took to load the database.
Based on the differences in how your patch performs on the 2GB & 3GB
systems, it appears that the more memory BDB has available to it *past* the
DB_CACHE, the better it runs. If there is too little space there, the
algorithm is substantially worse than the current BDB memory allocator. If
there is enough space, it is faster. I'm betting that the reason it is so
fast on your systems is because you have a very large amount of space
between the DB_CACHE and the upper memory limit. So on systems with a very
small memory pool available, this patch has the possibility of making
things worse, rather than better.
--Quanah
--
Quanah Gibson-Mount
Product Engineer
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>