[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#3851) Berkeley DB Scalability Patch
--On Thursday, July 28, 2005 3:26 PM -0400 Jong-Hyuk
<jongchoi@watson.ibm.com> wrote:
>
>> This isn't what I was saying. If it takes 15 minutes instead of 10
>> for 100K entries, and increases the time from 11 hours to 20 hours for
>> 4 million entries if you
>
> This is usually not the case in many engineering problems including
> scalability.
>
>> use a lot of indices, then its usefulness becomes suspect. Which is
>> why I'm curious if you have any data points using a large number of
>> indices that indicates
>
> Well, if indexing is turned out to be a real suspect, it will have to be
> handled as a separate, orthogonal issue.
I would say that indexing is definitely a serious problem with your patch.
I set up a 2 million entry LDIF file, and used a 1.68 GB cache. With only
3 indices, everything looked great:
Time w/o patch:
1609.570u 36.590s 29:30.58 92.9% 0+0k 0+0io 460966pf+0w
Time w/ patch:
749.850u 33.920s 16:03.15 81.3% 0+0k 0+0io 291633pf+0w
So a nearly 2:1 performance GAIN.
However, if you give it just a medium amount of indices (21), the
performance is atrocious. It was so bad, I had to halt the slapadd.
Time w/o patch (to get to 35%):
[############# ] ( 34%) 01h15m 687344 obj
[############# ] ( 34%) 01h16m 687952 obj
[############# ] ( 34%) 01h16m 688256 obj
[############# ] ( 34%) 01h17m 688864 obj
[############# ] ( 34%) 01h17m 689168 obj
[############# ] ( 34%) 01h17m 689472 obj
[############# ] ( 34%) 01h17m 689776 obj
[############# ] ( 34%) 01h17m 690080 obj
[############# ] ( 34%) 01h17m 690384 obj
[############# ] ( 34%) 01h17m 690688 obj
[############# ] ( 34%) 01h17m 690992 obj
[############# ] ( 34%) 01h17m 691296 obj
[############# ] ( 34%) 01h18m 692208 obj
[############# ] ( 34%) 01h18m 694640 obj
[############# ] ( 34%) 01h18m 695856 obj
[############# ] ( 34%) 01h18m 696464 obj
[############# ] ( 34%) 01h18m 697072 obj
[############# ] ( 34%) 01h19m 697680 obj
[############# ] ( 34%) 01h19m 697984 obj
[############# ] ( 34%) 01h19m 698896 obj
[############# ] ( 34%) 01h19m 699200 obj
[############# ] ( 34%) 01h19m 700416 obj
[############# ] ( 34%) 01h19m 701024 obj
[############## ] ( 35%) 01h19m 703152 obj
[############## ] ( 35%) 01h19m 703760 obj
[############## ] ( 35%) 01h20m 704368 obj
Time w/ patch (to get to 35%):
[############# ] ( 34%) 20h08m 691296 obj
[############# ] ( 34%) 20h11m 691600 obj
[############# ] ( 34%) 20h28m 693424 obj
[############# ] ( 34%) 20h40m 694640 obj
[############# ] ( 34%) 20h48m 695552 obj
[############# ] ( 34%) 21h08m 697680 obj
[############# ] ( 34%) 21h42m 701328 obj
[############## ] ( 35%) 21h59m 703152 obj
As you can see here, the performance LOSS is nearly 22:1 (22 hours to get
where the unpatched version of BDB gets in 1).
--Quanah
--
Quanah Gibson-Mount
Product Engineer
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>