Re: (ITS#3851) Berkeley DB Scalability Patch

[jongchoi@watson.ibm.com]

Moving this to ITS. Comments inline.

Quanah Gibson-Mount wrote:

>
>
> --On Tuesday, October 04, 2005 9:45 AM -0500 Eric Irrgang 
> <erici@motown.cc.utexas.edu> wrote:
>
>> I've been following this thread with great interest but I was 
>> wondering if
>> you could provide more clear numbers on the sizes of the data you are
>> working with.  I'm trying to figure out where I fit into these scaling
>> benchmarks.  Could you provide some information on the actual sizes of
>> your db files at the large number-of-entries, many indexes, db cache
>> size, etc?
>>
>> For instance, my id2entry file is 6GB and my indexes are an additional
>> 3GB for 2 million entries and 47 index files maintaining about twice as
>> many attribute/search-type indexes.  To get consistently good 
>> performance
>> I have to keep a db cache of at least 6 Gigabytes right now with OL 
>> 2.2.26
>> and I would be very interested in improving write performance or being
>> able to use less cache, but I want to get a better idea of what I should
>> be expecting.
>
>
> Hi Eric,
>
> This ITS mainly has to do with loading data into back-bdb, as opposed 
> to how it performs long term.  In my tests, I used a 1.7GB DB Cache, 
> indexing 21 attributes out of 28 attributes.

As I have shown its search / search-update performance in this message 
(http://www.openldap.org/its/index.cgi/Development?id=3851;selectid=3851#followup6), 
it has a significant impact on the normal search / update workload as 
well as the directory population.

>
> My /db dir shows:
>
> 58M     carLicense.bdb
> 648M    cn.bdb
> 56M     departmentNumber.bdb
> 444M    dn2id.bdb
> 7.7M    employeeType.bdb
> 335M    givenName.bdb
> 444M    homePhone.bdb
> 3.5G    id2entry.bdb
> 24M     l.bdb
> 57M     mail.bdb
> 445M    mobile.bdb
> 664K    o.bdb
> 2.6M    objectClass.bdb
> 664K    ou.bdb
> 444M    pager.bdb
> 42M     postalCode.bdb
> 20M     preferredLanguage.bdb
> 27M     roomNumber.bdb
> 419M    sn.bdb
> 20M     st.bdb
> 443M    telephoneNumber.bdb
> 56M     uid.bdb
> 664K    userPassword.bdb
>
>
>
> All of my testing indicates that your DB_CACHE while running (not 
> slapadding) must be at least as big as your id2entry bdb file, 
> regardless of this patch.


Again, numbers please.

>
> --Quanah
>
> -- 
> Quanah Gibson-Mount
> Product Engineer
> Symas Corporation
> Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
> <http://www.symas.com>
>
>