My test was the same for the 2 solutions : mass-provisioning of 1000 new subscriptions (1 subscription = 4 ldap entry in bdb1 backend + 3 ldap entry in bdb2 backend => cf. slapd.conf for backends definition)
Here's my results:
solution 1:
time to execute the provisioning = 6mn43s
only 2 running slapd with %CPU = 1-2% in average
db_stat with option -e, -m, -t, -c, -l in the 2 backend
seems to be correct
solution2:
time to execute the provisioning = 20mn28s
8 running slapd with %CPU = 20% in average (with for
one of them a peak = 32%CPU)
db_stat with options -m, -t, -c in the 2 backend seems to be
correct
db_stat with option -l gives a bad result: "1355 The number of region
locks granted after waiting." for bdb1 backend
and "3411 The number of region locks granted after waiting." for bdb2
backend
Cf. in attachment the result of db_stat command for solution n°2
(only bdb1 backend)
Cf. in attachment too the DB_CONFIG file of the bdb1 backend for example
So my question is: How doing to obtain best results with solution n°2
(with tuning BerkeleyDB ??)
the solution n°1 being disadvised ?
Best regards,
Helene
> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org [mailto:owner-openldap-
> software@OpenLDAP.org] On Behalf Of Helene.Lastennet@alcatel.fr
> Sent: Wednesday, November 19, 2003 9:48 AM
> To: Quanah Gibson-Mount
> Cc: openldap-software@OpenLDAP.org
> Subject: Re: [openldap 2.1.12 + BerkeleyDb 4.1.24] db_checkpoint pb
>
> So it's sure if I choose to set the checkpointing directive in my
> slapd.conf
> like that: "checkpoint 250 15" rather than to use db_checkpoint in a
> script
> executed every hour by crontab ?? (cf. slapd.conf + DB_CONFIG)
> Please, I would like to be sure because there is a FAQ which advises the
> use of
> the script rather than the checkpoint directive in the slapd.conf ?I don't know which FAQ you are referring to, so I can't comment on that. I
can tell you, though, that this is the checkpointing/archiving method we
recommend to our customers who are using our commercial builds, and they are
quite happy with it.Your checkpoint statement will cause a checkpoint every 15 minutes or 250K
of log data. This is four times more frequent than your crontab, so it may
have an adverse impact on performance. OTOH, it provides a bit better
recovery capability if your system should ever go down. The values you come
up with for checkpoint need to be those that work best for the way you are
using OpenLDAP.In the end, all opinions are worth what you paid for them and Your Mileage
May Vary ;-)Cheers,
Matthew Hardin
Symas Corporation
Packaged, certified, and supported LDAP software:
http://www.symas.net/download> > --On Wednesday, November 19, 2003 2:17 PM +0100
> Helene.Lastennet@alcatel.fr
> > wrote:
> >
> > >> Hi,
> > >>
> > >> Does this problem occurs if you use the checkpoint
> > >> directive in slapd.conf instead?
> > >
> > > I don't know ...
> > > I chose to use db_checkpoint of BerkeleyDB instead of checkpoint in
> > > slapd.conf
> > > because when you read the openldap doc (+ FAQ) it seems to be better
> to
> > > make tuning directly in the bdb-backend rather than to use the file of
> > > configuration slapd.conf ...?
> > > So what can I do to be sure ?
> >
> > Helene,
> >
> > You may wish to turn on the slapd checkpointing. Although we also
> > configure most of our BDB parameters via DB_CONFIG, we set the
> > checkpointing in slapd.conf.
> >
> > --Quanah
> >
> > --
> > Quanah Gibson-Mount
> > Principal Software Developer
> > ITSS/TSS/Computing Systems
> > ITSS/TSS/Infrastructure Operations
> > Stanford University
> > GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Attachment:
slapd.conf
Description: application/unknown-content-type-conf_auto_file
62MB 513KB 752B Total cache size. 1 Number of caches. 62MB 520KB Pool individual cache size. 0 Requested pages mapped into the process' address space. 140M Requested pages found in the cache (100%). 13146 Requested pages not found in the cache. 166173 Pages created in the cache. 10365 Pages read into the cache. 221412 Pages written from the cache to the backing file. 1619 Clean pages forced from the cache. 164312 Dirty pages forced from the cache. 0 Dirty pages written by trickle-sync thread. 10607 Current total page count. 10607 Current clean page count. 0 Current dirty page count. 8191 Number of hash buckets used for page location. 140M Total number of times hash chains searched for a page. 5 The longest hash chain searched for a page. 275M Total number of hash buckets examined for page location. 328M The number of hash bucket locks granted without waiting. 3 The number of hash bucket locks granted after waiting. 2 The maximum number of times any hash bucket lock was waited for. 808032 The number of region locks granted without waiting. 65 The number of region locks granted after waiting. 176573 The number of page allocations. 435241 The number of hash buckets examined during allocations 4643 The max number of hash buckets examined for an allocation 165931 The number of pages examined during allocations 2247 The max number of pages examined for an allocation =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: maillogin.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 6998676 Requested pages found in the cache (100%). 2486 Requested pages not found in the cache. 1119 Pages created in the cache. 1826 Pages read into the cache. 6529 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: submxid.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 11M Requested pages found in the cache (100%). 232 Requested pages not found in the cache. 328 Pages created in the cache. 232 Pages read into the cache. 4568 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: pnnumber.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 7018000 Requested pages found in the cache (100%). 2488 Requested pages not found in the cache. 889 Pages created in the cache. 2047 Pages read into the cache. 6382 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: uid.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 18M Requested pages found in the cache (100%). 7191 Requested pages not found in the cache. 2753 Pages created in the cache. 5511 Pages read into the cache. 16732 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: objectClass.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 43M Requested pages found in the cache (100%). 621 Requested pages not found in the cache. 818 Pages created in the cache. 621 Pages read into the cache. 12459 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: dn2id.bdb 4096 Page size. 0 Requested pages mapped into the process' address space. 48M Requested pages found in the cache (100%). 124 Requested pages not found in the cache. 112528 Pages created in the cache. 124 Pages read into the cache. 124681 Pages written from the cache to the backing file. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Pool File: id2entry.bdb 16384 Page size. 0 Requested pages mapped into the process' address space. 3940924 Requested pages found in the cache (100%). 4 Requested pages not found in the cache. 47738 Pages created in the cache. 4 Pages read into the cache. 50061 Pages written from the cache to the backing file.
40988 Log magic number. 7 Log version number. 2MB Log record cache size. 0600 Log file mode. 10Mb Current log file size. 3GB 775MB 293KB 368B Log bytes written. 3GB 775MB 293KB 368B Log bytes written since last checkpoint. 17002 Total log file writes. 1534 Total log file write due to overflow. 15474 Total log file flushes. 385 Current log file number. 8491134 Current log file offset. 385 On-disk log file number. 8491134 On-disk log file offset. 2 Max commits in a log flush. 0 Min commits in a log flush. 3MB Log region size. 1355 The number of region locks granted after waiting. 30M The number of region locks granted without waiting.
142 Last allocated locker ID. 2147M Current maximum unused locker ID. 9 Number of lock modes. 1000 Maximum number of locks possible. 1000 Maximum number of lockers possible. 1000 Maximum number of lock objects possible. 10 Number of current locks. 51 Maximum number of locks at any one time. 28 Number of current lockers. 63 Maximum number of lockers at any one time. 10 Number of current lock objects. 43 Maximum number of lock objects at any one time. 72M Total number of locks requested. 72M Total number of locks released. 0 Total number of lock requests failing because DB_LOCK_NOWAIT was set. 0 Total number of locks not immediately available due to conflicts. 0 Number of deadlocks. 0 Lock timeout value. 0 Number of locks that have timed out. 0 Transaction timeout value. 0 Number of transactions that have timed out. 360KB The size of the lock region.. 0 The number of region locks granted after waiting. 75M The number of region locks granted without waiting.
#
# Set the database in memory cache size.
#
set_cachesize 0 52428800 0
#
# Set database flags.
#
#set_flags DB_TXN_NOSYNC
#
# Set log values.
#
set_lg_regionmax 1048576
set_lg_max 10485760
set_lg_bsize 2097152
#set_lg_dir /opt/openldap/var/bdb-logs
#
# Set temporary file creation directory.
#
set_tmp_dir /tmp