[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: syncrepl consumer is slow
Howard Chu wrote:
Emmanuel Lécharny wrote:
Le 03/02/15 05:11, Howard Chu a écrit :
Another option here is simply to perform batching. Now that we have
the TXN api exposed in the backend interface, we could just batch up
e.g. 500 entries per txn. much like slapadd -q already does.
Ultimately we ought to be able to get syncrepl refresh to occur at
nearly the same speed as slapadd -q.
Batching is ok, except that you never know how many entries you'll going
to have, thus you will have to actually write the data after a period of
time, even if you don't have the 500 entries.
This isn't a problem - we know exactly when refresh completes, so we can
finish the batch regardless of how many entries are left over.
Testing this out with the experimental ITS#8040 patch - with lazy commit
the 2.8M entries (2.5GB data) takes ~10 minutes for the refresh to pull
them across. With batching 500 entries/txn+lazy commit it takes ~7
minutes, a decent improvement. It's still 2x slower than slapadd -q
though, which loads the data in 3-1/2 minutes.
In case anyone else wants to try this out, patch attached.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
commit 3c182823d606fba46620c23450f89d8ef981dabf
Author: Howard Chu <hyc@openldap.org>
Date: Tue Feb 3 08:21:13 2015 +0000
Use batched write txns in refresh
diff --git a/servers/slapd/back-mdb/back-mdb.h b/servers/slapd/back-mdb/back-mdb.h
index 224f05b..136fb53 100644
--- a/servers/slapd/back-mdb/back-mdb.h
+++ b/servers/slapd/back-mdb/back-mdb.h
@@ -115,6 +115,7 @@ typedef struct mdb_op_info {
} mdb_op_info;
#define MOI_READER 0x01
#define MOI_FREEIT 0x02
+#define MOI_KEEPER 0x04
/* Copy an ID "src" to pointer "dst" in big-endian byte order */
#define MDB_ID2DISK( src, dst ) \
diff --git a/servers/slapd/back-mdb/id2entry.c b/servers/slapd/back-mdb/id2entry.c
index 119c8fa..142f383 100644
--- a/servers/slapd/back-mdb/id2entry.c
+++ b/servers/slapd/back-mdb/id2entry.c
@@ -284,7 +284,7 @@ int mdb_entry_release(
mdb_entry_return( op, e );
moi = (mdb_op_info *)oex;
/* If it was setup by entry_get we should probably free it */
- if ( moi->moi_flag & MOI_FREEIT ) {
+ if (( moi->moi_flag & (MOI_FREEIT|MOI_KEEPER)) == MOI_FREEIT ) {
moi->moi_ref--;
if ( moi->moi_ref < 1 ) {
mdb_txn_reset( moi->moi_txn );
@@ -541,7 +541,12 @@ int mdb_txn( Operation *op, int txnop, OpExtra **ptr )
switch( txnop ) {
case SLAP_TXN_BEGIN:
- return mdb_opinfo_get( op, mdb, 0, moip );
+ rc = mdb_opinfo_get( op, mdb, 0, moip );
+ if ( !rc ) {
+ moi = *moip;
+ moi->moi_flag |= MOI_KEEPER;
+ }
+ return rc;
case SLAP_TXN_COMMIT:
rc = mdb_txn_commit( moi->moi_txn );
op->o_tmpfree( moi, op->o_tmpmemctx );
diff --git a/servers/slapd/syncrepl.c b/servers/slapd/syncrepl.c
index 23b638d..bb10789 100644
--- a/servers/slapd/syncrepl.c
+++ b/servers/slapd/syncrepl.c
@@ -110,6 +110,10 @@ typedef struct syncinfo_s {
int si_refreshDelete;
int si_refreshPresent;
int si_refreshDone;
+ int si_refreshCount;
+ time_t si_refreshBeg;
+ time_t si_refreshEnd;
+ OpExtra *si_refreshTxn;
int si_syncdata;
int si_logstate;
int si_lazyCommit;
@@ -736,6 +740,11 @@ do_syncrep1(
}
si->si_refreshDone = 0;
+ si->si_refreshBeg = slap_get_time();
+ si->si_refreshCount = 0;
+ si->si_refreshTxn = NULL;
+ Debug( LDAP_DEBUG_ANY, "do_syncrep1: %s starting refresh\n",
+ si->si_ridtxt, 0, 0 );
rc = ldap_sync_search( si, op->o_tmpmemctx );
@@ -1267,6 +1276,15 @@ do_syncrep2(
{
si->si_refreshDone = 1;
}
+ if ( si->si_refreshDone ) {
+ if ( si->si_refreshCount ) {
+ LDAP_SLIST_REMOVE( &op->o_extra, si->si_refreshTxn, OpExtra, oe_next );
+ op->o_bd->bd_info->bi_op_txn( op, SLAP_TXN_COMMIT, &si->si_refreshTxn );
+ }
+ si->si_refreshEnd = slap_get_time();
+ Debug( LDAP_DEBUG_ANY, "do_syncrep1: %s finished refresh\n",
+ si->si_ridtxt, 0, 0 );
+ }
ber_scanf( ber, /*"{"*/ "}" );
if ( abs(si->si_type) == LDAP_SYNC_REFRESH_AND_PERSIST &&
si->si_refreshDone )
@@ -2933,8 +2951,20 @@ syncrepl_entry(
slap_queue_csn( op, syncCSN );
}
- if ( !si->si_refreshDone && si->si_lazyCommit )
- op->o_lazyCommit = SLAP_CONTROL_NONCRITICAL;
+ if ( !si->si_refreshDone ) {
+ if ( si->si_lazyCommit )
+ op->o_lazyCommit = SLAP_CONTROL_NONCRITICAL;
+ if ( si->si_refreshCount == 500 ) {
+ LDAP_SLIST_REMOVE( &op->o_extra, si->si_refreshTxn, OpExtra, oe_next );
+ op->o_bd->bd_info->bi_op_txn( op, SLAP_TXN_COMMIT, &si->si_refreshTxn );
+ si->si_refreshCount = 0;
+ si->si_refreshTxn = NULL;
+ }
+ if ( !si->si_refreshCount ) {
+ op->o_bd->bd_info->bi_op_txn( op, SLAP_TXN_BEGIN, &si->si_refreshTxn );
+ }
+ si->si_refreshCount++;
+ }
slap_op_time( &op->o_time, &op->o_tincr );
switch ( syncstate ) {