Viewing Software Enhancements/4261
Full headers

Major security issue: yes  no

Notes:
replication
Notification:

Date: Thu, 15 Dec 2005 19:42:33 GMT
From: ando@sys-net.it
To: openldap-its@OpenLDAP.org
Subject: [enhancement] Factor replication code out of slapd

Full_Name: Pierangelo Masarati
Version: HEAD
OS: irrelevant
URL: http://www.sys-net.it/~ando/Download/slurpover-2005-12-15.1.patch.bz2
Submission from: (NULL) (81.72.89.40)
Submitted by: ando


This patch, for HEAD as of December 15th, 2005, factors replication code out of
slapd.  It's actually composed of two parts, which are tightly coupled and thus
it's submitted as just one.

It provides two overlays:

"slurprov" takes care of logging modifications to the replog, and of those
config statements that are actually used by slurpd: replica-{args,pid}file,
replica, replicationinterval.  It should be instantiated inside any master
database.

"consumer" is a generic consumer interface.  It handles generic consumer issues,
like rejecting non-replication modifications (unless set up as so-called
"multimaster") and, for syncrepl, it accounts for the configuration.

Both overlays support back-config.

Current issues:
- some existing overlays that are replog-aware are not handled yet.  As such,
slapo-refint(5) and slapo-ppolicy(5) cannot be compiled along with
slapo-slurprov(5);
- the overlays need to be explicitly instantiated; the rest of the configuration
is fully compatible with the current.  It would be nice to have them
automatically instantiated when the appropriate replication statements occur in
slapd.conf(5), I don't think it's going to be a big deal;
- slapo-consumer(5) must be global; thus, a single slapo-consumer(5) takes care
of all consumer database in the DSA.  As a consequence, the related
configuration is held in subordinate entries of the global overlay rather than
inside (or subordinate to) the related database.  This makes configuration via
back-config little intuitive
- slapo-consumer(5) doesn't yet support run-time configuration via back-config;
only EMIT is available.

The patch is provided for initial discussion, development and testing.  I plan
to commit it when it's a bit more mature, i.e. when the above issues are closed.
 Note though that (at least on my laptop ;) it passes all tests, and
significantly all replication-related ones.

p.

PS: no space left on device on ftp.openldap.org; I've placed it on my website.

Followup 1

Date: Thu, 15 Dec 2005 16:25:01 -0800
From: Howard Chu <hyc@symas.com>
To: ando@sys-net.it
CC: openldap-its@OpenLDAP.org
Subject: Re: (ITS#4261) [enhancement] Factor replication code out of slapd

ando@sys-net.it wrote:
> This patch, for HEAD as of December 15th, 2005, factors replication code
out of
> slapd.  It's actually composed of two parts, which are tightly coupled and
thus
> it's submitted as just one.
>
> It provides two overlays:
>
> "slurprov" takes care of logging modifications to the replog, and of those
> config statements that are actually used by slurpd: replica-{args,pid}file,
> replica, replicationinterval.  It should be instantiated inside any master
> database.
>
> "consumer" is a generic consumer interface.  It handles generic consumer
issues,
> like rejecting non-replication modifications (unless set up as so-called
> "multimaster") and, for syncrepl, it accounts for the configuration.
>
> Both overlays support back-config.
>
> Current issues:
>   

There's a lot of misc cleanup in the diff that appears unrelated to the 
current feature, perhaps you should commit those bits in advance. Also, 
don't include the diff for configure since that script will just get 
regenerated from configure.in.

> - some existing overlays that are replog-aware are not handled yet.  As
such,
> slapo-refint(5) and slapo-ppolicy(5) cannot be compiled along with
> slapo-slurprov(5);
>   

I've looked at refint and ppolicy; the reason they invoke replog_cb 
directly is due to the order of processing between that callback and the 
overlay callbacks. With slurprov moved into an overlay, the references 
to slap_replog_cb can be removed from these overlays. Then it will just 
be a matter of configuring the overlays in the proper order to insure 
that refint and ppolicy's own updates are propagated as desired. (In 
this case, slurprov must be below refint/ppolicy in the overlay stack.) 
That takes care of one issue; the other is that these overlays may need 
to know whether the current database is a shadow or not, and alter their 
behavior accordingly.

> - the overlays need to be explicitly instantiated; the rest of the
configuration
> is fully compatible with the current.  It would be nice to have them
> automatically instantiated when the appropriate replication statements
occur in
> slapd.conf(5), I don't think it's going to be a big deal;
> - slapo-consumer(5) must be global; thus, a single slapo-consumer(5) takes
care
> of all consumer database in the DSA.  As a consequence, the related
> configuration is held in subordinate entries of the global overlay rather
than
> inside (or subordinate to) the related database.  This makes configuration
via
> back-config little intuitive
> - slapo-consumer(5) doesn't yet support run-time configuration via
back-config;
> only EMIT is available.
>
> The patch is provided for initial discussion, development and testing.  I
plan
> to commit it when it's a bit more mature, i.e. when the above issues are
closed.
>  Note though that (at least on my laptop ;) it passes all tests, and
> significantly all replication-related ones.
-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Followup 2

Date: Sun, 05 Feb 2006 01:29:06 -0800
From: Howard Chu <hyc@symas.com>
To: openldap-its@openldap.org
Subject: ITS#4261 replication

This is a multi-part message in MIME format.
--------------000708000308030507070003
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

While waiting for this patch to get cleaned up, I wrote this up to allow 
configuring the multimaster mode.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/


--------------000708000308030507070003
Content-Type: text/plain;
 name="dif.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="dif.txt"

Index: add.c
===================================================================
RCS file: /var/CVSROOT/ldap23/servers/slapd/add.c,v
retrieving revision 1.9
retrieving revision 1.10
diff -u -r1.9 -r1.10
--- add.c	5 Feb 2006 01:04:53 -0000	1.9
+++ add.c	5 Feb 2006 08:16:40 -0000	1.10
@@ -284,9 +284,7 @@
 	if ( op->o_bd->be_add ) {
 		/* do the update here */
 		int repl_user = be_isupdate( op );
-#ifndef SLAPD_MULTIMASTER
-		if ( !SLAP_SHADOW(op->o_bd) || repl_user )
-#endif
+		if ( !SLAP_SINGLE_SHADOW(op->o_bd) || repl_user )
 		{
 			int		update = !BER_BVISEMPTY( &op->o_bd->be_update_ndn );
 			slap_callback	cb = { NULL, slap_replog_cb, NULL, NULL };
@@ -328,15 +326,11 @@
 					send_ldap_result( op, rs );
 					goto done;
 				}
-			}
 
-#ifdef SLAPD_MULTIMASTER
-			if ( !repl_user )
-#endif
-			{
 				cb.sc_next = op->o_callback;
 				op->o_callback = &cb;
 			}
+
 			rc = op->o_bd->be_add( op, rs );
 			if ( rc == LDAP_SUCCESS ) {
 				/* NOTE: be_entry_release_w() is
@@ -346,7 +340,6 @@
 				op->o_private = op->o_bd;
 			}
 
-#ifndef SLAPD_MULTIMASTER
 		} else {
 			BerVarray defref = NULL;
 
@@ -369,7 +362,6 @@
 					LDAP_UNWILLING_TO_PERFORM,
 					"shadow context; no update referral" );
 			}
-#endif /* SLAPD_MULTIMASTER */
 		}
 	} else {
 	    Debug( LDAP_DEBUG_ARGS, "	 do_add: no backend support\n", 0, 0, 0 );
Index: bconfig.c
===================================================================
RCS file: /var/CVSROOT/ldap23/servers/slapd/bconfig.c,v
retrieving revision 1.11
retrieving revision 1.12
diff -u -r1.11 -r1.12
--- bconfig.c	18 Jan 2006 14:40:35 -0000	1.11
+++ bconfig.c	5 Feb 2006 08:16:40 -0000	1.12
@@ -162,6 +162,7 @@
 	CFG_SSTR_IF_MAX,
 	CFG_SSTR_IF_MIN,
 	CFG_TTHREADS,
+	CFG_MULTIMASTER,
 
 	CFG_LAST
 };
@@ -361,6 +362,9 @@
 #endif
 		"( OLcfgGlAt:31 NAME 'olcModulePath' "
 			"SYNTAX OMsDirectoryString SINGLE-VALUE )", NULL, NULL },
+	{ "multimaster", "on|off", 2, 2, 0,
ARG_DB|ARG_ON_OFF|ARG_MAGIC|CFG_MULTIMASTER,
+		&config_generic, "( OLcfgDbAt:0.16 NAME 'olcMultiMaster' "
+			"SYNTAX OMsBoolean SINGLE-VALUE )", NULL, NULL },
 	{ "objectclass", "objectclass", 2, 0, 0,
ARG_PAREN|ARG_MAGIC|CFG_OC|ARG_NO_DELETE|ARG_NO_INSERT,
 		&config_generic, "( OLcfgGlAt:32 NAME 'olcObjectClasses' "
 		"DESC 'OpenLDAP object classes' "
@@ -667,7 +671,7 @@
 		 "olcMaxDerefDepth $ olcPlugin $ olcReadOnly $ olcReplica $ "
 		 "olcReplogFile $ olcRequires $ olcRestrict $ olcRootDN $ olcRootPW $ "
 		 "olcSchemaDN $ olcSecurity $ olcSizeLimit $ olcSyncrepl $ "
-		 "olcTimeLimit $ olcUpdateDN $ olcUpdateRef ) )",
+		 "olcTimeLimit $ olcUpdateDN $ olcUpdateRef $ olcMultiMaster ) )",
 		 	Cft_Database, NULL, cfAddDatabase },
 	{ "( OLcfgGlOc:5 "
 		"NAME 'olcOverlayConfig' "
@@ -875,6 +879,9 @@
 		case CFG_LASTMOD:
 			c->value_int = (SLAP_NOLASTMOD(c->be) == 0);
 			break;
+		case CFG_MULTIMASTER:
+			c->value_int = (SLAP_SINGLE_SHADOW(c->be) == 0);
+			break;
 		case CFG_SSTR_IF_MAX:
 			c->value_int = index_substr_if_maxlen;
 			break;
@@ -961,6 +968,7 @@
 		case CFG_AZPOLICY:
 		case CFG_DEPTH:
 		case CFG_LASTMOD:
+		case CFG_MULTIMASTER:
 		case CFG_SASLSECP:
 		case CFG_SSTR_IF_MAX:
 		case CFG_SSTR_IF_MIN:
@@ -1235,6 +1243,20 @@
 				SLAP_DBFLAGS(c->be) |= SLAP_DBFLAG_NOLASTMOD;
 			break;
 
+		case CFG_MULTIMASTER:
+			if(!SLAP_SHADOW(c->be)) {
+				snprintf( c->msg, sizeof( c->msg ), "<%s> database is not a
shadow",
+					c->argv[0] );
+				Debug(LDAP_DEBUG_ANY, "%s: %s\n",
+					c->log, c->msg, 0 );
+				return(1);
+			}
+			if(c->value_int)
+				SLAP_DBFLAGS(c->be) &= ~SLAP_DBFLAG_SINGLE_SHADOW;
+			else
+				SLAP_DBFLAGS(c->be) |= SLAP_DBFLAG_SINGLE_SHADOW;
+			break;
+
 		case CFG_SSTR_IF_MAX:
 			if (c->value_int < index_substr_if_minlen) {
 				snprintf( c->msg, sizeof( c->msg ), "<%s> invalid value",
c->argv[0] );
@@ -2576,7 +2598,7 @@
 		return 1;
 	}
 
-	SLAP_DBFLAGS(c->be) |= (SLAP_DBFLAG_SHADOW | flag);
+	SLAP_DBFLAGS(c->be) |= (SLAP_DBFLAG_SHADOW | SLAP_DBFLAG_SINGLE_SHADOW |
flag);
 
 	return 0;
 }
Index: delete.c
===================================================================
RCS file: /var/CVSROOT/ldap23/servers/slapd/delete.c,v
retrieving revision 1.6
retrieving revision 1.7
diff -u -r1.6 -r1.7
--- delete.c	1