Re: replication factored out of slapd

[Pierangelo Masarati <ando@sys-net.it>]

On Wed, 2005-12-14 at 15:02 -0800, Howard Chu wrote:

> Sounds great. The slurpd/replog support was already moved to callbacks, 

In fact, that was the easy part :-)

Now I'm somewhat stuck at a point where the "consumer" knows about both
"slurp" and "sync" replication; in principle, what it should do is "take
decisions", like reject modifies and generate referrals, or let them
pass thru.  But, to bring the factorization process to an extreme, I'm
considering the opportunity to move syncrepl into this overlay (the
configuration is already there, so there is no more be_syncinfo in the
database structure...).  Now the configuration, without any change in
slapd.conf syntax, looks like this:

dn: olcDatabase={-1}frontend,cn=config
# ...

dn: olcOverlay={0}consumer,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcConsumerOverlayConfig
olcOverlay: {0}consumer

dn: olcConsumer={0}sync,olcOverlay={0}consumer,olcDatabase={-1}frontend,cn=con
 fig
objectClass: olcSyncConsumerConfig
olcConsumer: {0}sync
olcDatabase: {2}bdb
olcSyncrepl: rid=001 provider=ldap://localhost:9012/ starttls=no bindmethod=si
 mple binddn="cn=Manager 2,dc=example,dc=com" credentials="secret" filter="(ob
 jectClass=*)" searchbase="ou=groups,dc=example,dc=com" scope=sub attrs="*,+"
 schemachecking=off type=refreshAndPersist retry="10 3 300 5"
olcMultiMaster: FALSE
olcUpdateRef: ldap://localhost:9012/

#...

dn: olcDatabase={1}bdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcBdbConfig
olcDatabase: {1}bdb
olcSuffix: ou=Information Technology Division,ou=People,dc=example,dc=com
olcSubordinate: TRUE
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=Manager 1,dc=example,dc=com
olcDbDirectory: ./testrun/db.1.a
olcDbCacheSize: 1000
olcDbNoSync: FALSE
olcDbDirtyRead: FALSE
olcDbIDLcacheSize: 0
olcDbIndex: objectClass eq
olcDbIndex: entryUUID pres
olcDbIndex: entryCSN pres
olcDbIndex: cn pres,eq,sub
olcDbIndex: uid pres,eq,sub
olcDbIndex: sn pres,eq,sub
olcDbLinearIndex: FALSE
olcDbMode: 384
olcDbSearchStack: 16
olcDbShmKey: 0
olcDbCacheFree: 0

so it's pretty much unfriendly: consumer info is now in a child of the
global overlay and nothing about that appears any longer in the
database; I'd really love to have the global "consumer" overlay append
its children below the appropriate database, but I don't know how to do
it (I don't think it's possible within the current framework, because
when the global overlay is processed the database entries do not exist;
but I'd love to be contradicted).

Another point is the need to add "overlay consumer" before any database,
and "overlay slurprov" before any replication-related instruction in the
master databases.  I think this can be worked around by forcing the
instantiation of the overlays when the appropriate directives appear.

> refint may be interesting. I assumed that it would only be needed on a 
> master, and that any resulting changes would be propagated normally.

But it directly uses the slap_replog_cb()...

>
> For ppolicy there are a number of status attributes that are currently only 
> maintained locally on each DSA. It may make sense to have a config flag 
> to control whether those status changes remain local-only or should be 
> chained/replicated to other DSAs. I suppose always chaining provides 
> tighter consistency, but it will have a notable performance cost.

Same as above...

> This also brings to mind another favorite topic - eliminating slurpd... 
> Right now we can eliminate slurpd using a separate slapd instance 
> configured with a back-ldap pointing at the slave, and a syncrepl 
> consumer pointing at the master. A separate slapd instance must be used 
> because the back-ldap suffix would be identical to the master's suffix, 
> and we don't allow multiple databases to share the same suffix. While 
> the current code already provides this functionality, it's inconvenient 
> because it requires a slapd instance per replica.
> 
> I was thinking about allowing certain databases to be configured as 
> "hidden/shadow" databases, with the same suffix as an existing database. 
> These hidden databases would never be selected by select_backend(), so 
> they would never be used to satisfy any incoming requests. They would 
> only serve as platforms for configuring syncrepl consumers. This would 
> allow multiple replication targets to be configured using a single 
> slapd. Any objections?

This will get a separate reply after I re-read and digest it a little
bit more.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------