[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: replication factored out of slapd

Pierangelo Masarati wrote: 
I have replication factored out in two overlays up and running right
now.  It's not 100% implemented, and it might get tricky at some point.
Most of all, the code is getting awful in terms of #ifdef's, so I'd like
to gather some consensus about this direction.

Basically, there are two overlays: "slurprov", which takes care of
logging modifications to replog, and "consumer", which takes care of
deciding whether a modification is to be accepted or not.

Sounds great. The slurpd/replog support was already moved to callbacks, I was intending to move this direction as well. Thanks for getting there first ;) 
"slurprov" is a per-database overlay which allows to specify a replog, a
replica-argsfile, a replica-pidfile, a replicationinterval and multiple
replica instances.  There's little more to say.  A global replog is
allowed right now.

"consumer" is a global overlay.  It had to be local to allow mucking
with no-user-mod stuff and so.  When it's instantiated, it keeps track
of what databases are shadow, what updateref they have, and what type of
shadow they are.  Syncrepl shadows do not accept modifications (unless
"multimaster", I kept that code in place although I haven't tested it
yet).  Slurpd shadows can only be modified by the updatedn.  Moreover,
multimaster can be switched on via slapd.conf, if enabled at compile
time.  If stacked right before a global instance of the "chain" overlay,
it allows to write to a slave having the updateref chained to the
master.

Both overlays support back-config in input; "slurprov" also in output,
while for "consumer" I have to work out an easy manner to allow a global
overlay to append entries below a database.

Right now, if the overlays are not compiled, the old code is used, so it
should be safe to commit them as they are now.  Only, the frontend code
that handles replication got quite hairy.

There might yet be issues with passwd exop, which I didn't test yet;
also, I haven't been able to cast slapo-refint and slapo-ppolicy into
this framework, essentially because I couldn't catch their logic at a
glance; I don't think it will be too hard to modify them accordingly.

refint may be interesting. I assumed that it would only be needed on a master, and that any resulting changes would be propagated normally. For ppolicy there are a number of status attributes that are currently only maintained locally on each DSA. It may make sense to have a config flag to control whether those status changes remain local-only or should be chained/replicated to other DSAs. I suppose always chaining provides tighter consistency, but it will have a notable performance cost. 
I'll play with these overlays a bit more and then commit to HEAD;
apologies if something won't work for a while.

This also brings to mind another favorite topic - eliminating slurpd... Right now we can eliminate slurpd using a separate slapd instance configured with a back-ldap pointing at the slave, and a syncrepl consumer pointing at the master. A separate slapd instance must be used because the back-ldap suffix would be identical to the master's suffix, and we don't allow multiple databases to share the same suffix. While the current code already provides this functionality, it's inconvenient because it requires a slapd instance per replica.

I was thinking about allowing certain databases to be configured as "hidden/shadow" databases, with the same suffix as an existing database. These hidden databases would never be selected by select_backend(), so they would never be used to satisfy any incoming requests. They would only serve as platforms for configuring syncrepl consumers. This would allow multiple replication targets to be configured using a single slapd. Any objections?

--
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/