contextCSN propagation problems

[Rein Tollevik <rein@basefarm.no>]

The worst problems we had after upgrading to 2.4.x seems to be over now, 
and the replication appear to work (at least mostly) as it should.  One 
problem that still remains is that the contextCSN attributes themselves 
don't propagate as I wish they would. Before I start coding or filing 
more ITSes I would like to have comments as to whether my analysis of 
the problem seems correct or not.

First, a description of our configuration is needed.  I think a test 
script for this configuration would be nice to have, so I'm going to 
create that before I do anything else.  I currently think of it as 
syncrepl-asymmetric, but suggestions for better names are welcome.

Now the configuration, which is centered around a central master server.
It has a glued database with a set of subordinates.  The central master 
is the (sole) master for most of these subordinates, but there are also 
a set of remote site-masters that each has one subordinate database that 
 they are the master for and that is replicated back to the central master.

The site-masters has a similar glued configuration, and they replicates 
the glue entry from the central master.  Different rootdn values on the
subordinates managed by syncrepl and the one the site-master is the 
master for prevents syncrepl from wiping out the content of that 
database during the present phase. None of the site-masters receives all 
the subordinate databases the central master has, and which they receive 
varies. This is controlled by acl rules on the central master.

On each of the sites (including where the central master is) there are 
search-only servers that replicates the glue suffix from their 
site-master (or the central master).  The search servers has a single 
database (for historical reasons), but their layout shouldn't matter 
very much.

All of the master servers uses syncprov on the glue database, and 
everyone except the central master uses syncrepl on the glue database. 
The central master cannot use it there, as that would have caused it to 
wipe out those subordinates that aren't on the site-master it replicates 
from during the refresh phase.  Different serverIDs are used on the 
master servers, so an updated contextCSN set should include as many 
values as there are master servers.

Now to the problem. If a modification is made (on the central master) to 
a subordinate database that isn't replicated to one of its consumers it 
will not receive the updated contextCSN value from the central master. 
Which means that I cannot monitor the contextCSN values to verify that 
the replication is working as it should.  And the consumers will (after 
a restart) present an outdated contextCSN set during the refresh phase, 
even though their database content is up to date. But this will be 
corrected when/if an update is made to a subordinate db that is 
replicated to the consumer.

A worse problem is when the modifications is made to a subordinate db 
the central master replicates from one of the site-masters.  In this 
case there will never be any updates from that site-master that should 
be propagated. So the consumers will, until it restarts, be stuck with 
the contextCSN value matching that remote site-master which it received 
after the present phase.

Or, oh well, that is not quite true...  Due to a bug in syncprov it 
fails (on the central master) to detect and filter out the contextCSN 
update from syncrepl on a subordinate db.  Which means that the central 
master will send an update of the glue entry to its consumers, so that 
the value is propagated from the central master to its immediate 
consumers. But it will not propagate further from the site-masters to 
the search servers on their site.  As this bug makes things work at 
least partly as I wish I'm a bit reluctant to have it fixed yet...

So far my proposed solution to this problem is that syncprov_matchops() 
should, when a modification fails to match the test_filter() nor is a 
deleted entry, send a sync info protocol message with the updated 
contextCSN in the newCookie field to its consumers.  Does this sound 
like a valid solution?  There seem to be support in syncrepl.c for 
receiving these messages, and in syncprov.c for sending them. But it 
never actually does it as far as I can see.

--
Rein Tollevik
Basefarm AS