Strange issue with contextCSN

[Pierangelo Masarati <ando@sys-net.it>]

I'm running concurrency tests of MMR, and I see some strange issues:

1) loss of sync after multi-concurrent load (multiple concurrent ops on 
each server, and modifications to the same data subset on all servers, 
in order to trigger conflicts).  I'm still trying to see if there is any 
pattern or clue about what failed (like finding some explanation in the 
logs).  This happens once in a while after many operations.  I don't 
expect this to be necessarily a bug; it might be the consequence of 
conflicts.  Of course, it would be nice if slapd allows to clearly 
identify where the conflict occurred, to support manual resolution.

2) loss of sync after single-concurrent load (multiple concurrent ops on 
a single server).  This is really inesplicable (to me), as there should 
be no conflict.  The only possible explanation I see (but need to 
investigate further) is that an entry is added on a server, sync'd to 
another one and, in the meanwhile, deleted on the first one before its 
own sync gets back.  This happens very seldom.

3) whay puzzled me a bit is that when I load a single server, I'd expect 
to end up with a single contextCSN containing the SID of that server. 
This is correct for the server I load, but the others, even when they 
get correctly sync'd, contain a contextCSN for each server in the MMR 
pool, and the contextCSN with the other SIDs don't get propagated to the 
server that was loaded.  It's not clear why those CSNs are generated, 
and how they get into the loop and propagate between servers that do not 
receive direct modifications.

4) another thing that puzzled me a bit is that in some cases, when all 
servers are loaded, and the contextCSNs are one for each SID and the 
same in all of the servers, they are sorted randomly, and differently; 
for example:

bash-3.2$ diff -u testrun/server2.out testrun/server3.out
--- testrun/server2.out	2008-11-22 17:17:28.000000000 +0100
+++ testrun/server3.out	2008-11-22 17:17:28.000000000 +0100
@@ -2497,8 +2497,8 @@
 associatedDomain: example.com
 entryCSN: 20081122161630.753152Z#000000#001#000000
 contextCSN: 20081122161708.935242Z#000000#001#000000
-contextCSN: 20081122161658.195350Z#000000#002#000000
 contextCSN: 20081122161658.193983Z#000000#003#000000
+contextCSN: 20081122161658.195350Z#000000#002#000000

Not a big deal (except for the need to sort values to compare them), but 
I'd expect them to be exactly in the same order...

I'm going to pack my suite of tests and put them on ftp.openldap.org 
(and eventually add them to OpenLDAP's test suite, specifically meant to 
test MMR), but first I need to polish them a little bit and enucleate 
those that present issues, in order to open specific ITSes.

p.


Ing. Pierangelo Masarati
OpenLDAP Core Team

SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
-----------------------------------
Office:  +39 02 23998309
Mobile:  +39 333 4963172
Fax:     +39 0382 476497
Email:   ando@sys-net.it
-----------------------------------