(ITS#7472) syncrepl conflicting write resolution trouble

[ebackes@symas.com]

Full_Name: Emily Backes
Version: 2.4.33+ITS#7427
OS: 
URL: 
Submission from: (NULL) (98.155.30.248)


We are seeing a condition under heavy conflicting-write load-- i.e. changes to
the same DN from multiple nodes in a syncrepl MMR cluster at the same time--
where the content of the entry is desynchronized between servers.

There is a non-reproduced case with incomplete logs where servers traded entry
versions, even with one CSN significantly (by one second) later than the other. 
I'll supply more info if that reappears.

Additionally, we have a -d logged case of a conflicting write resulting in
different changes applied on different servers while the same CSN gets applied
to both, which seems truly bizarre.

The expected case is that one server "wins" the conflict and its state is sent
to all of the others, eventually.

While most likely unrelated, we're also still seeing massive desynchronization
from conflicting writes involving modifies and deletes, where the deletes are
sent under heavy load such that the commits occur out of order and result in a
cookie-less syncrepl event-- these (Despite #7427) can still resurrect entries
later.

All of these results are using back-hdb.  More details soon, hopefully.