[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: (ITS#7104) refreshOnly syncrepl out of sync during bulk update/delete operations

mkrz@free.fr wrote:
> Full_Name: Martin Krzaczek
> Version: 2.4.28
> OS: CentOS 5.7
> URL:
> Submission from: (NULL) (
> Hello,
> I noticed an issue with "refreshOnly" syncrepl when a large number of records is
> added, updated and deleted. At the end of synchronization the contextCSN is
> equal on all servers, but the data is out of synchronization for some deleted
> records that remains undeleted on consumer/slave servers.
> Restarting the consumer or forcing a synchronization with "slapd ... -c
> 'rid=1,csn=0' " does not help.

This appears to caused by the sessionlog overflowing due to incoming modifies 
that occur after the refresh operation has begun. I would call this a 
misconfigured server, since the sessionlog can only ever be useful if you size 
it large enough to accomodate all of your update traffic.

I don't have any plans to fix this issue; fix the server configuration.
> A similar issue was submitted few weeks ago (ITS#7052), but I'm not sure the
> problem is the same.
> My configuration :
> - CentOS 5.7 + OpenLDAP 2.4.28 + BDB 5.1.29
>    (the same issue occurs with CentOS 5.7 + OpenLDAP 2.4.26 + BDB 5.1.25)
> - one main provider where all modification operations are performed
> - one 'first level' consumer synchronizing data on the main provider
>    and providing data for 'second level' consumers
> - one 'second level' consumer synchronizing data on the 'first level'
>    consumer/provider
> I tried several combinations of configurations (refreshOnly/refreshAndPersist,
> syncprov-sessionlog) and noticed that when refreshOnly mode is used, some
> records may be not deleted on consumer/slave servers. The number of undeleted
> records varies depending on the configuration.
> So with a higher value of "syncprov-sessionlog" (for example 5000 instead of
> 500) the number of lost records is reduced.
> I have an application that sequentially (i.e. in a single thread) adds about
> 12000 records, modifies about 25000 records and deletes about 1000 records.
> Add/update/delete operations are mixed i.e. it is possible a have sequences such
> as add/add/delete/update/update/update/delete/add/...
> With a small amount of records to add/update/delete the synchronization properly
> works without any data loss.
> So it seams that if all modifications may be cached and transmitted to consumers
> then deleted records are not lost.
> There is no data loss for added and updated records whatever is the number of
> operations.
> Regards,
> Martin

   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/