Viewing Software Bugs/5378
Full headers

Major security issue: yes  no

Notes:
fixed in HEAD/RE24/RE23
Notification:

Date: Wed, 13 Feb 2008 22:04:13 GMT
From: quanah@OpenLDAP.org
To: openldap-its@openldap.org
Subject: hot slapcat for slapadd issues with delta-sync

Full_Name: Quanah Gibson-Mount
Version: 2.3.40
OS: NA
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (24.23.156.219)


If someone is using delta-syncrepl for replication, and does a hot slapcat of a
server, and a change occurs during the slapcat, the resulting LDIF file cannot
be used on a replica.  This is because the replica will try to resume from the
CSN in the LDIF file, which is from before the change occurred.  Since the
change is in the LDIF file, when the replica contacts the master, it tries to
re-replicate the change (which obviously fails).  At that point, the replica is
stuck and unable to progress.  The only guaranteed way to get a viable slapcat
when using delta-syncrepl is to put the server being dumped into RO mode or to
stop it.

--Quanah

Followup 1

Date: Thu, 14 Feb 2008 01:26:54 -0800
From: Howard Chu <hyc@OpenLDAP.org>
To: quanah@openldap.org
CC: openldap-its@openldap.org
Subject: Re: (ITS#5378) hot slapcat for slapadd issues with delta-sync

quanah@OpenLDAP.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.3.40
> OS: NA
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (24.23.156.219)
>
>
> If someone is using delta-syncrepl for replication, and does a hot slapcat
of a
> server, and a change occurs during the slapcat, the resulting LDIF file
cannot
> be used on a replica.  This is because the replica will try to resume from
the
> CSN in the LDIF file, which is from before the change occurred.  Since the
> change is in the LDIF file, when the replica contacts the master, it tries
to
> re-replicate the change (which obviously fails).  At that point, the
replica is
> stuck and unable to progress.  The only guaranteed way to get a viable
slapcat
> when using delta-syncrepl is to put the server being dumped into RO mode or
to
> stop it.

Another possibility would be to filter out any entries whose entryCSNs are 
newer than the contextCSN in the LDIF before trying to slapadd it.

-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Followup 2

Date: Thu, 14 Feb 2008 10:39:00 +0100
From: Pierangelo Masarati <ando@sys-net.it>
To: hyc@openldap.org
CC: openldap-its@openldap.org
Subject: Re: (ITS#5378) hot slapcat for slapadd issues with delta-sync

hyc@OpenLDAP.org wrote:

> Another possibility would be to filter out any entries whose entryCSNs are 
> newer than the contextCSN in the LDIF before trying to slapadd it.

Should slapadd take care of it (perhaps if instructed to do so)?

p.



Ing. Pierangelo Masarati
OpenLDAP Core Team

SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
---------------------------------------
Office:  +39 02 23998309
Mobile:  +39 333 4963172
Email:   pierangelo.masarati@sys-net.it
---------------------------------------

Followup 3

Date: Thu, 14 Feb 2008 04:15:33 -0800
From: Howard Chu <hyc@symas.com>
To: Pierangelo Masarati <ando@sys-net.it>
CC: openldap-its@openldap.org
Subject: Re: (ITS#5378) hot slapcat for slapadd issues with delta-sync

Pierangelo Masarati wrote:
> hyc@OpenLDAP.org wrote:
>
>> Another possibility would be to filter out any entries whose entryCSNs
are
>> newer than the contextCSN in the LDIF before trying to slapadd it.
>
> Should slapadd take care of it (perhaps if instructed to do so)?

I guess that would be ok. It seems that doc/devel/toolargs isn't up to date 
now, dunno what option flag would make sense here.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Followup 4

Date: Thu, 14 Feb 2008 04:20:57 -0800
From: Howard Chu <hyc@symas.com>
To: Pierangelo Masarati <ando@sys-net.it>
CC: openldap-its@openldap.org
Subject: Re: (ITS#5378) hot slapcat for slapadd issues with delta-sync

Pierangelo Masarati wrote:
> hyc@OpenLDAP.org wrote:
>
>> Another possibility would be to filter out any entries whose entryCSNs
are
>> newer than the contextCSN in the LDIF before trying to slapadd it.
>
> Should slapadd take care of it (perhaps if instructed to do so)?

Of course it might be better to change the delta-sync consumer to check the 
target's entryCSN after a modification fails, to see if it matches the current 
modification attempt. If so, then the failure can be safely ignored.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Followup 5

Date: Thu, 14 Feb 2008 16:19:44 -0800
From: Quanah Gibson-Mount <quanah@zimbra.com>
To: hyc@symas.com, openldap-its@openldap.org
Subject: Re: (ITS#5378) hot slapcat for slapadd issues with delta-sync

--On February 14, 2008 12:21:00 PM +0000 hyc@symas.com wrote:

> Pierangelo Masarati wrote:
>> hyc@OpenLDAP.org wrote:
>>
>>> Another possibility would be to filter out any entries whose
entryCSNs
>>> are newer than the contextCSN in the LDIF before trying to slapadd
it.
>>
>> Should slapadd take care of it (perhaps if instructed to do so)?
>
> Of course it might be better to change the delta-sync consumer to check
> the  target's entryCSN after a modification fails, to see if it matches
> the current  modification attempt. If so, then the failure can be safely
> ignored. --

Note:

It should be if it matches or is greater.   For example, if it was the very 
last entry in the DB, thus the last to be added to the resulting LDIF, it 
is entirely possible that multiple modifications could be done from the 
time the dump started and the dump ended, meaning the CSN would be greater 
than the one encountered in the accesslog DB.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration

Followup 6

Date: Wed, 09 Apr 2008 03:06:34 -0700
From: Howard Chu <hyc@symas.com>
To: openldap-its@openldap.org
Subject: ITS#5378 delta-sync issues

A fix is in HEAD. delta-syncrepl will now fallback to regular refresh in 
additional cases - LDAP_ALREADY_EXISTS, which may occur for old Adds,
LDAP_NO_SUCH_OBJECT, for cases already covered in ITS#5376,
LDAP_NO_SUCH_ATTRIBUTE, for old Modifies, and LDAP_TYPE_OR_VALUE_EXISTS
also for old Modifies. Pretty sure that covers all of the possible out-of-sync 
cases. Please test.
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/