RE: (ITS#6503) syncrepl provider sends entryUUIDs of all LDAP records to consumer when consumer slapd is started

[bcolston@xtec.com]

Provider slapd configuration (partial):
serverID 000
database	bdb
suffix		"dc=authentx"
rootdn          "SUPPRESSED"
rootpw          SUPPRESSED
directory	/authentx/db/ldap/authentx-sync1

overlay syncprov
syncprov-checkpoint 100 15
syncprov-sessionlog 5000

checkpoint      128     20
index		objectClass			eq,pres 
index		entryCSN,entryUUID	eq
cachesize	100000
dncachesize	200000
idlcachesize	1000
sizelimit	500000
timelimit	36000

Consumer slapd configuration (partial):
serverID 001
database	bdb
suffix		"dc=authentx"
rootdn          "SUPPRESSED"
rootpw          SUPPRESSED
directory	/authentx/db/ldap/authentx-sync2

syncrepl	rid=001
	provider=ldap://localhost:3891
	type=refreshAndPersist
	retry="300 60 60 +"
	searchbase="dc=authentx"
	scope=sub
	schemachecking=off
	bindmethod=simple
	binddn="SUPPRESSED"
	credentials="SUPPRESSED"

checkpoint      128     20
index		objectClass			eq,pres
index		entryCSN,entryUUID	eq
cachesize	100000
dncachesize	200000
idlcachesize	1000
sizelimit	500000
timelimit	36000

Barry

-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com] 
Sent: Wednesday, March 31, 2010 9:17 PM
To: bcolston@xtec.com
Cc: openldap-its@openldap.org
Subject: Re: (ITS#6503) syncrepl provider sends entryUUIDs of all LDAP
records to consumer when consumer slapd is started

bcolston@xtec.com wrote:
> Full_Name: Barry Colston
> Version: 2.4.21
> OS: Fedora 12
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (209.255.208.210)
>
>
> When a consumer slapd is started, if the provider slapd has had only Add
> operations performed while the consumer slapd was down, the provider slapd
sends
> the consumer slapd the entryUUIDs of all records in the provider database.
> Note: this is not a list of all records that have been
added/changed/deleted
> since the previous synchronization based on the consumer's contextCSN, it
is a
> list of all records in the LDAP database (for some of our customers, this
is
> 10,000,000 entryUUIDs that are returned to the consumer slapd.)

That is the normal behavior of syncrepl. Read the Admin Guide and/or
RFC4533.

> The scenario to reproduce this problem is:
> 1. Start provider slapd
> 2. Start consumer slapd
> 3. Ensure that provider contextCSN matches consumer contextCSN
> 4. Shutdown consumer slapd
> 5. Add records to provider slapd (1 record is sufficient to create the
problem)
> 6. Start consumer slapd with debug turned on (so the messages being sent
can be
> viewed).  The consumer slapd receives 1 to N LDAP_RES_INTERMEDIATE
SYNC_ID_SETs
> (message type = 121) containing the entryUUIDs of the records in the
provider's
> database, followed by a LDAP_RES_SEARCH_ENTRY, LDAP_SYNC_ADD (message type
=
> 100) message for each record added to the provider, then finally a
> LDAP_RES_INTERMEDIATE, REFRESH_PRESENT message (message type = 121)
message
> containing the provider's sync cookie.
>
> I am running OpenLDAP 2.4.21 with Berkeley 4.6.21 (with all patches) on
Fedora
> 12.  Running sync replication in RefreshAndPersist mode.
>
> Note: if at least 1 Modify or Delete operation is performed against the
provider
> slapd while the consumer slapd is down, when the consumer slapd is
started, the
> provider slapd does NOT send the Sync ID sets containing the entryUUIDs of
all
> records in the provider database.

Using the syncprov_sessionlog can allow the provider to shortcut this. Since

you haven't show any details of your configuration, we can't say if this 
difference in behavior is a bug or a config error.
-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/