(ITS#8672) syncrepl with openldap 2.4.{40,42} and mdb backend

[remy.dernat@umontpellier.fr]

Full_Name: Dernat R.my
Version: 2.4.40+dfsg-1+deb8u3 and 2.4.42+dfsg-2ubuntu3.2
OS: Debian and Ubuntu
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (162.38.181.76)


Hi,

Since I moved my OpenLDAP to another server my replication between 2 ldap
servers through syncrepl does not work anymore. I tested many many things.
Finally, I decided to backup the database and restore it on another server (so,
I have 3 ldap servers) and  (...) it worked.

After many other tests, I was able to determine the source of this issue. With a
HDB backend on my provider my replication works, while it did not work with a
MDB backend on the provider.

I had this kind of logs on the provider (with MDB):
=============================================================================
Jun  8 12:22:03 ldap2 slapd[15083]: send_search_entry: conn 20855  ber write
failed.
Jun  8 12:24:03 ldap2 slapd[15083]: send_search_entry: conn 20888  ber write
failed.
...
=============================================================================
While, on the slave, I get:
=============================================================================
Jun  8 09:33:32 ldap3-bis slapd[88560]: do_syncrepl: rid=010 rc -1 retrying
Jun  8 09:38:32 ldap3-bis slapd[88560]: do_syncrep2: rid=010 got search entry
without Sync State control (dc=my,dc=domain,dc=com)
Jun  8 09:38:32 ldap3-bis slapd[88560]: do_syncrepl: rid=010 rc -1 retrying
Jun  8 09:43:32 ldap3-bis slapd[88560]: do_syncrep2: rid=010 got search entry
without Sync State control (dc=my,dc=domain,dc=com)
...
=============================================================================

I am able to reproduce the bug quite easily.


I added only two schemas : autofs and quota.
With 
=============================================================================
ldapadd -Q -Y EXTERNAL -H ldapi:/// -f autofs.ldif
ldapadd -Q -Y EXTERNAL -H ldapi:/// -f quota.ldif
=============================================================================

I also loaded accesslog module (I am creating a specific directory for
accesslog(*)) on the provider and the syncprov module on both sides.

=============================================================================
(*)
mdkir /var/lib/ldap/accesslog
chown openldap:openldap /var/lib/ldap/accesslog
=============================================================================

Here is what I am doing to setup the syncrepl :

ldapadd -Q -Y EXTERNAL -H ldapi:/// -f file.ldif


With file.ldif, on the provider site (replication.ldif ; replacing HDB with MDB
to test with a MDB backend):

=============================================================================
#Load the syncprov and accesslog modules.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov
-
add: olcModuleLoad
olcModuleLoad: accesslog

# Accesslog database definitions
dn: olcDatabase={2}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {2}hdb
olcDbDirectory: /var/lib/ldap/accesslog
olcSuffix: cn=accesslog
olcRootDN: cn=XXXX,dc=YYYY,dc=ZZ
olcDbIndex: default eq
olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart

# Accesslog db syncprov.
dn: olcOverlay=syncprov,olcDatabase={2}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpNoPresent: TRUE
olcSpReloadHint: TRUE

# syncrepl Provider for primary db
dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpNoPresent: TRUE

# accesslog overlay definitions for primary db
dn: olcOverlay=accesslog,olcDatabase={1}hdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: accesslog
olcAccessLogDB: cn=accesslog
olcAccessLogOps: writes
olcAccessLogSuccess: TRUE
# scan the accesslog DB every day, and purge entries older than 7 days
olcAccessLogPurge: 07+00:00 01+00:00
=============================================================================


On the consumer (with a unique rid, and by replacing HDB with MDB to test with a
MDB backend), the file.ldif looks like:

=============================================================================
#Load the syncprov module.
dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov

# syncrepl specific indices
dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcDbIndex
olcDbIndex: entryUUID eq
-
add: olcSyncRepl
olcSyncRepl: rid=1
    provider=ldaps://consumer.mydomain.fr
    bindmethod=simple
    binddn="cn=XXXXX,dc=mydomain,dc=fr"
    credentials=XXXX
    searchbase="dc=mydomain,dc=fr" logbase="cn=accesslog"
    logfilter="(&(objectClass=auditWriteObject)(reqResult=0))
    schemachecking=off
    type=refreshAndPersist retry="60 +"
    syncdata=accesslog
-
add: olcUpdateRef
olcUpdateRef: ldaps://consumer.mydomain.fr

=============================================================================
On the provider I am using:

=============================================================================
lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.8 (jessie)
Release:        8.8
Codename:       jessie

dpkg -l slapd
ii  slapd                   2.4.40+dfsg-1+de amd64            OpenLDAP server
(slapd)

=============================================================================

Same configuration on one slave, and on the other slave, I am using:

=============================================================================
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial

dpkg -l slapd
ii  slapd                   2.4.42+dfsg-2ubu amd64            OpenLDAP server
(slapd)

=============================================================================


Best regards,
Rémy