[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: syncrepl and mass deletes



Well,

To resume my  tests:

provider and consumer in 2.4.24 -> same behaviour
Get rid of session log -> same behaviour

If I removed one of the providers of my consumer, everything went fine. Even, if I stop the consumer during mass deletes, it sync very fast as soon as it starts again
That's related to several providers for one consumer and may be something else I don't see.
The second provider (first rid in my conf) has less that 15 entries and no updates at all.

I changed the order of the providers in my slapd.conf. There is no difference.

Dom

Mar 24 11:36:32 ldaprelay slapd[5444]: slapd starting
Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET
Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET
Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE
Mar 24 11:36:32 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET
Mar 24 11:36:33 ldaprelay slapd[5444]: last message repeated 200 times
Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT
Mar 24 11:36:33 ldaprelay slapd[5444]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110324103606.185647Z#000000#020#000000
Mar 24 11:36:33 ldaprelay slapd[5444]: slap_queue_csn: queing 0x227dfe0 20110324103606.185647Z#000000#020#000000
Mar 24 11:36:33 ldaprelay slapd[5444]: slap_graduate_commit_csn: removing 0x227e0a0 20110324103606.185647Z#000000#020#000000


# consumer
# big provider and mass deletes
serverid 0x20 ldap://anutest.univ.fr/
syncrepl rid=020
    provider=ldap://anutest.univ.fr/
    type=refreshAndPersist
    searchbase="dc=univ,dc=fr"
    retry="60 10 300 +"
    scope=sub
    filter="(objectClass=*)"
    attrs="*,+"
    schemachecking=off
    bindmethod=simple

# small provider
serverid 0x40 ldap://ldapmaitre.univ-a.fr/
syncrepl rid=040
    provider=ldap://ldapmaitre.univ-a.fr/
    type=refreshAndPersist
    searchbase="dc=univ-a,dc=fr"
    retry="60 10 300 +"
    scope=sub
    filter="(objectClass=*)"
    attrs="*,+"
    schemachecking=off
    bindmethod=simple

context on the servers

# univ.fr
dn: dc=univ,dc=fr
contextCSN: 20110324103606.185647Z#000000#020#000000

# univ-a.fr
dn: dc=univ-a,dc=fr
contextCSN: 20110323101636.221613Z#000000#040#000000

dn: dc=fr
contextCSN: 20110324103606.185647Z#000000#020#000000
contextCSN: 20110323101636.221613Z#000000#040#000000



2011/3/24 LALOT Dominique <dom.lalot@gmail.com>
Here is my provider, and we use syncprov-sessionlog

database bdb
suffix "dc=univ-yy,dc=fr"
directory /var/ldap/base/
sizelimit -1
cachesize 50000
dbconfig set_cachesize 0 140000000 1
dbconfig set_flags DB_LOG_AUTOREMOVE
dbconfig set_tas_spins 1
dbconfig set_lg_regionmax 262144
idlcachesize 150000
checkpoint 1024 5
# flush tous les 1Mo ou 5 minutes
#dbnosync
syncprov-checkpoint 100 10
syncprov-sessionlog 100
index objectClass,entryCSN,entryUUID eq

...

Should we use a session log? I'll try without. That's tricky to understand each parameter.
Would be great to have something anaylizing more than syntax, relations between parameters and database cache optimization.


Dom



2011/3/23 Howard Chu <hyc@symas.com>
LALOT Dominique wrote:
Howard,

I was obliged to remove slapd package on consumer. Then compile in 2.4.24 and
restart. doing the same tests, there was nothing diffferent. provider is still
2.4.23
My test: delete 30000 entries, stop consumer when deleting; start again
consumer when it's finished. Consumer is then out of sync.

You haven't posted your provider config. It appears you're using syncprov-sessionlog. Obviously both ends of this system are vital to solving the puzzle.

Mar 23 17:27:50 ldaprelay slapd[4358]: slapd stopped.
Mar 23 17:34:45 ldaprelay slapd[4577]: @(#) $OpenLDAP: slapd 2.4.24 (Mar 23
2011 16:52:04)
$#012#011root@ldaprelay.univ-amu.fr:/usr/local/src/openldap-2.4.24/servers/slapd
Mar 23 17:34:45 ldaprelay slapd[4579]: slapd starting
Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=040
LDAP_RES_INTERMEDIATE - REFRESH_DELETE
Mar 23 17:34:45 ldaprelay slapd[4579]: do_syncrep2: rid=020
LDAP_RES_INTERMEDIATE - SYNC_ID_SET
Mar 23 17:34:48 ldaprelay slapd[4579]: last message repeated 175 times
Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020
LDAP_RES_INTERMEDIATE - REFRESH_PRESENT
Mar 23 17:34:48 ldaprelay slapd[4579]: do_syncrep2: rid=020
cookie=rid=020,sid=020,csn=20110323163416.105518Z#000000#020#000000
Mar 23 17:34:48 ldaprelay slapd[4579]: slap_queue_csn: queing 0x1a35560
20110323163416.105518Z#000000#020#000000
Mar 23 17:34:48 ldaprelay slapd[4579]: slap_graduate_commit_csn: removing
0x1a35620 20110323163416.105518Z#000000#020#000000

I changed the provider to 2.4.24 that makes deletes. Hopefully this one was
built on tar.gz
That makes no difference, after restarting the consumer, it does not delete
extra entries

Dom

2011/3/23 LALOT Dominique <dom.lalot@gmail.com <mailto:dom.lalot@gmail.com>>


   Hi Howard,

   We were told to migrate to 2.4.23 sometimes ago, and we did some work to
   update our production servers. Can I try 2.4.24 only on the consumer side?
   It would be a pain to migrate all servers to 2.4.24 without package.

   is this related to the last fixes?

   Fixed slapd syncrepl reuse of presence list (ITS#6707)
   Fixed slapd syncrepl uninitialized return code (ITS#6719)
   Fixed slapd syncrepl variable initialization (ITS#6739)


   Fixed slapd syncrepl refresh to use complete cookie (ITS#6807)

   Thanks

   Dom


   2011/3/23 Howard Chu <hyc@symas.com <mailto:hyc@symas.com>>


       LALOT Dominique wrote:

           Hello,

           I am testing the replication feature in a multimaster environment
           replicating
           into a single database. As stated before, I added serverid to my
           providers. I
           just have two providers for test purpose.
           I tested mass updates on a provider, stopped my replica during
           updates, then
           start again and it's OK, it updates the entries
           If I do the same for mass deletes. I deleted 40000 entries while
           stopping the
           consumer. My consumer is still with 30000 undeleted entries. I
           left the
           consumer for hours, restarting it twice.
           It seems there is no regular compare between consumer or provider
           in such
           situation. I'll simplify to test in a single provider setup, to
           see if it works.

           All servers are 2.4.23


       Please try your test with 2.4.24 instead.


--
 -- Howard Chu
 CTO, Symas Corp.           http://www.symas.com
 Director, Highland Sun     http://highlandsun.com/hyc/
 Chief Architect, OpenLDAP  http://www.openldap.org/project/



--
Dominique LALOT
Ingénieur Systèmes et Réseaux
http://annuaire.univmed.fr/showuser.php?uid=lalot



--
Dominique LALOT
Ingénieur Systèmes et Réseaux
http://annuaire.univmed.fr/showuser.php?uid=lalot