[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

To: "openldap-technical@openldap.org" <openldap-technical@openldap.org>
Subject: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
From: "Borresen, John - 0444 - MITLL" <John.Borresen@ll.mit.edu>
Date: Mon, 16 May 2016 17:21:54 +0000
Accept-language: en-US
Content-language: en-US
In-reply-to: <306F04FEF2912A92FE6F33CB@[192.168.1.19]>
References: <D1ABA0029F01AE4EA344F208D704EA532D610769@LLE2K10-MBX01.mitll.ad.local> <D1ABA0029F01AE4EA344F208D704EA532D611B08@LLE2K10-MBX01.mitll.ad.local> <306F04FEF2912A92FE6F33CB@[192.168.1.19]>
Thread-index: AdGtJyzCzE8kQwUjTAyNwhcCi3KaZQCVrj/QAAu/MQAABfMOMA==
Thread-topic: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

I've run db_recover on ldapserver2, to no avail:

# db_recover -v -h /var/lib/openldap/accesslog
Finding last valid log LSN: file: 5 offset 4659816
Recovery starting from [5][4655494]
Recovery complete at Mon May 16 13:03:19 2016
Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]

# db_recover -v -h /var/lib/openldap/openldap-data
Finding last valid log LSN: file: 3 offset 7169862
Recovery starting from [3][7169734]
Recovery complete at Mon May 16 13:03:33 2016
Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]

Same error:
# /usr/local/openldap/libexec/slapd -u ldap -h
ldap://ldapserver2.example.come -F /usr/local/openldap/etc/openldap/slapd.d
-d 256
5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
	
clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.40/serv
ers/slapd
5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown
detected; attempting recovery.
5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected;
attempting recovery.
5739fe11 slapd starting
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted

I found references to remove *.bdb in the accesslog and openldap-data
directories and the __db.*.  Could I run a slapcat of the dbase on
ldapserver1 and copy that over to ldapserver2?   Is that a viable option?



John D. Borresen (Dave)
Ph: (781) 981-1609
Email: john.borresen@ll.mit.edu


-----Original Message-----
From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] 
Sent: Monday, May 16, 2016 11:55 AM
To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org
Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS

That was specific to back-mdb.  Your logs showed corruption with BDB.  Are
you using mdb, bdb, or both?

--Quanah

--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" 
<John.Borresen@ll.mit.edu> wrote:

>
>
> I've noticed this error/warning that keeps standing out when starting 
> slapd on ldapserver2:
>
>
>
> slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
>
>
>
> In my google searches I found this post from Quanah as a possible bug 
> in
> 2.4.44 (we're running 2.4.40):
>
>
>
>
>
> This list is for discussing reported issues in OpenLDAP Software ()
>
> headers
>
> quanah | 27 Apr 16:56 2016
>
> (ITS#8413) Assertion in back-mdb/search.c during replication
>
>
>
> Full_Name: Quanah Gibson-Mount
>
> Version: 2.4.44
>
> OS: Linux
>
> URL: ftp://ftp.openldap.org/incoming/
>
> Submission from: (NULL) (75.111.52.177)
>
>
>
> During replication from the accesslog DB, in a 4-way MMR setup, 
> various masters
>
> periodically crash with slapd: search.c:1246: oc_filter: Assertion `f 
> != ((void
>
> *)0)' failed.
>
>
>
> This is back-mdb/search.c, not slapd-search.c
>
>
>
> This is triggered when a NULL filter is passed through.  However, it 
> should be
>
> impossible for the filter generated by str2filter to ever fail.
>
>
>
> Permalink | Reply |
>
> Navigate
>
> Go to gmane.network.openldap.bugs.
>
> Topic
>
> Go to the topic.
>
> Advertisement
>
> Project Web Page
>
> This list is for discussing reported issues in OpenLDAP Software ()
>
> Search Archive
>
>
>
> Language
>
> Change language
>
> Options
>
> Current view: Threads only / Showing whole messages / Not hiding cited 
> text.
>
> Change to All messages, shortened messages, or hide cited text.
>
>
>
> Post a message
>
> NNTP Newsgroup
>
> Classic Gmane web interface
>
> XML RSS Feed
>
> List Information
>
>
>
> About Gmane
>
>
>
> Gmane
>
>
>
> Again, if anyone has any suggestions as to a workaround or a 
> resolution that would be most appreciative.
>
>
>
> Thanks,
>
>
>
> John D. Borresen (Dave)
>
> Email: john.borresen@ll.mit.edu
>
>
>
>
> From: openldap-technical 
> [mailto:openldap-technical-bounces@openldap.org]
> On Behalf Of Borresen, John - 0444 - MITLL
> Sent: Friday, May 13, 2016 11:13 AM
> To: openldap-technical@openldap.org
> Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
>
>
>
> We have a 3-way multimaster configuration running on CentOS 5.11, 
> OpenLDAP 2.4.40.  All three have been up for years, until the other day:
>
>
>
> Slapd is running on two of the three (server names:  ldapserver1, 
> ldapserver2, and ldapserver3).  Slapd stopped and won't restart on 
> ldapserver2.
>
>
>
> From Logs on ldapserver2:
>
> May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 
> operations/tasks to finish
>
> May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database
> "cn=accesslog": database already in use.
>
> May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one 
> (type=bdb,
> suffix="cn=accesslog"): bi_db_open failed! (-1)
>
> May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
>
> May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
>
>
>
> When attempting to restart slapd on server2:
>
> May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
>
> May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) 
> Content Sync Refresh Required
>
> May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) 
> Content Sync Refresh Required
>
> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put 
> id
> failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
>
> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: 
> parent
> (cn=accesslog) insert failed: -30995
>
> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del 
> id
> failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
>
> May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50:
> parent (cn=accesslog) delete failed: -30995
>
> May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database
> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>
> May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database
> "cn=accesslog": unclean shutdown detected; attempting recovery.
>
> May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
>
> May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
> "dc=example,dc=com": unclean shutdown detected; attempting recovery.
>
> May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database
> "cn=accesslog": unclean shutdown detected; attempting recovery.
>
> May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database
> "cn=accesslog": unclean shutdown detected; attempting recovery.
>
> May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: 
> do_start_tls
> failed:stat=-1
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to 
> LDAP server ldap://ldapserver1.example.com
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
> "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database
> "cn=accesslog": unclean shutdown detected; attempting recovery.
>
> May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
>
> May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 
> (Sep 30 2014 16:49:45) 
> $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl
> dap
> -2.4.40/servers/slapd
>
>
>
> Attempting to restart slapd from the command-line:
>
> 5735ed50 slapd starting
>
> 5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
>
> 5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
>
> 5735ed50 bdb_idl_fetch_key: %cn=accesslog
>
> 5735ed50 bdb_idl_fetch_key: [b49d1940]
>
> 5735ed50 bdb_idl_fetch_key:
>
> 5735ed50 send_ldap_result: err=0 matched="" text=""
>
> 5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
>
> 5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
>
> ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN 
> reqDeleteOldRDN reqNewSuperior entryCSN
>
> ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN 
> reqDeleteOldRDN reqNewSuperior entryCSN
>
> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>
> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_dn2bv(272)
>
> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_dn2bv(272)
>
> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>
> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_dn2bv(272)
>
> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
>
> <= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> => ldap_dn2bv(272)
>
> <= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
>
> 5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
>
> 5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
>
> slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
>
> Aborted
>
>
>
>
>
> I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
>
>
>
> Does anyone have any suggestions?
>
>
>
> Thank you in advance for any assistance.
>
>
>
>
>
>
>
> John D. Borresen (Dave)
>
> Linux/Unix Systems Administrator
>
> MIT  Lincoln Laboratory
>
> Humanitarian Assistance and Disaster Relief (HADR) Systems
>
> 244 Wood St
>
> Lexington, MA  02420
>
> Email: john.borresen@ll.mit.edu
>
>



--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration A division
of Synacor, Inc

Attachment: smime.p7s
Description: S/MIME cryptographic signature

Follow-Ups:
- RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
  - From: Quanah Gibson-Mount <quanah@zimbra.com>

References:
- SLAPD WON'T START ON ONE OF THE MULTIMASTERS
  - From: "Borresen, John - 0444 - MITLL" <John.Borresen@ll.mit.edu>
- SLAPD WON'T START ON ONE OF THE MULTIMASTERS
  - From: "Borresen, John - 0444 - MITLL" <John.Borresen@ll.mit.edu>
- Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
  - From: Quanah Gibson-Mount <quanah@zimbra.com>

Prev by Date: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
Next by Date: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
Index(es):
- Chronological
- Thread