ldapsearch returning failure to import cert

[John Rouillard <rouilj@renesys.com>]

Hi all:

I am running Scientific Linux 6 (a Red Hat enterprise
repackage). Until recently these machines were interacting fine with
our ldap setup. We use a self signed cert for the ldap servers and
deploy the CA cert in /etc/openldap/cacert.pem.

However after the last series of updates ldapsearch has been failing
in an interesting way and our sssd caching daemons are failing to
connect to our ldaps servers. I am hoping that they are both having the
same issue.

The relevant installed packages are:

  openldap-2.4.23-26.el6_3.2.x86_64
  openssl-1.0.0-27.el6_4.2.x86_64
  nss-util-3.14.0.0-2.el6.x86_64
  nss-3.14.0.0-12.el6.x86_64

I am using the command (lightly obscured):

  ldapsearch -d -1 -v -x -b
     uid=user,ou=people,dc=staff,dc=example,dc=com -D
     uid=user,ou=people,dc=staff,dc=example,dc=com -W -H
     ldaps://auth.staff.example.com/

This fails with the error:

  TLS: error: connect - force handshake failure: errno 21 - moznss error
-8054
  TLS: can't connect: TLS error -8054:You are attempting to import a
  cert with the same issuer/serial as an existing cert, but that is not
  the same cert..
  ldap_err2string
  ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Where is ldapsearch "importing" a cert? Where is it getting its other
certs from? I ran strace on ldapsearch and the only cert file I can
see it accessing is /etc/openldap/cacert.pem as specified in
/etc/openldap/ldap.conf (not counting the /usr/lib64/libnssckbi.so
file). The cert in cacert.pem is identical to the one retrieved by
running:

   openssl s_client -connect auth.staff.example.com:636 </dev/null \
    2>/dev/null | sed -ne '/BEGIN CERTIFICATE/,/END CERTIFICATE/p'

Here is where it gets a little more interesting:

I have a previous CA cert (that used an md5 message digest).  If I
install that as the CA, ldapsearch works for 2 of my 3 ldap servers.

I have used openssl x509 -in ... -text to compare the certificates for
my 3 ldap server and they look identical except where they shouldn't
be (subject name, subject name digests...). The issuer, issuer digest
... fields are the same.

If I use

  openssl verify -CAfile /etc/openldap/cacert.pem -purpose sslserver
      -issuer_checks ldap

where ldap is the cert retrieved using s_client it validates for all
three servers regardless of whether the CAfile is the older md5 or
newer cert.

Just to add more into the mix, our CentOS 5 boxes have no issues
with any of the servers (IIUC they have an entirely different tls/cert
level since they do not use Mozilla nss).

Thanks for any insight or questions as the answer didn't come to me
while I was writing this email 8-).

--
                                -- rouilj

John Rouillard       System Administrator
Renesys Corporation  603-244-9084 (cell)  603-643-9300 x 111