[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#8382) Activation of idlcache leads to wrong ldapsearch results

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#8382) Activation of idlcache leads to wrong ldapsearch results
From: fdopheide@iponweb.net
Date: Wed, 09 Mar 2016 10:40:27 +0000
Auto-submitted: auto-generated (OpenLDAP-ITS)
Hi,

I am also adding the guide to replicate the error as plain text:=20

Step-by-step guide

The following guide was created on an Ubuntu Linux V. 14.04.3 LTS, but =
it should be executeable on an Debian based Linux operating systems. You =
should be logged in as root user or adopt all commands with sudo.

1. Add hostname
    echo 'ldap-debug.example.com' > /etc/hostname && hostname `cat =
/etc/hostname` && hostname
    echo '52.49.174.211 ldap-debug.example.com ldap-debug' >> /etc/hosts

2. Download latest Sources
    wget =
http://www.openldap.org/software/download/OpenLDAP/openldap-release/openld=
ap-2.4.44.tgz

3. Validate Sources(compare md5 provided on website with generated md5 =
hash)
    md5sum openldap-2.4.44.tgz

4.    Make archive executable and extract sources and change into =
extracted directory
    chmod +x openldap-2.4.44.tgz && tar xzf openldap-2.4.44.tgz && cd =
openldap-2.4.44

5. Fullfil most important software requirements (Dependencies)
    apt-get update
    apt-get install libdb-dev
    apt-get install openssl
    apt-get install gnutls-bin
    apt-get install gcc
    apt-get install make

6. Run configure with default settings and add a target dir only
    ./configure --prefix=3D/usr/local/openldap

7. Run make and install sources to target directory
    make depend
    make
    make test
    make install

8. Stop the daemon, because slapd was already started by make.
    pkill slapd

9. Edit main config file in /usr/local/openldap/etc/openldap/slapd.conf =
and add the following lines:
    include         /usr/local/openldap/etc/openldap/schema/core.schema
    pidfile         /usr/local/openldap/var/run/slapd.pid
    argsfile        /usr/local/openldap/var/run/slapd.args
    modulepath      /usr/lib/ldap
    moduleload      back_hdb
    backend         hdb
    database        hdb
    suffix          "dc=3Dexample,dc=3Dcom"
    rootdn          "cn=3DManager,dc=3Dexample,dc=3Dcom"
    rootpw          secret
    directory       /var/lib/ldap
    index           objectClass eq

10. Call slappasswd and set rootpw to secret
    slappasswd

11. Create data directory for hdb files
    mkdir /var/lib/ldap

12. We will add a symbolic link to ease slapd commands
    ln -s /usr/local/openldap/libexec/slapd /usr/local/bin/slapd

13. Add the path to OpenLDAP bin and sbin dirs and activate the new path
    echo "export =
PATH=3D\$PATH:/usr/local/openldap/bin/:/usr/local/openldap/sbin/" >> =
/root/.bashrc && source /root/.bashrc

14. Start the slapd daemon with -f flag for location of the config file
    slapd -f /usr/local/openldap/etc/openldap/slapd.conf

15. Create initial DIT file with an editor of your choice, i.e. vi =
/root/create_dit.ldif and copy paste the following lines into this file.
    dn: dc=3Dexample,dc=3Dcom
    changetype: add
    objectClass: dcObject
    objectClass: organization
    objectClass: top
    dc: example
    o: example.com
    =20
    dn: ou=3DProjects,dc=3Dexample,dc=3Dcom
    changetype: add
    ou: Projects
    objectClass: organizationalUnit
    =20
    dn: ou=3DTrash,dc=3Dexample,dc=3Dcom
    changetype: add
    ou: Trash
    objectClass: organizationalUnit
    =20
    dn: ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
    changetype: add
    ou: test1
    objectClass: organizationalUnit
    =20
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
    changetype: add
    ou: test2
    objectClass: organizationalUnit

16. Add the initial DIT
    ldapadd -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dcom' =
-w secret -f /root/create_dit.ldif

17. Execute modrdn and assign directory Trash as a new superior of test1 =
directory
    ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"

18. Execute following ldapsearch and save the result somewhere
    for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
    Result:
    =3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom

19. We see until here everything is working as expected, but now we will =
enable cache and idlcache as recommended in OpenLDAP docs and =
Performance Guides and problems will appear soon.
    echo -e "cachesize\t10000\nidlcachesize\t30000" >> =
/usr/local/openldap/etc/openldap/slapd.conf

20. Remove existing DIT
    ldapdelete -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dco=
m' -w secret -r "dc=3Dexample,dc=3Dcom" -v

21. Restart daemon and recreate initial DIT
    pkill slapd
    slapd -f /usr/local/openldap/etc/openldap/slapd.conf
    ldapadd -H 'ldap://localhost' -D 'cn=3DManager,dc=3Dexample,dc=3Dcom' =
-w secret -f /root/create_dit.ldif

22. Execute modrdn
    ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"

23. Check result again with ldapsearch
    for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
    Result:
    =3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
24. Once again the result was fine and as expected. Keep in mind cache =
and idlcache were empty as we've restarted the daemon. Now we are =
performing same actions again while we have cached results.

25. Remove directories in ou =E2=80=9ETrash=E2=80=9C
    ldapdelete -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom"

26. Add directories to ou =E2=80=9EProjects=E2=80=9C again
    ldapmodify -D  "cn=3DManager,dc=3Dexample,dc=3Dcom" -W
    Type in password
    Enter LDAP Password:
    Copy paste following ldif and be sure emtpy lines are copied, too!
    dn: ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
    changetype: add
    objectClass: organizationalUnit
    =20
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom
    changetype: add
    objectClass: organizationalUnit

27. Now we have same DIT structure again and can rerun modrdn command
    ldapmodrdn -x -w secret -D "cn=3DManager,dc=3Dexample,dc=3Dcom" -s =
"ou=3DTrash,dc=3Dexample,dc=3Dcom" -r =
"ou=3Dtest1,ou=3DProjects,dc=3Dexample,dc=3Dcom" "ou=3Dtest1"

28. If we now check result with ldapsearch again the idlcache error will =
appear
    for i in {,{,{,ou=3Dtest2\,}ou=3Dtest1\,}ou=3DTrash\,}dc=3Dexample,dc=3D=
com; do echo -e "\n=3D=3D=3D=3D=3D> basedn: $i <=3D=3D=3D=3D=3D\n"; =
ldapsearch -LLL -s 'sub' -H 'ldap://localhost:389' -D =
'cn=3DManager,dc=3Dexample,dc=3Dcom' -w secret -b "$i" '(ou=3Dtest*)' =
dn; done
    Result:
     =3D=3D=3D=3D=3D> basedn: dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=
=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom =
<=3D=3D=3D=3D=3D
    dn: ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom
    =3D=3D=3D=3D=3D> basedn: =
ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom <=3D=3D=3D=3D=3D
    dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3Dcom

29. The search result of basedn: ou=3DTrash,dc=3Dexample,dc=3Dcom is =
wrong! The line of dn: ou=3Dtest2,ou=3Dtest1,ou=3DTrash,dc=3Dexample,dc=3D=
com is missing within the resultset.
30. After a restart of the daemon the missing line will appear again. =
Actually the one and only solution is to switch of idlcachesize and set =
it explicitely to "0"

> Am 07.03.2016 um 11:45 schrieb fdopheide@iponweb.net:
>=20
> Full_Name: Frank Dopheide
> Version: slapd 2.4.44 (Mar  3 2016 10:27:05)
> OS: Ubuntu 14.04.3 LTS
> URL: http://52.48.8.167/idlcache_error.pdf
> Submission from: (NULL) (87.139.29.59)
>=20
>=20
> Good morning Gentlemen,
>=20
> we've found a quite serious bug in slapd 2.4.44 on an Ubuntu 14.04.3 =
LTS server.
> As soon as we enable the idlcache the ldapsearch results will be wrong =
after
> moddn and modrdn commands. First result after idlcachesize activation =
is still
> correct as the cache is still empty and the daemon reads from db, but =
as soon as
> it starts reading from idlcache the latest modifications are missing =
until slapd
> will be restarted!
>=20
> We've created a small howto to replicatehe e error. The howto is =
available via
> http: http://52.48.8.167/idlcache_error.pdf
>=20
> Please let us now if you require any additional information!
>=20
> Thx,
> Frank
>=20
>=20
>=20
Prev by Date: (ITS#8383) msys build error "portable.h:1116:19: error: two or more data types in declaration specifiers"
Next by Date: (ITS#8384) LDAP Servers in Multi-master configuration locked up and unresponsive
Index(es):
- Chronological
- Thread