Full_Name: Mark Cairney Version: 2.4.35-RE24 OS: SL 6.4 64-bit URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (129.215.200.23) Having migrated the database of our production service running 2.4.35 from BDB to MDB alias de-referencing stopped working completely (this has already been reported in ITS# 7557). Upgrading to the RE 24 now has alias derefencing working but it is considerably slower than BDB was for the same search. To confirm this I downgraded one of our nodes (we run 4 nodes in multi-master) and on that node the search results are once more acceptable: With MDB: time ldapsearch -H ldaps://alder.authorise.is.ed.ac.uk:636 -b 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" # numResponses: 425 # numEntries: 424 real 0m26.490s user 0m0.028s sys 0m0.032s with BDB: time ldapsearch -H ldaps://oak.authorise.is.ed.ac.uk:636 -b 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" real 0m2.605s user 0m0.054s sys 0m0.006s If there's anything additional I can supply (log files, strace output) let me know. Our OpenLDAP DB currently has around 400,000 user accounts if that's relevant.
Mark.Cairney@ed.ac.uk wrote: > Full_Name: Mark Cairney > Version: 2.4.35-RE24 > OS: SL 6.4 64-bit > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (129.215.200.23) > > > Having migrated the database of our production service running 2.4.35 from BDB > to MDB alias de-referencing stopped working completely (this has already been > reported in ITS# 7557). > > Upgrading to the RE 24 now has alias derefencing working but it is considerably > slower than BDB was for the same search. > > To confirm this I downgraded one of our nodes (we run 4 nodes in multi-master) > and on that node the search results are once more acceptable: > > With MDB: > time ldapsearch -H ldaps://alder.authorise.is.ed.ac.uk:636 -b > 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always > "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" > > # numResponses: 425 > # numEntries: 424 > > real 0m26.490s > user 0m0.028s > sys 0m0.032s > > > with BDB: > > time ldapsearch -H ldaps://oak.authorise.is.ed.ac.uk:636 -b > 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always > "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" > > real 0m2.605s > user 0m0.054s > sys 0m0.006s > > If there's anything additional I can supply (log files, strace output) let me > know. > > Our OpenLDAP DB currently has around 400,000 user accounts if that's relevant. How many aliases? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--On August 8, 2013 3:02:51 PM +0000 Mark.Cairney@ed.ac.uk wrote: > Full_Name: Mark Cairney > Version: 2.4.35-RE24 > OS: SL 6.4 64-bit > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (129.215.200.23) > > > Having migrated the database of our production service running 2.4.35 > from BDB to MDB alias de-referencing stopped working completely (this has > already been reported in ITS# 7557). back-bdb or back-hdb? "BDB" is just the storage database for those two backends. Thanks, Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
On 08/08/13 16:32, Quanah Gibson-Mount wrote: > > > --On August 8, 2013 3:02:51 PM +0000 Mark.Cairney@ed.ac.uk wrote: > >> Full_Name: Mark Cairney >> Version: 2.4.35-RE24 >> OS: SL 6.4 64-bit >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (129.215.200.23) >> >> >> Having migrated the database of our production service running 2.4.35 >> from BDB to MDB alias de-referencing stopped working completely (this has >> already been reported in ITS# 7557). > > back-bdb or back-hdb? "BDB" is just the storage database for those two > backends. back-bdb. > > Thanks, > Quanah > -- /**************************** Mark Cairney ITI UNIX Section Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
On 08/08/13 16:27, Howard Chu wrote: > > How many aliases? > In that OU 435, in total across the whole database: 29179. -- /**************************** Mark Cairney ITI UNIX Section Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
--On August 8, 2013 4:34:32 PM +0100 Mark Cairney <Mark.Cairney@ed.ac.uk> wrote: >> back-bdb or back-hdb? "BDB" is just the storage database for those two >> backends. > > > back-bdb. Thanks. I would note that back-mdb is based on back-hdb, and that can have significant differences in performance profile for some operations as compared to back-bdb. If it were possible, I'd be curious to know if you see the same issue with back-hdb as you do with back-mdb. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi Quanah, I've transferred one of the nodes to back-hdb and the performance appears to broadly match back-bdb. Is there a simple test I can perform to confirm that it is indeed using back-hdb? For completeness here's the result of my timed search: bash-4.1$ time ldapsearch -H ldaps://rowan.authorise.is.ed.ac.uk:636 -b 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" # numResponses: 425 # numEntries: 424 real 0m0.902s user 0m0.054s sys 0m0.025s On 08/08/13 16:45, Quanah Gibson-Mount wrote: > --On August 8, 2013 4:34:32 PM +0100 Mark Cairney > <Mark.Cairney@ed.ac.uk> wrote: > > >>> back-bdb or back-hdb? "BDB" is just the storage database for those two >>> backends. >> >> >> back-bdb. > > Thanks. I would note that back-mdb is based on back-hdb, and that can > have significant differences in performance profile for some operations > as compared to back-bdb. If it were possible, I'd be curious to know if > you see the same issue with back-hdb as you do with back-mdb. > > --Quanah > -- /**************************** Mark Cairney ITI UNIX Section Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
--On August 8, 2013 5:12:50 PM +0100 Mark Cairney <Mark.Cairney@ed.ac.uk> wrote: > Hi Quanah, > > I've transferred one of the nodes to back-hdb and the performance appears > to broadly match back-bdb. > > Is there a simple test I can perform to confirm that it is indeed using > back-hdb? > > For completeness here's the result of my timed search: > > bash-4.1$ time ldapsearch -H ldaps://rowan.authorise.is.ed.ac.uk:636 -b > 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always > "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" > ># numResponses: 425 ># numEntries: 424 > > real 0m0.902s > user 0m0.054s > sys 0m0.025s No, that's enough, thanks. So it is MDB specific then. ;) --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration
Hi, Finally got round to porting the tree with the aliases to our Test server running 2.4.36 release with the same result in terms of performance issues. Perhaps not surprising as there was nothing in the changelog mentioning this issue although the ITS I did see that I think is related was (ITS #7577). I've attached the strace output but I can't see anything obvious or helpful. It might be worth mentioning that the CPU utilisation on one thread spikes to 100% while the search is underway. On 08/08/13 17:26, Quanah Gibson-Mount wrote: > > > --On August 8, 2013 5:12:50 PM +0100 Mark Cairney > <Mark.Cairney@ed.ac.uk> wrote: > >> Hi Quanah, >> >> I've transferred one of the nodes to back-hdb and the performance appears >> to broadly match back-bdb. >> >> Is there a simple test I can perform to confirm that it is indeed using >> back-hdb? >> >> For completeness here's the result of my timed search: >> >> bash-4.1$ time ldapsearch -H ldaps://rowan.authorise.is.ed.ac.uk:636 -b >> 'ou=staff,ou=law,ou=hss,dc=authorise,dc=ed,dc=ac,dc=uk' -a always >> "(&(eduniSchoolCode=S26)(eduniCategory=101)(eduniIDStatus=300))" >> >> # numResponses: 425 >> # numEntries: 424 >> >> real 0m0.902s >> user 0m0.054s >> sys 0m0.025s > > No, that's enough, thanks. So it is MDB specific then. ;) > > --Quanah > -- /**************************** Mark Cairney ITI UNIX Section Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Hi, Apologies it's taken a while for me to get back to you. Attached are my config and a very simple script I've used to generate OUs filled with aliases based on given subsets of users. The config is from our Test LDAP service which is 2 nodes using delta-syncrepl MMR and a MDB backend running 2.4.38 with Google gperftools with the following ./configure to build it: ./configure --prefix=/usr/local/authz ': ./configure --prefix=/usr/local/authz --enable-meta --enable-ldap --enable-translucent --enable-bdb --enable-mdb --enable-hdb --enable-syncprov --enable-memberof --enable-dyngroup --enable-dynlist --enable-refint --with-threads --with-tls --with-cyrus-sasl --enable-syslog --enable-accesslog --enable-spasswd The 2 nodes are identical bar the olcsyncrepl option (for obvious reasons). I do have a lot of "custom" attributes and a lot of indexed attributes but I don't think that this should make a difference. To replicate the problem a search like: time /usr/local/authz/bin/ldapsearch -x -H ldaps://`hostname`:636 -b "ou=staff,ou=sps,ou=hss,dc=authorise-test,dc=ed,dc=ac,dc=uk" "(objectclass=*)" "*" -a always >output gives: real 0m49.916s user 0m0.037s sys 0m0.009s Without de-referencing gives: time /usr/local/authz/bin/ldapsearch -x -H ldaps://`hostname`:636 -b "ou=staff,ou=sps,ou=hss,dc=authorise-test,dc=ed,dc=ac,dc=uk" "(objectclass=*)" "*" -a never >output real 0m0.047s user 0m0.025s sys 0m0.010s This is on a fairly small (436) number of entries in the given OU. Doing the same on Live (which is using a HDB backend) yields a relatively small difference between alias de-referencing being enabled or not. -a never real 0m0.080s user 0m0.028s sys 0m0.014s -a always real 0m0.197s user 0m0.084s sys 0m0.019s In all cases I've redirected output to a file to eliminate any delays from stdout going to the screen. Hopefully this is enough info for you to reproduce the problem and apologies in advance for any oddities in my config :-) -- /**************************** Mark Cairney ITI UNIX Section Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
changed notes moved from Incoming to Software Bugs
changed notes
changed notes changed state Open to Release
Fyi, we recently revisited this issue and are seeing improved results with commit e90e8c7d3c12d897bb0584ba04dc519d4f23acf9 in master.
Hi Howard, Brilliant- I notice that there's a 2.4.48 RC due out shortly so Ill roll that out on a Dev box and see how it performs. Kind regards, Mark On 15/07/2019 17:07, Howard Chu wrote: > Fyi, we recently revisited this issue and are seeing improved results with > commit e90e8c7d3c12d897bb0584ba04dc519d4f23acf9 in master. > -- /**************************** Mark Cairney ITI Enterprise Services Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk PGP: 0x435A9621 *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Fixed in master Fixed in RE24 (2.4.48) alias deref in back-mdb has terrible perf problems See also ITS#8875 See also ITS#7567
changed notes changed state Release to Closed
Hi Howard, Just to confirm that I'm seeing significantly better results with the 2.4.48 RC release for querying OUs full of aliases and de-referencing them: Comparing 2 servers with the same data on them (one 2.4.47, the other 2.4.48 with the new patch) and just after restarting slapd yield the following: 2.4.48: 0.52s user 0.09s system 46% cpu 1.315 total (with a small spike in CPU) 2.4.47: 0.46s user 0.94s system 0% cpu 3:39.89 total (with the CPU spiking to 100% on the server for the duration of the query) This is for an OU with 11932 entries in it (all aliasedObjects) On 15/07/2019 17:07, Howard Chu wrote: > Fyi, we recently revisited this issue and are seeing improved results with > commit e90e8c7d3c12d897bb0584ba04dc519d4f23acf9 in master. > -- /**************************** Mark Cairney ITI Enterprise Services Information Services University of Edinburgh Tel: 0131 650 6565 Email: Mark.Cairney@ed.ac.uk PGP: 0x435A9621 *******************************/ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
Mark.Cairney@ed.ac.uk wrote: > Hi Howard, Hi Mark, thanks for the feedback. For completeness' sake, the tool I used to generate our test dataset is in http://www.openldap.org/its/index.cgi?findid=8875#followup14 > > Just to confirm that I'm seeing significantly better results with the > 2.4.48 RC release for querying OUs full of aliases and de-referencing them: > > Comparing 2 servers with the same data on them (one 2.4.47, the other > 2.4.48 with the new patch) and just after restarting slapd yield the > following: > > 2.4.48: 0.52s user 0.09s system 46% cpu 1.315 total (with a small spike > in CPU) > > 2.4.47: 0.46s user 0.94s system 0% cpu 3:39.89 total (with the CPU > spiking to 100% on the server for the duration of the query) > > > This is for an OU with 11932 entries in it (all aliasedObjects) > > On 15/07/2019 17:07, Howard Chu wrote: >> Fyi, we recently revisited this issue and are seeing improved results with >> commit e90e8c7d3c12d897bb0584ba04dc519d4f23acf9 in master. >> > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/