Full_Name: Adam Heath Version: 2.4.44 OS: debian stretch URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (99.146.168.62) I have configured slapd to proxy to a remote server. Using ldapsearch, I can talk directly to that remote server, and using the pr=200/noprompt option, I get back 2900 results. Pointing ldapsearch at localhost, *without* pcache, I get the same set of results(pages, and the final count is correct). When I enabled slapo-pcache, with *no* attribute sets, then the paging options are removed, and I get only 2000 results(the max-size from the remote server). Command: docker exec -ti caching-ad ldapsearch -E pr=200/noprompt -o ldif-wrap=no -z none -w 'XXXX' -b 'XXXX' -D 'XXXX' '(objectclass=group)' dn -h localhost|grep -A 5 '# search result' slapd.conf: == egrep -v '^(#|$)' ../app-hosting/brainfood-docker-image-recipes/openldap/files/caching-slapd.conf include /etc/ldap/schema/core.schema include /etc/ldap/schema/cosine.schema include /etc/ldap/schema/nis.schema include /etc/ldap/schema/inetorgperson.schema attributetype ( 1.2.840.113556.1.4.221 NAME 'sAMAccountName' SYNTAX '1.3.6.1.4.1.1466.115.121.1.15' SINGLE-VALUE ) pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args loglevel none modulepath /usr/lib/ldap moduleload back_ldap.la moduleload back_hdb.la moduleload back_bdb.la moduleload rwm moduleload pcache.la moduleload memberof.la sizelimit 5000 tool-threads 1 backend ldap readonly yes database ldap protocol-version 3 rebind-as-user norefs yes chase-referrals no uri "ldap://192.168.178.2:389" suffix "dc=ldap01,dc=com" rootdn "cn=admin,dc=ldap01,dc=com" rootpw "foobar" overlay memberof overlay pcache proxyCache hdb 100000 3 1000 100 pcachePersist TRUE cachesize 200 directory /var/lib/ldap == I expect pcache to not truncate the results from the remote server.
--On Wednesday, September 06, 2017 6:15 PM +0000 adam@brainfood.com wrote: > Full_Name: Adam Heath > Version: 2.4.44 > OS: debian stretch > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (99.146.168.62) > > > I have configured slapd to proxy to a remote server. > > Using ldapsearch, I can talk directly to that remote server, and using the > pr=200/noprompt option, I get back 2900 results. > > Pointing ldapsearch at localhost, *without* pcache, I get the same set of > results(pages, and the final count is correct). > > When I enabled slapo-pcache, with *no* attribute sets, then the paging > options are removed, and I get only 2000 results(the max-size from the > remote server). Hi Adam, slapo-pcahce is acting in the correct fashion. It would appear that your remote system is Active Directory, which in typical Microsoft fashion, deliberately mis-implements paged results so that it incorrectly ignores the maxsize setting when paged results are in use (contrary to specifications). I would generally suggest talking to the AD administrator so that the bind identity of the pcache database is not subject to the maxsize limitation. This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
changed notes changed state Open to Closed
No, it's a pcache bug. 10.10.55.128(remote active directory) works localhost(without pcache) works localhost(with pcache) breaks. Paging of the results *does* work with AD. And works with back-ldap, pointed at AD. It's only when pcache is added that the paging options are ignored. On Wed, Sep 6, 2017 at 12:48 PM, Quanah Gibson-Mount <quanah@symas.com> wrote: > --On Wednesday, September 06, 2017 6:15 PM +0000 adam@brainfood.com wrote: > >> Full_Name: Adam Heath >> Version: 2.4.44 >> OS: debian stretch >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (99.146.168.62) >> >> >> I have configured slapd to proxy to a remote server. >> >> Using ldapsearch, I can talk directly to that remote server, and using the >> pr=200/noprompt option, I get back 2900 results. >> >> Pointing ldapsearch at localhost, *without* pcache, I get the same set of >> results(pages, and the final count is correct). >> >> When I enabled slapo-pcache, with *no* attribute sets, then the paging >> options are removed, and I get only 2000 results(the max-size from the >> remote server). > > > Hi Adam, > > slapo-pcahce is acting in the correct fashion. It would appear that your > remote system is Active Directory, which in typical Microsoft fashion, > deliberately mis-implements paged results so that it incorrectly ignores the > maxsize setting when paged results are in use (contrary to specifications). > I would generally suggest talking to the AD administrator so that the bind > identity of the pcache database is not subject to the maxsize limitation. > > This ITS will be closed. > > Regards, > Quanah > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> >
I'll say it another way. ldapsearch is asking for 200 items at a time. But it gets 2000 from the local slapd, when pcache is enabled. What the local client is asking of slapd should be separate from how it talks to the back-ldap. If the back-ldap returns more results then the requested page size, then slapd should handle that. On Wed, Sep 6, 2017 at 12:59 PM, Adam Heath <adam@brainfood.com> wrote: > No, it's a pcache bug. > > 10.10.55.128(remote active directory) works > localhost(without pcache) works > localhost(with pcache) breaks. > > Paging of the results *does* work with AD. And works with back-ldap, > pointed at AD. It's only when pcache is added that the paging options > are ignored. > > On Wed, Sep 6, 2017 at 12:48 PM, Quanah Gibson-Mount <quanah@symas.com> wrote: >> --On Wednesday, September 06, 2017 6:15 PM +0000 adam@brainfood.com wrote: >> >>> Full_Name: Adam Heath >>> Version: 2.4.44 >>> OS: debian stretch >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (99.146.168.62) >>> >>> >>> I have configured slapd to proxy to a remote server. >>> >>> Using ldapsearch, I can talk directly to that remote server, and using the >>> pr=200/noprompt option, I get back 2900 results. >>> >>> Pointing ldapsearch at localhost, *without* pcache, I get the same set of >>> results(pages, and the final count is correct). >>> >>> When I enabled slapo-pcache, with *no* attribute sets, then the paging >>> options are removed, and I get only 2000 results(the max-size from the >>> remote server). >> >> >> Hi Adam, >> >> slapo-pcahce is acting in the correct fashion. It would appear that your >> remote system is Active Directory, which in typical Microsoft fashion, >> deliberately mis-implements paged results so that it incorrectly ignores the >> maxsize setting when paged results are in use (contrary to specifications). >> I would generally suggest talking to the AD administrator so that the bind >> identity of the pcache database is not subject to the maxsize limitation. >> >> This ITS will be closed. >> >> Regards, >> Quanah >> >> >> -- >> >> Quanah Gibson-Mount >> Product Architect >> Symas Corporation >> Packaged, certified, and supported LDAP solutions powered by OpenLDAP: >> <http://www.symas.com> >>
Patches welcome
changed notes changed state Closed to Open
Hi Adam, I would note that pcache is not back-ldap, so their behavior is not the same. slapo-pcache strips out the paged results control, and has since 2005 (ec532ce88590c5454025cbc030d9df65d2df634f). So in the 12 years since then, you're the first to have an issue with the behavior. If you'd like to see a change in relation to this, patches welcome. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com> -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
I just cloned the source, and found where it removes the paging control. == /* NOTE: this is a quick workaround to let pcache minimally interact * with pagedResults. A more articulated solutions would be to * perform the remote query without control and cache all results, * performing the pagedResults search only within the client * and the proxy. This requires pcache to understand pagedResults. */ static int pcache_chk_controls( Operation *op, SlapReply *rs ) == Well, then since you are confirming the bug, I would say this ticket should *not* be closed. I might just have to patch this to make it work. The client software I'm actually working with is nextcloud, and it is *very* *very* *very* ldap chatty. And with 2900+ objectclass=group, and 3300 objectclass=person, it runs super dog slow. Their suggestion is to install a local ldap instance configured as a caching proxy, which has led us to here. I've already fixed several issues in nextcloud locally(the first one I ran into was $limit=400 hard-coded in the group search function). This kind of paged result handling needs to be dealt the same way an http proxy would deal with partial content caches. On Wed, Sep 6, 2017 at 4:22 PM, Quanah Gibson-Mount <quanah@symas.com> wrote: > Hi Adam, > > I would note that pcache is not back-ldap, so their behavior is not the > same. > > slapo-pcache strips out the paged results control, and has since 2005 > (ec532ce88590c5454025cbc030d9df65d2df634f). So in the 12 years since then, > you're the first to have an issue with the behavior. If you'd like to see a > change in relation to this, patches welcome. > > --Quanah > > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> >
--On Wednesday, September 06, 2017 7:06 PM -0500 Adam Heath <adam@brainfood.com> wrote: > Well, then since you are confirming the bug, I would say this ticket > should *not* be closed. I already reopened it. ;) > I might just have to patch this to make it work. Make sure you follow the contribution guidelines: <https://www.openldap.org/devel/contributing.html> Thanks, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
Cool. I've looked at pcache.c, and realized I'm a noob with this code. Basically, my thoughts, is that pcache should ignore the incoming search request, and recreate it's own with it's own options, always asking for paging. Then page through all the results from the backend, then store the full set in the backing store. While that is going on, the cache entry should be marked with a bit flag saying that it's in the process of being built, but otherwise can still store the entries fetched thus far. Meanwhile, the frontend request can begin to return results from this partially resolved backend entry stored in the cache, utilizing whatever pagination value(and or offset/limit) has been requested. Of course, that's just the high-level description, not even really pseudo-code ready. As an aside, and this s/b a separate bug, if an error occurs from the backend this does *not* get cached. Aka, during this investigation, my backend is returning 2000 entries, err=4. This takes 7s each time(I'm in Dallas, while the backend is in California, and I'm communicating over a tunnel). It's not cached, so the request is passed through each time. I've configured the templates correctly, and the debug shows that the request is CACHEABLE. On Wed, Sep 6, 2017 at 6:22 PM, Quanah Gibson-Mount <quanah@symas.com> wrote: > --On Wednesday, September 06, 2017 7:06 PM -0500 Adam Heath > <adam@brainfood.com> wrote: > > >> Well, then since you are confirming the bug, I would say this ticket >> should *not* be closed. > > > I already reopened it. ;) > >> I might just have to patch this to make it work. > > > Make sure you follow the contribution guidelines: > > <https://www.openldap.org/devel/contributing.html> > > Thanks, > > Quanah > > -- > > Quanah Gibson-Mount > Product Architect > Symas Corporation > Packaged, certified, and supported LDAP solutions powered by OpenLDAP: > <http://www.symas.com> >
May be documentation update as pcache requires the full result set to properly function which is why the paged results control is stripped
https://git.openldap.org/openldap/openldap/-/merge_requests/266
Commits: • 932d18fd by Quanah Gibson-Mount at 2021-03-04T21:44:38+00:00 ITS#8724 - Note that paged results is stripped