Issue 5092 - page size value of 1.2.840.113556.1.4.319 control
Summary: page size value of 1.2.840.113556.1.4.319 control
Status: VERIFIED FIXED
Alias: None
Product: OpenLDAP
Classification: Unclassified
Component: historical (show other issues)
Version: unspecified
Hardware: All All
: --- normal
Target Milestone: ---
Assignee: OpenLDAP project
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-08-15 14:21 UTC by randolf.werner@sap.com
Modified: 2019-11-20 18:03 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description randolf.werner@sap.com 2007-08-15 14:21:29 UTC
Full_Name: Randolf Werner 
Version: 2.3.35
OS: Windows XP
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (155.56.68.221)


Hi,
i just have found the following compatibilty issue when using the
1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt) supported
by slapd. According to Microsoft and IBM LDAP API ldap_create_page_control

http://msdn2.microsoft.com/en-us/library/aa366547.aspx
http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis/ldap_create_page_control.htm

the page size parameter should be an 32 Bit unsigned integer, however the RFC
specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses 2**32-1
as default page size value. This works fine with Microsoft Active Directroy for
many years but fails with "Protocol Error" (Error 2) when using slapd. Our
workaround is to use 2**31-1 as default value in our client. I am not sure which
implementation is correct, anyway it is a compatibility issue for ldap clients
and might be "fixed" to improve compatibility.

Best regards
    Randolf

Comment 1 Kurt Zeilenga 2007-08-15 16:34:28 UTC
On Aug 15, 2007, at 2:21 PM, randolf.werner@sap.com wrote:

> Full_Name: Randolf Werner
> Version: 2.3.35
> OS: Windows XP
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (155.56.68.221)
>
>
> Hi,
> i just have found the following compatibilty issue when using the
> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/ 
> rfc2696.txt) supported
> by slapd. According to Microsoft and IBM LDAP API  
> ldap_create_page_control
>
> http://msdn2.microsoft.com/en-us/library/aa366547.aspx
> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp? 
> topic=/apis/ldap_create_page_control.htm
>
> the page size parameter should be an 32 Bit unsigned integer,  
> however the RFC
> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client  
> uses 2**32-1
> as default page size value. This works fine with Microsoft Active  
> Directroy for
> many years but fails with "Protocol Error" (Error 2) when using  
> slapd. Our
> workaround is to use 2**31-1 as default value in our client. I am  
> not sure which
> implementation is correct, anyway it is a compatibility issue for  
> ldap clients
> and might be "fixed" to improve compatibility.

Given maxInt is defined in RFC 4511 as:
         maxInt INTEGER ::= 2147483647 -- (2^^31 - 1) --

this doesn't appear to be a bug in OpenLDAP Software.

-- Kurt

Comment 2 Howard Chu 2007-08-15 16:42:08 UTC
randolf.werner@sap.com wrote:
> Full_Name: Randolf Werner 
> Version: 2.3.35
> OS: Windows XP
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (155.56.68.221)
> 
> 
> Hi,
> i just have found the following compatibilty issue when using the
> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt) supported
> by slapd. According to Microsoft and IBM LDAP API ldap_create_page_control
> 
> http://msdn2.microsoft.com/en-us/library/aa366547.aspx
> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis/ldap_create_page_control.htm
> 
> the page size parameter should be an 32 Bit unsigned integer, however the RFC
> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses 2**32-1
> as default page size value. This works fine with Microsoft Active Directroy for
> many years but fails with "Protocol Error" (Error 2) when using slapd. Our
> workaround is to use 2**31-1 as default value in our client. I am not sure which
> implementation is correct, anyway it is a compatibility issue for ldap clients
> and might be "fixed" to improve compatibility.

I suggest you file bug reports with Microsoft and IBM to fix their broken 
implementations then. RFC4511 clearly defines maxInt to be 2^31-1. That value 
has not changed since RFC2251 either.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Comment 3 ando@openldap.org 2007-08-15 18:56:24 UTC
> randolf.werner@sap.com wrote:
>> Full_Name: Randolf Werner
>> Version: 2.3.35
>> OS: Windows XP
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (155.56.68.221)
>>
>>
>> Hi,
>> i just have found the following compatibilty issue when using the
>> 1.2.840.113556.1.4.319 control (http://www.ietf.org/rfc/rfc2696.txt)
>> supported
>> by slapd. According to Microsoft and IBM LDAP API
>> ldap_create_page_control
>>
>> http://msdn2.microsoft.com/en-us/library/aa366547.aspx
>> http://publib.boulder.ibm.com/infocenter/iseries/v5r3/index.jsp?topic=/apis/ldap_create_page_control.htm
>>
>> the page size parameter should be an 32 Bit unsigned integer, however
>> the RFC
>> specifies it as "(INTEGER (0..maxInt)". Therefore our LDAP client uses
>> 2**32-1
>> as default page size value. This works fine with Microsoft Active
>> Directroy for
>> many years but fails with "Protocol Error" (Error 2) when using slapd.
>> Our
>> workaround is to use 2**31-1 as default value in our client. I am not
>> sure which
>> implementation is correct, anyway it is a compatibility issue for ldap
>> clients
>> and might be "fixed" to improve compatibility.
>
> I suggest you file bug reports with Microsoft and IBM to fix their broken
> implementations then. RFC4511 clearly defines maxInt to be 2^31-1. That
> value
> has not changed since RFC2251 either.

Or, fix your clients so that they don't use pagedResults when not
necessary.  2**31-1 is a pretty large number.

p.



Ing. Pierangelo Masarati
OpenLDAP Core Team

SysNet s.r.l.
via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
---------------------------------------
Office:  +39 02 23998309
Mobile:  +39 333 4963172
Email:   pierangelo.masarati@sys-net.it
---------------------------------------


Comment 4 randolf.werner@sap.com 2007-08-16 07:17:45 UTC
Hi,
 
thanks for your replies. Anyway i did not opened this message to discuss
who implented RFC 2696 correctly and who not.  The point is something
else:
 
1) I tried to share my experience (and solution) with other LDAP
application programmers, since i am pretty sure there are/will be more
clients running into this issue (e.g. ldp.exe from Windows support tools
seems to have the same problem when enabling 1.2.840.113556.1.4.319 /
pages search type again slapd).
2) Try to point out how it can easily happen to write a client working
with Active Directory but not with slapd and suggested an easy but
perhaps not 100% RFC 2696 compliant slapd change.
 
Perhaps i should explain 2) a little bit more from my application
develeopers point of view. Here is my rough application history:
 
a) In ~1999 we started building a LDAP V3 client for SAP that had to
work on a very wide variety of OS platforms and as much as possible LDAP
V3 compliant directories.
b) It was first shipped in 2000/2001 and customers were happy.
c) Data volume at customer side increazed and it stopped working for
Active Directory customers.
d) We learned about size limits, page size and the fact you simply have
to use 1.2.840.113556.1.4.319  to get more than 1000 entries from Active
Directroy. We implemented it and everybody was happy again. Implenting
ldap_create_page_control for all OS platforms was not that trivial at
that time.
e) In ~2004 the first customer liked to use slapd. We noticed some minor
issues (e.g. setting V3 protocol version explicitly) fixed it and once
more everybody was happy.
f) Yesterday a slapd 2.3.35 slapd customer reported a problem similar to
e) ("Protocol Error"). So we analyzed it and fixed it again by reducing
default page size to 2**31-1.
 
Here is why i believe others might run into the same issue. If you need
to create a ldap client dealing with big search results for various LDAP
directories you had to use 1.2.840.113556.1.4.319 even if you are only
interested in collecting the entire result otherwise you would fail with
Active Directory. You look at the ldap_create_page_control API spec
(developers oftern use API specs and not protocol RFCs) and notice page
size to be an usinged integer. Since you are not really interested in
pages you specify the largest possible value 2**32-1. This way you end
up with client working with Active Directory, all directories not
supporting 1.2.840.113556.1.4.319 but not with slapd supporting
1.2.840.113556.1.4.319.
 
I am not interested in discussing who is right and who is wrong, i am
just interested in making as much as possible working together in the
real world with as little as possible customer pain. Therefore i am
almost 100% sure Active Directory will not break existing clients by
using signed instead of unsigned integer page size just to be more RFC
2696 compliant.On the other hand slapd has a chance to detect negavtive
page sizes and assume some reasonable page size in that situation
instead of "Protocol Error" without breaking any existing client. For my
customers it will not make much difference since the patch will be
rolled out soon, but perhaps others can benefit.
 
Best regards
    Randolf
 
 
Comment 5 ando@openldap.org 2007-08-16 07:49:50 UTC
changed notes
Comment 6 Hallvard Furuseth 2007-08-16 16:00:10 UTC
randolf.werner@sap.com writes:
> Active Directory. You look at the ldap_create_page_control API spec
> (developers oftern use API specs and not protocol RFCs) and notice page
> size to be an usinged integer. Since you are not really interested in
> pages you specify the largest possible value 2**32-1.

So that API spec is bogus, it should have specified the range.  Which
data type is being used is not relevant to what the protocol accepts, in
particular since the width of data types sometimes change.  E.g. if you
move from a 32-bit to a 64-bit host.

> I am not interested in discussing who is right and who is wrong, i am
> just interested in making as much as possible working together in the
> real world with as little as possible customer pain.

Notice however that your first problem happened because AD breaks the
standard, and your second problem because you rewrote it by bogus
Microsoft documentation and it worked because of another way AD breaks
the standard.  (Though less seriously in the second case, it follows the
"be liberal in what you accept" way.)

If OpenLDAP is changed to accept wider limits, it might also encourage
more people to write bogus code like you did.  OpenLDAP has been going
in the other direction over the years, doing ever more strict error
checking.

That said, pretending that Microsoft does not exist isn't always a
winning strategy, so it might make sense to implement a "relaxed" mode
which accepts common ways to break the standard.  But I don't think it
should be turned on by default.

> Therefore i am almost 100% sure Active Directory will not break
> existing clients by using signed instead of unsigned integer page size
> just to be more RFC 2696 compliant.

In addition to fixing the doc, they they could fix that in API by
reducing a passed value >= 2**31 to the max valid value.  Not that they
seem to care much about standards anyway.

One note - I'm curious why you did not use page size 0, which the
Microsoft API pages you posted say overrides such page sizes.  Though
maybe I misunderstood and that doesn't actually help.

-- 
Regards,
Hallvard

Comment 7 Howard Chu 2007-08-16 16:12:11 UTC
randolf.werner@sap.com wrote:
> Hi,
> =20
> thanks for your replies. Anyway i did not opened this message to discuss
> who implented RFC 2696 correctly and who not.  The point is something
> else:

The point is that there is a specific definition of the protocol. When vendors 
implement the protocol incorrectly, interoperability suffers. The fix for those 
interoperability problems is to take steps to ensure that vendors don't deviate 
from the published specifications. Likewise, application developers must pay 
attention to the published specs and write compliant code. Otherwise the entire 
concept of published standards is meaningless.

> Here is why i believe others might run into the same issue. If you need
> to create a ldap client dealing with big search results for various LDAP
> directories you had to use 1.2.840.113556.1.4.319 even if you are only
> interested in collecting the entire result otherwise you would fail with
> Active Directory.

Nonsense. You simply ask your AD administrator to raise the default sizelimit, 
the same way you would ask an OpenLDAP administrator to do the same thing.

> I am not interested in discussing who is right and who is wrong, i am
> just interested in making as much as possible working together in the
> real world with as little as possible customer pain.

In the real world, if you ignore the specs, your programs break. If you adhere 
to the specs, your programs keep working.

What you're asking for is the equivalent of this: "My Microsoft building has a 
nail sticking out on the floor and I keep tripping over it when I walk by. Your 
OpenLDAP building is missing this nail; please pull a nail out so I can trip 
over it the same way."
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Comment 8 Howard Chu 2007-08-16 16:22:16 UTC
h.b.furuseth@usit.uio.no wrote:
> That said, pretending that Microsoft does not exist isn't always a
> winning strategy, so it might make sense to implement a "relaxed" mode
> which accepts common ways to break the standard.  But I don't think it
> should be turned on by default.

We don't pretend Microsoft doesn't exist. But we *do* expect them to mean what 
they say; since 3 Microsoft employees worked on RFC2696 you would sort of 
expect them to get it right.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Comment 9 randolf.werner@sap.com 2007-08-16 17:14:25 UTC
Hi,

oh my god what have i started here. Please don't try to convince me that
standards are important and slapd pricecly implents RFC 2696, i was
already convinced of that before i started the discussion. Anyway i like
to resond to Howard Chu because he mentioned something dangerous and
wrong regarding Active Directory (somewhat of topic here) but i like to
avaoid people following his suggestion:

Howard Chu <hyc@symas.com> wrote:
>> Here is why i believe others might run into the same issue. If you
need
>> to create a ldap client dealing with big search results for various
LDAP
>> directories you had to use 1.2.840.113556.1.4.319 even if you are
only
>> interested in collecting the entire result otherwise you would fail
with
>> Active Directory.
>
>Nonsense. You simply ask your AD administrator to raise the default
sizelimit, 
>the same way you would ask an OpenLDAP administrator to do the same
thing.

Well about 6 years ago when i first noticed this problem, i asked
Microsoft about that limit and yes you can use ntdsutil.exe to set
MaxPageSize to a higher limit:

http://support.microsoft.com/kb/315071/en-us

But it is strongly recommend to not change it!!!! Over the years i have
seen quite some Active Directories customers and many of them are very
afraid when you ask them you change some configuration or extending the
schema. In 2001 SAP's IT set MaxPageSize=10000 for a few weeks as a
temporary workaround until i was able to provide a real solution. That
was a long time ago and today SAP's and probably most other companies
Active Directories are running with default values.
1.2.840.113556.1.4.319 and MaxPageSize had not been invented to make our
life more difficult. A good article about it is e.g.
http://searchwinit.techtarget.com/tip/0,289483,sid1_gci1265206,00.html.
I like to close with Gary Olsen's conclusion "The best practice for
setting MaxPageSize is to leave it alone."

Best regards
    Randolf  

Comment 10 Howard Chu 2007-08-16 17:50:28 UTC
randolf.werner@sap.com wrote:
> Over the years i have
> seen quite some Active Directories customers and many of them are very
> afraid when you ask them you change some configuration or extending the
> schema.

Doesn't it seem odd to you that people would voluntarily use software that they 
are afraid of?

Never mind, this discussion has gone far enough off-topic.
-- 
   -- Howard Chu
   Chief Architect, Symas Corp.  http://www.symas.com
   Director, Highland Sun        http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP     http://www.openldap.org/project/

Comment 11 Howard Chu 2007-08-16 17:50:59 UTC
changed state Open to Suspended
Comment 12 ando@openldap.org 2007-08-30 08:55:49 UTC
changed notes
moved from Incoming to Historical
Comment 13 OpenLDAP project 2019-11-20 18:03:24 UTC
not ours
proposed change rejected
Comment 14 Quanah Gibson-Mount 2019-11-20 18:03:24 UTC
changed notes
changed state Suspended to Closed