Re: sasl_encode type conflict in 64 bit builds (ITS#3212)

[Kurt@OpenLDAP.org]

Your fix, I believe, is correct.  I've committed changes
based upon it to HEAD.  Please test.

Kurt

At 12:18 AM 6/29/2004, jeff.lewis29@verizon.net wrote:
>Full_Name: Jeff Lewis
>Version: 2.1.17
>OS: SunOS 5.8
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (192.127.94.7)
>
>
>I'm an open source novice and have run into a problem building the OpenLDAP
>clients on SunOS.  This very well could be a cockpit error on my part and I'd be
>delighted to learn that it's my error.
>
>I think I've stumbled upon a type conflict that's keeping the ldap client tools
>from working.  I just built the clients on my SunOS platform with is a big
>endian 64 bit machine.  When I run ldapsearch (and ldapwhoami...haven't tried
>the others but I suspect they'll do something similar), I get the following
>results:
>
>  $ ./ldapsearch -b "" -s base -h tnt80 -U djl -Y DIGEST-MD5
>  SASL/DIGEST-MD5 authentication started
>  Please enter your password:
>  SASL username: djl
>  SASL SSF: 128
>  SASL installing layers
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <> with scope base
>  # filter: (objectclass=*)
>  # requesting: ALL
>  #
>
>  ldapsearch: ldap_search_ext: Can't contact LDAP server (81)
>  $
>
>I did a network trace and found that the bind was successful, but the very next
>write/send to the server carrying the search request had "run away" and was
>sending a massive amount of data.  After some study, I found my way into
>sb_sasl_write in the libraries/libldap/cyrus.c source module.  In it is a call
>to sasl_encode that is failing for me.  What's happening is the sasl_encode()
>call wants a pointer to an "unsigned int" and sb_sasl_write passes a pointer to
>a ber_len_t cast as "unsigned *".  The ber_len_t type wants to be "unsigned
>long" which is 8 bytes wide on my 64 bit machine.  The bottom line is
>sasl_encode stores 4 bytes of unsigned data into an 8 byte area, and since my
>machine is big endian, the resulting value is 2^^32 times bigger than it should
>be and the next write tries to send a huge amount of data.  If my machine were
>small endian, I wouldn't have noticed a problem.
>
>I figured that if there were a sasl_encode() call in the OpenLDAP source,
>there's probably a sasl_decode() call as well.  I found it in sb_sasl_read and
>sure enough, it suffers from the same problem.
>
>Knowing that I'm not running the latest OpenLDAP package, I downloaded the
>latest stable release (2.2.13, I think), configured it and examined the source. 
>I did not build the package.  But from desk checking the code and type
>definitions, it appears that this little problem of mine is still there.
>
>For now, I've added some code to copy the lengths into and out of an unsigned
>temp variable and passed a pointer to the temp to sasl_encode/decode.  The code
>is much happier and produces good results for me.
>
>Here's the original call to sasl_encode:
>
>        ret = sasl_encode( p->sasl_context, buf, len,
>                (SASL_CONST char **)&p->buf_out.buf_base,
>                (unsigned *)&p->buf_out.buf_size );
>
>Here's my kludged call:
>
>        temp = p->buf_out.buf_size;
>        ret = sasl_encode( p->sasl_context, buf, len,
>                (SASL_CONST char **)&p->buf_out.buf_base,
>                (unsigned *)&temp );
>        p->buf_out.buf_size = temp;
>
>I did something similar to the sasl_decode call in sb_sasl_read.
>
>If you're interested in the diffs:
>
>  $ diff cyrus.orig cyrus.c
>  227a228
>  >       unsigned                temp;
>  292a294
>  >       temp = p->buf_in.buf_end;
>  296c298,299
>  <               (unsigned *)&p->buf_in.buf_end );
>  ---
>  >               (unsigned *)&temp );
>  >         p->buf_in.buf_end = temp;
>  321a325
>  >       unsigned                temp;
>  350a355
>  >         temp = p->buf_out.buf_size;
>  353c358,359
>  <               (unsigned *)&p->buf_out.buf_size );
>  ---
>  >               (unsigned *)&temp );
>  >         p->buf_out.buf_size = temp;
>  $
>
>
>My ldapsearch is much much better after the changes, but I doubt seriously you'd
>want these particular changes in your code.  They're a bit of a kludge and are
>completely wasteful on 32 bit platforms.  But after making the changes above,
>here's what ldapsearch does for me (correctly, I presume):
>
>  $ ./ldapsearch -b "" -s base -h tnt80 -U djl -Y DIGEST-MD5
>  SASL/DIGEST-MD5 authentication started
>  Please enter your password:
>  SASL username: djl
>  SASL SSF: 128
>  SASL installing layers
>  # extended LDIF
>  #
>  # LDAPv3
>  # base <> with scope base
>  # filter: (objectclass=*)
>  # requesting: ALL
>  #
>
>  #
>  dn:
>  objectClass: top
>  objectClass: OpenLDAProotDSE
>
>  # search result
>  search: 3
>  result: 0 Success
>
>  # numResponses: 2
>  # numEntries: 1
>  $
>
>I've searched the archives and did not see anything that caused me to think
>someone else was having a similar problem.  But then, being an open source
>novice, I could have very easily missed something important.  :-)  I do hope
>this is enough information for you to go on.  If not, email me and I'll fill in
>any missing blanks.  Like I said, I'd be very happy to learn that the error is
>mine.