Note that this change was made as the previous versioning,
including what still used on the 2.1 branch, is broken.
It was (is) allowing incompatible versions of the libraries
to be picked up dynamically (which is causing the usual
problems).

Use of -release appears to be the only mechanism available to
us that:
        a) ensures applications can link in OpenLDAP libraries
        using -lldap -llber and pick up the latest (static or
        shared) libraries.
        b) ensures conflicting shared libraries are not picked
        up dynamically, and
        c) ensures the Project is able to update the ABIs as
        needed (including applying fixes that effect the ABI
        as patch releases, including on older branches).

The only approach I know which meets this requirement
would be to use of sonames which encode that effectively
encode the version (e.g., 20207).  This was rejected as
libtool only supports sonames as large as 999.  It is
also questionable if all dynamic linkers support sonames
larger than 255.

If you know of some other approach which meets the
project requirements, please feel free to suggest it on
the developers list.  However, reverting to the old broken
way of doing things is not an option.

Kurt


At 12:28 AM 3/23/2004, kean@armory.com wrote:
>Full_Name: Kean Johnston
>Version: 2.2.7
>OS: Multiple
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (67.114.177.38)
>
>
>OpenLDAP 2.2.7 introduced a change to how libtool is being used. build/top.mk
>changes from using -version-info to using -release, and build/version.var
>changes the format of ol_api_lib accordingly. This has serious packaging
>implications.
>
>The upshot of this change is that every time a new version of OpenLDAP comes
>out, there is a new set of libraries that is produced. By using -release, file
>names like liblber-2.2.7.so are used. The problem with this is that if you ever
>distribute, say version 2.2.7 of OpenLDAP, then you have to ship that version
>forever more, as there may be applications that now have that shared library as
>a dependency. The purpose of -version-info is to fix exactly this, so that you
>only need to ship old .so's for releases where the binary interface really has
>changed. I am sure it was not your intention to make package vendors ship every
>singel release of OpenLDAP thats ever been produced (or that a package
>distributor may have published on the web).
>
>Can you please go back to using -version-info? Many thanks.

changed notes
changed state Open to Suspended

> It was (is) allowing incompatible versions of the libraries
> to be picked up dynamically (which is causing the usual
> problems).
There are ways to address that. See below.

>         c) ensures the Project is able to update the ABIs as
>         needed (including applying fixes that effect the ABI
>         as patch releases, including on older branches).
I think that this requirement is almost impossible to meet even using 
the -release, in any sensible way, without some effort. I also think 
you're using the term ABI slightly incorrectly, and really mean API. 
Consider this. Suppose you have release 2.1.30. It has a defined API, 
that is expressed in header files and implemented in the shared library. 
That implementation defines the ABI, as it is what applications that 
link against that version of the library will expect. Lets say this 
library was installed with an SONAME of libldap.so.2.1. For argument's 
sake, lets say there was a function int some_ldap_fn (int). Now you are 
producing version 2.1.31, and you find that this function absolutely 
must take an extra argument, lets say it now becomes int some_ldap_fn 
(int, const char *).

You are now faced with two choices. When fixing this, you can address it 
at the API level and leave the ABI untouched (requires some extra work) 
or you can 'full steam ahead and f*** the icebergs' and change both the 
API and the ABI (requires no work beyond the change itself). The first 
approach is, obviously, preferable. Here's one way of doing it. Suppose 
this function was declared in ldap.h. Instead of changing:
    extern int some_ldap_fn(int);
to
    extern int some_ldap_fn(int, const char *);

you would instead do something like:
    extern int some_ldap_fn_2(int, const char *);
    #define some_ldap_fn(A,B) some_ldap_fn_2((A),(B))

In which ever file implements some_ldap_fn you have to of course still have:
int some_ldap_fn (int x)
{
   return old_behaviour();
}

but I bet that in most cases this can simply be a stub that calls the 
new one, passing in the extra argument as a NULL, perhaps as:
int some_ldap_fn (int x)
{
   return some_ldap_fn_2 (x, NULL);
}

The details will vary from case to case, of course.

This ensures that for those users that are recompiling code, that they 
get the new API definition, and will result in their application using 
the new ABI. But for those applications that are not being recompiled, 
and are simply linked against the library, nothing will change, because 
at the time they were compiled the function did just take one argument, 
and that is preserved.

This approach means that through the lifetime of, say, OpenLDAP 2.1.x, 
you can keep the same shared library name. When you change the API and 
ABI in ways that cannot practically be handled above, you then change to 
OpenLDAP 2.2.x, and change the shared library to libopenldap.so.2.2.

This has a distinct advantage for OS vendors and package maintainers. It 
means that for the lifetime of release 2.1, they do not have to 
recompile every application in the OS distribution that uses OpenLDAP. 
They can continue to update OpenLDAP itself, and it will continue to 
serve all the applications that were linked against it. It also means 
that for backwards compatiblity, the package maintainers only need to 
provide one version of the shared library for each major release cycle 
of OpenLDAP. Consider the alternative.

Suppose you stick with the -release mechanism. Today, the release is 
2.2.7. So anything I link against OpenLDAP will have a dependency (at 
the ELF level) on a shared library clled libldap-2.2.7.so. Lets say that 
today I aolso compile PHP, and it uses OpenLDAP. I now package both and 
release them to the world. 2 weeks later there is a new version of 
OpenLDAP, but PHP has unchanged. I would like to release just a new 
OpenLDAP. However, I can no longer do that. Well, I can, but I have to 
preserve the old libldap-2.2.7.so. So now my OpenLDAP package contains 
botht hat version and libldap-2.2.8.so. And so on and so on ... you will 
end up in the situation where OS vendors will need to ship 20 or 30 
versions of the same library, all very subtly different. The only 
alternative is to track down absolutely every package that may have used 
it, and recompile them and redistribute them. And yet you will *STILL* 
have to provide all the old versions becuase you have no idea what the 
end user has done, and which versions they have linked against. This is 
the *HUGE* problem with using -release.

> encode the version (e.g., 20207).  This was rejected as
> libtool only supports sonames as large as 999.  It is
> also questionable if all dynamic linkers support sonames
> larger than 255.
Thats a libtool limitation then, but not required. You can (and really 
should try to) keep one shared library version per OpenLDAP series. For 
example, keep libldap.so.2.1 as the soname throughout the lifetime of 
the OpenLDAP 2.1.x series, and libldap.so.2.2 through the lifetime of 
OpenLDAP 2.2.x etc. Also note that the version number that appears in a 
DT_SONAME is numeric by convention only, and the RTLD's (on all the OSes 
Ive ever worked on) pay no attention to it except as a string value. Its 
how its defined in the ELF standards. There is absolutely no 
significance to the name whatsoever - it either matches what a binary 
has in its DT_NEEDED or it doesn't. If it does, the RTLD will use it. 
But the point is moot if you use smaller, more human readable versions 
like 2.1 and 2.2 that reflect the core software the library is implementing.

> If you know of some other approach which meets the
> project requirements, please feel free to suggest it on
> the developers list.  However, reverting to the old broken
> way of doing things is not an option.
For the record, the old was was only broken for how it was being used, 
not intrinsically. libtool has a sophisticated (read: obtuse and 
convoluted) algorithm for determining the SONAME to use, and that is 
controlled by the arguments to -version-info. I think you were using 
that option incorrectly in the last. If you changed the ABI one of the 
three values (I forget which) should have changed too. BVut that is a 
much worse implementation than what I described above.

Kean

* kean@armory.com (kean@armory.com) wrote:
> > It was (is) allowing incompatible versions of the libraries
> > to be picked up dynamically (which is causing the usual
> > problems).
> There are ways to address that. See below.

If I understand correctly, these are actually not the only options
available, thankfully.  A much better option available on halfway recent
systems is to use symbol versioning.  This allows you to have multiple
symbols defined in the same library but have different versions for
them.  This means you can support, inside the same library and SONAME,
multiple ABIs.  This is exactly what glibc does, in fact, and they
havn't changed their SONAME since 1997 or so.  The difficulty with
implementing symbol versioning correctly is that it requires what some
may call 'overhead' in that you have to be aware of when you're making
an ABI change, or an API change and you have to update your versioning
script correctly when such changes are made.  It also means you have to
be very careful when changing code to fix a bug in an older version and
make sure that you don't change behaviour that was depended upon.

Certainly, symbol versioning makes packaging much easier and makes it
possible to link in multiple versions of a library safely (which is
happening more and more often as the complexity of programs increase).
Unfortunately, it's not supported on every Unix that ever lived.  Please
consider using it where it's available though and let the old Unixes
continue to do what they do today.

As one of the Debian package maintainers, if this isn't done by the
OpenLDAP core team we will likely have to do it ourselves, though we
won't be able to track every ABI/API change and will probably just use
the SONAME (which should correspond to OpenLDAP 2.1 or OpenLDAP 2.2) for
the version.  This isn't as clean or efficient but it will at least
handle the problem where two versions of the library are linked in at
the same time.  If OpenLDAP does decide to move to using '2.1.21' or
similar we'd probably be forced to correct that and manage the ABI
changes ourselves- the alternatives are much worse.

	Stephen

At 09:35 AM 3/23/2004, kean@armory.com wrote:
>> It was (is) allowing incompatible versions of the libraries
>> to be picked up dynamically (which is causing the usual
>> problems).
>There are ways to address that. See below.

Your comments focus mostly on how to avoid making incompatible
changes.  However, there will be incompatible changes.  We
need to ensure that when multiple libraries are available to the
dynamic linker that incorrect ones are not chosen.  This is
what using -release does for us.

The only alternative that seems acceptable, if it where
technically feasible, is to have per-minor release major
interface numbers that update normally within a minor
release, but updated by some step between minor releases.

That is, 2.2 would use numbers like:
        20201, 20202, 20203, ...
        (where the least significant digit was tied to interface
        changes, not patch numbers).
and 2.3 would use numbers like:
        20301, 20302, 20303, ...
and 3.0 would use numbers like:
        30001, 30002, 30003, ...

However, this does not appear to be technical feasible as
        1) libtool restrictions,
        2) dynamic linker restrictions.

Now, you say there are not problems with dynamic linkers.
However, I have my doubts here.  I rather not make the
assumption that 1) is okay until the libtool community
was convinced that it was okay and hence lifted 2).

Kurt

* Kurt@OpenLDAP.org (Kurt@OpenLDAP.org) wrote:
> Your comments focus mostly on how to avoid making incompatible
> changes.  However, there will be incompatible changes.  We
> need to ensure that when multiple libraries are available to the
> dynamic linker that incorrect ones are not chosen.  This is
> what using -release does for us.

If you can't manage to not make incompatible changes for a reasonable
amount of time then you shouldn't be making a library.  For development
it's fine- have a development branch and just use the same SONAME.  For
releases, no, it's not reasonable or acceptable.  What you are ensuring
by doing this is that users will *much* more often end up trying to run
binaries which link in multiple incompatible versions of a library.
This is exactly the same problem we've run into w/ OpenSSL and it's been
a huge pain in the ass because they aren't competent enough to figure
out what an ABI change is.

The way you basically screw your users over a barrel by doing this is
through indirect linking, which happens quite often, *especially* for
such a low-level library as libldap.  It goes like this:

A builds libldap 2.1.25
B builds libX which uses libldap 2.1.25
C builds appY which uses libX and libldap 2.1.25.

As long as libX compiles against the same libldap that appY does, you're
fine.  This doesn't last very long though-

OpenLDAP 2.1.26 comes out
A builds libldap 2.1.26
New version of appY comes out
C rebuilds appY against the unchanged libX and libldap 2.1.26

Yay, now appY segfaults all over the place because both libldap 2.1.25
and libldap 2.1.26 get loaded and the sybmols collide.

Now, this is mitigated slightly by slowing down the rate of ABI changes
so that it's only changed between major releases.  The problem still
exists, but at least you don't run into it as often and when you do you
just have to recompile everything.  Very annoying, but much more doable
when it's rare.

Of course, the correct way to handle this is to use versioned symbols.
That way you can have a single library/SONAME which supports both old
programs/libraries and new programs/libraries.  If you can't handle that
then you should at least stamp a version on each major library/SONAME so
that when the two libraries get pulled in they don't collide.

Doing this for every minor version would be it's own hell, of course, 
because you'd have to have a package for every minor reversion of
OpenLDAP and you'd have to *support* every minor revision of OpenLDAP.
This gets *very* ugly and makes maintenance a nightmare.  This would
basically be equivilant to staticlly linking.  Consider a security bug
in version 2.2.45, it'd probably be in 2.2.1-2.2.44 too and you'd have
to issue an update for *all* of them.

This also would grow the memory usage of applications because each
library it linked to might be loading in it's own version of libldap.
These things are not acceptable.

Please try to understand and to consider the implications of your
change.  Almost certainly we would need to revert it in order to be able
to continue to maintain OpenLDAP reasonably.  This would likely involve
a fork of OpenLDAP which I'd be lothe to do except as a last resort.
Your solution also certainly does not solve the problem you are 
outlining.

	Thanks,

		Stephen

Reducing ABI changes in released versions is all well and good.
However, some ABI changes are unavoidable.  That's why ABI
versioning exists.  However, at present, we seem to having
problems coming up with a workable scheme.

My question to those who prefer we go back to a version-info
scheme is to provide a suggested numbering scheme (with no
changes to the library names (see requirements in my previous
post)) for 2.2, 2.3, and 2.x (and if you are really forward
thinker, 3.x) branches which support inter-branch ABI changes.

Kurt

changed notes

* Kurt D. Zeilenga (Kurt@OpenLDAP.org) wrote:
> Reducing ABI changes in released versions is all well and good.
> However, some ABI changes are unavoidable.  That's why ABI
> versioning exists.  However, at present, we seem to having
> problems coming up with a workable scheme.

I'm glad to hear that it doesn't sound like your intent is to break
binary compatibility between each released version.  Having to change
the ABI on a released version every once in a while is understandable
when there's a long release cycle for major releases.

> My question to those who prefer we go back to a version-info
> scheme is to provide a suggested numbering scheme (with no
> changes to the library names (see requirements in my previous
> post)) for 2.2, 2.3, and 2.x (and if you are really forward
> thinker, 3.x) branches which support inter-branch ABI changes.

I'm guessing you mean 'intra-branch' ABI changes, as in those inside a
given major release (ie: 2.2).  Certainly one option, as I mentioned
previously, would be to use versioned symbols.  I realize this isn't
possible on all architectures but it'd be a great thing to have on the
architectures where it's available.  I expect we'll probably actually
implement symbol versioning for the Debian OpenLDAP packages as soon as
libtool's upstream includes the patch to automate symbol versioning.
It's not as nice as if OpenLDAP upstream did it themselves on a
per-symbol basis similar to what glibc does but it will help in the
situations I outlined previously.

As for the numbering scheme, I'm actually inclined to suggest a very
simple one- a simple counter that's not derived from the OpenLDAP
version number.  Chances are it'll be a very long time before you get to
the '255' number that's apparently causing some concern.  It wouldn't be
as obvious by looking at just the SONAME what OpenLDAP version it came
from but certainly a cross-reference could be maintained easily.

This is just off the top of my head, I'll discuss this with the other
Debian OpenLDAP maintainers and Debian release managers who have to deal
with these library issues across a large set of packages much more often
than I do and see what suggestions they have and get back to you.

	Stephen

At 07:18 PM 3/23/2004, sfrost@snowman.net wrote:
>As for the numbering scheme, I'm actually inclined to suggest a very
>simple one- a simple counter that's not derived from the OpenLDAP
>version number.  Chances are it'll be a very long time before you get to
>the '255' number that's apparently causing some concern.  It wouldn't be
>as obvious by looking at just the SONAME what OpenLDAP version it came
>=66rom but certainly a cross-reference could be maintained easily. 

Note that the versioning scheme must take into account we
have multiple release branches all sharing the same library
names.  That is, -lldap -llber.  Which means that 255 has to
somehow divided between the possible branches.

Kurt 

Does anybody have experience with doing something like:
	-release 2.2 -version-info 1:0:0

That is, do such constructions works on most platforms.
Libtool seems to construct:
  # Names of this library.
  library_names='liblber-2.2.so.1 liblber-2.2.so liblber.so'

  # The name of the static archive.
  old_library='liblber.a'

If this works, this might met our requirements.  Comments?

Kurt

Kurt,

Your last suggestion looks good to me.  The current versioning method will 
definitely break things here for Stanford -- We rely on the ability to be 
able to stow/unstow packages for upgrading a particular package in-place on 
systems without rebuilding software.

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITSS/TSS/Computing Systems
ITSS/TSS/Infrastructure Operations
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

* Kurt D. Zeilenga (Kurt@OpenLDAP.org) wrote:
> Does anybody have experience with doing something like:
> 	-release 2.2 -version-info 1:0:0
> 
> That is, do such constructions works on most platforms.
> Libtool seems to construct:
>   # Names of this library.
>   library_names='liblber-2.2.so.1 liblber-2.2.so liblber.so'
> 
>   # The name of the static archive.
>   old_library='liblber.a'
> 
> If this works, this might met our requirements.  Comments?

Alright, this doesn't really work because it means that -llber could
actually mean any of the versions which is generally not what you really
want.  You should be making a distinction between the major releases in
OpenLDAP because in general you don't expect people to be able to move
from 2.0 to 2.2 w/o  changing something (API differences).  What you
probably want to do is close though, and is very similar to what the
Sleepcat (libdb) folks do now-

Use a different library name for the major versions (2.0, 2.1, 2.2, 3.0,
etc) and then keep the API the same through those but change the ABI as
necessary and update *that* using the regular -version-info method.
This will give you libraries like:
liblber-2.2.so.0, liblber-2.2.so.1 with a link for liblber-2.2.so
liblber-2.3.so.0, liblber-2.3.so.1 with a link for liblber-2.3.so, etc.

Note that you *won't* have a 'liblber.so' link and therefore won't be
able to do '-llber', but you don't really want people doing that anyway
because they're different APIs.  You also won't be using '-release' at
all.

Your idea above probably wouldn't actually be a problem for Debian since
we don't allow people to try and have multiple -dev packages installed
at the same time (because of the .so symlink) anyway, but it wouldn't
really be much fun for us to deal with in the packaging.  It's also not
something we'd recommend.

What I think we probably *would* do is introduce symbol versioning which
then followed the SONAME, so something like- LDAP_2_2_SO_0, 
LDAP_2_2_SO_1.  This would mean that a given application could link 
against a library which linked against LDAP_2_2_0 and link itself 
against LDAP_2_2_SO_1 and not have any problems.  It should also make it
clear that the symbol version is derived from the SONAME (as opposted to
the release version).

Of course, changing the ABI in a maintenance release should still be
pretty rare, and made *very* clear (it should really be noted in the
release notes, at the very least) when it does happen.  It's also going
to be something of a pain for the Debian maintainers to deal with
because we're going to have to support both versions which also means
we're going to be having seperate source packages for each ABI version.

It'd be much nicer to either use versioned symbols or just require that
the ABI version not change in maintenance releases.

	Stephen

I did some research last night and found no issue with
going that route I last suggested.  Hence, I plan on
moving to -release 2.2 -version-info x in the next 2.2
release.

Kurt

At 10:29 PM 3/23/2004, quanah@stanford.edu wrote:
>Kurt,
>
>Your last suggestion looks good to me.  The current versioning method will 
>definitely break things here for Stanford -- We rely on the ability to be 
>able to stow/unstow packages for upgrading a particular package in-place on 
>systems without rebuilding software.
>
>--Quanah
>
>--
>Quanah Gibson-Mount
>Principal Software Developer
>ITSS/TSS/Computing Systems
>ITSS/TSS/Infrastructure Operations
>Stanford University
>GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

At 06:44 AM 3/24/2004, sfrost@snowman.net wrote:

>--sHhmzMatm85/eqYu
>Content-Type: text/plain; charset=us-ascii
>Content-Disposition: inline
>Content-Transfer-Encoding: quoted-printable
>
>* Kurt D. Zeilenga (Kurt@OpenLDAP.org) wrote:
>> Does anybody have experience with doing something like:
>>       -release 2.2 -version-info 1:0:0
>>=20
>> That is, do such constructions works on most platforms.
>> Libtool seems to construct:
>>   # Names of this library.
>>   library_names=3D'liblber-2.2.so.1 liblber-2.2.so liblber.so'
>>=20
>>   # The name of the static archive.
>>   old_library=3D'liblber.a'
>>=20
>> If this works, this might met our requirements.  Comments?
>
>Alright, this doesn't really work because it means that -llber could
>actually mean any of the versions which is generally not what you really
>want.

Of the choices, this choice is the best fit for the project.
We want -llber to continue to work.

>You should be making a distinction between the major releases in
>OpenLDAP because in general you don't expect people to be able to move
>=66rom 2.0 to 2.2 w/o changing something (API differences).

I expect people to be able to make such moves.  An application designed
for 2.0 should compile just fine with 2.2.

>Use a different library name for the major versions (2.0, 2.1, 2.2, 3.0,
>etc)...

No.

>Your idea above probably wouldn't actually be a problem for Debian since
>we don't allow people to try and have multiple -dev packages installed
>at the same time (because of the .so symlink) anyway, but it wouldn't
>really be much fun for us to deal with in the packaging.  It's also not
>something we'd recommend.

Funny... I actually got the idea from the Debian Library Packaging Guide. :-)

>What I think we probably *would* do is introduce symbol versioning which
>then followed the SONAME, so something like- LDAP_2_2_SO_0,=20
>LDAP_2_2_SO_1.

I don't think symbol versioning is supported broadly enough for us
to use it (in official releases).  But if you want to, have fun.

Kurt 

* Kurt D. Zeilenga (Kurt@OpenLDAP.org) wrote:
> I expect people to be able to make such moves.  An application designed
> for 2.0 should compile just fine with 2.2.

Is this true between 1.x, 2.x and 3.x?

> Funny... I actually got the idea from the Debian Library Packaging Guide. :-)

Yeah, the much-changed-never-right-not-official 'Debian Library
Packaging Guide'.  I've spoken with the author a number of times about
the problems with his 'Guide', it's gotten better but it's still very
far from being right.  I've been considering starting up essentially
another one which corrects and clarifies some things in it.

> >What I think we probably *would* do is introduce symbol versioning which
> >then followed the SONAME, so something like- LDAP_2_2_SO_0,=20
> >LDAP_2_2_SO_1.
> 
> I don't think symbol versioning is supported broadly enough for us
> to use it (in official releases).  But if you want to, have fun.

I wish you'd consider using it where it's available.  On those platforms
you could have a single .so which supported everything for a given
release (2.1, 2.2).  On platforms which didn't support it they'd have to
do whatever it is they've been doing, which probably involves
recompiling when a new ABI comes out, on whichever tree is being used.

I'm considering looking into doing this for the Debian packages which
would make things alot simpler I think.  I expect the ABI changes to be
small and distant enough that we might be able to pull it off reasonably
and it'd probably be nicer than having different source packages and
forcing recompiles of lots of things.

	Stephen

At 08:17 AM 3/24/2004, Stephen Frost wrote:
>* Kurt D. Zeilenga (Kurt@OpenLDAP.org) wrote:
>> I expect people to be able to make such moves.  An application designed
>> for 2.0 should compile just fine with 2.2.
>
>Is this true between 1.x, 2.x and 3.x?

I just got e-mail from someone today who able to compile a 1.2
application using 2.2 libraries.  The application appears to
behave just as it would if built with 1.2.

For 3.x, well, that depends on whether it includes -lldap -llber
or not.  If it does include a -lldap & -llber, then I generally
expect someone with a 1.x or 2.x application to be able to compile
against it.

>> Funny... I actually got the idea from the Debian Library Packaging Guide. :-)
>
>Yeah, the much-changed-never-right-not-official 'Debian Library
>Packaging Guide'.  I've spoken with the author a number of times about
>the problems with his 'Guide', it's gotten better but it's still very
>far from being right.  I've been considering starting up essentially
>another one which corrects and clarifies some things in it.

Okay... I wouldn't mind seeing some of this get into libtool
documents (especially generic bits like: -release
and -version-info are not necessarily mutually exclusive
concepts).


Kurt 

changed notes
changed state Suspended to Release
moved from Incoming to Build

changed state Release to Closed

moved from Build to Archive.Build

fix in re22