Issue 4387 - slapd-ldap backend leaks descriptors on closed connections on x86_64
Summary: slapd-ldap backend leaks descriptors on closed connections on x86_64
Status: VERIFIED FIXED
Alias: None
Product: OpenLDAP
Classification: Unclassified
Component: slapd (show other issues)
Version: unspecified
Hardware: All All
: --- normal
Target Milestone: ---
Assignee: OpenLDAP project
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-02-03 13:12 UTC by aleksander.adamowski@gmail.com
Modified: 2014-08-01 21:06 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this issue.
Description aleksander.adamowski@gmail.com 2006-02-03 13:12:38 UTC
Full_Name: Aleksander Adamowski
Version: 2.2.29, 2.3.19
OS: Fedora Core 4
URL: http://infra.altkom.pl/openldap/slapd-ldap_descriptor_leak-configs.tar.gz
Submission from: (NULL) (85.128.15.81)


In our company, we use OpenLDAP for our main mail server (they are closely
integrated).

Some time ago, to workaround BDB stability problems in OpenLDAP 2.1, we've
devised a scheme where there are 4 instances of slapd running  on 2 physical
machines, 2 instances per machine.

The BDB instance would store the actual database on disk using the slapd-bdb
backend, and would listen on non-standard ports (different than 389/636).

The LDAP instance would proxy the BDB instance, and would listen on standard
ports (389 and 636), and would forward all queries to BDB instance, and if it
doesn't work, to another BDB instance on a backup machine (using two values for
the "uri" configuration attribute, separated with a space).

The backup machine would have a similar scheme (LDAP instance proxying a BDB
instance listening on a non standard port).

Primary BDB would be replicated to the secondary BDB instance using slurpd.

All worked well, but after migrating the whole configuration from Fedora Core 1
running on x86 (an SMP Xeon system) to Fedora Core 4 running on x86_64 (an SMP
dual core Opteron system), we've found out that the LDAP instance that uses only
slapd-ldap backend starts leaking descriptors from its client connections to the
BDB instance. When we switch to using the BDB instance directly, all is OK
(slapd-bdb doesn't leak any descriptors).

After slapd-ldap process reaches a 1024 descriptor limit (a static per-process
limit compiled into kernel), it stops working and logs the following error to
syslog:

slapd[31619]: daemon: 1026 beyond descriptor table size 1024
.....
slapd[31619]: daemon: 1027 beyond descriptor table size 1024
... etc.

All global system limits on open file descriptors, as well as per-user limits
are set high above 100000 of descriptors, but the 1024 limit is compiled in.
Besides, raising it makes no sense since slapd-ldap will allocate as many
desciptors as it can.

Concerning a similar problem, I've read this thread on the mailin list:
http://www.openldap.org/lists/openldap-software/200303/msg00865.html.

According to that thread, I've set the idle connection timeout for the
slapd-ldap instance to as low as 16 seconds, but there is still something wrong,
long after switching all clients to use the BDB instance on non-standard port,
when there are no queries sent to the LDAP instance for several minutes, it
still holds descriptors for those connections and doesn't free them. It should
close them after 16 seconds of inactivity, but it doesn't. Those connections are
still established, as visible in output from "netstat -anp" (port 392 is the
listening port of the BDB instance):

tcp        0      0 127.0.0.1:43189             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43188             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43193             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43195             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43197             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43196             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43198             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43137             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43139             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43138             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43141             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43140             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43143             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43142             127.0.0.1:392              
ESTABLISHED 17220/slapd         
tcp        0      0 127.0.0.1:43147             127.0.0.1:392              
ESTABLISHED 17220/slapd         


This can also be seen by looking up the PID of the slapd process of slapd-ldap
instance, and looking into its /proc/<PID>/fd. There are 1024 entries there, and
they don't disappear over several minutes - only after slapd restart.

So I've concluded that there's something wrong with the slapd-ldap instance:

1) While operating, slapd-ldap connection caching logic (described in slapd-ldap
manpage) doesn't work properly - it opens much more connections to the proxied
slapd-bdb instance, than needed for sharing cached connections. It seems
connections aren't reused. There definitely aren't more that 30 simultaneous
queries executed, and most of the time there are less than 10.

2) After queries stop coming in to slapd-ldap instance, it doesn't timeout idle
connections to the proxied slapd-bdb instance after configured 16 seconds. They
are kept indefinitely for several minutes, until the instance is stopped
forcibly.

3) If the number of connections hits the per-process 1024 descriptor limit and
tries to exceed it, there's also a problem when the slapd-ldap instance
terminates. When it receives the TERM signal, it gets stuck waiting for some
threads to terminate:

....
Jan 30 10:37:35 nmail slapd[16859]: conn=710 fd=999 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=714 fd=1003 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=717 fd=1008 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=723 fd=1010 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=727 fd=1015 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=728 fd=1017 closed 
Jan 30 10:37:35 nmail slapd[16859]: conn=730 fd=1019 closed 
Jan 30 10:37:35 nmail slapd[16859]: slapd shutdown: waiting for 45 threads to
terminate 

It doesn't exit until forcibly killed with SIGKILL.

A package with our config files (with anonymized passwords and addresses) is
available under the URL provided with this ITS.

Comment 1 aleksander.adamowski@gmail.com 2006-02-03 14:04:22 UTC
Additional details:

When I also configure the slapd-bdb instance to timeout idle
connections, by placing "idletimeout 33" in the global section, the
connections are forcibly closed on slapd-bdb side, but slapd-ldap
instance still keeps their descriptors.

In that case, "netstat -anp" show connections in CLOSE_WAIT state,
instead of ESTABLISHED:

tcp        1      0 127.0.0.1:40768             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40769             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40778             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40777             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40827             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40806             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40807             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40804             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40805             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40802             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40803             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd
tcp        1      0 127.0.0.1:40815             127.0.0.1:392         
     CLOSE_WAIT  1374/slapd

Comment 2 ando@openldap.org 2006-02-03 16:43:11 UTC
On Fri, 2006-02-03 at 14:04 +0000, aleksander.adamowski@gmail.com wrote:
> Additional details:
> 
> When I also configure the slapd-bdb instance to timeout idle
> connections, by placing "idletimeout 33" in the global section, the
> connections are forcibly closed on slapd-bdb side, but slapd-ldap
> instance still keeps their descriptors.

Use idle-timeout to timeout idle connections from the proxy's side.  See
slapd-ldap(5) (2.3 only) for details.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 3 aleksander.adamowski@gmail.com 2006-02-03 21:55:00 UTC
On 2/3/06, Pierangelo Masarati <ando@sys-net.it> wrote:

> Use idle-timeout to timeout idle connections from the proxy's side.  See
> slapd-ldap(5) (2.3 only) for details.

You are right, I placed idle-timeout only in one of the slapd-ldap
backends running in the LDAP instance.

However, do you think that it's correct for slapd-ldap backend to do
the following:
1) Not get rid of descriptors for connections closed by the other side
(CLOSE_WAIT state)
2) Not reuse cached connections queries, but open more and more new connections?

In my opinion this behaviour is not correct.

Best Regards,
--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 4 ando@openldap.org 2006-02-04 11:07:47 UTC
On Fri, 2006-02-03 at 21:55 +0000, aleksander.adamowski@gmail.com wrote:
> On 2/3/06, Pierangelo Masarati <ando@sys-net.it> wrote:
> 
> > Use idle-timeout to timeout idle connections from the proxy's side.  See
> > slapd-ldap(5) (2.3 only) for details.
> 
> You are right, I placed idle-timeout only in one of the slapd-ldap
> backends running in the LDAP instance.
> 
> However, do you think that it's correct for slapd-ldap backend to do
> the following:
> 1) Not get rid of descriptors for connections closed by the other side
> (CLOSE_WAIT state)
> 2) Not reuse cached connections queries, but open more and more new connections?
> 
> In my opinion this behaviour is not correct.

I haven't noticed the behavior you describe; I don't understand how it
could happen (and I'm not 100% sure I understood what's actually
happening; that's why I didn't answer this point).

A new connection between proxy and remote server is established when no
appropriate cached connection exists or when a bind occurs on an
existing cached connection.  Cached connections are closed when the
client disconnect, except for the anonymous connection which remains
open and is shared by all anonymous clients.

Establishing a new connection when a bind occurs was a recent change
(ITS#4315, I believe); this might be causing resource "leaking" (not
really a leak; see below), because the existing cached connection is
ignored, and a new one is created.

This is not actually a leak because the connection remains accessible in
the connection tree, but it might not be intercepted by the
connection_destroy() hook; so it's only destroyed when the connection
tree is freed, i.e. at shutdown.  This is a guess, I haven't
investigated the issue deep enough yet.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 5 ando@openldap.org 2006-02-04 15:16:00 UTC
I've investigated this issue a little bit more.  An unusual growth in
the number of cached connections seems to occur when the client uses the
same connection to repeatedly bind with different identities.  In that
case, each time a new identity binds on the same connection, a new
connection between the proxy and the remote server is created.  This
partially makes sense, from the proxy's point of view, because after a
successful bind that client->proxy connection assumes the new identity,
so a new proxy->remote connection needs be created.  However, a side
effect of this operation sequence is that the client->proxy connection
can no longer act with the old identity, so it should be treated as if a
connection_destroy() was requested.  We could take different behaviors
in this case to prevent an excessive resource use; one that sounds
reasonable to me consists in adding a task that routinely shuts down the
idle connections (much like the client->server "idletimeout") instead of
waiting that a connection is used to check if it timed out.  Let me
point out that in my opinion reusing a bound connection to rebind with a
different identity sounds like a poor client design.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 6 ando@openldap.org 2006-02-04 15:39:05 UTC
A better approach would be to destroy all cached connections with a
given connid regardless of the DN they're saved with.  This would be
consistent with the fact that after a connid is closed, that cached
connection couldn't be used any longer.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 7 aleksander.adamowski@gmail.com 2006-02-05 17:46:42 UTC
On 2/4/06, Pierangelo Masarati <ando@sys-net.it> wrote:
> On Fri, 2006-02-03 at 21:55 +0000, aleksander.adamowski@gmail.com wrote:
> > However, do you think that it's correct for slapd-ldap backend to do
> > the following:
> > 1) Not get rid of descriptors for connections closed by the other side
> > (CLOSE_WAIT state)
> > 2) Not reuse cached connections queries, but open more and more new connections?
> >
> > In my opinion this behaviour is not correct.
>
> I haven't noticed the behavior you describe; I don't understand how it
> could happen (and I'm not 100% sure I understood what's actually
> happening; that's why I didn't answer this point).
>
> A new connection between proxy and remote server is established when no
> appropriate cached connection exists or when a bind occurs on an
> existing cached connection.

This would explain lack of connection reuse to some extent....

Almost all connections come from the Courier MTA - its authldap and
ldapalias daemons.

They bind as the user cn=Courier,o=...etc..., so practically 98% of
connections are associated with a bind operation.

>  Cached connections are closed when the
> client disconnect, except for the anonymous connection which remains
> open and is shared by all anonymous clients.

It seems that in this case cached connections weren't being closed
after client disconnection.
That's because I'm almost certain that the clients didn't open as much
simultaneous connections to slapd-ldap backend, as the slapd-ldap
backend was opening to the slapd-bdb backend.
First of all, the sum of limits on process count for ldapalias daemon
and authdaemon was much lower (max. 24 and 28 processes,
respectively). Those daemons don't open more than a single connection
per process AFAIK. So the whole Courier server shouldn't open more
than 52 connections to LDAP.

There are some application on our network that connect to LDAP
independently, but I'd estimate that they together open at most 20
simultaneous connections (during peak hours).

So there definately weren't more that 100 client simultaneous
connections to the server, while slapd-ldap has managed to open over
1024 connections to the backend.

Seems like there were lots of unnecessary connections that slapd-ldap
had kept without a cause.

> Establishing a new connection when a bind occurs was a recent change
> (ITS#4315, I believe); this might be causing resource "leaking" (not
> really a leak; see below), because the existing cached connection is
> ignored, and a new one is created.
>
> This is not actually a leak because the connection remains accessible in
> the connection tree, but it might not be intercepted by the
> connection_destroy() hook; so it's only destroyed when the connection
> tree is freed, i.e. at shutdown.  This is a guess, I haven't
> investigated the issue deep enough yet.

This would go along with my observations - connections from slapd-ldap
to slapd-bdb were not destroyed; most of connections to slapd-ldap
(and the resulting connections from slapd-ldap to slapd-bdb) weren't
anonymous.

Best Regards,
--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 8 aleksander.adamowski@gmail.com 2006-02-05 18:00:17 UTC
> I've investigated this issue a little bit more.  An unusual growth in
> the number of cached connections seems to occur when the client uses the
> same connection to repeatedly bind with different identities.

That's right, the Courier authdaemon uses a single connectioncheck the
password validity of users - it does this by performing bind
operations on this connection.

>Let me
> point out that in my opinion reusing a bound connection to rebind with a
>different identity sounds like a poor client design.

In my opinion it's a very good design in the given case - the role ot
Courier's authdaemon (among other roles, e.g. account lookup) is to
check validity of user credentials.

It does multiple credential checks on a single LDAP connection to
conserve resources - to test a password, it doesn't need to open a new
LDAP connection every time and then close it .

This is a good design from the performance perspective, and I'd
speculate that LDAP protocol permits multiple bind operations per
connection because its designers have foreseen this usage scenario
(LDAP was designed as a network authentication protocol, among other
uses).

Best Regards,
--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 9 Howard Chu 2006-02-05 18:25:18 UTC
aleksander.adamowski@gmail.com wrote:
> On 2/4/06, Pierangelo Masarati <ando@sys-net.it> wrote:
>   
>> On Fri, 2006-02-03 at 21:55 +0000, aleksander.adamowski@gmail.com wrote:
>>     
>>> However, do you think that it's correct for slapd-ldap backend to do
>>> the following:
>>> 1) Not get rid of descriptors for connections closed by the other side
>>> (CLOSE_WAIT state)
>>> 2) Not reuse cached connections queries, but open more and more new connections?
>>>
>>> In my opinion this behaviour is not correct.
>>>       
>> I haven't noticed the behavior you describe; I don't understand how it
>> could happen (and I'm not 100% sure I understood what's actually
>> happening; that's why I didn't answer this point).
>>
>> A new connection between proxy and remote server is established when no
>> appropriate cached connection exists or when a bind occurs on an
>> existing cached connection.
>>     
>
> This would explain lack of connection reuse to some extent....
>
> Almost all connections come from the Courier MTA - its authldap and
> ldapalias daemons.
>
> They bind as the user cn=Courier,o=...etc..., so practically 98% of
> connections are associated with a bind operation.

Explicit binds to the back-ldap database always use a new connection. 
But if you Bind to a user in some other local database and then use that 
identity when searching back-ldap, then a connection for that identity 
will be cached. This behavior is already described in the 2nd paragraph 
of the slapd-ldap(5) manpage. You should rework your configuration if 
you want to take full advantage of the connection caching.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Comment 10 Howard Chu 2006-02-05 18:46:32 UTC
aleksander.adamowski@gmail.com wrote:
>> Let me
>> point out that in my opinion reusing a bound connection to rebind with a
>> different identity sounds like a poor client design.
>>     
>
> In my opinion it's a very good design in the given case - the role ot
> Courier's authdaemon (among other roles, e.g. account lookup) is to
> check validity of user credentials.
>
> It does multiple credential checks on a single LDAP connection to
> conserve resources - to test a password, it doesn't need to open a new
> LDAP connection every time and then close it .
>
> This is a good design from the performance perspective, and I'd
> speculate that LDAP protocol permits multiple bind operations per
> connection because its designers have foreseen this usage scenario
> (LDAP was designed as a network authentication protocol, among other
> uses).

Actually not, but it has been used as such simply because it accomodates 
such a rich set of authentication mechanisms.

There's a simple tradeoff here - if you use a single connection for all 
Binds, you must fully serialize the procedure, because the receipt of 
any Bind request automatically aborts any other outstanding requests on 
the connection. If you use multiple connections, you can have multiple 
authentication attempts in progress at once. For back-ldap, we felt it 
was more important to support high concurrency.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Comment 11 aleksander.adamowski@gmail.com 2006-02-05 18:50:15 UTC
On 2/5/06, Howard Chu <hyc@symas.com> wrote:
> Explicit binds to the back-ldap database always use a new connection.
> But if you Bind to a user in some other local database and then use that
> identity when searching back-ldap, then a connection for that identity
> will be cached. This behavior is already described in the 2nd paragraph
> of the slapd-ldap(5) manpage. You should rework your configuration if
> you want to take full advantage of the connection caching.

Understood. Still, it doesn't justify slapd-ldap leaking unused
connections to the target server.

And I think that theoretically, connection caching would be possible
even with explicit binds to back-ldap.
It would require some sort of an associative array (a mapping similar
to: (bound identity) --> (connection)), with associations being
changed upon each bind operation for each connection.
A bind operation on proxy's side would check whether a connection to
the target server exists for given bound identity and operation's base
DN, and it such a connection exists, it would use it, otherwise it
would open a new one.
Idle connections would be closed after some time (BTW, the
slapd-ldap's idle-timeout configuration parameter that's responsible
for this could use some reasonable default value instead of the
current "no limit", which makes little sense and is dangerous as I've
discovered myself).

I understand that implementing such connection caching logic would
require a lot of work and analysing lots of edge cases - I just think
this is a good idea for future.

Best Regards,
--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 12 aleksander.adamowski@gmail.com 2006-02-05 18:55:34 UTC
On 2/5/06, Howard Chu <hyc@symas.com> wrote:
> aleksander.adamowski@gmail.com wrote:
> > This is a good design from the performance perspective, and I'd
> > speculate that LDAP protocol permits multiple bind operations per
> > connection because its designers have foreseen this usage scenario
> > (LDAP was designed as a network authentication protocol, among other
> > uses).
>
> Actually not, but it has been used as such simply because it accomodates
> such a rich set of authentication mechanisms.
>
> There's a simple tradeoff here - if you use a single connection for all
> Binds, you must fully serialize the procedure, because the receipt of
> any Bind request automatically aborts any other outstanding requests on
> the connection. If you use multiple connections, you can have multiple
> authentication attempts in progress at once.

This doesn't rule out using each of those multiple connections for
multiple binds, which will give the most throughput and ocncurrency
possible.

And this is exactly what the Courier authdaemon does - it opens
multiple LDAP connections (28 in my case), and binds several times on
each of them.
This enables hich concurrency with minimum overhead stemming from
opening/closing LDAP connections.

--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 13 Howard Chu 2006-02-05 18:56:51 UTC
Aleksander Adamowski wrote:
> On 2/5/06, Howard Chu <hyc@symas.com> wrote:
>   
>> Explicit binds to the back-ldap database always use a new connection.
>> But if you Bind to a user in some other local database and then use that
>> identity when searching back-ldap, then a connection for that identity
>> will be cached. This behavior is already described in the 2nd paragraph
>> of the slapd-ldap(5) manpage. You should rework your configuration if
>> you want to take full advantage of the connection caching.
>>     
>
> Understood. Still, it doesn't justify slapd-ldap leaking unused
> connections to the target server.
>   

Right, the leak needs to be fixed.

> And I think that theoretically, connection caching would be possible
> even with explicit binds to back-ldap.
> It would require some sort of an associative array (a mapping similar
> to: (bound identity) --> (connection)), with associations being
> changed upon each bind operation for each connection.
>   

Associations are already changed upon completion of a Bind. But 
obviously you cannot be sharing a single connection across multiple 
clients when a Bind occurs, because any outstanding operations will be 
(and must be) invalidated.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Comment 14 aleksander.adamowski@gmail.com 2006-02-05 19:13:49 UTC
On 2/5/06, Howard Chu <hyc@symas.com> wrote:

> Associations are already changed upon completion of a Bind. But
> obviously you cannot be sharing a single connection across multiple
> clients when a Bind occurs, because any outstanding operations will be
> (and must be) invalidated.

Yes that's obvious. But a connection without any outstanding
operations can be considered for sharing - another client may perform
a simple quick operation and get the results, before the original
client decides to start another operation.

If the original client decides to start another operation before the
second client's operation has completed, a new connection needs to be
transparently created to the target server and used for that.


--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 15 ando@openldap.org 2006-02-05 21:04:35 UTC
On Sun, 2006-02-05 at 18:57 +0000, hyc@symas.com wrote:
> Associations are already changed upon completion of a Bind. But 
> obviously you cannot be sharing a single connection across multiple 
> clients when a Bind occurs, because any outstanding operations will be 
> (and must be) invalidated.

I think a reasonable design would be to allow caching all of those
connections and, on connection destroy, destroy all cached connections
with that connid, i.e. don't limit connection destroy to the pair
connid/authzdn.  This way, if for the same connection the same DN is
used more than once, the cached connection can be exploited.  Mechanisms
like "idle-timeout" and "conn-ttl" (not released yet) can be used to
limit the amount of idle connections out there.  Unfortunately, they
both act when a to be renewed connection is requested, not when it
expires; the latter could be implemented by adding a periodic task that
cycles over the connection tree and takes care of expired connections in
cache.  I'd defer the latter, and provide the former ASAP.  I think a
reasonable approach would be to have ldap_back_conn_cmp() compare on
lc_conn first, then on lc_local_dn (i.e. reverse the current behavior).
ldap_back_conn_destroy() would use a different implementation that just
compares lc_conn, and avl_delete() should be called with it repeatedly,
until no hit occurs.  Reverting the testing sequence above would allow
ordered lookups to be correct in both cases, since cache entry storing
would use the connid/DN check.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 16 ando@openldap.org 2006-02-05 22:47:56 UTC
On Sun, 2006-02-05 at 18:00 +0000, aleksander.adamowski@gmail.com wrote:
> >Let me
> > point out that in my opinion reusing a bound connection to rebind with a
> >different identity sounds like a poor client design.
> 
> In my opinion it's a very good design in the given case - the role ot
> Courier's authdaemon (among other roles, e.g. account lookup) is to
> check validity of user credentials.
> 
> It does multiple credential checks on a single LDAP connection to
> conserve resources - to test a password, it doesn't need to open a new
> LDAP connection every time and then close it .
> 
> This is a good design from the performance perspective, and I'd
> speculate that LDAP protocol permits multiple bind operations per
> connection because its designers have foreseen this usage scenario
> (LDAP was designed as a network authentication protocol, among other
> uses).

I mean: if all it performs is binds then I'd agree; but if it performs
anything else in between, then it's a poor design, because the identity
that's used for the other operations would change after each
(successful) bind.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 17 aleksander.adamowski@gmail.com 2006-02-05 22:52:03 UTC
On 2/5/06, Pierangelo Masarati <ando@sys-net.it> wrote:
> I mean: if all it performs is binds then I'd agree; but if it performs
> anything else in between, then it's a poor design, because the identity
> that's used for the other operations would change after each
> (successful) bind.

From the logs I can see, that it binds as the special mail server user
when it needs to look up account information, and binds as other users
when it needs to test their password.

So it typically looks like this:

connection1:
bind as user bob
bind as user alice
bind as user CourierMTA
search for mail=gregory@example.com attributes homeDirectory, cn,
uidNumber, gidNumber etc.
bind as user bob
...

connection2:
bind as user CourierMTA
search for mail=bob@example.com attributes homeDirectory, cn,
uidNumber, gidNumber etc.
bind as user alice
bind as user alice
...

Seems like it behaves correctly.

--
    Aleksander Adamowski
        Jabber JID: olo@jabber.altkom.pl
        GG#: 274614
        ICQ UIN: 19780575
        http://olo.ab.altkom.pl

Comment 18 ando@openldap.org 2006-02-06 07:22:17 UTC
On Sun, 2006-02-05 at 22:52 +0000, aleksander.adamowski@gmail.com wrote:
> On 2/5/06, Pierangelo Masarati <ando@sys-net.it> wrote:
> > I mean: if all it performs is binds then I'd agree; but if it performs
> > anything else in between, then it's a poor design, because the identity
> > that's used for the other operations would change after each
> > (successful) bind.
> 
> >From the logs I can see, that it binds as the special mail server user
> when it needs to look up account information, and binds as other users
> when it needs to test their password.
> 
> So it typically looks like this:
> 
> connection1:
> bind as user bob
> bind as user alice
> bind as user CourierMTA
> search for mail=gregory@example.com attributes homeDirectory, cn,
> uidNumber, gidNumber etc.
> bind as user bob
> ...
> 
> connection2:
> bind as user CourierMTA
> search for mail=bob@example.com attributes homeDirectory, cn,
> uidNumber, gidNumber etc.
> bind as user alice
> bind as user alice
> ...
> 
> Seems like it behaves correctly.

You see, that's (almost) exactly what I mean: a (set of) separate
connection, bound as the CourierMTA, could be maintained for lookups,
saving a lot of useless binds.  The other connections could be used just
for repeated binds as different identities, so your problem would still
be there.  I'll work at it as soon as I can spare some time; I think
ITS#4390 is related.  In HEAD, slapd-bind has already been modified to
be reproduce the multiple bind behavior, this will help in tracking the
issues.

p.




Ing. Pierangelo Masarati
Responsabile Open Solution
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309          
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------

Comment 19 ando@openldap.org 2006-02-06 20:40:34 UTC
changed notes
changed state Open to Feedback
moved from Incoming to Software Bugs
Comment 20 ando@openldap.org 2006-02-17 18:24:12 UTC
changed state Feedback to Test
Comment 21 ando@openldap.org 2006-02-17 21:36:53 UTC
changed notes
changed state Test to Release
Comment 22 Kurt Zeilenga 2006-02-18 18:42:10 UTC
changed state Release to Closed
Comment 23 Howard Chu 2009-02-17 05:17:20 UTC
moved from Software Bugs to Archive.Software Bugs
Comment 24 OpenLDAP project 2014-08-01 21:06:42 UTC
partial fix in HEAD/re23