[Date Prev][Date Next] [Chronological] [Thread] [Top]

Long-lived connections with 2.3.x stop returning attributes?



I have a cluster of several LDAP servers currently running 2.1.30 behind a
hardware load balancer. I'm working on upgrading them to 2.3.25, but I've
encountered a problem with long-lived connections. Currently, I only have
one machine upgraded to 2.3.25.

We have a Perl-based daemon (using Net::LDAP) which can keep a single LDAP
connection open for several days, or even weeks. It performs synchronous
searches on this connection at the rate of one or two per second. All of its
searches match a single entry and it requests two attributes.

After this daemon has been connected to the 2.3.x machine in the cluster for
about a day, it begins receiving no values in response to its searches. The
search succeeds and a result (including the found entry's DN) is returned,
but there are no attributes present.

Here is strace(1) output for some representative searches that *work*:

write(4, "0_\2\2\0\211cY\4\no=frontier\n\1\2\n\1\2\2\1\2\2\1\5"..., 97) = 97
 | 00000  30 5f 02 02 00 89 63 59  04 0a 6f 3d 66 72 6f 6e  0_....cY ..o=fron |
 | 00010  74 69 65 72 0a 01 02 0a  01 02 02 01 02 02 01 05  tier.... ........ |
 | 00020  01 01 00 a3 25 04 04 6d  61 69 6c 04 1d 72 6f 77  ....%..m ail..row |
 | 00030  6c 61 6e 64 66 61 6d 69  6c 79 40 66 72 6f 6e 74  landfami ly@front |
 | 00040  69 65 72 6e 65 74 2e 6e  65 74 30 15 04 0b 73 65  iernet.n et0...se |
 | 00050  72 76 69 63 65 54 79 70  65 04 06 70 61 72 65 6e  rviceTyp e..paren |
 | 00060  74                                                t                 |
read(4, "0z", 2)                        = 2
 | 00000  30 7a                                             0z                |
read(4, "\2\2\0\211dt\4Muid=rowlandfamily,dc=fro"..., 122) = 122
 | 00000  02 02 00 89 64 74 04 4d  75 69 64 3d 72 6f 77 6c  ....dt.M uid=rowl |
 | 00010  61 6e 64 66 61 6d 69 6c  79 2c 64 63 3d 66 72 6f  andfamil y,dc=fro |
 | 00020  6e 74 69 65 72 6e 65 74  2c 64 63 3d 6e 65 74 2c  ntiernet ,dc=net, |
 | 00030  6f 75 3d 66 72 6f 6e 74  69 65 72 6e 65 74 2c 6f  ou=front iernet,o |
 | 00040  75 3d 73 65 72 76 69 63  65 73 2c 6f 3d 66 72 6f  u=servic es,o=fro |
 | 00050  6e 74 69 65 72 30 23 30  21 04 0b 73 65 72 76 69  ntier0#0 !..servi |
 | 00060  63 65 54 79 70 65 31 12  04 10 46 72 6f 6e 74 69  ceType1. ..Fronti |
 | 00070  65 72 20 44 53 4c 20 4d  41 58                    er DSL M AX       |
read(4, "0\r", 2)                       = 2
 | 00000  30 0d                                             0.                |
read(4, "\2\2\0\211e\7\n\1\0\4\0\4\0", 13) = 13
 | 00000  02 02 00 89 65 07 0a 01  00 04 00 04 00           ....e... .....    |

write(4, "0Z\2\2\0\212cT\4\no=frontier\n\1\2\n\1\2\2\1\2\2\1\5"..., 92) = 92
 | 00000  30 5a 02 02 00 8a 63 54  04 0a 6f 3d 66 72 6f 6e  0Z....cT ..o=fron |
 | 00010  74 69 65 72 0a 01 02 0a  01 02 02 01 02 02 01 05  tier.... ........ |
 | 00020  01 01 00 a3 20 04 04 6d  61 69 6c 04 18 6d 65 72  .... ..m ail..mer |
 | 00030  74 6b 61 74 79 40 66 72  6f 6e 74 69 65 72 6e 65  tkaty@fr ontierne |
 | 00040  74 2e 6e 65 74 30 15 04  0b 73 65 72 76 69 63 65  t.net0.. .service |
 | 00050  54 79 70 65 04 06 70 61  72 65 6e 74              Type..pa rent     |
read(4, "0}", 2)                        = 2
 | 00000  30 7d                                             0}                |
read(4, "\2\2\0\212dw\4Huid=mertkaty,dc=frontier"..., 125) = 125
 | 00000  02 02 00 8a 64 77 04 48  75 69 64 3d 6d 65 72 74  ....dw.H uid=mert |
 | 00010  6b 61 74 79 2c 64 63 3d  66 72 6f 6e 74 69 65 72  katy,dc= frontier |
 | 00020  6e 65 74 2c 64 63 3d 6e  65 74 2c 6f 75 3d 66 72  net,dc=n et,ou=fr |
 | 00030  6f 6e 74 69 65 72 6e 65  74 2c 6f 75 3d 73 65 72  ontierne t,ou=ser |
 | 00040  76 69 63 65 73 2c 6f 3d  66 72 6f 6e 74 69 65 72  vices,o= frontier |
 | 00050  30 2b 30 29 04 0b 73 65  72 76 69 63 65 54 79 70  0+0)..se rviceTyp |
 | 00060  65 31 1a 04 18 44 69 61  6c 2d 55 70 20 41 63 63  e1...Dia l-Up Acc |
 | 00070  65 73 73 20 55 6e 6c 69  6d 69 74 65 64           ess Unli mited    |
read(4, "0\r", 2)                       = 2
 | 00000  30 0d                                             0.                |
read(4, "\2\2\0\212e\7\n\1\0\4\0\4\0", 13) = 13
 | 00000  02 02 00 8a 65 07 0a 01  00 04 00 04 00           ....e... .....    |

Here is strace(1) output for some representative searches that return *no*
attributes:

write(7, "0Z\2\3\0\232<cS\4\no=frontier\n\1\2\n\1\2\2\1\2\2\1"..., 92) = 92
 | 00000  30 5a 02 03 00 9a 3c 63  53 04 0a 6f 3d 66 72 6f  0Z....<c S..o=fro |
 | 00010  6e 74 69 65 72 0a 01 02  0a 01 02 02 01 02 02 01  ntier... ........ |
 | 00020  05 01 01 00 a3 1f 04 04  6d 61 69 6c 04 17 6d 61  ........ mail..ma |
 | 00030  72 67 69 65 34 40 66 72  6f 6e 74 69 65 72 6e 65  rgie4@fr ontierne |
 | 00040  74 2e 6e 65 74 30 15 04  0b 73 65 72 76 69 63 65  t.net0.. .service |
 | 00050  54 79 70 65 04 06 70 61  72 65 6e 74              Type..pa rent     |
read(7, "0R", 2)                        = 2
 | 00000  30 52                                             0R                |
read(7, "\2\3\0\232<dK\4Guid=margie4,dc=frontier"..., 82) = 82
 | 00000  02 03 00 9a 3c 64 4b 04  47 75 69 64 3d 6d 61 72  ....<dK. Guid=mar |
 | 00010  67 69 65 34 2c 64 63 3d  66 72 6f 6e 74 69 65 72  gie4,dc= frontier |
 | 00020  6e 65 74 2c 64 63 3d 6e  65 74 2c 6f 75 3d 66 72  net,dc=n et,ou=fr |
 | 00030  6f 6e 74 69 65 72 6e 65  74 2c 6f 75 3d 73 65 72  ontierne t,ou=ser |
 | 00040  76 69 63 65 73 2c 6f 3d  66 72 6f 6e 74 69 65 72  vices,o= frontier |
 | 00050  30 00                                             0.                |
read(7, "0\16", 2)                      = 2
 | 00000  30 0e                                             0.                |
read(7, "\2\3\0\232<e\7\n\1\0\4\0\4\0", 14) = 14
 | 00000  02 03 00 9a 3c 65 07 0a  01 00 04 00 04 00        ....<e.. ......   |

write(7, "0X\2\3\0\232=cQ\4\no=frontier\n\1\2\n\1\2\2\1\2\2\1"..., 90) = 90
 | 00000  30 58 02 03 00 9a 3d 63  51 04 0a 6f 3d 66 72 6f  0X....=c Q..o=fro |
 | 00010  6e 74 69 65 72 0a 01 02  0a 01 02 02 01 02 02 01  ntier... ........ |
 | 00020  05 01 01 00 a3 1d 04 04  6d 61 69 6c 04 15 6b 62  ........ mail..kb |
 | 00030  63 67 6f 40 66 72 6f 6e  74 69 65 72 6e 65 74 2e  cgo@fron tiernet. |
 | 00040  6e 65 74 30 15 04 0b 73  65 72 76 69 63 65 54 79  net0...s erviceTy |
 | 00050  70 65 04 06 70 61 72 65  6e 74                    pe..pare nt       |
read(7, "0P", 2)                        = 2
 | 00000  30 50                                             0P                |
read(7, "\2\3\0\232=dI\4Euid=kbcgo,dc=frontierne"..., 80) = 80
 | 00000  02 03 00 9a 3d 64 49 04  45 75 69 64 3d 6b 62 63  ....=dI. Euid=kbc |
 | 00010  67 6f 2c 64 63 3d 66 72  6f 6e 74 69 65 72 6e 65  go,dc=fr ontierne |
 | 00020  74 2c 64 63 3d 6e 65 74  2c 6f 75 3d 66 72 6f 6e  t,dc=net ,ou=fron |
 | 00030  74 69 65 72 6e 65 74 2c  6f 75 3d 73 65 72 76 69  tiernet, ou=servi |
 | 00040  63 65 73 2c 6f 3d 66 72  6f 6e 74 69 65 72 30 00  ces,o=fr ontier0. |
read(7, "0\16", 2)                      = 2
 | 00000  30 0e                                             0.                |
read(7, "\2\3\0\232=e\7\n\1\0\4\0\4\0", 14) = 14
 | 00000  02 03 00 9a 3d 65 07 0a  01 00 04 00 04 00        ....=e.. ......   |

Even though this connection has reached the point where the daemon is
receiving no values, identical/similar searches against the same server with
ldapsearch(1) (same filter, returned entry, and requested attributes) work
fine. Other types of daemons running at the same time are also unaffected.

It seems that the long-running connection is the key here, or perhaps the
number of operations during the connection's lifetime. This is the only
daemon we have that routinely holds a connection open for longer than a day.
At about the day or day and a half point, all searches on this connection
find the correct entry, but do not contain any values in the response. We've
never seen this behavior with 2.1.x, which we've been running for a few
years.

Also, I was originally running 2.3.24 and upgraded to 2.3.25 to make sure
this hadn't been fixed in that release, but it didn't make any difference.

Any thoughts?

thanks,
john
-- 
John Morrissey          _o            /\         ----  __o
jwm@horde.net        _-< \_          /  \       ----  <  \,
www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__