Re: [ldap-nis] getgrent is slow

["Karl O. Pinc" <kpinc@artic.edu>]

The problem is in openldap.   Calls to ldap_result() with "all" = 0 (return
1 result) have the server transfer the entire query result to the client
for each call.

Although caching in nss_ldap might help, that kind of overhead is clearly a
no-no.

Anybody know if there's any chance of this being fixed in openldap?  (I'm
using 1.2.9 now.)

Regards,

Karl <kop@meme.com>



>>On 16 Mar, Karl O. Pinc wrote:
>>>>On 15 Mar, Karl O. Pinc wrote:
>>>>> Very slow.  To the point that looping through the group file is unusable.
>>>>> Does anybody have any solutions?
>>>>>
>>>>> I'm using openldap, and I think it's compiled to be ldap version 2
>>>>> compliant.  Does ldap 3 solve this?
>>>>
>>>>Have you added any indexes? How many groups do you have?
>>>
>>> About 6000.  And yup, I've indexes.  The indexes speeded up calls like
>>> getgrnam, but getgrent is for looping through the whole group table.  It
>>> takes about 7 seconds per getgrent/row.
>>
>>Actually, an index on objectclass should speed-up the getgrent calls
>>as well. The query ends up being:
>>
>>filter="(objectclass=POSIXGROUP)"
>
>I've had an index on objectclass.  (FWIW, when I put indexes in, I had:
>
>  index {dn,uid,memberUid,gidNumber,objectClass,uidNumber} eq
>
>ldif2ldbm didn't seem to make the uidNumber index.
>
>  index {dn,uid,memberUid,gidNumber,uidNumber,objectClass} eq
>
>worked.
>
>I think this was with version 105.  For some reason I've the feeling that
>it made objectClass indexes even before I asked it to... But maybe not.
>)
>
>
>>Is it possible to get a copy of your openldap logs during a query? Is
>>the ldap server local or remote?
>
>I did a bit of logging, it appears that each getgrent call returns _all_
>the posixGroup entries.  (In absence of limits on the number of rows
>returned by slapd.  I tried to set sizelimit to 0 to turn it off, that just
>exceeded the limit on all queries.  -1 turns off the limit, but I suspect
>that that's due to int/signed int wraparound, which is a bad thing to rely
>on.  I settled on setting it to a million.)  After about 10 minutes or so
>the login fails, I think the slapd server per-query timelimit is reached.
>
>I've been working with 2 boxes, a client and a server, as that's our
>production environment.  But sometimes I just do queries from slapd box.
>Our secondary goal is to convert the slapd server box to also act as a
>client, so that's in the mix too.
>
>If you still want log copies, let me know.  I'll do a run with loglevel set
>to whatever you want.
>
>>My current nss-ldap hack work has been designing a new async subsystem
>>with caching which should help this type of situation. Of course, it's
>>been just been of hack interest and hence low priority for me but if
>>there is public interest I will bump the priority a couple notches <g>
>
>That'd be nice.  I certainally had to beat my head against the problem.
>Thanks.  I think the current software is just good enough to do what we
>need, which is good because I couldn't talk them into using non-released
>code.  I probably won't be able to talk them into letting me spend time
>testing such code either as I'm close to the deadline and have to get stuff
>working, but we'd be sure to use it eventually.  So would everybody else.
>
>FYI: There's comments in the nss_ldap getgrent related code that say that
>it should really be using the asyncronous ldap calls...\

Recent versions of nss_ldap use asyncronous calls.

Karl

May the Legos (TM) always be swept from your path in the night.