moved from Incoming to Software Enhancements

moved from Software Enhancements to Contrib

changed notes
changed state Open to Feedback

changed notes

changed state Feedback to Suspended

Hi,

Based on your feedback on the contribution: "LDAP proxy cache extension for
OpenLDAP" released as a patch to OpenLDAP2.1.4, I have submitted a new
patch incorporating the suggestions. The patch is for OpenLDAP2.1.9.  It
can be accessed at:

ftp://ftp.openldap.org/incoming/apurva-kumar-pkg-021210.tgz

The doc ldapcache.html in the tarball contains design, implementation,
usage info.

This release has the following features:

1) Semantic caching of positive conjunctive LDAP queries
   - Answering of repeat and contained queries.
   - Support for answering queries with equality,
     GE, LE and substring assertions.
   - Answering of queries corresponding to specified
     query templates eg. (cn=), &(cn=)(c=).
2) Attribute level caching.
   - Only required attributes of an entry are cached to
     improve cache utilization.
3) Consistency support
   - TTL based weak consistency support provided.
4) Support for multiple backend types
5) Caching operations implemented using backend APIs.
6) Support for caching multiple directories
7) Can function as a meta-directory cache.
8) Support for multiple database instances for a single
   cache directory tree.

With respect to the initial release, the following major changes have been
made:
1) Implementing the proxy cache using the back-meta rather than back-ldap.
This enables the cache for meta directory caching.
2) TTL based weak consistency support added.
3) Using callback mechanism to make the solution backend independent
(requires a control to be added).
4) Support for access control.

The caching operations are implemented using add, modify, search, delete
backend APIs and the callback mechanism. However as discussed in the
forwarded mail, certain checks in the add, modify, delete and search
functions need to be relaxed.  This is done by adding a control in the
Operation struct which is used in the backends supporting LDAP caching.
Thus minor modifications are required in the backends supporting caching.
The changes required are similar for all backends. A patch for LDBM backend
for these changes is included in the release.

Weak consistency is provided, by associating a TTL value with a query type
(template). After the TTL is over, the query (and data) is removed from the
cache.

Only read permissions need to be specified in the ACL since write
operations pass through the cache.

The glue backend is used to glue together multiple database instances
serving the cache directory .

Further details are provided in ldapcache.html included in the tarball.
Will greatly appreciate comments/feedback on the contribution.

Special thanks to Howard, Kurt and Pierangelo for their suggestions which I
have tried to incorporate.

Thanks,
Apurva Kumar,
Research Staff Member,
IBM India Research Lab
Phone: +91-11-6861100
Fax: +91-11-6861555

-------------------------------------

Hi,

Thanks for your suggestions on the proxy cache code.

The suggestion to use callback facility to support all the backends without
modifying their codes can save a lot of work. However I am trying to figure
out how to do the following operations required in the cache with this
facility.

1) adding an entry without a parent.
2) deleting an entry with children (without deleting the children).
3) making a search with the search base not in the cache.

These operations are encountered while doing the following:
1) adding to the cache, an entry returned from the backend server, which
does not have its parent in the cache.
2) removing an entry whose corresponding queries have been removed by cache
replacement.
3) while searching the local cache for an answerable query with base entry
not in the cache.

For LDBM backend I could achieve the above by implementing  three
additional interfaces for adding/merging, searching and removing. I am not
sure if all of these can be taken care of by the existing interfaces for
search/add/delete/modify.

1) can probably be achieved by adding as root. For 3) the only solution I
could think of was to use the backend's suffix as the search base for all
the cache searches and filter out the entries not in the subtree using the
callback function for send_search_entry. This is not  very efficient.

Would greatly appreciate any help in solving this problem.

Thanks,
Apurva Kumar,
Research Staff Member,
IBM India Research Lab
Phone: +91-11-6861100
Fax: +91-11-6861555






                      "Howard Chu"

                      <hyc@symas.com>               To:       Apurva
Kumar/India/IBM@IBMIN, <openldap-devel@OpenLDAP.org>
                      Sent by:                      cc:

                      owner-openldap-devel@O        Subject:  RE: Proxy
cache extension for OpenLDAP
                      penLDAP.org



                      09/06/02 03:38 PM






> -----Original Message-----
> From: Apurva Kumar [mailto:kapurva@in.ibm.com]
>
> LDAP proxy cache docs in HTML.

Thanks. It's a fascinating idea. The effect of ACLs on cached results isn't
considered though; I guess you assume that all clients of the proxy will
have
equal privileges on the remote server. (That's a fair enough assumption for
many scenarios, it just needs to be stated.)

You can implement your cache_backend APIs without directly modifying
back-ldbm. Look at the callback facility, see backglue.c for an idea of how
to use it. If you use the callback approach that backglue uses, your cache
can interface to any of the existing backends without needing to modify
their
code in any way. sasl.c and saslauthz.c also contain a number of examples
of
how to perform searches using callbacks to execute a variety of different
operations. (This callback facility is very powerful. We used it
extensively
in Connexitor to do a lot of things that normal X.500-based directories
can't
do.) I would very much like to see a version of this code that doesn't rely
on directly extending back-ldbm or any other backends (besides back-ldap).

Also, there should be some kind of cache aging parameter to eliminate stale
data from the cache. A trickle-refresh scheme might be interesting as well,
but that's probably not necessary right away.

It's a very good effort. I think the query containment and query template
approach makes sense. Hopefully some more folks will examine this patch and
chime in.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

Hi,

I have uploaded a new patch for "Proxy cache extension for OpenLDAP,
ITS#2062". The extensions are all behind #ifdef LDAP_CACHING. The tarball
below contains the proxy cache patch for OpenLDAP-2.1.12 and a document
containing design, implementation and usage info (ldapcache.html).

It can be accessed at:
ftp://ftp.openldap.org/incoming/apurva-kumar-pkg-030214.tgz

Thanks,
Apurva Kumar,
Research Staff Member,
IBM India Research Lab
Phone: +91-11-26861100
Fax: +91-11-26861555

changed notes
changed state Suspended to Feedback

changed notes
changed state Feedback to Closed

I have uploaded a patch for the proxy cache contribution. The modifications
take into account the recent  callback, backend interface changes in slapd.

ftp://ftp.openldap.org/incoming/apurva-kumar-030519.patch

Apurva

moved from Contrib to Archive.Contrib

new patch provided
integrated into HEAD