[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Proxy Cache Contribution

To: openldap-devel@OpenLDAP.org
Subject: Re: Proxy Cache Contribution
From: "Apurva Kumar" <kapurva@in.ibm.com>
Date: Thu, 13 Mar 2003 10:52:10 +0530
Content-disposition: inline

Hi,

The attached doc provides a brief desrciption of the LDAP proxy cache,
syntaxes of cache specific directives and an example configuration file.
Comments/feedbacks on the proxy cache are welcome.

(See attached file: cache_manual.html)
Thanks,
Apurva Kumar,
Research Staff Member,
IBM India Research Lab
Phone: +91-11-26861100
Fax: +91-11-26861555


                                                                                                                                            
                      Jonghyuk Choi                                                                                                         
                      <jongchoi@us.ibm.com>         To:       openldap-devel@OpenLDAP.org                                                   
                      Sent by:                      cc:                                                                                     
                      owner-openldap-devel@O        Subject:  Proxy Cache Contribution                                                      
                      penLDAP.org                                                                                                           
                                                                                                                                            
                                                                                                                                            
                      03/13/03 12:14 AM                                                                                                     
                                                                                                                                            
                                                                                                                                            








The proxy cache code from Apurva Kumar of IBM India Research was committed
to HEAD recently.

The proxy cache provides the following features :

    1. Semantic caching of positive conjunctive LDAP queries
    2. Attribute level caching.
    3. Consistency support
    4. Support for multiple backend types
    5. Support for caching multiple directories
    6. Support for multiple database instances for a single cache directory
       tree.




Please refer to the attached document and ITS#2062 for further information
and history.
Any comments and feedbacks to the developers' list are welcome.

(See attached file: ldapcache.html)

------------------------
Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979


#### ldapcache.html has been removed from this note on March 13 2003 by
Apurva Kumar

Title: LDAP proxy cache

LDAP Proxy Cache Extension for OpenLDAP

Brief Description

The LDAP cache extension to back-meta allows answering of LDAP queries from previously cached queries. The meta backend is associated with one or more "cache backend databases" which cache entries corresponding to the cached queries. Currenlty LDBM backend can be used as a cache backend.

Caching is based on the concept of queries and templates. LDAP search requests are referred to as 'queries'. The base, scope, filter and projected attributes parameters of a query are referred to as 'metadata'. Templates are prototypes used by typical LDAP applications for forming LDAP queries. A template is defined by a template string and a set of projected attributes. Template string is represented by omitting the 'value' from the RFC 2254 representation of LDAP filters. An example of template used to generate queries like (sn=smith), (sn=parker*) is (sn=). The template does not differentiate between substring and equality filters. Presence filters are represented without any change. For e.g. filter &(sn=smith*)(c=*) belongs to the template &(sn=)(c=*).

At configuration time, a list of cacheable templates is specified. Queries belonging to only these templates are cached. An incoming query is checked for containment in the cached queries belonging to templates with the same template string. If an incoming query can not be answered from the cache, the results are obtained from the backend directory server and sent to the client. If the number of entries returned is within a cacheable limit, the entries are added to the cache backend and the meta data for the query is added.

Query containment algorithm provides answering of positive conjunctive attributes with equality, range (>=/<=) and substring assertions. Answering of both equality and substring queries from substring assertions of cached queries is supported. For e.g. both the queries (sn=richards*) and (sn=richardson) can be answered by a cached query (sn=richard*).

Cache replacement uses a simple LRU policy. The metadata of the least recently used query and entries belonging only to that query are removed. A query is considered to be used when it answers an incoming query. Cache replacement is invoked when the cache size crosses a threshold (hi-thresh) and continues till it is greater than a lo-thresh.

A time to live (TTL) value is associated with queries. The value is same for all queries of the same template. After the time to live expires the query (and entries belonging to only that query) can be removed.

Cache configuration specific directives

Three cache specific directives are added:

cacheparams: Used to set various cache parameters. The syntax is:
```
		
cacheparams <lo_thresh> <hi_thresh> <numattrsets> 
<max entries> <consistency cycle time>

		
	    
```
The first two parameters are used for cache replacement. Cache replacement starts when the cache size crosses the hi_thresh bytes and continues till the cache size is greater than lo_thresh bytes . Total number of attributes sets (as specified by the attrset directive) is given by numattrsets. Queries are cached only if they correspond to a cacheable template (specified by the addtemplate directive) and the number of entries returned is less than max entries .

Support for weak consistency is provided by associating a time to live (TTL) with each query. The TTL is specified while adding a template using the addtemplate directive. The consistency checks are performed every consistency cycle time seconds. Stale queries and entries belonging to only those queries are removed.

Example:
```
 
	cacheparams 100000 120000 3 5 1000
		
```
attrset: Used to associate a set of attributes to an index.
```
 
		
attrset <index> <attr1> <attr2> <attr3> ...
		
	    
```
Each attribute set is associated with an index number from 0 to numattrsets-1. These indices are used by the addtemplate directive to define cacheable templates.

Example:
```
 
	attrset 0 cn sn mail 
		
```
addtemplate: Adds a cacheable template with a given TTL.
```
 
		
addtemplate <template string> <attr set index> <ttl>
		
	    
```
template string representation is similar to RFC 2254 with any values in simple/substring filters omitted. Some examples of template strings are: (sn=), (age >=), &(sn=)(c=*). First template could have queries with equality and substring assertions. Second template represents queries with a simple >= assertion and the (c=*) in the third template represents a presence filter. The attr set index is used to associate one of the attribute sets defined by the attrset directive with a template. TTL (in seconds) is used to associate a time to live with queries of the template.

Example:
```
 
	addtemplate &(sn=)(givenname=) 0 3600
		
```

Example slapd.conf

        #schema file (same or subset of backend server)  
	include		/home/apurva/schema/<schema_file>

        #other general directives
        #

	#######################################################################
	# cache databases
	#######################################################################
        # sub database 1
	database ldbm
	subordinate 

	suffix "ou=people,o=University of Michigan,c=US,cn=cache" 
	directory /usr/local/var/openldap-data/<database_dir>
	cachesize 1000
        # other LDBM directives 
	#

        # other sub databases
	#

        # 
	# parent database
        
	database ldbm
	suffix "o=University of Michigan,c=US,cn=cache"
	directory /usr/local/var/openldap-data/<parent_database_dir>
	cachesize 1000
        # other LDBM directives 

 
        #######################################################################
        # meta cache  
        #######################################################################
	database	meta
	rewriteEngine   on
        # rule for rewriting DNs of entries to be cached.   
	rewriteContext cacheResult
	rewriteRule "(.*)o=University of Michigan,c=US" "%1o=University of Michigan,c=US,cn=cache" ":"

        # rule for rewriting the base for searching the cache for answerable queries
	rewriteContext cacheBase
	rewriteRule "(.*)o=University of Michigan,c=US" "%1o=University of Michigan,c=US,cn=cache" ":"

        # rule for rewriting DNs of cached entries before sending the result to the client. 
	rewriteContext cacheReturn 
	rewriteRule "(.*)o=University of Michigan,c=US,cn=cache" "%1o=University of Michigan,c=US" ":"
	 

	suffix          "o=University of Michigan,c=US"
	uri		ldap://<backend_hostport>/o=University of Michigan,c=US
       
	cacheparams	10000 15000 4 50 1000

	attrset 0 cn title
	attrset 1 homephone pager telephonenumber mail
	attrset 2 postaladdress  
	attrset 3 member 

	addtemplate   (uid=) 0 3600
	addtemplate   (sn=) 1 3600
	addtemplate   (sn=) 2 3600 
	addtemplate   &(objectclass=)(cn=) 3 100000

Note: To have the caching feature enabled, the following are required

#define LDAP_CACHING 1 in slap.h
Cache specific directives in the slapd configuration file.

Important: For correct operation of the proxy cache, database directories of the cache databases should be cleared before starting slapd.

Author: Apurva Kumar kapurva@in.ibm.com

Prev by Date: Proxy Cache Contribution
Next by Date: Re: config backend
Index(es):
- Chronological
- Thread