RE: New Proxy cache overlay

["Howard Chu" <hyc@highlandsun.com>]

> -----Original Message-----
> From: Apurva Kumar [mailto:kapurva@in.ibm.com]

> Howard,
>
> That's really great news!
>
> I haven't seen the new code yet

Consider it an attempt at further validating the overlay concept. Testing
this code has highlighted some of the uglier aspects of the current
slap_callback mechanism. That really needs to be smoothed out.

> but moving out caching logic from the
> backend was something I very much wanted to do. Since
> back-meta and proxy
> cache were really backends for accessing multiple backend
> servers it does
> make sense for them to be implemented as an overlay over
> proxy (back-ldap)
> backends serving the backend servers. Implementing proxy cache as an
> overlay also solves the problem of having to map between the
> proxied naming
> context and the cache databases naming context. The new
> design also makes
> easier to implement some enhancements to the proxy cache.

Looking forward to hearing about your enhancement ideas.

I removed the byte-counting in the cache manager and left it with just
counting entries instead. It seems to me that given an explicit attrset in
the replies, most cached entries will be "close" to the same size anyway, so
the overhead of counting individual entry sizes is unnecessary.

There's an obvious hole in the current behavior - the information about what
queries are present in the cache is only maintained in memory, but the cache
is stored on-disk. After a server restart, the cache will contain stale data
that the cache manager doesn't know about. The consistency_check function
also only removes entries based on the in-memory query list, so it doesn't
currently fix things. There are a couple of possible approaches to fix this:
	1) in db_open, wipe out the data directory before calling the database
db_open.
	2) in db_open, do a subtree search over the entire tree, and delete all
entries whose createTimeStamp is older than the cache TTL.
	3) store the templatestr in the cached entries. in db_open, do a subtree
search over the tree, delete stale entries, and regenerate query data for any
entries that are still "new enough."

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support