[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Dnyamic Re: commit: ldap/servers/slapd entry.c proto-slap.h slap.h zn_malloc.c

Thanks, that was interesting. CKRM looks like just the thing for this.

I implemented a dynamic cache sizing mechanism in an
application along similar lines and generally it's been
successful. That one used throughput rather than latency as
the feedback variable, because in its case there was a constant load
and therefore it was easy to measure throughput.

Since the kernel knows who has what memory, it seems like
it'd be easier to just have it tell the applications when to
grow/shrink their usage.

It is shown in the experiment that the BDB cache exhibits significantly
better performance than the entry cache once swapping occurs. It is not easy
to pinpoint the exact differences because I'm not BDB expert. The main
difference I attribute to the difference in performance is the difference in
the locality of page references in BDB and entry caches. The actual working
set size of the BDB cache seems a lot smaller than that of the entry cache.

BDB hashes the pages in the mpool. You could try hashing rather than AVL trees in the entry cache. Entries in the entry cache aren't optimized for storage efficiency, so it's reasonable to expect the memory footprint per entry to be larger always.

The other difference is that paging would
happen, but to the region backing file (in the case that you're not using
a private region). Perhaps that paging doesn't show up in the stats you
measured ?

The new entry cache design in the adaptive caching also reduces the working
set size of the entry cache since the zone memory heap only contains Entry
and DBT structures separating them from the EntryInfo AVL tree in the normal
heap. Since it becomes efficient to resize the entry cache with the
zone-based adaptive cache, one can rely more on the entry cache by shifting
memory from BDB cache to the entry cache in configurations where it is not
easy to avoid swapping from occurring.

Reminds me a bit of the old object databases and pointer swizling. Not sure those things ever performed very well though.

Would it be useful to be able to re-size the BDB mpool via the CKRM
policy ? (to avoid the shutdown and re-start of the database : although
your paper says that the overhead is low, if the region has to be shared
the underlying file will need to be created and on some OS'es that can
be quite expensive). BDB has the capability to have multiple mpools
in use, and I suspect it wouldn't be too hard to have it create new ones
on-the-fly. Shrinking its cache presumably can be done by simply
avoiding touching a subset of the already comitted pages (unless the goal
is to reduce page file usage too).