[Q] Logging to a directory

["Alex Zeltser" <alex_zeltser@securecomputing.com>]

I'd like to ask a general LDAP question which may make LDAP purists cringe.
 We're currently working with Netscape's directory (v.4.x) and have an
application that needs to maintain a detailed audit trail of events that
have taken place over a period of time.  The volume of events is fairly
high and each log entry has a fair amount of detail.  We're currently using
the same directory that we use for everything else as the audit log
repository.  This is very convenient in some respects: it lets us use the
same calls for storing logs as for everything else, it lets us do very
flexible searches for audit logs easily, and it offers replication through
Netscape's built-in replication mechanism.  (The replication seems a little
flakey, BTW, but that's a different issue.  ;-) ).

Unfortunately, it also presents some concerns.  These mainly have to do
with deleting log entries that are no longer needed from the directory.  We
found that: 1) It takes a _really_ long time to delete a large number of
entries (I think on the order of minutes for a few thousand entries), and
2) Even after the entries are deleted the size of the physical DB files
doesn't decrease.

I understand that directories were never really meant to be used as a
storage mechanism for volatile data (and that's why I think LDAP purists
would want to crucify me ;-) ), but am wondering if anyone has run into
similar issues and how they dealt with them.  So:

- Is it really a bad, bad idea to log to a directory when the anticipated
log volume is high?
- Can anything be done to improve performance of batch delete's?  (We do
have to index several attributes in the log entries, but what can we do
short of not indexing?)
- Are there any calls (or even standalone tools) for compacting the DB
files?
- (As an aside question, does anyone know if the space in these DB files is
reclaimed later when new objects are allocated, and so their size can be
expected to remain more or less consistent after a certain point?)
- Does anyone have any general words of wisdom when it comes to logging in
directory-enabled apps?  Log to the directory?  Don't?  Log, but with
certain constraints?  Tips?

Thanks in advance,

Alex Zeltser