Re: transaction logging

[Will Ballantyne <Will.Ballantyne@gems1.gov.bc.ca>]

"Mark D. Anderson" wrote:
> You should consider what your use case is -- do you want this
> to build replication (that is what inspired CHANGELOG), or do
> want this for auditing (SIGOPS), or for some other reason?

I would like to do both. We have a requirement to be able to easily tell
who changed what and when, and it would also be very useful if the
replication was a little safer. 

I would be a little concerned with putting the transactions in the
directory itself as CHANGELOG indicates.  Export of the directory using
ldbmcat would export the entire change history as well.  That could be
huge and add unnecessary work for sync runs to external systems that
require checking the whole directory image.

> Neither of the drafts above properly deals with the consequences
> of proxy authentication via SASL -- in theory you'd like to know
> both the proxy id and the user the proxy is acting for.

Yes, good point.  I have not looked into SASL but I suspect there may be
additional fields that may become relevant later.  It might be useful to
have the log record be extensible.
 
> I think Netscape used CHANGELOG in their replication implementation in their
> directory server v3 (g. good went to work for netscape). I haven't been
> tracking the LDUP debates, so i don't know how good a fit it is anymore.
> 
> You have to think about what to do about "automatic" attributes like
> modifiersName and modifyTime -- do you also log the changes to them.

I would think so.  If we replicate the data then the remote site may
need these or we would introduce disparaties...Hm, anyone have opinions
on that?  I seem to recollect v3 states the client (in this case the
replicator) would not be able to change those.

> Note that Innosoft uses the capability for multiple "backends" in their
> LDAP server specifically for the changelog case -- it allows the changelog
> entries to be placed not only in a different part of the DIT, but to be
> done by a separate engine on a different disk partition, which aids in
> performance. Innosoft is based on the umich sources; so far as i know
> a similar capability could be added in openldap.

We have the Innosoft idds product and will be using it for some things.
I had noticed it behaved remarkably like the umich slapd. 

The idea of using multiple back-ends is very good.  We could put the
changelog implementation on its own back-end and avoid having ldbmcat
export the whole change history with it.

> A related capability to changelog is the ability to log a text file in
> ldif format which contains all the operations regardless of whether
> it is an add, update, or delete. In principle, this file could be directly
> imported with ldap_modify, if ldap_modify can handle anything besides adds.

Yes, that sounds like a really good way to store the changes.  
 
> You want to watch your use of the word "transaction", as making the
> ldap server truly transactional (in the ACID sense) is an entirely
> different project from merely change logging.

Well, I would rather state "ACID transactional" if that's what I meant. 
By transaction I mean the dictionary term: a communicative action
involving two parties that influence each other.  

Thanks for the pointers and comments.  Very useful.

-- 
Will Ballantyne             GEMS Technical Architect
mailto:Will.Ballantyne@gems1.gov.bc.ca