Compact log format for faster searching?

[Hallvard Breien Furuseth <h.b.furuseth@usit.uio.no>]

Have anyone come up with a searchable, compact slapd-log format,
so the logs can be searched quicker than the current verbose format?

That is, the search tool would understand the compact format without
expanding it (fully) first, and only expand the final output.
Our syslog from loglevel "stats" compresses to 1/20 of the original,
but that's no help when we must search the entire uncompressed log.

It's easy to halve the log and still keep it mostly readable: Remove
syslog cruft and write it only when it changes, replace "conn= <op/fd>="
with base-32 "conn.<o/f>op", join up multiple SRCH attr= lines, etc.

Or down to 1/3 of the original in our case by replacing a few common
operations (filters, suffixes, etc), but that quickly makes the result
unreadable without a tool to translate back and forth.  So beyond that
something is needed to translate back and forth, or an entirely new,
human-readable, compact format.

BTW, "perl -lne '/uid=xyzzy/i && print' log" is 10-15 times faster than
GNU  "grep -i 'uid=xyzzy' log" on my Linux box.
And the initial zcat of a compressed log is faster than the perl/grep.

-- 
Hallvard