Re: Problems with log entries order

[Dario Zanzico <dario@dariozanzico.com>]

On Tue, Jan 26, 2016, at 05:24 PM, Hallvard Breien Furuseth wrote:
> Maybe it would work to move Statslog( ... ACCEPT ...)
> above ldap_pvt_thread_mutex_unlock( &c->c_mutex );
> at the end of servers/slapd/connection.c:connection_init().
> Holding on to the mutex longer would make slapd a bit
> less responsive, though.  No idea how much.

That's kinda what I initially expected openldap to do :D

> Another variant: Put the cost in the log parser instead.

yep, that's the way I worked around this problem

> When you see an ACCEPT, delay processing log lines until
> you've seen some more lines, in case they are out of order.
> I've got an old tweak to contrib/slapd-tools/statslog
> which does that and some other stuff, here:
> <http://folk.uio.no/hbf/OpenLDAP/statslog.reorder>

I was sending my logs through 
syslog=>logstash=>elasticsearch
correlating binds and accepts in logstash (whose filters prevented
me to easily implement delays)
So, considering the rarity of the mixed log entries, I decided to:
1) save both accept and bind entries in elasticsearch
2) keep correlating events in passing through logstash, if possible
3) schedule a script which "correlates uncorrelated" binds and deletes
old 
connections data from elasticsearch

> Don't remember why I haven't committed it. Maybe
> it isn't quite finished or I hoped to optimize it,
> since it does slow the script down.  I'd be glad to
> hear of any improvements from you:-)

Perl scares me :)

> Come to think of it, statslog is quite old.  It
> might need some tweaks to catch up with whatever has
> been done to OpenLDAP logs since it was written.

I suppose it would be nice to have an option to force Openldap to
write ordered logs (obviously with a performance penalty).
Or maybe enabling logging the real operation timestamp (as in: 
generated just before the operation itself), enabling a simple filter to
reorder logs.

Thanks

> Hallvard

dario