Test suite overhaul

[Ondřej Kuzník <ondra@mistotebe.net>]

This is a follow-up to my email to -technical, this time more into the
future of OpenLDAP's test suite and tools we could build to make
managing OpenLDAP deployments much easier for everyone.

Historically, there has been a decent coverage in the test suite and
that's what's being run before anyone pushes, but it's not been enough
to capture some of the issues. It also takes ages to finish, and is very
limited in what can be tested, mostly because we can only rely on POSIX
shell and command line tools.

That means a lot of boilerplate (and since shell functions are not
available, that boilerplate has to be copy-pasted everywhere) and we can
never actively do two things at the same time, let alone submit several
operations over the same connection. There is no way we can get notified
when things happen (servers have started up, all members think they're
in sync, ...) so we have to rely on polling and manual delays which
slows it down significantly, or introduces false positives on hosts
where these delays are too short. The tests can't be run in parallel
either, not from the same build environment.

None of the above can be resolved without a major overhaul. I've been
thinking about all of that for a while now and want to get started
improving the situation.

If we switched to an actual programming language to write the tests in,
something like Python 3, Golang or Rust, we could build a support
library including a harness to manage slapd through, tools to configure
and interact with the server in a high level way (continue as soon as
the server has started up, or once an MMR environment settles, ...). All
of this should then make it much easier to write new tests and maintain
and understand existing ones.

I have already started the part that people without intimate experience
with OpenLDAP internals would struggle with - a loadable module that
knows how to connect back to the program that started it and informs it
of the port it's listening on and when it has become ready to process
traffic. That code is here:
https://github.com/mistotebe/openldap/tree/mod_harness

To see it in action, just run `nc -l -p <port>`, then start slapd with
`-h ldap://localhost:0` (yes, an explicit zero) and make sure this is in
slapd's configuration:

```
moduleload path/to/mod_harness

backend harness
host localhost
port <your port above>
```

I would appreciate if people joined to help define the testing
infrastructure. My preference is Python 3 as the host language, but
anything that can express concurrency well and has a good LDAP client
implementation should be acceptable if there are enough volunteers. We
will also have to look into defining the protocol the test suite would
use to communicate with the slapd module as the existing plain text
protocol is just a placeholder for easy experimentation.

This would do well for testing in the 2.5 lifecycle. In the future, I
want to extend this to support fine grained testing where the inner
workings of slapd are much more transparent to the test code. Eventually
going as far as making it possible to unit-test any OpenLDAP code. The
first step in that direction is probably going to be learning how to
host a C compiler within the loadable module so we can compile, link and
inject code and callbacks into slapd as and when needed to meld the test
suite with the server much better. Clang (the LLVM C/C++ compiler) seems
to be capable of this.

All of the above will introduce new helper APIs, so in the end it will
overlap with efforts to develop new user-facing tools to configure,
monitor and manage OpenLDAP deployments:
- user friendly tools to manage cn=config
- high-level tools to set up and manage common replication topologies
- tools to monitor replication progress
- many more as we progress with making OpenLDAP user friendly...

(Please don't turn this thread into another flame war on slapd.conf vs.
cn=config, start another one if you really can't resist.)

Again, thanks for being brave and getting this far, let us know what you
think. This is mostly vaporware yet, giving everyone a great opportunity
to shape the project and leave their mark - remember that without your
help, it will take a long while before it's ready!

Regards,

-- 
Ondřej Kuzník
Senior Software Engineer
Symas Corporation                       http://www.symas.com
Packaged, certified, and supported LDAP solutions powered by OpenLDAP