Re: Updating the test suite

[Hallvard B Furuseth <h.b.furuseth@usit.uio.no>]

Pierangelo Masarati writes:
> test040 is currently specific to back-hdb because only back-hdb
> supports subtree rename (it could be moved to a backend-specific
> add-on to test005)

That's true.  Though if back-newWonderfulDatabase is implemented someday
and also supports subtree rename, test005 would have to be renamed back.

I wasn't even thinking of such cases, I was looking at the sql tests and
passwd-search - these backends cannot use the general tests.  Different
config files, different database contents.

Same for back-dnssrv, back-null, and others.  For back-perl and
back-shell one would have to implement a lot of OpenLDAP as perl or
shell scripts in order to run the full test suite, I doubt anyone would
bother:-)

Yet now that I think of it, there ought to be some general tests which
both back-sql and these other backends can run.

> test035 and test036 are specific to back-meta, but use any of
> bdb,hdb,ldbm as remote storage
>
> test028, test029 and test039 are specific to back-ldap; same as above

So we don't rename them.  If they are made backend-specific we get into
trouble, but if we leave them as they are they work fine.
Same for test020 and test030 below.

> test020 is specific to slapo-pcache, and uses back-ldap as proxy and
> any of bdb, hdb or ldbm as storages; it should be modified to run with
> back-meta (and possibly with back-sql, as it may benefit from local
> caching) as proxies

Hmm.  My scheme still wouldn't run this test for back-sql.
Maybe tests named "scripts/all-test*" should be run for any backend.
Then add a more convenient scheme for specifying which backends support
which features, and filter out which tests to run that way.  Could
handle at least test005 above the same way.

One important feature would be whether or not slapadd is supported.

Come to think of it, which backends can run a test chould be seen as
just another feature, maybe 

(Or most other tests could be renamed dbtest* or something which would
only be run for bdb, ldbm and hdb, but that seems more invasive...)

> test030 can test back-relay, back-ldap and back-meta, using any of bdb,
> hdb or ldbm as storages; this currently tests all 9 combinations, so ot's
> pretty exaustive; maybe too much, in some cases.

> I mean: when only one backend type is involved, everything is fine; when
> more are involved, we should allow the full test matrix as well as partial
> ones, trading testing time versus exaustion.  In principle, since the test
> suite is mainly intended as a means to trace regressions, exaustion should
> be the preferred choice.

Yes.

-- 
Hallvard