Re: Updating the test suite

["Pierangelo Masarati" <ando@sys-net.it>]

> Pierangelo Masarati writes:
>>Hallvard B Furuseth writes:
>>> * Factor a lot of code out to shell functions in defines.sh.  (...)
>>>   Shell functions are fairly portable now, and in any case OpenLDAP
>>>   uses build/shtool which uses shell functions.  (...)
>>
>> You seem to be fairly optimistic about this.  I have no real arguments
>> against; I hope you're correct.
>
> I wasn't optimistic until I looked around a bit - and then noticed the
> shell functions in shtool:-)  Info node (autoconf)Shellology says
>
>      While most (at least most System V's) do have a Bourne shell that
>      accepts shell functions most vendor `/bin/sh' programs are not the
>      POSIX shell.
>
>      So while most modern systems do have a shell _somewhere_ that
>      meets the POSIX standard, the challenge is to find it.
>
> Mascheck says this section is outdated and only Ultrix is still
> relevant.  Since there are no relevant ITSes about shtool either,
> that seems to be right at least for OpenLDAP.
>
> Still, we chould allow the user to set a variable=<a shell which
> supports functions> (both for shtool and tests), and if a user arrives
> with such a need, see if we can include a configure test which finds
> that shell.  I can ask on the autoconf list about that.
>
>>> * Let ./run and scripts/all look for backend-specific as well as
>>>   general tests.  Remove scripts/sql-all.
>>>
>>>   "./run -b foo all" (used by "make test" for backend foo) would
>>>   first run scripts/foo-test*.  Then for bdb, ldbm, hdb or if a
>>>   "-general" (-g) flag is given, also run scripts/test*.
>>>
>>>   scripts/all would first do ". $SRCDIR/scripts/foo-setup" or
>>>   something, so we have some place to put the message in
>>>   scripts/sql-all.
>>>
>>>   Maybe this will encourage someone to write tests for more
>>>   backends:-)
>>
>> I like the idea in principle; one objection is that, for example, proxy
>> testing may use different backends for storage.  Currently, they get
>> tested against all configured storage types, while according to the
>> suggested partitioning they would be tested by default only against one
>> type (the "default"?  selected by the user?  what if all have to be
>> tested?).
>
> No, this is about what happens once you have chosen a backend, not
> about which backend to choose.
>
> There is no change in the default behaviour with the existing tests:
> "make test" would still walk through "./run -b <backend> all" for all
> backends supported by the test suite.  There are currently no
> backend-specific tests for BDB, LDBM and HDB, while 'make test' for
> SQL curently only runs the sql-specific tests.

test040 is currently specific to back-hdb because only back-hdb supports
subtree rename (it could be moved to a backend-specific add-on to test005)

test035 and test036 are specific to back-meta, but use any of bdb,hdb,ldbm
as remote storage

test028, test029 and test039 are specific to back-ldap; same as above

test020 is specific to slapo-pcache, and uses back-ldap as proxy and any
of bdb, hdb or ldbm as storages; it should be modified to run with
back-meta (and possibly with back-sql, as it may benefit from local
caching) as proxies

test030 can test back-relay, back-ldap and back-meta, using any of bdb,
hdb or ldbm as storages; this currently tests all 9 combinations, so ot's
pretty exaustive; maybe too much, in some cases.

I mean: when only one backend type is involved, everything is fine; when
more are involved, we should allow the full test matrix as well as partial
ones, trading testing time versus exaustion.  In principle, since the test
suite is mainly intended as a means to trace regressions, exaustion should
be the preferred choice.

p.

-- 
Pierangelo Masarati
mailto:pierangelo.masarati@sys-net.it


    SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497