Re: Updating the test suite

["Pierangelo Masarati" <ando@sys-net.it>]

> Pierangelo Masarati writes:
>> In any case this is mainly a hack to allow importing the project
>> under AEGIS; I don't either like having to source defines.sh twice
>> only to get few directories.  Maybe directories should simply be
>> set up in run and passed to defines.sh thru the environment?
>
> I suppose ./run could run the scripts by sourcing them instead:
>   test -x "$SCRIPT" && (. "$SCRIPT")
> Then the script will see variables from ./run, not just environment
> variables.  It gives me a slightly uncomforable feel but should
> work fine.  At least on Unix, don't know about other systems.
> E.g. skipping the 'test -x' would probably be a bad idea, so it
> should not be ported to a system where one cannot do that.
>
>
> I'd like to update the test suite in various ways anyway.
> Comments welcome.
>
> (Note that I only know Unix, not Cygwin or Mingw.  Or AEGIS,
> whatever that is.)

A concurrent development system.  ask Walter <walter@sys-net.it> for
details...  I think the issue he's having is that to accept a commit, the
package must pass some set of project manager defined checks, and he's
trying to run the test suite among the configured checks.

>
> * Factor a lot of code out to shell functions in defines.sh.  Then
>   we can also easily move code between ./run and the test scripts.
>   With functions to handle the details we'll also make fewer errors
>   like sometimes forgetting to kill the daemons before exit.
>
>   Tests with specific needs can define their own functions for
>   duplicated code.
>
>   Shell functions are fairly portable now, and in any case OpenLDAP
>   uses build/shtool which uses shell functions.  In comp.unix.shell,
>   Sven Mascheck writes in message <dh4658Uv0kL1@news.in-ulm.de>:
>
>   > The only relevant system nowadays which comes with a /bin/sh
>   > without functions is Ultrix - but there, "sh5" provides them.
>
>   (I think Ultrix does not support "test -x" either, which "make
>   test" also uses.  No ITS mentions Ultrix so far.

You seem to be fairly optimistic about this.  I have no real arguments
against; I hope you're correct.

>
> * Let ./run and scripts/all look for backend-specific as well as
>   general tests.  Remove scripts/sql-all.
>
>   "./run -b foo all" (used by "make test" for backend foo) would
>   first run scripts/foo-test*.  Then for bdb, ldbm, hdb or if a
>   "-general" (-g) flag is given, also run scripts/test*.
>
>   scripts/all would first do ". $SRCDIR/scripts/foo-setup" or
>   something, so we have some place to put the message in
>   scripts/sql-all.
>
>   Maybe this will encourage someone to write tests for more
>   backends:-)

I like the idea in principle; one objection is that, for example, proxy
testing may use different backends for storage.  Currently, they get
tested against all configured storage types, while according to the
suggested partitioning they would be tested by default only against one
type (the "default"?  selected by the user?  what if all have to be
tested?).

>
> * Run each LDAP client and daemon as something like
>     $LDAP_TESTER <program> <args>...
>   where the user can set the (normally unset) $LDAP_TESTER
>   environment variable to e.g. valgrind or "xterm -e gdb --args".
>   Actually, we could use 4 variables:
>   $LDAP_RETCODE_TESTER for clients whose return code (other than
>   success/failure?) is important, defaulting to:
>   $LDAP_FG_TESTER for foreground processes, defaulting to:
>   $LDAP_TESTER for any process, and
>   $LDAP_BG_TESTER for background processes (default $LDAP_TESTER).
>
> * Give ./run an "-ignore" (-i) argument, to ignore some errors and
>   keep running the tests.  scripts/all would not abort if a test
>   fails, and specific checks in each script could be marked as
>   "soft" (keep going under ./run -i) or "hard" (always abort).
>
> * The tests should wait for daemons to exit after killing them, and
>   fail if they failed.  Then we can also omit the sleeps after each
>   test in scripts/all, and a few others.
>
> * While waiting for slapd to start,
>   - sleep briefly after first failure and longer in each iteration.
>   - If ldapsearch fails, abort the loop if slapd is not running.
>   - Do not sleep 5 seconds before exiting the loop and failing.
>
> * Print LDAP result code names instead of numbers in error messages.
>
> * Move bookkeeping about pids into the functions.  Give each pid a
>   name to be used in messages ("master slapd", "slave slapd", etc).
>
> * "./run all" should only run tests matching "scripts/*[a-zA-Z0-9]"
>   or something, so Emacs backup files "testxyz-foo.~19~" are not run.
>
>   What are the backup file names to avoid from other editors?

Most of the ones I know about simply append a "~".

>
>   Does Cygwin support filename expansion like *[a-zA-Z0-9], or must
>   this be handled with a case statement in scripts/all?
>
> * Allow "./run scripts/scriptname" and not just "./run scriptname",
>   so one  can use filename expansion when typing a test command:
>   "./run scripts/<start-of-filename><tab>"

p.

-- 
Pierangelo Masarati
mailto:pierangelo.masarati@sys-net.it


    SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497