[Date Prev][Date Next] [Chronological] [Thread] [Top]

Updating the test suite (Was: commit: ldap/tests run.in)

Pierangelo Masarati writes:
> In any case this is mainly a hack to allow importing the project
> under AEGIS; I don't either like having to source defines.sh twice
> only to get few directories.  Maybe directories should simply be
> set up in run and passed to defines.sh thru the environment?

I suppose ./run could run the scripts by sourcing them instead:
  test -x "$SCRIPT" && (. "$SCRIPT")
Then the script will see variables from ./run, not just environment
variables.  It gives me a slightly uncomforable feel but should
work fine.  At least on Unix, don't know about other systems.
E.g. skipping the 'test -x' would probably be a bad idea, so it
should not be ported to a system where one cannot do that.

I'd like to update the test suite in various ways anyway.
Comments welcome.

(Note that I only know Unix, not Cygwin or Mingw.  Or AEGIS,
whatever that is.)

* Factor a lot of code out to shell functions in defines.sh.  Then
  we can also easily move code between ./run and the test scripts.
  With functions to handle the details we'll also make fewer errors
  like sometimes forgetting to kill the daemons before exit.

  Tests with specific needs can define their own functions for
  duplicated code.

  Shell functions are fairly portable now, and in any case OpenLDAP
  uses build/shtool which uses shell functions.  In comp.unix.shell,
  Sven Mascheck writes in message <dh4658Uv0kL1@news.in-ulm.de>:

  > The only relevant system nowadays which comes with a /bin/sh
  > without functions is Ultrix - but there, "sh5" provides them.

  (I think Ultrix does not support "test -x" either, which "make
  test" also uses.  No ITS mentions Ultrix so far.

* Let ./run and scripts/all look for backend-specific as well as
  general tests.  Remove scripts/sql-all.

  "./run -b foo all" (used by "make test" for backend foo) would
  first run scripts/foo-test*.  Then for bdb, ldbm, hdb or if a
  "-general" (-g) flag is given, also run scripts/test*.

  scripts/all would first do ". $SRCDIR/scripts/foo-setup" or
  something, so we have some place to put the message in

  Maybe this will encourage someone to write tests for more

* Run each LDAP client and daemon as something like
    $LDAP_TESTER <program> <args>...
  where the user can set the (normally unset) $LDAP_TESTER
  environment variable to e.g. valgrind or "xterm -e gdb --args".
  Actually, we could use 4 variables:
  $LDAP_RETCODE_TESTER for clients whose return code (other than
  success/failure?) is important, defaulting to:
  $LDAP_FG_TESTER for foreground processes, defaulting to:
  $LDAP_TESTER for any process, and
  $LDAP_BG_TESTER for background processes (default $LDAP_TESTER).

* Give ./run an "-ignore" (-i) argument, to ignore some errors and
  keep running the tests.  scripts/all would not abort if a test
  fails, and specific checks in each script could be marked as
  "soft" (keep going under ./run -i) or "hard" (always abort).

* The tests should wait for daemons to exit after killing them, and
  fail if they failed.  Then we can also omit the sleeps after each
  test in scripts/all, and a few others.

* While waiting for slapd to start,
  - sleep briefly after first failure and longer in each iteration.
  - If ldapsearch fails, abort the loop if slapd is not running.
  - Do not sleep 5 seconds before exiting the loop and failing.

* Print LDAP result code names instead of numbers in error messages.

* Move bookkeeping about pids into the functions.  Give each pid a
  name to be used in messages ("master slapd", "slave slapd", etc).

* "./run all" should only run tests matching "scripts/*[a-zA-Z0-9]"
  or something, so Emacs backup files "testxyz-foo.~19~" are not run.

  What are the backup file names to avoid from other editors?

  Does Cygwin support filename expansion like *[a-zA-Z0-9], or must
  this be handled with a case statement in scripts/all?

* Allow "./run scripts/scriptname" and not just "./run scriptname",
  so one  can use filename expansion when typing a test command:
  "./run scripts/<start-of-filename><tab>"


Example - test011-glue-slapadd using functions:

echo "running defines.sh"
. $SRCDIR/scripts/defines.sh

Init_testdir $DBDIR1A $DBDIR1B $DBDIR1C || exit $?

echo "Running slapadd to build glued slapd databases..."
Demand_RC $? -eq 0 --text "slapadd failed ($?)!" || exit $?

echo "Starting slapd on TCP/IP port $PORT1..."
$SLAPD -f $CONF1 -h $URI1 -d $LVL $TIMING > $LOG1 2>&1 &
Set_pidinfo PID 'slapd'

echo "Using ldapsearch to retrieve all the entries..."
Wait_slapd --pid $PID --out $SEARCHOUT $LDAPSEARCH -b "$BASEDN" \
Demand_RC $? -eq 0 --op "ldapsearch" || exit $?

echo "Filtering ldapsearch results..."
echo "Filtering original ldif used to create database..."
echo "Comparing filter output..."
Demand_RC $? -eq 0 \
	"comparison failed - database was not created correctly" \
	-- '$DIFF $SEARCHFLT $LDIFFLT' || exit $?

echo "Testing sizelimit..."
$LDAPSEARCH -b "$BASEDN" -h $LOCALHOST -p $PORT1 -s one -z 2 > $SEARCHOUT 2>&1
Demand_RC $? -eq 4 --op "search with sizelimit at end" || exit $?

	objectclass=OpenLDAPPerson > $SEARCHOUT 2>&1
Demand_RC $? -eq 4 --op "search with sizelimit at middle" || exit $?

Kill_all || exit $?

echo ">>>>> Test succeeded"
exit 0


Draft addition to defines.sh:

# Functions.
# They use internal variables $fV_*.

# Init_testdir [--more] [subdirectory...]
# Clean up and then create $TESTDIR unless --more is given, then
# create the subdirectories.
# (--more is for doing Init_testdir several times in one script.)
Init_testdir () {
	if test "x$1" = x--more; then
		# Remove cruft from prior test, then create dir
		if test "$PRESERVE" = yes ; then
			/bin/rm -rf $TESTDIR/db.*
			/bin/rm -rf $TESTDIR
		test -d "$TESTDIR" || mkdir -p $TESTDIR || return $?
	test $# -eq 0 || mkdir "$@"
	return $?

# Set_pidinfo [--nowait] variable-name 'textual process name'
# Registers information about $! - the most recent background process.
# Sets ${variable-name}=$!, and some internal state.
# Under ./run -w ($WAIT), waits for user to press Enter unless --nowait.
# No return value.
Set_pidinfo () {
	case $1,$WAIT in
	--nowait,*) shift;;
	*,0 | *,) :;;
	*) $SRCDIR/../build/shtool echo -n "Started $2 (pid $!). Press Enter: "
	   read fV_dummy;;
	eval "$1=\$! fV_PIDNAME$!=\$2 fV_PID$!=\$!"

# Wait_slapd [--pid slapd-pid] [--out outfile] test-command arg...
# Waits for a slapd (registered with Set_pidinfo) to start.
# Runs 'test-command arg... >outfile(or /dev/null) 2>&1' until success.
# Gives up after ~25 seconds, or if a Set_pidinfo process died unexpectedly.
# (The pid parameter is currently only used for reporting.)
# Returns the final exit status from the command.
Wait_slapd () {
	while :; do case $1 in
		--pid)	eval "fV_name=\${fV_PIDNAME$2-child}"; shift 2;;
		--out)	fV_out=$2; shift 2;;
		*)	break;;
	esac; done
	eval "fV_pids=\"$fV_PIDVARS\""
	for fV_i in 0 1 2 4 5 6 7; do
		# Expects $fV_WAIT_START -le 7
		if test $fV_WAIT_START -lt $fV_i; then
			echo "Waiting $fV_i seconds for $fV_name to start..."
			sleep $fV_i
		elif test $fV_i -lt $fV_WAIT_START; then
		fV_rc=0 fV_WAIT_START=$fV_i
		"$@" >$fV_out 2>&1 && break
		for fV_p in $fV_pids; do
			kill -0 $fV_p 2>/dev/null || break 2
	return $fV_rc

# Kill_all
# Kills and waits for remaining Set_pidinfo processes.
# Returns nonzero iff any process returns failure or had died.
# Does nothing under ./run -k ($KILLSERVERS).
Kill_all () {
	eval "fV_pids=\"$fV_PIDVARS\""
	test "x$KILLSERVERS" != xno && case $fV_pids in *[0-9]*)
		echo 'Stopping and waiting for subprocesses.'
		kill -HUP $fV_pids || fV_rc=99
		for fV_p in $fV_pids; do
			wait $fV_p && continue
			eval echo \
 "\"\${fV_PIDNAME$fV_p-child} (pid $fV_p) failed ($?)!\"" >&2
	return $fV_rc

# Kill_named <pid> || exit $?
# Kills and waits for one process.
# If the process returns failure or had died, runs Kill_all as well.
# Returns 0 iff the process was running and returned success when killed.
Kill_named () {
	eval "fV_name=\${fV_PIDNAME$1-child} fV_PID$1="
	echo "Stopping and waiting for $fV_name."
	kill -HUP $1 && wait $1 && return 0
	echo "$fV_name (pid $1) failed ($?)!" >&2
	return 99

# Check_RC [--ignore|--soft] status {-eq|-ne} expect {msg|--op name} [-- stmt]
# Checks that the return code <status> from a program is -eq|-ne <expect>.
# Otherwise complains, evals <stmt> (should be quoted) and runs Kill_all.
# --op reports the error for <name> using LDAP ASN.1 result code names.
# --ignore refrains from running Kill_all, and reports the error as ignored.
# --soft under ./run -i ($IGNORE_ERRORS) will --ignore and also return success.
# Returns 0 at success, otherwise <status> if that is nonzero, otherwise 99.

# This function is getting too complex, and also may need two return
# values: One for whether the caller should abort or not, and one
# with the real exit value in the case of a --soft error.
# Not sure what kind of args to give it either.
# E.g., maybe --soft should be default and instad a --force flag
# should mean not to keep running even under $IGNORE_ERRORS.

Check_RC () {
	case $1 in
		--ignore | --soft) fV_ignore=$1; shift ;;
		*) fV_ignore= ;;
	test "$1" "$2" "$3" && return 0
	case $fV_ignore,$IGNORE_ERRORS in
		--ignore,*)	fV_ignore=" - IGNORED" ;;
		--soft,yes)	fV_ignore=" - IGNORED" fV_ret=0 ;;
	if test "x$4" = x--op; then
		fV_status=`$PROGDIR/ldaperror "$1"`
		fV_expect=`$PROGDIR/ldaperror "$3"`
		case $1,$2,$3 in
		  *,-eq,0) fV_msg="failed ($fV_status)";;
		  *,-ne,0) fV_msg="should have failed";;
		  0,-eq,*) fV_msg="should have failed with $fV_expect";;
		  *,-eq,*) fV_msg="returned $fV_status, expected $fV_expect";;
		  *)       fV_msg="returned wrong result ($fV_status)";;
		echo "$5 $fV_msg!$fV_ignore" >&2
		echo "$4$fV_ignore" >&2
	test "x$5" = x--	&& shift 5 && eval "$@"
	test "$fV_ignore"	&& Kill_all
	test "$fV_ret" -eq 0	&& return 99
	return $fV_ret

# $fV_<uppercase> are internal state variables for the functions:
#   $fV_PIDNAME<pid> = textual process name of <pid> from Set_pidinfo.
#   $fV_PID<pid> = <pid> after Set_pidinfo <pid>, "" after Kill_named <pid>.
#   $fV_PIDVARS = $fV_PID<pid> of all Set_pidinfo pids since last Kill_all.
#   Thus, eval "x=\"$fV_PIDVARS\"" sets x=<all started and not killed pids>.
#   $fV_WAIT_START = index in the Wait_slapd loop to start from.
# $fV_<lowercase> are not used after a function call, but they are not locals.
# Note: $1..$<n> might not be locals either, function calls may clobber them.