test suite: don't consider skipped individual tests as failing

It isn't completely clear what we want to do here, but

1) We currently don't fail if we skip a whole test file (mainly because
we neglect to count those skipped tests properly). This change at least
makes the two kinds of skipping consistent.

2) Automated build environments may have good reasons for building with
a minimal set of prereqs, and we don't want to discourage running our
test suite by breaking builds.

diff --git a/test/README b/test/README
index ce403cef298dd8aaeac60ab94b49a029d7839ed4..e54e36b795e852e536dab96256420edbfee84368 100644 (file)
--- a/test/README
+++ b/test/README
@@ -117,6 +117,13 @@ Note that some tests in the existing test suite rely on previous test
 items, so you cannot arbitrarily skip any test and expect the
 remaining tests to be unaffected.
 
+Currently we do not consider skipped tests as build failures. For
+maximum robustness, when setting up automated build processes, you
+should explicitely skip tests, rather than relying on notmuch's
+detection of missing prerequisites. In the future we may treat tests
+unable to run because of missing prerequisites, but not explicitely
+skipped by the user, as failures.
+
 Writing Tests
 -------------
 The test script is written as a shell script.  It should start with

diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh
index b016edb9117108523642b17862b20846f4fbc89c..6322854675fdae93cabbf55ac9a7cb531513b00d 100755 (executable)
--- a/test/aggregate-results.sh
+++ b/test/aggregate-results.sh
@@ -82,7 +82,10 @@ if [ "$skipped" != "0" ]; then
     echo "$skipped $tests skipped."
 fi
 
-if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]
+# Note that we currently do not consider skipped tests as failing the
+# build.
+
+if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ]
 then
     exit 0
 else