[PATCH v2] test: exit with nonzero value when not all tests completed successfully

diff --git a/3c/a4abdb898a148226514197596cb3a6bf587ded b/3c/a4abdb898a148226514197596cb3a6bf587ded
new file mode 100644 (file)
index 0000000..cdb7948
--- /dev/null
+++ b/3c/a4abdb898a148226514197596cb3a6bf587ded
@@ -0,0 +1,100 @@
+Return-Path: <too@guru-group.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 51757431FBC\r
+       for <notmuch@notmuchmail.org>; Sun,  8 Sep 2013 08:53:45 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+       autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id HabxIQnpw6mA for <notmuch@notmuchmail.org>;\r
+       Sun,  8 Sep 2013 08:53:38 -0700 (PDT)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+       by olra.theworths.org (Postfix) with ESMTP id AC749431FAE\r
+       for <notmuch@notmuchmail.org>; Sun,  8 Sep 2013 08:53:38 -0700 (PDT)\r
+Received: by guru.guru-group.fi (Postfix, from userid 501)\r
+       id 85213100093; Sun,  8 Sep 2013 18:53:32 +0300 (EEST)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: notmuch@notmuchmail.org\r
+Subject: [PATCH v2] test: exit with nonzero value when not all tests completed\r
+       successfully\r
+Date: Sun,  8 Sep 2013 18:53:30 +0300\r
+Message-Id: <1378655610-21497-1-git-send-email-tomi.ollila@iki.fi>\r
+X-Mailer: git-send-email 1.8.0\r
+Cc: tomi.ollila@iki.fi\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Sun, 08 Sep 2013 15:53:45 -0000\r
+\r
+If any of the tests in our test system is not passing the execution\r
+of the test suite completes with nonzero exit value.\r
+\r
+It is better to rely on the exit value of the test system instead\r
+of some arbitrary strings in test output (or use both).\r
+---\r
+\r
+This is v2 of id:1378479935-26312-1-git-send-email-tomi.ollila@iki.fi\r
+thanks to Mark's observation in id:87d2olg9bz.fsf@qmul.ac.uk\r
+which widened my thought region :D\r
+\r
+Tomi\r
+\r
+ test/aggregate-results.sh | 9 +++++++++\r
+ test/notmuch-test         | 3 +++\r
+ 2 files changed, 12 insertions(+)\r
+\r
+diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh\r
+index 732d6ca..b016edb 100755\r
+--- a/test/aggregate-results.sh\r
++++ b/test/aggregate-results.sh\r
+@@ -1,5 +1,7 @@\r
+ #!/usr/bin/env bash\r
+ \r
++set -eu\r
++\r
+ fixed=0\r
+ success=0\r
+ failed=0\r
+@@ -79,3 +81,10 @@ if [ "$skipped" != "0" ]; then\r
+     tests=$(pluralize "test" $skipped)\r
+     echo "$skipped $tests skipped."\r
+ fi\r
++\r
++if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ]\r
++then\r
++    exit 0\r
++else\r
++    exit 1\r
++fi\r
+diff --git a/test/notmuch-test b/test/notmuch-test\r
+index 6db7979..aa28bb0 100755\r
+--- a/test/notmuch-test\r
++++ b/test/notmuch-test\r
+@@ -97,6 +97,9 @@ trap - HUP INT TERM\r
+ \r
+ # Report results\r
+ ./aggregate-results.sh test-results/*\r
++ev=$?\r
+ \r
+ # Clean up\r
+ rm -rf test-results corpus.mail\r
++\r
++exit $ev\r
+-- \r
+1.8.0\r
+\r