Re: [PATCH 2/3] test: improve known broken tests support

diff --git a/80/14ee7c20dea1164756c13108c3e9302bb0ea36 b/80/14ee7c20dea1164756c13108c3e9302bb0ea36
new file mode 100644 (file)
index 0000000..c95a292
--- /dev/null
+++ b/80/14ee7c20dea1164756c13108c3e9302bb0ea36
@@ -0,0 +1,252 @@
+Return-Path: <amdragon@gmail.com>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+       by olra.theworths.org (Postfix) with ESMTP id 2845B431FD0\r
+       for <notmuch@notmuchmail.org>; Sun,  3 Jul 2011 20:42:16 -0700 (PDT)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.699\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.699 tagged_above=-999 required=5\r
+       tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001,\r
+       RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+       by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+       with ESMTP id WZsA8IDmIQsm for <notmuch@notmuchmail.org>;\r
+       Sun,  3 Jul 2011 20:42:15 -0700 (PDT)\r
+Received: from mail-qy0-f181.google.com (mail-qy0-f181.google.com\r
+       [209.85.216.181]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+       (No client certificate requested)\r
+       by olra.theworths.org (Postfix) with ESMTPS id ECE64431FB6\r
+       for <notmuch@notmuchmail.org>; Sun,  3 Jul 2011 20:42:14 -0700 (PDT)\r
+Received: by qyk9 with SMTP id 9so3224310qyk.5\r
+       for <notmuch@notmuchmail.org>; Sun, 03 Jul 2011 20:42:14 -0700 (PDT)\r
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;\r
+       h=mime-version:sender:in-reply-to:references:date\r
+       :x-google-sender-auth:message-id:subject:from:to:cc:content-type\r
+       :content-transfer-encoding;\r
+       bh=W6ge2dTNoDe1XlDUS23AIq8QGQnEUaX2+hdKL8Zl4iQ=;\r
+       b=UhhtF7WPli4Bn9jRt1byNTSh5auQYpcyUAS25Y4rF6vvp21EAzTXRk9i/L4xoQ5de0\r
+       f964nXGd8k+xrOLVC3XbtuzcSvsW1zZhLtzsDhaZOKJTprX0CjbzZulT4ovffidEcgme\r
+       Udxcc8YRmynYahb2Nk+5xHCm3HM8vLwoA55S8=\r
+MIME-Version: 1.0\r
+Received: by 10.224.188.146 with SMTP id da18mr4523195qab.124.1309750934019;\r
+       Sun, 03 Jul 2011 20:42:14 -0700 (PDT)\r
+Sender: amdragon@gmail.com\r
+Received: by 10.229.229.211 with HTTP; Sun, 3 Jul 2011 20:42:13 -0700 (PDT)\r
+In-Reply-To: <1309744743-8556-2-git-send-email-dmitry.kurochkin@gmail.com>\r
+References: <1309744743-8556-1-git-send-email-dmitry.kurochkin@gmail.com>\r
+       <1309744743-8556-2-git-send-email-dmitry.kurochkin@gmail.com>\r
+Date: Sun, 3 Jul 2011 23:42:13 -0400\r
+X-Google-Sender-Auth: 7_W2JV2V_d4bwMTBKzqUavpscWQ\r
+Message-ID:\r
+ <CAH-f9WuKrBpEjBaBet0d95oHXo1kWzGv6oa7zz7Zqe4JK=4N+Q@mail.gmail.com>\r
+Subject: Re: [PATCH 2/3] test: improve known broken tests support\r
+From: Austin Clements <amdragon@mit.edu>\r
+To: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>\r
+Content-Type: text/plain; charset=ISO-8859-1\r
+Content-Transfer-Encoding: quoted-printable\r
+Cc: notmuch@notmuchmail.org\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+       <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+       <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 04 Jul 2011 03:42:16 -0000\r
+\r
+Great idea!\r
+\r
+Just a heads-up, this will conflict with the first two patches in the\r
+atomicity series (though git may not detect the second conflict).\r
+Both are trivial patches, though, so the merge should be easy.\r
+\r
+Three minor comments below.\r
+\r
+On Sun, Jul 3, 2011 at 9:59 PM, Dmitry Kurochkin\r
+<dmitry.kurochkin@gmail.com> wrote:\r
+> There is existing support for broken tests. =A0But it is not convenient\r
+> to use. =A0The primary issue is that we have to maintain a set of\r
+> test_expect_*_failure functions which are equivalent to the normal\r
+> test_expect_* counterparts except for what functions are called for\r
+> result reporting. =A0The patch adds test_subtest_known_broken function\r
+> which marks a subset as broken, making the normal test_expect_*\r
+> functions behave as test_expect_*_failure. =A0All test_expect_*_failure\r
+> functions are removed. =A0Test_known_broken_failure_ is changed to\r
+> format details the same way as test_failure_ does.\r
+>\r
+> Another benefit of this change is that the diff when a broken test is\r
+> fixed would be small and nice.\r
+>\r
+> Documentation is updated accordingly.\r
+> ---\r
+> =A0test/README =A0 =A0 =A0| =A0 17 ++++++++---------\r
+> =A0test/test-lib.sh | =A0 53 +++++++++++++++-----------------------------=\r
+---------\r
+> =A02 files changed, 23 insertions(+), 47 deletions(-)\r
+>\r
+> diff --git a/test/README b/test/README\r
+> index a245bf1..f926b9f 100644\r
+> --- a/test/README\r
+> +++ b/test/README\r
+> @@ -132,20 +132,19 @@ library for your script to use.\r
+> =A0 =A0<script>. =A0If it yields success, test is considered\r
+> =A0 =A0successful. =A0<message> should state what it is testing.\r
+>\r
+> - test_expect_failure <message> <script>\r
+> -\r
+> - =A0 This is NOT the opposite of test_expect_success, but is used\r
+> - =A0 to mark a test that demonstrates a known breakage. =A0Unlike\r
+> - =A0 the usual test_expect_success tests, which say "ok" on\r
+> - =A0 success and "FAIL" on failure, this will say "FIXED" on\r
+> - =A0 success and "still broken" on failure. =A0Failures from these\r
+> - =A0 tests won't cause -i (immediate) to stop.\r
+> -\r
+> =A0test_begin_subtest <message>\r
+>\r
+> =A0 =A0Set the test description message for a subsequent test_expect_equa=\r
+l\r
+> =A0 =A0invocation (see below).\r
+>\r
+> + test_subtest_known_broken\r
+> +\r
+> + =A0 Mark the current test as broken. =A0Such tests are expected to fail=\r
+.\r
+> + =A0 Unlike the normal tests, which say "PASS" on success and "FAIL" on\r
+> + =A0 failure, these will say "FIXED" on success and "BROKEN" on failure.\r
+> + =A0 Failures from these tests won't cause -i (immediate) to stop. =A0Th=\r
+is\r
+> + =A0 must be called before any test_expect_* function.\r
+\r
+Perhaps "A test must call this before any test_expect_* function"?\r
+There can be earlier test_expect_* calls from other tests.\r
+\r
+> +\r
+> =A0test_expect_equal <output> <expected>\r
+>\r
+> =A0 =A0This is an often-used convenience function built on top of\r
+> diff --git a/test/test-lib.sh b/test/test-lib.sh\r
+> index 22e387e..0cd4170 100755\r
+> --- a/test/test-lib.sh\r
+> +++ b/test/test-lib.sh\r
+> @@ -424,6 +424,7 @@ test_begin_subtest ()\r
+> =A0 =A0 =A0 =A0error "bug in test script: Missing test_expect_equal in ${=\r
+BASH_SOURCE[1]}:${BASH_LINENO[0]}"\r
+> =A0 =A0 fi\r
+> =A0 =A0 test_subtest_name=3D"$1"\r
+> + =A0 =A0test_subtest_known_broken_=3D\r
+> =A0 =A0 # Remember stdout and stderr file descriptors and redirect test\r
+> =A0 =A0 # output to the previously prepared file descriptors 3 and 4 (see\r
+> =A0 =A0 # below)\r
+> @@ -484,29 +485,6 @@ test_expect_equal_file ()\r
+> =A0 =A0 fi\r
+> =A0}\r
+>\r
+> -test_expect_equal_failure ()\r
+> -{\r
+> - =A0 =A0 =A0 exec 1>&6 2>&7 =A0 =A0 =A0 =A0 =A0# Restore stdout and stde=\r
+rr\r
+> - =A0 =A0 =A0 inside_subtest=3D\r
+> - =A0 =A0 =A0 test "$#" =3D 3 && { prereq=3D$1; shift; } || prereq=3D\r
+> - =A0 =A0 =A0 test "$#" =3D 2 ||\r
+> - =A0 =A0 =A0 error "bug in the test script: not 2 or 3 parameters to tes=\r
+t_expect_equal"\r
+> -\r
+> - =A0 =A0 =A0 output=3D"$1"\r
+> - =A0 =A0 =A0 expected=3D"$2"\r
+> - =A0 =A0 =A0 if ! test_skip "$@"\r
+> - =A0 =A0 =A0 then\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 if [ "$output" =3D "$expected" ]; then\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_ok_ "$tes=\r
+t_subtest_name"\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 else\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_failure_ =\r
+"$test_subtest_name"\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 testname=3D$this_test.$test=\r
+_count\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 echo "$expected" > $testnam=\r
+e.expected\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 echo "$output" > $testname.=\r
+output\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 fi\r
+> - =A0 =A0fi\r
+> -}\r
+> -\r
+> =A0NOTMUCH_NEW ()\r
+> =A0{\r
+> =A0 =A0 notmuch new | grep -v -E -e '^Processed [0-9]*( total)? file|Foun=\r
+d [0-9]* total file'\r
+> @@ -568,12 +546,20 @@ test_have_prereq () {\r
+> =A0# the text_expect_* functions instead.\r
+>\r
+> =A0test_ok_ () {\r
+> + =A0 =A0 =A0 if [ "$test_subtest_known_broken_" =3D 1 ]; then\r
+> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_ok_ "$@"\r
+> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return\r
+> + =A0 =A0 =A0 fi\r
+> =A0 =A0 =A0 =A0test_success=3D$(($test_success + 1))\r
+> =A0 =A0 =A0 =A0say_color pass "%-6s" "PASS"\r
+> =A0 =A0 =A0 =A0echo " $@"\r
+> =A0}\r
+>\r
+> =A0test_failure_ () {\r
+> + =A0 =A0 =A0 if [ "$test_subtest_known_broken_" =3D 1 ]; then\r
+> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_failure_ "$@"\r
+> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return\r
+> + =A0 =A0 =A0 fi\r
+> =A0 =A0 =A0 =A0test_failure=3D$(($test_failure + 1))\r
+> =A0 =A0 =A0 =A0say_color error "%-6s" "FAIL"\r
+> =A0 =A0 =A0 =A0echo " $1"\r
+> @@ -592,7 +578,10 @@ test_known_broken_ok_ () {\r
+> =A0test_known_broken_failure_ () {\r
+> =A0 =A0 =A0 =A0test_broken=3D$(($test_broken+1))\r
+> =A0 =A0 =A0 =A0say_color pass "%-6s" "BROKEN"\r
+> - =A0 =A0 =A0 echo " $@"\r
+> + =A0 =A0 =A0 echo " $1"\r
+> + =A0 =A0 =A0 shift\r
+> + =A0 =A0 =A0 echo "$@" | sed -e 's/^/ =A0 =A0 =A0 =A0/'\r
+> + =A0 =A0 =A0 if test "$verbose" !=3D "t"; then cat test.output; fi\r
+\r
+Would it be possible to keep just the one copy of the failure printing\r
+code in test_failure_?\r
+\r
+> =A0}\r
+>\r
+> =A0test_debug () {\r
+> @@ -636,20 +625,8 @@ test_skip () {\r
+> =A0 =A0 =A0 =A0esac\r
+> =A0}\r
+>\r
+> -test_expect_failure () {\r
+> - =A0 =A0 =A0 test "$#" =3D 3 && { prereq=3D$1; shift; } || prereq=3D\r
+> - =A0 =A0 =A0 test "$#" =3D 2 ||\r
+> - =A0 =A0 =A0 error "bug in the test script: not 2 or 3 parameters to tes=\r
+t-expect-failure"\r
+> - =A0 =A0 =A0 if ! test_skip "$@"\r
+> - =A0 =A0 =A0 then\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_run_ "$2"\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 if [ "$?" =3D 0 -a "$eval_ret" =3D 0 ]\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 then\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_ok_ "$1"\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 else\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 test_known_broken_failure_ =\r
+"$1"\r
+> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 fi\r
+> - =A0 =A0 =A0 fi\r
+> +test_subtest_known_broken () {\r
+> + =A0 =A0 =A0 test_subtest_known_broken_=3D1\r
+\r
+The convention in test-lib appears to be "t" for true variables.\r
+\r
+> =A0}\r
+>\r
+> =A0test_expect_success () {\r
+> --\r
+> 1.7.5.4\r