From: Austin Clements Date: Wed, 4 Dec 2013 20:15:07 +0000 (+1900) Subject: Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=9a2c43422bd6a3c976d2140ffe1f21feaf69cdc9;p=notmuch-archives.git Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage --- diff --git a/ac/7e103d438da3295e761b2d2efe534170f1958e b/ac/7e103d438da3295e761b2d2efe534170f1958e new file mode 100644 index 000000000..bc304ddac --- /dev/null +++ b/ac/7e103d438da3295e761b2d2efe534170f1958e @@ -0,0 +1,204 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 20147431FD2 + for ; Wed, 4 Dec 2013 12:15:21 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id yTSzDuYJ03tz for ; + Wed, 4 Dec 2013 12:15:13 -0800 (PST) +Received: from dmz-mailsec-scanner-4.mit.edu (dmz-mailsec-scanner-4.mit.edu + [18.9.25.15]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 1F89D431FD0 + for ; Wed, 4 Dec 2013 12:15:13 -0800 (PST) +X-AuditID: 1209190f-b7fb86d000000c36-e5-529f8d5014b2 +Received: from mailhub-auth-4.mit.edu ( [18.7.62.39]) + (using TLS with cipher AES256-SHA (256/256 bits)) + (Client did not present a certificate) + by dmz-mailsec-scanner-4.mit.edu (Symantec Messaging Gateway) with SMTP + id C5.4B.03126.05D8F925; Wed, 4 Dec 2013 15:15:12 -0500 (EST) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id rB4KFAb2009161; + Wed, 4 Dec 2013 15:15:11 -0500 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id rB4KF7u4012916 + (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); + Wed, 4 Dec 2013 15:15:10 -0500 +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) + (envelope-from ) + id 1VoIqh-0001pe-DK; Wed, 04 Dec 2013 15:15:07 -0500 +From: Austin Clements +To: Tomi Ollila , notmuch@notmuchmail.org +Subject: Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET + variable usage +In-Reply-To: <87vbz437yu.fsf@awakening.csail.mit.edu> +References: <1384288868-23903-1-git-send-email-tomi.ollila@iki.fi> + <1385399299-12936-1-git-send-email-tomi.ollila@iki.fi> + <1385399299-12936-3-git-send-email-tomi.ollila@iki.fi> + <87vbz437yu.fsf@awakening.csail.mit.edu> +User-Agent: Notmuch/0.16+154~g96c0ce2 (http://notmuchmail.org) Emacs/23.4.1 + (i486-pc-linux-gnu) +Date: Wed, 04 Dec 2013 15:15:07 -0500 +Message-ID: <87siu82x10.fsf@awakening.csail.mit.edu> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFnrNIsWRmVeSWpSXmKPExsUixG6nrhvQOz/IYN02DovrN2cyW7xZOY/V + gcnj8NeFLB7PVt1iDmCK4rJJSc3JLEst0rdL4Mp42bOcseCMcsXGewoNjF9kuhg5OSQETCRO + 3vvGDmGLSVy4t56ti5GLQ0hgNpPE/onrmCCcDYwSGxfvhcqcYpI48P88VGYJo8SKwxvA+tkE + NCS27V/OCGKLCNhKXFlykrWLkYODGWhu+7wQkLCwQLjE0r7jYCWcAqYSvSvXsoDYQgI3GSX+ + rdQBsUUF4iWmLdzJDGKzCKhK/Nz2khlkDC/QqT9uxIOEeQUEJU7OfALWyiygJXHj30umCYyC + s5CkZiFJLWBkWsUom5JbpZubmJlTnJqsW5ycmJeXWqRropebWaKXmlK6iREUppyS/DsYvx1U + OsQowMGoxMPrkDI/SIg1say4MvcQoyQHk5Ior349UIgvKT+lMiOxOCO+qDQntfgQowQHs5II + 778aoBxvSmJlVWpRPkxKmoNFSZz3Jod9kJBAemJJanZqakFqEUxWhoNDSYI3sQeoUbAoNT21 + Ii0zpwQhzcTBCTKcB2j4qW6Q4cUFibnFmekQ+VOMilLivHUgzQIgiYzSPLheWBp5xSgO9Iow + bytIFQ8wBcF1vwIazAQ0uPnBPJDBJYkIKakGxgBFwW/2wmu6nx59HbysIX2L0owvMw2udyR1 + R/mIN+YEb+mc1PVU7vbfQ1ffB+SnPKl7EmpSylNxY5kmd8KbuLPde89Pnm85Z9Gsx9nfedba + 8TQ5/aq03DV9i27lZjt90zeJySc/r5ew9ZlmdjF16q4TyYtki+IZxFa+UhKxdy77u+7zxris + vUosxRmJhlrMRcWJAMA6etH+AgAA +Cc: tomi.ollila@iki.fi +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Wed, 04 Dec 2013 20:15:21 -0000 + +On Wed, 04 Dec 2013, Austin Clements wrote: +> I just tried to use this and realized it hadn't been pushed yet. +> +> This series LGTM except one minor nit below and the fact that it +> introduces a lot of tab-indented code in sections of test-lib.sh that +> appear to be space-indented. Given that test-lib.sh is already a mess +> of indentation styles, I don't know if we care, but it would be nice if +> its entropy were at least non-increasing. + +Ignore this comment. Apparently I was confused by show-mode +transforming leading tabs into spaces. + +> On Mon, 25 Nov 2013, Tomi Ollila wrote: +>> When NOTMUCH_TEST_QUIET environment variable is set to non-null value +>> messages when new test script starts and when test PASSes are disabled. +>> This eases picking the cases when tests FAIL (as those are still printed). +>> --- +>> test/README | 8 ++++++++ +>> test/basic | 4 ++-- +>> test/test-lib.sh | 11 ++++++++++- +>> 3 files changed, 20 insertions(+), 3 deletions(-) +>> +>> diff --git a/test/README b/test/README +>> index d12cff2..79a9b1b 100644 +>> --- a/test/README +>> +++ b/test/README +>> @@ -74,10 +74,18 @@ the tests in one of the following ways. +>> +>> TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient make test +>> TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs +>> make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient +>> +>> +Quiet Execution +>> +--------------- +>> + +>> +Normally, when new script starts and when test PASSes you get a message +>> +printed on screen. This printing can be disabled by setting the +>> +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test +>> +failures and skips are still printed. +>> + +>> Skipping Tests +>> -------------- +>> If, for any reason, you need to skip one or more tests, you can do so +>> by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more +>> sections of tests. +>> diff --git a/test/basic b/test/basic +>> index 64eb7d7..f7eed32 100755 +>> --- a/test/basic +>> +++ b/test/basic +>> @@ -72,16 +72,16 @@ suppress_diff_date() { +>> sed -e 's/\(.*\-\-\- test-verbose\.4\.\expected\).*/\1/' \ +>> -e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/' +>> } +>> +>> test_begin_subtest "Ensure that test output is suppressed unless the test fails" +>> -output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date) +>> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose 2>&1 | suppress_diff_date) +>> expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date) +>> test_expect_equal "$output" "$expected" +>> +>> test_begin_subtest "Ensure that -v does not suppress test output" +>> -output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date) +>> +output=$(cd $TEST_DIRECTORY; NOTMUCH_TEST_QUIET= ./test-verbose -v 2>&1 | suppress_diff_date) +>> expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date) +>> # Do not include the results of test-verbose in totals +>> rm $TEST_DIRECTORY/test-results/test-verbose +>> rm -r $TEST_DIRECTORY/tmp.test-verbose +>> test_expect_equal "$output" "$expected" +>> diff --git a/test/test-lib.sh b/test/test-lib.sh +>> index 34e0db6..9d4a807 100644 +>> --- a/test/test-lib.sh +>> +++ b/test/test-lib.sh +>> @@ -196,11 +196,14 @@ print_test_description () +>> test -z "$test_description_printed" || return 0 +>> echo +>> echo $this_test: "Testing ${test_description}" +>> test_description_printed=1 +>> } +>> -print_test_description +>> +if [ -z "$NOTMUCH_TEST_QUIET" ] +>> +then +>> + print_test_description +>> +fi +>> +>> exec 5>&1 +>> +>> test_failure=0 +>> test_count=0 +>> @@ -715,20 +718,26 @@ test_ok_ () { +>> if test "$test_subtest_known_broken_" = "t"; then +>> test_known_broken_ok_ +>> return +>> fi +>> test_success=$(($test_success + 1)) +>> + if test -n "$NOTMUCH_TEST_QUIET"; then +>> + return 0 +>> + fi +>> say_color pass "%-6s" "PASS" +>> echo " $test_subtest_name" +>> } +>> +>> test_failure_ () { +>> if test "$test_subtest_known_broken_" = "t"; then +>> test_known_broken_failure_ "$@" +>> return +>> fi +>> test_failure=$(($test_failure + 1)) +>> + if test -n "$NOTMUCH_TEST_QUIET"; then +> +> Strictly speaking, this test isn't necessary, right? +> +>> + print_test_description +>> + fi +>> test_failure_message_ "FAIL" "$test_subtest_name" "$@" +>> test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; } +>> return 1 +>> } +>> +>> -- +>> 1.8.4.2