From: Austin Clements Date: Tue, 12 Nov 2013 23:02:59 +0000 (+1900) Subject: Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage X-Git-Url: http://git.tremily.us/?a=commitdiff_plain;h=c446b8d3cc5e6a945b7d55e88a653cc90c778f93;p=notmuch-archives.git Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET variable usage --- diff --git a/e6/f25a0e61e81915d9e522da0c9ed4298671c5d7 b/e6/f25a0e61e81915d9e522da0c9ed4298671c5d7 new file mode 100644 index 000000000..90ebaf277 --- /dev/null +++ b/e6/f25a0e61e81915d9e522da0c9ed4298671c5d7 @@ -0,0 +1,249 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id DD988431FD7 + for ; Tue, 12 Nov 2013 15:03:08 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id thGm7jeXgGLB for ; + Tue, 12 Nov 2013 15:03:03 -0800 (PST) +Received: from dmz-mailsec-scanner-1.mit.edu (dmz-mailsec-scanner-1.mit.edu + [18.9.25.12]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id E9A74431FCF + for ; Tue, 12 Nov 2013 15:03:02 -0800 (PST) +X-AuditID: 1209190c-b7f7f6d000000bbd-e8-5282b3a6c922 +Received: from mailhub-auth-3.mit.edu ( [18.9.21.43]) + (using TLS with cipher AES256-SHA (256/256 bits)) + (Client did not present a certificate) + by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP + id 22.0B.03005.6A3B2825; Tue, 12 Nov 2013 18:03:02 -0500 (EST) +Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) + by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id rACN31Xb027243; + Tue, 12 Nov 2013 18:03:01 -0500 +Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91]) + (authenticated bits=0) + (User authenticated as amdragon@ATHENA.MIT.EDU) + by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id rACN2xvS028631 + (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); + Tue, 12 Nov 2013 18:03:00 -0500 +Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80) + (envelope-from ) + id 1VgMz5-000784-AB; Tue, 12 Nov 2013 18:02:59 -0500 +Date: Tue, 12 Nov 2013 18:02:59 -0500 +From: Austin Clements +To: Tomi Ollila +Subject: Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET + variable usage +Message-ID: <20131112230259.GD13399@mit.edu> +References: <1384288868-23903-1-git-send-email-tomi.ollila@iki.fi> + <1384288868-23903-3-git-send-email-tomi.ollila@iki.fi> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +Content-Disposition: inline +In-Reply-To: <1384288868-23903-3-git-send-email-tomi.ollila@iki.fi> +User-Agent: Mutt/1.5.21 (2010-09-15) +X-Brightmail-Tracker: + H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IR4hTV1l22uSnI4OkLYYvrN2cyW7xZOY/V + gcnj8NeFLB7PVt1iDmCK4rJJSc3JLEst0rdL4Mr42/iasWCXfsWWzauYGxh71LoYOTkkBEwk + dn15wgxhi0lcuLeerYuRi0NIYDaTxNmpDewQzkZGiTdXzjFBOKeZJG7de8kC4SxhlPg+/SA7 + SD+LgKpEw9JbjCA2m4CGxLb9y8FsEQEViQdt61lBbGYBaYlvv5uZQGxhgUiJN3c3gu3mFdCR + mDypmwXEFhKolpgw9SAjRFxQ4uTMJywQvVoSN/69BOrlAJuz/B8HSJhTwFli/qH3YOWiQKum + nNzGNoFRaBaS7llIumchdC9gZF7FKJuSW6Wbm5iZU5yarFucnJiXl1qka6iXm1mil5pSuokR + HNiSPDsY3xxUOsQowMGoxMP7ILIpSIg1say4MvcQoyQHk5Iob8RaoBBfUn5KZUZicUZ8UWlO + avEhRgkOZiURXuk1QDnelMTKqtSifJiUNAeLkjjvTQ77ICGB9MSS1OzU1ILUIpisDAeHkgTv + qU1AjYJFqempFWmZOSUIaSYOTpDhPEDDm0BqeIsLEnOLM9Mh8qcYFaXEeR+BJARAEhmleXC9 + sMTzilEc6BVh3h0gVTzApAXX/QpoMBPQYItisMEliQgpqQZG6xfN++aHXr29luNhi7bQJWmh + 9RErmIrZ7D52mvW/nJY3xZ1ly+QOSea7Beci1SVY79w/2po4OWext28fZ2yz8U5hr0l7PZ53 + Me5v+p4UIv5FPM2s5aLbkk1GLzXfvLAucAra9eXeCpHtpcUaMpbHl30RLzobvKZb+LvsvybX + 5QquTy91mT1QYinOSDTUYi4qTgQAI7vsExcDAAA= +Cc: notmuch@notmuchmail.org +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 12 Nov 2013 23:03:09 -0000 + +Quoth Tomi Ollila on Nov 12 at 10:41 pm: +> When NOTMUCH_TEST_QUIET environment variable is set to non-null value +> messages when new test script starts and when test PASSes are disabled. +> This eases picking the cases when tests FAIL (as those are still printed). +> --- +> test/README | 8 ++++++++ +> test/basic | 12 ++++++++++-- +> test/test-lib.sh | 11 ++++++++++- +> test/test.expected-output/test-quiet-verbose-no | 20 ++++++++++++++++++++ +> test/test.expected-output/test-quiet-verbose-yes | 24 ++++++++++++++++++++++++ +> 5 files changed, 72 insertions(+), 3 deletions(-) +> create mode 100644 test/test.expected-output/test-quiet-verbose-no +> create mode 100644 test/test.expected-output/test-quiet-verbose-yes +> +> diff --git a/test/README b/test/README +> index d12cff2..79a9b1b 100644 +> --- a/test/README +> +++ b/test/README +> @@ -76,6 +76,14 @@ the tests in one of the following ways. +> TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs +> make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient +> +> +Quiet Execution +> +--------------- +> + +> +Normally, when new script starts and when test PASSes you get a message +> +printed on screen. This printing can be disabled by setting the +> +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test +> +failures and skips are still printed. +> + +> Skipping Tests +> -------------- +> If, for any reason, you need to skip one or more tests, you can do so +> diff --git a/test/basic b/test/basic +> index 64eb7d7..3b7668b 100755 +> --- a/test/basic +> +++ b/test/basic +> @@ -73,14 +73,22 @@ suppress_diff_date() { +> -e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/' +> } +> +> +if [ -z "$NOTMUCH_TEST_QUIET" ] +> +then +> + test_verbose_no=$EXPECTED/test-verbose-no +> + test_verbose_yes=$EXPECTED/test-verbose-yes +> +else +> + test_verbose_no=$EXPECTED/test-quiet-verbose-no +> + test_verbose_yes=$EXPECTED/test-quiet-verbose-yes +> +fi +> test_begin_subtest "Ensure that test output is suppressed unless the test fails" +> output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date) +> -expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date) +> +expected=$(cat ${test_verbose_no} | suppress_diff_date) +> test_expect_equal "$output" "$expected" +> +> test_begin_subtest "Ensure that -v does not suppress test output" +> output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date) +> -expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date) +> +expected=$(cat ${test_verbose_yes} | suppress_diff_date) +> # Do not include the results of test-verbose in totals +> rm $TEST_DIRECTORY/test-results/test-verbose +> rm -r $TEST_DIRECTORY/tmp.test-verbose +> diff --git a/test/test-lib.sh b/test/test-lib.sh +> index e022e46..4b342ac 100644 +> --- a/test/test-lib.sh +> +++ b/test/test-lib.sh +> @@ -195,7 +195,10 @@ print_test_description () +> echo +> echo $this_test: "Testing ${test_description}" +> } +> -print_test_description +> +if [ -z "$NOTMUCH_TEST_QUIET" ] +> +then +> + print_test_description +> +fi +> +> exec 5>&1 +> +> @@ -703,6 +706,9 @@ test_ok_ () { +> return +> fi +> test_success=$(($test_success + 1)) +> + if test -n "$NOTMUCH_TEST_QUIET"; then +> + return 0 +> + fi +> say_color pass "%-6s" "PASS" +> echo " $test_subtest_name" +> } +> @@ -713,6 +719,9 @@ test_failure_ () { +> return +> fi +> test_failure=$(($test_failure + 1)) +> + if test -n "$NOTMUCH_TEST_QUIET"; then +> + print_test_description + +This prints the test description for *every* failing test. Was that +intentional? I would think that, ideally, it would be only printed +before the first failing subtest in a test (maybe by setting a +variable in print_test_description on the first call and making it +return immediately if this variable is set? Then you wouldn't even +need the condition here, just the call to print_test_description.) + +> + fi +> test_failure_message_ "FAIL" "$test_subtest_name" "$@" +> test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; } +> return 1 + +Stylistic nit: The three if's above use two different styles ([ vs +test and hanging 'then'). OTOH, maybe this is consistent with +test-lib's inconsistent style. + +> diff --git a/test/test.expected-output/test-quiet-verbose-no b/test/test.expected-output/test-quiet-verbose-no +> new file mode 100644 +> index 0000000..74840b9 +> --- /dev/null +> +++ b/test/test.expected-output/test-quiet-verbose-no +> @@ -0,0 +1,20 @@ +> + +> +test-verbose: Testing the verbosity options of the test framework itself. +> + FAIL print something in test_expect_success and fail +> + +> + echo "hello stdout" && +> + echo "hello stderr" >&2 && +> + false +> + +> +hello stdout +> +hello stderr +> + +> +test-verbose: Testing the verbosity options of the test framework itself. +> + FAIL print something test_begin_subtest and test_expect_equal and fail +> + --- test-verbose.4.expected +> + +++ test-verbose.4.output +> + @@ -1 +1 @@ +> + -b +> + +a +> +hello stdout +> +hello stderr +> diff --git a/test/test.expected-output/test-quiet-verbose-yes b/test/test.expected-output/test-quiet-verbose-yes +> new file mode 100644 +> index 0000000..51e759d +> --- /dev/null +> +++ b/test/test.expected-output/test-quiet-verbose-yes +> @@ -0,0 +1,24 @@ +> +hello stdout +> +hello stderr +> +hello stdout +> +hello stderr +> + +> +test-verbose: Testing the verbosity options of the test framework itself. +> + FAIL print something in test_expect_success and fail +> + +> + echo "hello stdout" && +> + echo "hello stderr" >&2 && +> + false +> + +> +hello stdout +> +hello stderr +> +hello stdout +> +hello stderr +> + +> +test-verbose: Testing the verbosity options of the test framework itself. +> + FAIL print something test_begin_subtest and test_expect_equal and fail +> + --- test-verbose.4.expected +> + +++ test-verbose.4.output +> + @@ -1 +1 @@ +> + -b +> + +a