e6/f25a0e61e81915d9e522da0c9ed4298671c5d7

   1 Return-Path: <amdragon@mit.edu>\r
   2 X-Original-To: notmuch@notmuchmail.org\r
   3 Delivered-To: notmuch@notmuchmail.org\r
   4 Received: from localhost (localhost [127.0.0.1])\r
   5         by olra.theworths.org (Postfix) with ESMTP id DD988431FD7\r
   6         for <notmuch@notmuchmail.org>; Tue, 12 Nov 2013 15:03:08 -0800 (PST)\r
   7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
   8 X-Spam-Flag: NO\r
   9 X-Spam-Score: -0.7\r
  10 X-Spam-Level: \r
  11 X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
  12         tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
  13 Received: from olra.theworths.org ([127.0.0.1])\r
  14         by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
  15         with ESMTP id thGm7jeXgGLB for <notmuch@notmuchmail.org>;\r
  16         Tue, 12 Nov 2013 15:03:03 -0800 (PST)\r
  17 Received: from dmz-mailsec-scanner-1.mit.edu (dmz-mailsec-scanner-1.mit.edu\r
  18         [18.9.25.12])\r
  19         (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))\r
  20         (No client certificate requested)\r
  21         by olra.theworths.org (Postfix) with ESMTPS id E9A74431FCF\r
  22         for <notmuch@notmuchmail.org>; Tue, 12 Nov 2013 15:03:02 -0800 (PST)\r
  23 X-AuditID: 1209190c-b7f7f6d000000bbd-e8-5282b3a6c922\r
  24 Received: from mailhub-auth-3.mit.edu ( [18.9.21.43])\r
  25         (using TLS with cipher AES256-SHA (256/256 bits))\r
  26         (Client did not present a certificate)\r
  27         by dmz-mailsec-scanner-1.mit.edu (Symantec Messaging Gateway) with SMTP\r
  28         id 22.0B.03005.6A3B2825; Tue, 12 Nov 2013 18:03:02 -0500 (EST)\r
  29 Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11])\r
  30         by mailhub-auth-3.mit.edu (8.13.8/8.9.2) with ESMTP id rACN31Xb027243; \r
  31         Tue, 12 Nov 2013 18:03:01 -0500\r
  32 Received: from awakening.csail.mit.edu (awakening.csail.mit.edu [18.26.4.91])\r
  33         (authenticated bits=0)\r
  34         (User authenticated as amdragon@ATHENA.MIT.EDU)\r
  35         by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id rACN2xvS028631\r
  36         (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);\r
  37         Tue, 12 Nov 2013 18:03:00 -0500\r
  38 Received: from amthrax by awakening.csail.mit.edu with local (Exim 4.80)\r
  39         (envelope-from <amdragon@mit.edu>)\r
  40         id 1VgMz5-000784-AB; Tue, 12 Nov 2013 18:02:59 -0500\r
  41 Date: Tue, 12 Nov 2013 18:02:59 -0500\r
  42 From: Austin Clements <amdragon@MIT.EDU>\r
  43 To: Tomi Ollila <tomi.ollila@iki.fi>\r
  44 Subject: Re: [PATCH 3/3] test: implement and document NOTMUCH_TEST_QUIET\r
  45         variable usage\r
  46 Message-ID: <20131112230259.GD13399@mit.edu>\r
  47 References: <1384288868-23903-1-git-send-email-tomi.ollila@iki.fi>\r
  48         <1384288868-23903-3-git-send-email-tomi.ollila@iki.fi>\r
  49 MIME-Version: 1.0\r
  50 Content-Type: text/plain; charset=us-ascii\r
  51 Content-Disposition: inline\r
  52 In-Reply-To: <1384288868-23903-3-git-send-email-tomi.ollila@iki.fi>\r
  53 User-Agent: Mutt/1.5.21 (2010-09-15)\r
  54 X-Brightmail-Tracker:\r
  55  H4sIAAAAAAAAA+NgFmphleLIzCtJLcpLzFFi42IR4hTV1l22uSnI4OkLYYvrN2cyW7xZOY/V\r
  56         gcnj8NeFLB7PVt1iDmCK4rJJSc3JLEst0rdL4Mr42/iasWCXfsWWzauYGxh71LoYOTkkBEwk\r
  57         dn15wgxhi0lcuLeerYuRi0NIYDaTxNmpDewQzkZGiTdXzjFBOKeZJG7de8kC4SxhlPg+/SA7\r
  58         SD+LgKpEw9JbjCA2m4CGxLb9y8FsEQEViQdt61lBbGYBaYlvv5uZQGxhgUiJN3c3gu3mFdCR\r
  59         mDypmwXEFhKolpgw9SAjRFxQ4uTMJywQvVoSN/69BOrlAJuz/B8HSJhTwFli/qH3YOWiQKum\r
  60         nNzGNoFRaBaS7llIumchdC9gZF7FKJuSW6Wbm5iZU5yarFucnJiXl1qka6iXm1mil5pSuokR\r
  61         HNiSPDsY3xxUOsQowMGoxMP7ILIpSIg1say4MvcQoyQHk5Iob8RaoBBfUn5KZUZicUZ8UWlO\r
  62         avEhRgkOZiURXuk1QDnelMTKqtSifJiUNAeLkjjvTQ77ICGB9MSS1OzU1ILUIpisDAeHkgTv\r
  63         qU1AjYJFqempFWmZOSUIaSYOTpDhPEDDm0BqeIsLEnOLM9Mh8qcYFaXEeR+BJARAEhmleXC9\r
  64         sMTzilEc6BVh3h0gVTzApAXX/QpoMBPQYItisMEliQgpqQZG6xfN++aHXr29luNhi7bQJWmh\r
  65         9RErmIrZ7D52mvW/nJY3xZ1ly+QOSea7Beci1SVY79w/2po4OWext28fZ2yz8U5hr0l7PZ53\r
  66         Me5v+p4UIv5FPM2s5aLbkk1GLzXfvLAucAra9eXeCpHtpcUaMpbHl30RLzobvKZb+LvsvybX\r
  67         5QquTy91mT1QYinOSDTUYi4qTgQAI7vsExcDAAA=\r
  68 Cc: notmuch@notmuchmail.org\r
  69 X-BeenThere: notmuch@notmuchmail.org\r
  70 X-Mailman-Version: 2.1.13\r
  71 Precedence: list\r
  72 List-Id: "Use and development of the notmuch mail system."\r
  73         <notmuch.notmuchmail.org>\r
  74 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
  75         <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
  76 List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
  77 List-Post: <mailto:notmuch@notmuchmail.org>\r
  78 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
  79 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
  80         <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
  81 X-List-Received-Date: Tue, 12 Nov 2013 23:03:09 -0000\r
  82 \r
  83 Quoth Tomi Ollila on Nov 12 at 10:41 pm:\r
  84 > When NOTMUCH_TEST_QUIET environment variable is set to non-null value\r
  85 > messages when new test script starts and when test PASSes are disabled.\r
  86 > This eases picking the cases when tests FAIL (as those are still printed).\r
  87 > ---\r
  88 >  test/README                                      |  8 ++++++++\r
  89 >  test/basic                                       | 12 ++++++++++--\r
  90 >  test/test-lib.sh                                 | 11 ++++++++++-\r
  91 >  test/test.expected-output/test-quiet-verbose-no  | 20 ++++++++++++++++++++\r
  92 >  test/test.expected-output/test-quiet-verbose-yes | 24 ++++++++++++++++++++++++\r
  93 >  5 files changed, 72 insertions(+), 3 deletions(-)\r
  94 >  create mode 100644 test/test.expected-output/test-quiet-verbose-no\r
  95 >  create mode 100644 test/test.expected-output/test-quiet-verbose-yes\r
  96 > \r
  97 > diff --git a/test/README b/test/README\r
  98 > index d12cff2..79a9b1b 100644\r
  99 > --- a/test/README\r
 100 > +++ b/test/README\r
 101 > @@ -76,6 +76,14 @@ the tests in one of the following ways.\r
 102 >       TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient ./emacs\r
 103 >       make test TEST_EMACS=my-special-emacs TEST_EMACSCLIENT=my-emacsclient\r
 104 >  \r
 105 > +Quiet Execution\r
 106 > +---------------\r
 107 > +\r
 108 > +Normally, when new script starts and when test PASSes you get a message\r
 109 > +printed on screen. This printing can be disabled by setting the\r
 110 > +NOTMUCH_TEST_QUIET variable to a non-null value. Message on test\r
 111 > +failures and skips are still printed.\r
 112 > +\r
 113 >  Skipping Tests\r
 114 >  --------------\r
 115 >  If, for any reason, you need to skip one or more tests, you can do so\r
 116 > diff --git a/test/basic b/test/basic\r
 117 > index 64eb7d7..3b7668b 100755\r
 118 > --- a/test/basic\r
 119 > +++ b/test/basic\r
 120 > @@ -73,14 +73,22 @@ suppress_diff_date() {\r
 121 >       -e 's/\(.*\+\+\+ test-verbose\.4\.\output\).*/\1/'\r
 122 >  }\r
 123 >  \r
 124 > +if [ -z "$NOTMUCH_TEST_QUIET" ]\r
 125 > +then\r
 126 > +     test_verbose_no=$EXPECTED/test-verbose-no\r
 127 > +     test_verbose_yes=$EXPECTED/test-verbose-yes\r
 128 > +else\r
 129 > +     test_verbose_no=$EXPECTED/test-quiet-verbose-no\r
 130 > +     test_verbose_yes=$EXPECTED/test-quiet-verbose-yes\r
 131 > +fi\r
 132 >  test_begin_subtest "Ensure that test output is suppressed unless the test fails"\r
 133 >  output=$(cd $TEST_DIRECTORY; ./test-verbose 2>&1 | suppress_diff_date)\r
 134 > -expected=$(cat $EXPECTED/test-verbose-no | suppress_diff_date)\r
 135 > +expected=$(cat ${test_verbose_no} | suppress_diff_date)\r
 136 >  test_expect_equal "$output" "$expected"\r
 137 >  \r
 138 >  test_begin_subtest "Ensure that -v does not suppress test output"\r
 139 >  output=$(cd $TEST_DIRECTORY; ./test-verbose -v 2>&1 | suppress_diff_date)\r
 140 > -expected=$(cat $EXPECTED/test-verbose-yes | suppress_diff_date)\r
 141 > +expected=$(cat ${test_verbose_yes} | suppress_diff_date)\r
 142 >  # Do not include the results of test-verbose in totals\r
 143 >  rm $TEST_DIRECTORY/test-results/test-verbose\r
 144 >  rm -r $TEST_DIRECTORY/tmp.test-verbose\r
 145 > diff --git a/test/test-lib.sh b/test/test-lib.sh\r
 146 > index e022e46..4b342ac 100644\r
 147 > --- a/test/test-lib.sh\r
 148 > +++ b/test/test-lib.sh\r
 149 > @@ -195,7 +195,10 @@ print_test_description ()\r
 150 >       echo\r
 151 >       echo $this_test: "Testing ${test_description}"\r
 152 >  }\r
 153 > -print_test_description\r
 154 > +if [ -z "$NOTMUCH_TEST_QUIET" ]\r
 155 > +then\r
 156 > +     print_test_description\r
 157 > +fi\r
 158 >  \r
 159 >  exec 5>&1\r
 160 >  \r
 161 > @@ -703,6 +706,9 @@ test_ok_ () {\r
 162 >               return\r
 163 >       fi\r
 164 >       test_success=$(($test_success + 1))\r
 165 > +     if test -n "$NOTMUCH_TEST_QUIET"; then\r
 166 > +             return 0\r
 167 > +     fi\r
 168 >       say_color pass "%-6s" "PASS"\r
 169 >       echo " $test_subtest_name"\r
 170 >  }\r
 171 > @@ -713,6 +719,9 @@ test_failure_ () {\r
 172 >               return\r
 173 >       fi\r
 174 >       test_failure=$(($test_failure + 1))\r
 175 > +     if test -n "$NOTMUCH_TEST_QUIET"; then\r
 176 > +             print_test_description\r
 177 \r
 178 This prints the test description for *every* failing test.  Was that\r
 179 intentional?  I would think that, ideally, it would be only printed\r
 180 before the first failing subtest in a test (maybe by setting a\r
 181 variable in print_test_description on the first call and making it\r
 182 return immediately if this variable is set?  Then you wouldn't even\r
 183 need the condition here, just the call to print_test_description.)\r
 184 \r
 185 > +     fi\r
 186 >       test_failure_message_ "FAIL" "$test_subtest_name" "$@"\r
 187 >       test "$immediate" = "" || { GIT_EXIT_OK=t; exit 1; }\r
 188 >       return 1\r
 189 \r
 190 Stylistic nit: The three if's above use two different styles ([ vs\r
 191 test and hanging 'then').  OTOH, maybe this is consistent with\r
 192 test-lib's inconsistent style.\r
 193 \r
 194 > diff --git a/test/test.expected-output/test-quiet-verbose-no b/test/test.expected-output/test-quiet-verbose-no\r
 195 > new file mode 100644\r
 196 > index 0000000..74840b9\r
 197 > --- /dev/null\r
 198 > +++ b/test/test.expected-output/test-quiet-verbose-no\r
 199 > @@ -0,0 +1,20 @@\r
 200 > +\r
 201 > +test-verbose: Testing the verbosity options of the test framework itself.\r
 202 > + FAIL   print something in test_expect_success and fail\r
 203 > +     \r
 204 > +       echo "hello stdout" &&\r
 205 > +       echo "hello stderr" >&2 &&\r
 206 > +       false\r
 207 > +     \r
 208 > +hello stdout\r
 209 > +hello stderr\r
 210 > +\r
 211 > +test-verbose: Testing the verbosity options of the test framework itself.\r
 212 > + FAIL   print something test_begin_subtest and test_expect_equal and fail\r
 213 > +     --- test-verbose.4.expected\r
 214 > +     +++ test-verbose.4.output\r
 215 > +     @@ -1 +1 @@\r
 216 > +     -b\r
 217 > +     +a\r
 218 > +hello stdout\r
 219 > +hello stderr\r
 220 > diff --git a/test/test.expected-output/test-quiet-verbose-yes b/test/test.expected-output/test-quiet-verbose-yes\r
 221 > new file mode 100644\r
 222 > index 0000000..51e759d\r
 223 > --- /dev/null\r
 224 > +++ b/test/test.expected-output/test-quiet-verbose-yes\r
 225 > @@ -0,0 +1,24 @@\r
 226 > +hello stdout\r
 227 > +hello stderr\r
 228 > +hello stdout\r
 229 > +hello stderr\r
 230 > +\r
 231 > +test-verbose: Testing the verbosity options of the test framework itself.\r
 232 > + FAIL   print something in test_expect_success and fail\r
 233 > +     \r
 234 > +       echo "hello stdout" &&\r
 235 > +       echo "hello stderr" >&2 &&\r
 236 > +       false\r
 237 > +     \r
 238 > +hello stdout\r
 239 > +hello stderr\r
 240 > +hello stdout\r
 241 > +hello stderr\r
 242 > +\r
 243 > +test-verbose: Testing the verbosity options of the test framework itself.\r
 244 > + FAIL   print something test_begin_subtest and test_expect_equal and fail\r
 245 > +     --- test-verbose.4.expected\r
 246 > +     +++ test-verbose.4.output\r
 247 > +     @@ -1 +1 @@\r
 248 > +     -b\r
 249 > +     +a\r