--- /dev/null
+Return-Path: <sojkam1@fel.cvut.cz>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id 6EC7F431FBD\r
+ for <notmuch@notmuchmail.org>; Mon, 8 Feb 2010 07:14:31 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -2.32\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-2.32 tagged_above=-999 required=5 tests=[AWL=0.279, \r
+ BAYES_00=-2.599] autolearn=ham\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id cbWCtjm0g--i for <notmuch@notmuchmail.org>;\r
+ Mon, 8 Feb 2010 07:14:30 -0800 (PST)\r
+Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36])\r
+ by olra.theworths.org (Postfix) with ESMTP id BCFC7431FBC\r
+ for <notmuch@notmuchmail.org>; Mon, 8 Feb 2010 07:14:29 -0800 (PST)\r
+Received: from localhost (unknown [192.168.200.4])\r
+ by max.feld.cvut.cz (Postfix) with ESMTP id 25A6A19F3373;\r
+ Mon, 8 Feb 2010 16:14:29 +0100 (CET)\r
+X-Virus-Scanned: IMAP AMAVIS\r
+Received: from max.feld.cvut.cz ([192.168.200.1])\r
+ by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new,\r
+ port 10044)\r
+ with ESMTP id df4GTmIgEi7A; Mon, 8 Feb 2010 16:14:24 +0100 (CET)\r
+Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34])\r
+ by max.feld.cvut.cz (Postfix) with ESMTP id B51B019F2F45;\r
+ Mon, 8 Feb 2010 16:14:24 +0100 (CET)\r
+Received: from steelpick.localnet (k335-30.felk.cvut.cz [147.32.86.30])\r
+ (Authenticated sender: sojkam1)\r
+ by imap.feld.cvut.cz (Postfix) with ESMTPSA id AEFEC15C052;\r
+ Mon, 8 Feb 2010 16:14:24 +0100 (CET)\r
+From: Michal Sojka <sojkam1@fel.cvut.cz>\r
+To: notmuch@notmuchmail.org\r
+Date: Mon, 8 Feb 2010 16:14:24 +0100\r
+User-Agent: KMail/1.12.4 (Linux/2.6.31.11-amd64; KDE/4.3.4; x86_64; ; )\r
+References: <87ljf8pvxx.fsf@yoom.home.cworth.org>\r
+ <87hbpwpoko.fsf@yoom.home.cworth.org>\r
+ <5641883d1002060727ia4e6c16lf800a92fc8735430@mail.gmail.com>\r
+In-Reply-To: <5641883d1002060727ia4e6c16lf800a92fc8735430@mail.gmail.com>\r
+MIME-Version: 1.0\r
+Content-Type: Text/Plain;\r
+ charset="iso-8859-1"\r
+Content-Transfer-Encoding: 7bit\r
+Message-Id: <201002081614.24284.sojkam1@fel.cvut.cz>\r
+Subject: Re: [notmuch] A functional (but rudimentary) test suite for notmuch\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Mon, 08 Feb 2010 15:14:31 -0000\r
+\r
+On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:\r
+> The test suite is still extremely rudimentary. Here are some things I'd\r
+> like to improve about it:\r
+\r
+I converted the actual version of notmuch-test to git test framework.\r
+The result is in the followup patches.\r
+\r
+I'd like to know opinion of others. If Carl agrees that it could be\r
+merged I can do additional work as I describe below.\r
+\r
+The conversion was not as straightforward as I expected mainly because\r
+of problems with quoting. There are several sources of quotation problems\r
+one being Carl's hashed array parameters. I thing it would be\r
+sufficient to use plain variables for passing additional parameters.\r
+Instead of:\r
+ add_message [from]="\"Sender <sender@example.com>\"" \\r
+ [to]=test_suite@notmuchmail.org \\r
+ [subject]=notmuch-reply-test \\r
+ [date]="\"Tue, 05 Jan 2010 15:43:56 -0800\"" \\r
+ [body]="\"basic reply test\"" &&\r
+I'd do:\r
+ (\r
+ msg_from="Sender <sender@example.com>"\r
+ msg_to=test_suite@notmuchmail.org \r
+ msg_subject=notmuch-reply-test\r
+ msg_date="Tue, 05 Jan 2010 15:43:56 -0800"\r
+ msg_body="basic reply test"\r
+ add_message\r
+ )\r
+\r
+A possible additional improvement is elimination of\r
+execute_expecting(). Combination of action (running notmuch) and\r
+testing of a result in a single function makes it hard to distinguish\r
+whether the problem is in the action or in the output. For example, if\r
+notmuch is killed because of segfault, it would look like that no\r
+output was produced. So instead of:\r
+\r
+ execute_expecting new "No new mail. Removed 3 messages."\r
+\r
+I'd write something like:\r
+\r
+ echo "No new mail. Removed 3 messages." > expected\r
+\r
+ test_expect_success 'Run notmuch' \r
+ 'notmuch_filter_output new > actual'\r
+ test_expect_success 'Compare actual and correct output' \r
+ 'test_cmp expected actual'\r
+\r
+where test_cmp is basically diff -u. This has also the advantage that\r
+longer output (e.g. as in t0003-reply.sh) is more readable if there is\r
+a difference.\r
+\r
+On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:\r
+> Carl, have you considered outputting the test suite in the same format\r
+> as the test anything protocol? [1] I only mention this because it\r
+> might be a nice way to easily do some reporting (or perhaps even\r
+> continuous integration) notmuch, with trivial effort.\r
+\r
+According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project \r
+Git test output does not conform to TAP, but from my quick look it could be \r
+easily changed to conform.\r
+\r
+\r
+On Friday 05 of February 2010 00:29:27 Carl Worth wrote:\r
+> Looking at TAP, one thing I don't like is that it prints the\r
+> success/failure of the test first, before the description of the\r
+> test. That's not so nice in the case of a long-running (perhaps\r
+> infinitely running) test where you might need to interrupt it, but you'd\r
+> still want to know *what* was running for so long.\r
+\r
+Git test framework also outputs first the result and then the test. It can be \r
+easily changed by -v switch.\r
+\r
+To conclude, the output of running make in test/ directory now looks like \r
+this:\r
+*** t0000-basic.sh ***\r
+* ok 1: test that mail store was created\r
+* ok 2: mail store should be empty\r
+* ok 3: NOTMUCH_CONFIG is set and points to an existing file\r
+* ok 4: PATH is set to this repository\r
+* ok 5: success is reported like this\r
+* still broken 6: pretend we have a known breakage\r
+* FIXED 7: pretend we have fixed a known breakage\r
+* ok 8: test runs if prerequisite is satisfied\r
+* skip 9: unmet prerequisite causes test to be skipped\r
+* fixed 1 known breakage(s)\r
+* still have 1 known breakage(s)\r
+* passed all remaining 8 test(s)\r
+*** t0001-new.sh ***\r
+* ok 1: No new messages\r
+* ok 2: Single new message\r
+* ok 3: Multiple new messages\r
+* ok 4: No new messages (non-empty DB)\r
+* ok 5: New directories\r
+* ok 6: Alternate inode order\r
+* ok 7: Message moved in\r
+* ok 8: Renamed message\r
+* ok 9: Deleted message\r
+* ok 10: Renamed directory\r
+* ok 11: Deleted directory\r
+* ok 12: New directory (at end of list)\r
+* ok 13: Deleted directory (end of list)\r
+* ok 14: New symlink to directory\r
+* ok 15: New symlink to a file\r
+* ok 16: New two-level directory\r
+* ok 17: Deleted two-level directory\r
+* passed all 17 test(s)\r
+*** t0002-search.sh ***\r
+* ok 1: Search body\r
+* ok 2: Search by from:\r
+* ok 3: Search by to:\r
+* ok 4: Search by subject:\r
+* ok 5: Search by id:\r
+* ok 6: Search by tag:\r
+* ok 7: Search by thread:\r
+* ok 8: Search body (phrase)\r
+* ok 9: Search by from: (address)\r
+* ok 10: Search by from: (name)\r
+* ok 11: Search by to: (address)\r
+* ok 12: Search by to: (name)\r
+* ok 13: Search by subject: (phrase)\r
+* passed all 13 test(s)\r
+*** t0003-reply.sh ***\r
+* ok 1: Basic reply\r
+* ok 2: Multiple recipients\r
+* ok 3: Reply with CC\r
+* ok 4: Reply from alternate address\r
+* ok 5: Support for Reply-To\r
+* ok 6: Un-munging Reply-To\r
+* passed all 6 test(s)\r
+*** t0004-uuencoded-data.sh ***\r
+* ok 1: Generate message\r
+* ok 2: Ensure content before uu data is indexed\r
+* ok 3: Ensure uu data is not indexed\r
+* ok 4: Ensure content after uu data is indexed\r
+* passed all 4 test(s)\r
+*** t0005-dump-restore.sh ***\r
+* ok 1: Generate some message\r
+* ok 2: Dumping all tags\r
+* ok 3: Clearing all tags\r
+* ok 4: Restoring original tags\r
+* ok 5: Restore with nothing to do\r
+* passed all 5 test(s)\r
+fixed 1\r
+success 51\r
+failed 0\r
+broken 1\r
+total 54\r
+\r
+Michal\r
projects / notmuch-archives.git / commitdiff