1 Return-Path: <sojkam1@fel.cvut.cz>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id 6EC7F431FBD
\r
6 for <notmuch@notmuchmail.org>; Mon, 8 Feb 2010 07:14:31 -0800 (PST)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
11 X-Spam-Status: No, score=-2.32 tagged_above=-999 required=5 tests=[AWL=0.279,
\r
12 BAYES_00=-2.599] autolearn=ham
\r
13 Received: from olra.theworths.org ([127.0.0.1])
\r
14 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
15 with ESMTP id cbWCtjm0g--i for <notmuch@notmuchmail.org>;
\r
16 Mon, 8 Feb 2010 07:14:30 -0800 (PST)
\r
17 Received: from max.feld.cvut.cz (max.feld.cvut.cz [147.32.192.36])
\r
18 by olra.theworths.org (Postfix) with ESMTP id BCFC7431FBC
\r
19 for <notmuch@notmuchmail.org>; Mon, 8 Feb 2010 07:14:29 -0800 (PST)
\r
20 Received: from localhost (unknown [192.168.200.4])
\r
21 by max.feld.cvut.cz (Postfix) with ESMTP id 25A6A19F3373;
\r
22 Mon, 8 Feb 2010 16:14:29 +0100 (CET)
\r
23 X-Virus-Scanned: IMAP AMAVIS
\r
24 Received: from max.feld.cvut.cz ([192.168.200.1])
\r
25 by localhost (styx.feld.cvut.cz [192.168.200.4]) (amavisd-new,
\r
27 with ESMTP id df4GTmIgEi7A; Mon, 8 Feb 2010 16:14:24 +0100 (CET)
\r
28 Received: from imap.feld.cvut.cz (imap.feld.cvut.cz [147.32.192.34])
\r
29 by max.feld.cvut.cz (Postfix) with ESMTP id B51B019F2F45;
\r
30 Mon, 8 Feb 2010 16:14:24 +0100 (CET)
\r
31 Received: from steelpick.localnet (k335-30.felk.cvut.cz [147.32.86.30])
\r
32 (Authenticated sender: sojkam1)
\r
33 by imap.feld.cvut.cz (Postfix) with ESMTPSA id AEFEC15C052;
\r
34 Mon, 8 Feb 2010 16:14:24 +0100 (CET)
\r
35 From: Michal Sojka <sojkam1@fel.cvut.cz>
\r
36 To: notmuch@notmuchmail.org
\r
37 Date: Mon, 8 Feb 2010 16:14:24 +0100
\r
38 User-Agent: KMail/1.12.4 (Linux/2.6.31.11-amd64; KDE/4.3.4; x86_64; ; )
\r
39 References: <87ljf8pvxx.fsf@yoom.home.cworth.org>
\r
40 <87hbpwpoko.fsf@yoom.home.cworth.org>
\r
41 <5641883d1002060727ia4e6c16lf800a92fc8735430@mail.gmail.com>
\r
42 In-Reply-To: <5641883d1002060727ia4e6c16lf800a92fc8735430@mail.gmail.com>
\r
44 Content-Type: Text/Plain;
\r
45 charset="iso-8859-1"
\r
46 Content-Transfer-Encoding: 7bit
\r
47 Message-Id: <201002081614.24284.sojkam1@fel.cvut.cz>
\r
48 Subject: Re: [notmuch] A functional (but rudimentary) test suite for notmuch
\r
49 X-BeenThere: notmuch@notmuchmail.org
\r
50 X-Mailman-Version: 2.1.13
\r
52 List-Id: "Use and development of the notmuch mail system."
\r
53 <notmuch.notmuchmail.org>
\r
54 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
55 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
56 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
57 List-Post: <mailto:notmuch@notmuchmail.org>
\r
58 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
59 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
60 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
61 X-List-Received-Date: Mon, 08 Feb 2010 15:14:31 -0000
\r
63 On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
\r
64 > The test suite is still extremely rudimentary. Here are some things I'd
\r
65 > like to improve about it:
\r
67 I converted the actual version of notmuch-test to git test framework.
\r
68 The result is in the followup patches.
\r
70 I'd like to know opinion of others. If Carl agrees that it could be
\r
71 merged I can do additional work as I describe below.
\r
73 The conversion was not as straightforward as I expected mainly because
\r
74 of problems with quoting. There are several sources of quotation problems
\r
75 one being Carl's hashed array parameters. I thing it would be
\r
76 sufficient to use plain variables for passing additional parameters.
\r
78 add_message [from]="\"Sender <sender@example.com>\"" \
\r
79 [to]=test_suite@notmuchmail.org \
\r
80 [subject]=notmuch-reply-test \
\r
81 [date]="\"Tue, 05 Jan 2010 15:43:56 -0800\"" \
\r
82 [body]="\"basic reply test\"" &&
\r
85 msg_from="Sender <sender@example.com>"
\r
86 msg_to=test_suite@notmuchmail.org
\r
87 msg_subject=notmuch-reply-test
\r
88 msg_date="Tue, 05 Jan 2010 15:43:56 -0800"
\r
89 msg_body="basic reply test"
\r
93 A possible additional improvement is elimination of
\r
94 execute_expecting(). Combination of action (running notmuch) and
\r
95 testing of a result in a single function makes it hard to distinguish
\r
96 whether the problem is in the action or in the output. For example, if
\r
97 notmuch is killed because of segfault, it would look like that no
\r
98 output was produced. So instead of:
\r
100 execute_expecting new "No new mail. Removed 3 messages."
\r
102 I'd write something like:
\r
104 echo "No new mail. Removed 3 messages." > expected
\r
106 test_expect_success 'Run notmuch'
\r
107 'notmuch_filter_output new > actual'
\r
108 test_expect_success 'Compare actual and correct output'
\r
109 'test_cmp expected actual'
\r
111 where test_cmp is basically diff -u. This has also the advantage that
\r
112 longer output (e.g. as in t0003-reply.sh) is more readable if there is
\r
115 On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
\r
116 > Carl, have you considered outputting the test suite in the same format
\r
117 > as the test anything protocol? [1] I only mention this because it
\r
118 > might be a nice way to easily do some reporting (or perhaps even
\r
119 > continuous integration) notmuch, with trivial effort.
\r
121 According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project
\r
122 Git test output does not conform to TAP, but from my quick look it could be
\r
123 easily changed to conform.
\r
126 On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
\r
127 > Looking at TAP, one thing I don't like is that it prints the
\r
128 > success/failure of the test first, before the description of the
\r
129 > test. That's not so nice in the case of a long-running (perhaps
\r
130 > infinitely running) test where you might need to interrupt it, but you'd
\r
131 > still want to know *what* was running for so long.
\r
133 Git test framework also outputs first the result and then the test. It can be
\r
134 easily changed by -v switch.
\r
136 To conclude, the output of running make in test/ directory now looks like
\r
138 *** t0000-basic.sh ***
\r
139 * ok 1: test that mail store was created
\r
140 * ok 2: mail store should be empty
\r
141 * ok 3: NOTMUCH_CONFIG is set and points to an existing file
\r
142 * ok 4: PATH is set to this repository
\r
143 * ok 5: success is reported like this
\r
144 * still broken 6: pretend we have a known breakage
\r
145 * FIXED 7: pretend we have fixed a known breakage
\r
146 * ok 8: test runs if prerequisite is satisfied
\r
147 * skip 9: unmet prerequisite causes test to be skipped
\r
148 * fixed 1 known breakage(s)
\r
149 * still have 1 known breakage(s)
\r
150 * passed all remaining 8 test(s)
\r
151 *** t0001-new.sh ***
\r
152 * ok 1: No new messages
\r
153 * ok 2: Single new message
\r
154 * ok 3: Multiple new messages
\r
155 * ok 4: No new messages (non-empty DB)
\r
156 * ok 5: New directories
\r
157 * ok 6: Alternate inode order
\r
158 * ok 7: Message moved in
\r
159 * ok 8: Renamed message
\r
160 * ok 9: Deleted message
\r
161 * ok 10: Renamed directory
\r
162 * ok 11: Deleted directory
\r
163 * ok 12: New directory (at end of list)
\r
164 * ok 13: Deleted directory (end of list)
\r
165 * ok 14: New symlink to directory
\r
166 * ok 15: New symlink to a file
\r
167 * ok 16: New two-level directory
\r
168 * ok 17: Deleted two-level directory
\r
169 * passed all 17 test(s)
\r
170 *** t0002-search.sh ***
\r
171 * ok 1: Search body
\r
172 * ok 2: Search by from:
\r
173 * ok 3: Search by to:
\r
174 * ok 4: Search by subject:
\r
175 * ok 5: Search by id:
\r
176 * ok 6: Search by tag:
\r
177 * ok 7: Search by thread:
\r
178 * ok 8: Search body (phrase)
\r
179 * ok 9: Search by from: (address)
\r
180 * ok 10: Search by from: (name)
\r
181 * ok 11: Search by to: (address)
\r
182 * ok 12: Search by to: (name)
\r
183 * ok 13: Search by subject: (phrase)
\r
184 * passed all 13 test(s)
\r
185 *** t0003-reply.sh ***
\r
186 * ok 1: Basic reply
\r
187 * ok 2: Multiple recipients
\r
188 * ok 3: Reply with CC
\r
189 * ok 4: Reply from alternate address
\r
190 * ok 5: Support for Reply-To
\r
191 * ok 6: Un-munging Reply-To
\r
192 * passed all 6 test(s)
\r
193 *** t0004-uuencoded-data.sh ***
\r
194 * ok 1: Generate message
\r
195 * ok 2: Ensure content before uu data is indexed
\r
196 * ok 3: Ensure uu data is not indexed
\r
197 * ok 4: Ensure content after uu data is indexed
\r
198 * passed all 4 test(s)
\r
199 *** t0005-dump-restore.sh ***
\r
200 * ok 1: Generate some message
\r
201 * ok 2: Dumping all tags
\r
202 * ok 3: Clearing all tags
\r
203 * ok 4: Restoring original tags
\r
204 * ok 5: Restore with nothing to do
\r
205 * passed all 5 test(s)
\r
projects / notmuch-archives.git / blob