1 Return-Path: <dmitry.kurochkin@gmail.com>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id 6A3B9431FD0
\r
6 for <notmuch@notmuchmail.org>; Sun, 26 Jun 2011 20:54:36 -0700 (PDT)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
11 X-Spam-Status: No, score=-0.799 tagged_above=-999 required=5
\r
12 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
\r
13 FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled
\r
14 Received: from olra.theworths.org ([127.0.0.1])
\r
15 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
16 with ESMTP id EWcP3vxenp2k for <notmuch@notmuchmail.org>;
\r
17 Sun, 26 Jun 2011 20:54:32 -0700 (PDT)
\r
18 Received: from mail-bw0-f53.google.com (mail-bw0-f53.google.com
\r
19 [209.85.214.53]) (using TLSv1 with cipher RC4-SHA (128/128 bits))
\r
20 (No client certificate requested)
\r
21 by olra.theworths.org (Postfix) with ESMTPS id BB1DE429E2E
\r
22 for <notmuch@notmuchmail.org>; Sun, 26 Jun 2011 20:54:29 -0700 (PDT)
\r
23 Received: by mail-bw0-f53.google.com with SMTP id 12so3975366bwg.26
\r
24 for <notmuch@notmuchmail.org>; Sun, 26 Jun 2011 20:54:29 -0700 (PDT)
\r
25 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
\r
26 h=domainkey-signature:from:to:cc:subject:date:message-id:x-mailer
\r
27 :in-reply-to:references;
\r
28 bh=0kVO65XjIsZBCrElxskjQc3VCs1GUHtR7ueA4x19ncY=;
\r
29 b=Xnq5MJXiNyM4AmLbaOWjzvxPB74AwSugQcoGiqOuM+OUBkUtY9fag5GMXo42sDTpp5
\r
30 y+03EBL0N08hvI82UKj4dNjbn+MEuClyNB5nRK4y752rp9eHC7qBz9FdqxP9DOgRWZdK
\r
31 QSElsxc3b7eLSPvvpnHitJChHINPDWdV90mys=
\r
32 DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
\r
33 h=from:to:cc:subject:date:message-id:x-mailer:in-reply-to:references;
\r
34 b=nxgbPRn7ep8+dXvdZ2yxq3EsGN9Aspin5+ZUqbV5Ximk4CpKhabPxh3HsO1mMLtzr7
\r
35 yHbHpiGOoU1N+6HeglT7FXQtxqnJ8K/0iNbsxqgKgyseHgvMA2EGyyVYThfQ5CN2s/NP
\r
36 hZto6MLUMdiBvil9vb3Dqwtp4cm/GhVgbsoWk=
\r
37 Received: by 10.204.19.70 with SMTP id z6mr3864940bka.204.1309146869077;
\r
38 Sun, 26 Jun 2011 20:54:29 -0700 (PDT)
\r
39 Received: from localhost ([91.144.186.21])
\r
40 by mx.google.com with ESMTPS id n3sm3774843bka.16.2011.06.26.20.54.27
\r
41 (version=TLSv1/SSLv3 cipher=OTHER);
\r
42 Sun, 26 Jun 2011 20:54:28 -0700 (PDT)
\r
43 From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
\r
44 To: notmuch@notmuchmail.org
\r
45 Subject: [PATCH 2/2] test: use emacsclient(1) for Emacs tests
\r
46 Date: Mon, 27 Jun 2011 07:54:06 +0400
\r
47 Message-Id: <1309146846-30991-2-git-send-email-dmitry.kurochkin@gmail.com>
\r
48 X-Mailer: git-send-email 1.7.5.4
\r
49 In-Reply-To: <1309146846-30991-1-git-send-email-dmitry.kurochkin@gmail.com>
\r
50 References: <1309132379-24089-1-git-send-email-dmitry.kurochkin@gmail.com>
\r
51 <1309146846-30991-1-git-send-email-dmitry.kurochkin@gmail.com>
\r
52 X-BeenThere: notmuch@notmuchmail.org
\r
53 X-Mailman-Version: 2.1.13
\r
55 List-Id: "Use and development of the notmuch mail system."
\r
56 <notmuch.notmuchmail.org>
\r
57 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
58 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
59 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
60 List-Post: <mailto:notmuch@notmuchmail.org>
\r
61 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
62 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
63 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
64 X-List-Received-Date: Mon, 27 Jun 2011 03:54:36 -0000
\r
66 Before the change, every Emacs tests ran in a separate Emacs
\r
67 instance. Starting Emacs many times wastes considerable time and
\r
68 it gets worse as the test suite grows. The patch solves this by
\r
69 using a single Emacs server and emacsclient(1) to run multiple
\r
70 tests. Emacs server is started on the first test_emacs call and
\r
71 stopped when test_done is called or the test is killed by a
\r
72 signal. Several auxiliary scripts useful for debugging and test
\r
73 development are generated instead of the run_emacs script:
\r
75 * emacs_server_start - start Emacs server
\r
76 * emacs_server_stop - stop Emacs server
\r
77 * emacs_start - start Emacs
\r
78 * emacs_run - execute ELisp expressions in running Emacs server
\r
80 Since multiple tests are run in a single Emacs instance, they
\r
81 must not change Emacs environment because it may affect other
\r
82 tests. For now, the only Emacs environment modifications done by
\r
83 the tests are variable settings. Before the change, variables
\r
84 were set with `setq' which affected other tests. The patch
\r
85 changes all variables to use `let', so the scope of the change is
\r
86 limited to a single test.
\r
88 test/emacs | 74 +++++++++++++-------------
\r
89 test/test-lib.el | 6 ++
\r
90 test/test-lib.sh | 149 ++++++++++++++++++++++++++++++++++++++++++-----------
\r
91 3 files changed, 161 insertions(+), 68 deletions(-)
\r
93 diff --git a/test/emacs b/test/emacs
\r
94 index 4f16b41..f1939dc 100755
\r
97 @@ -12,20 +12,20 @@ test_emacs '(notmuch-hello)
\r
98 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello
\r
100 test_begin_subtest "Saved search with 0 results"
\r
101 -test_emacs '(setq notmuch-show-empty-saved-searches t)
\r
102 - (setq notmuch-saved-searches
\r
103 - '\''(("inbox" . "tag:inbox")
\r
104 - ("unread" . "tag:unread")
\r
105 - ("empty" . "tag:doesnotexist")))
\r
108 +test_emacs '(let ((notmuch-show-empty-saved-searches t)
\r
109 + (notmuch-saved-searches
\r
110 + '\''(("inbox" . "tag:inbox")
\r
111 + ("unread" . "tag:unread")
\r
112 + ("empty" . "tag:doesnotexist"))))
\r
115 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-with-empty
\r
117 test_begin_subtest "No saved searches displayed (all with 0 results)"
\r
118 -test_emacs '(setq notmuch-saved-searches
\r
119 - '\''(("empty" . "tag:doesnotexist")))
\r
122 +test_emacs '(let ((notmuch-saved-searches
\r
123 + '\''(("empty" . "tag:doesnotexist"))))
\r
126 test_expect_equal_file OUTPUT $EXPECTED/notmuch-hello-no-saved-searches
\r
128 test_begin_subtest "Basic notmuch-search view in emacs"
\r
129 @@ -147,9 +147,9 @@ output=$(notmuch search 'subject:"testing message sent via SMTP"' | notmuch_sear
\r
130 test_expect_equal "$output" "thread:XXX 2000-01-01 [1/1] Notmuch Test Suite; Testing message sent via SMTP (inbox)"
\r
132 test_begin_subtest "notmuch-fcc-dirs set to nil"
\r
133 -test_emacs "(setq notmuch-fcc-dirs nil)
\r
134 - (notmuch-mua-mail)
\r
136 +test_emacs "(let ((notmuch-fcc-dirs nil))
\r
137 + (notmuch-mua-mail)
\r
139 cat <<EOF >EXPECTED
\r
140 From: Notmuch Test Suite <test_suite@notmuchmail.org>
\r
142 @@ -164,9 +164,9 @@ mkdir -p mail/sent-string/new
\r
143 mkdir -p mail/sent-string/tmp
\r
145 test_begin_subtest "notmuch-fcc-dirs set to a string"
\r
146 -test_emacs "(setq notmuch-fcc-dirs \"sent-string\")
\r
147 - (notmuch-mua-mail)
\r
149 +test_emacs "(let ((notmuch-fcc-dirs \"sent-string\"))
\r
150 + (notmuch-mua-mail)
\r
152 cat <<EOF >EXPECTED
\r
153 From: Notmuch Test Suite <test_suite@notmuchmail.org>
\r
155 @@ -185,11 +185,11 @@ mkdir -p mail/failure/new
\r
156 mkdir -p mail/failure/tmp
\r
158 test_begin_subtest "notmuch-fcc-dirs set to a list (with match)"
\r
159 -test_emacs "(setq notmuch-fcc-dirs
\r
160 - '((\"notmuchmail.org\" . \"sent-list-match\")
\r
161 - (\".*\" . \"failure\")))
\r
162 - (notmuch-mua-mail)
\r
164 +test_emacs "(let ((notmuch-fcc-dirs
\r
165 + '((\"notmuchmail.org\" . \"sent-list-match\")
\r
166 + (\".*\" . \"failure\"))))
\r
167 + (notmuch-mua-mail)
\r
169 cat <<EOF >EXPECTED
\r
170 From: Notmuch Test Suite <test_suite@notmuchmail.org>
\r
172 @@ -205,11 +205,11 @@ mkdir -p mail/sent-list-catch-all/new
\r
173 mkdir -p mail/sent-list-catch-all/tmp
\r
175 test_begin_subtest "notmuch-fcc-dirs set to a list (catch-all)"
\r
176 -test_emacs "(setq notmuch-fcc-dirs
\r
177 - '((\"example.com\" . \"failure\")
\r
178 - (\".*\" . \"sent-list-catch-all\")))
\r
179 - (notmuch-mua-mail)
\r
181 +test_emacs "(let ((notmuch-fcc-dirs
\r
182 + '((\"example.com\" . \"failure\")
\r
183 + (\".*\" . \"sent-list-catch-all\"))))
\r
184 + (notmuch-mua-mail)
\r
186 cat <<EOF >EXPECTED
\r
187 From: Notmuch Test Suite <test_suite@notmuchmail.org>
\r
189 @@ -220,11 +220,11 @@ EOF
\r
190 test_expect_equal_file OUTPUT EXPECTED
\r
192 test_begin_subtest "notmuch-fcc-dirs set to a list (no match)"
\r
193 -test_emacs "(setq notmuch-fcc-dirs
\r
194 - '((\"example.com\" . \"failure\")
\r
195 - (\"nomatchhere.net\" . \"failure\")))
\r
196 - (notmuch-mua-mail)
\r
198 +test_emacs "(let ((notmuch-fcc-dirs
\r
199 + '((\"example.com\" . \"failure\")
\r
200 + (\"nomatchhere.net\" . \"failure\"))))
\r
201 + (notmuch-mua-mail)
\r
203 cat <<EOF >EXPECTED
\r
204 From: Notmuch Test Suite <test_suite@notmuchmail.org>
\r
206 @@ -253,15 +253,15 @@ test_expect_equal_file OUTPUT EXPECTED
\r
208 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-attachments"
\r
209 # save as archive to test that Emacs does not re-compress .gz
\r
210 -echo ./attachment1.gz |
\r
211 -test_emacs '(notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
\r
212 - (notmuch-show-save-attachments)' > /dev/null 2>&1
\r
213 +test_emacs '(let ((standard-input "\"attachment1.gz\""))
\r
214 + (notmuch-show "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com")
\r
215 + (notmuch-show-save-attachments))'
\r
216 test_expect_equal_file "$EXPECTED/attachment" attachment1.gz
\r
218 test_begin_subtest "Save attachment from within emacs using notmuch-show-save-part"
\r
219 # save as archive to test that Emacs does not re-compress .gz
\r
220 -echo ./attachment2.gz |
\r
221 -test_emacs '(notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5)' > /dev/null 2>&1
\r
222 +test_emacs '(let ((standard-input "\"attachment2.gz\""))
\r
223 + (notmuch-show-save-part "id:cf0c4d610911171136h1713aa59w9cf9aa31f052ad0a@mail.gmail.com" 5))' > /dev/null 2>&1
\r
224 test_expect_equal_file "$EXPECTED/attachment" attachment2.gz
\r
226 test_begin_subtest "View raw message within emacs"
\r
227 diff --git a/test/test-lib.el b/test/test-lib.el
\r
228 index 4e7f5cf..a5a3125 100644
\r
229 --- a/test/test-lib.el
\r
230 +++ b/test/test-lib.el
\r
232 ;; avoid crazy 10-column default of --batch
\r
233 (set-frame-width (window-frame (get-buffer-window)) 80)
\r
235 +;; `read-file-name' by default uses `completing-read' function to read
\r
236 +;; user input. It does not respect `standard-input' variable which we
\r
237 +;; use in tests to provide user input. So replace it with a plain
\r
239 +(setq read-file-name-function (lambda (&rest _) (read)))
\r
241 (defun notmuch-test-wait ()
\r
242 "Wait for process completion."
\r
243 (while (get-buffer-process (current-buffer))
\r
244 diff --git a/test/test-lib.sh b/test/test-lib.sh
\r
245 index ad1506c..1c1581b 100755
\r
246 --- a/test/test-lib.sh
\r
247 +++ b/test/test-lib.sh
\r
248 @@ -57,6 +57,9 @@ unset CDPATH
\r
252 +# PID of running Emacs server
\r
257 # A regexp to match 5 and 40 hexdigits
\r
258 @@ -174,6 +177,7 @@ test_success=0
\r
262 + emacs_server_stop
\r
263 if test -n "$GIT_EXIT_OK"
\r
266 @@ -394,19 +398,20 @@ emacs_deliver_message ()
\r
267 mkdir -p "$MAIL_DIR"/sent/{cur,new,tmp}
\r
268 ../smtp-dummy sent_message &
\r
270 - test_emacs "(setq message-send-mail-function 'message-smtpmail-send-it)
\r
271 - (setq smtpmail-smtp-server \"localhost\")
\r
272 - (setq smtpmail-smtp-service \"25025\")
\r
274 - (notmuch-mua-mail)
\r
275 - (message-goto-to)
\r
276 - (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
\r
277 - (message-goto-subject)
\r
278 - (insert \"${subject}\")
\r
279 - (message-goto-body)
\r
280 - (insert \"${body}\")
\r
282 - (message-send-and-exit)" >/dev/null 2>&1
\r
284 + "(let ((message-send-mail-function 'message-smtpmail-send-it)
\r
285 + (smtpmail-smtp-server \"localhost\")
\r
286 + (smtpmail-smtp-service \"25025\"))
\r
288 + (notmuch-mua-mail)
\r
289 + (message-goto-to)
\r
290 + (insert \"test_suite@notmuchmail.org\nDate: 01 Jan 2000 12:00:00 -0000\")
\r
291 + (message-goto-subject)
\r
292 + (insert \"${subject}\")
\r
293 + (message-goto-body)
\r
294 + (insert \"${body}\")
\r
296 + (message-send-and-exit))" >/dev/null 2>&1
\r
297 wait ${smtp_dummy_pid}
\r
298 notmuch new >/dev/null
\r
300 @@ -828,6 +833,8 @@ test_done () {
\r
304 + emacs_server_stop
\r
306 if [ "$test_failure" = "0" ]; then
\r
307 if [ "$test_broken" = "0" ]; then
\r
308 rm -rf "$remove_tmp"
\r
309 @@ -838,24 +845,26 @@ test_done () {
\r
314 - # Construct a little test script here for the benefit of the user,
\r
315 - # (who can easily run "run_emacs" to get the same emacs environment
\r
316 - # for investigating any failures).
\r
317 - cat <<EOF > run_emacs
\r
318 +# Generate some scripts for running Emacs tests. These scripts are
\r
319 +# used by Emacs tests and help investigating failures. The following
\r
320 +# scripts are generated:
\r
322 +# * emacs_server_start - start Emacs server
\r
323 +# * emacs_server_stop - stop Emacs server
\r
324 +# * emacs_start - start Emacs
\r
325 +# * emacs_run - execute ELisp expressions in running Emacs server
\r
326 +emacs_generate_scripts ()
\r
328 + server_name="notmuch-test-suite-$$"
\r
330 + cat <<EOF > "$TMP_DIRECTORY/emacs_server_start"
\r
333 export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
\r
335 -# We assume that the user will give a command-line argument only if
\r
336 -# wanting to run in batch mode.
\r
337 -if [ \$# -gt 0 ]; then
\r
341 # Here's what we are using here:
\r
343 -# --batch: Quit after given commands and print all (messages)
\r
344 +# --daemon Start Emacs as a daemon
\r
346 # --no-init-file Don't load users ~/.emacs
\r
348 @@ -865,13 +874,90 @@ fi
\r
350 # --load Force loading of notmuch.el and test-lib.el
\r
352 -emacs \$BATCH --no-init-file --no-site-file \
\r
353 - --directory ../../emacs --load notmuch.el \
\r
354 - --directory .. --load test-lib.el \
\r
355 - --eval "(progn \$@)"
\r
356 +emacs --daemon --no-init-file --no-site-file \
\r
357 + --directory "$TEST_DIRECTORY/../emacs" --load notmuch.el \
\r
358 + --directory "$TEST_DIRECTORY" --load test-lib.el \
\r
359 + --eval '(setq server-name "$server_name")'
\r
361 + chmod a+x "$TMP_DIRECTORY/emacs_server_start"
\r
363 + cat <<EOF > "$TMP_DIRECTORY/emacs_server_stop"
\r
366 +dir=\$(dirname "\$0")
\r
367 +"\$dir"/emacs_run '(kill-emacs)'
\r
369 - chmod a+x ./run_emacs
\r
371 + chmod a+x "$TMP_DIRECTORY/emacs_server_stop"
\r
373 + cat <<EOF > "$TMP_DIRECTORY/emacs_start"
\r
376 +export NOTMUCH_CONFIG=$NOTMUCH_CONFIG
\r
378 +# Here's what we are using here:
\r
380 +# --no-init-file Don't load users ~/.emacs
\r
382 +# --no-site-file Don't load the site-wide startup stuff
\r
384 +# --directory Ensure that the local elisp sources are found
\r
386 +# --load Force loading of notmuch.el and test-lib.el
\r
388 +emacs --no-init-file --no-site-file \
\r
389 + --directory "$TEST_DIRECTORY/../emacs" --load notmuch.el \
\r
390 + --directory "$TEST_DIRECTORY" --load test-lib.el
\r
392 + chmod a+x "$TMP_DIRECTORY/emacs_start"
\r
394 + cat <<EOF > "$TMP_DIRECTORY/emacs_run"
\r
397 +# Here's what we are using here:
\r
399 +# --socket-name Emacs server name
\r
401 +# --eval Evaluate ELisp expressions
\r
403 +emacsclient --socket-name "$server_name" --eval "(progn \$@)"
\r
405 + chmod a+x "$TMP_DIRECTORY/emacs_run"
\r
408 +# Start Emacs server if it is not running.
\r
409 +emacs_server_start ()
\r
411 + [ -n "$emacs_server_pid" ] && return
\r
413 + output=$("$TMP_DIRECTORY/emacs_server_start" 2>&1)
\r
414 + if [ "$?" -ne 0 ]; then
\r
419 + emacs_server_pid=$("$TMP_DIRECTORY/emacs_run" '(emacs-pid)')
\r
420 + [ "$?" -eq 0 -a -n "$emacs_server_pid" ]
\r
423 +# Stop Emacs server if it is running.
\r
424 +emacs_server_stop ()
\r
426 + [ -z "$emacs_server_pid" ] && return
\r
428 + emacs_server_pid=
\r
429 + output=$("$TMP_DIRECTORY/emacs_server_stop" 2>&1)
\r
430 + if [ "$?" -ne 0 ]; then
\r
436 +# Evaluate ELisp expressions in Emacs server. Server is started if it
\r
439 + emacs_server_start || return
\r
441 + "$TMP_DIRECTORY/emacs_run" "$@"
\r
445 @@ -999,6 +1085,7 @@ primary_email=test_suite@notmuchmail.org
\r
446 other_email=test_suite_other@notmuchmail.org;test_suite@otherdomain.org
\r
449 +emacs_generate_scripts
\r
451 # Use -P to resolve symlinks in our working directory so that the cwd
\r
452 # in subprocesses like git equals our $PWD (for pathname comparisons).
\r