1 Return-Path: <m.walters@qmul.ac.uk>
\r
2 X-Original-To: notmuch@notmuchmail.org
\r
3 Delivered-To: notmuch@notmuchmail.org
\r
4 Received: from localhost (localhost [127.0.0.1])
\r
5 by olra.theworths.org (Postfix) with ESMTP id CEDC2431FBC
\r
6 for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 15:43:45 -0800 (PST)
\r
7 X-Virus-Scanned: Debian amavisd-new at olra.theworths.org
\r
11 X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5
\r
12 tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001,
\r
13 NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled
\r
14 Received: from olra.theworths.org ([127.0.0.1])
\r
15 by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)
\r
16 with ESMTP id G6h2ZyYz4Ygw for <notmuch@notmuchmail.org>;
\r
17 Tue, 15 Jan 2013 15:43:45 -0800 (PST)
\r
18 Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6])
\r
19 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
\r
20 (No client certificate requested)
\r
21 by olra.theworths.org (Postfix) with ESMTPS id DEF3B431FAE
\r
22 for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 15:43:44 -0800 (PST)
\r
23 Received: from smtp.qmul.ac.uk ([138.37.6.40])
\r
24 by mail2.qmul.ac.uk with esmtp (Exim 4.71)
\r
25 (envelope-from <m.walters@qmul.ac.uk>)
\r
26 id 1TvGAM-0003nV-Ra; Tue, 15 Jan 2013 23:43:41 +0000
\r
27 Received: from 93-97-24-31.zone5.bethere.co.uk ([93.97.24.31] helo=localhost)
\r
28 by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69)
\r
29 (envelope-from <m.walters@qmul.ac.uk>)
\r
30 id 1TvGAM-0001ET-Ev; Tue, 15 Jan 2013 23:43:38 +0000
\r
31 From: Mark Walters <markwalters1009@gmail.com>
\r
32 To: Jani Nikula <jani@nikula.org>, notmuch@notmuchmail.org
\r
33 Subject: Re: [PATCH 0/5] notmuch batch count
\r
34 In-Reply-To: <cover.1358273133.git.jani@nikula.org>
\r
35 References: <cover.1358273133.git.jani@nikula.org>
\r
36 User-Agent: Notmuch/0.14+255~gff3cc55 (http://notmuchmail.org) Emacs/23.4.1
\r
37 (x86_64-pc-linux-gnu)
\r
38 Date: Tue, 15 Jan 2013 23:43:41 +0000
\r
39 Message-ID: <8738y2ui4y.fsf@qmul.ac.uk>
\r
41 Content-Type: text/plain; charset=us-ascii
\r
42 X-Sender-Host-Address: 93.97.24.31
\r
43 X-QM-SPAM-Info: Sender has good ham record. :)
\r
44 X-QM-Body-MD5: c33bdbdeee0d570020f8373a98493d2c (of first 20000 bytes)
\r
45 X-SpamAssassin-Score: -1.8
\r
46 X-SpamAssassin-SpamBar: -
\r
47 X-SpamAssassin-Report: The QM spam filters have analysed this message to
\r
49 spam. We require at least 5.0 points to mark a message as spam.
\r
50 This message scored -1.8 points.
\r
51 Summary of the scoring:
\r
52 * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/,
\r
54 * [138.37.6.40 listed in list.dnswl.org]
\r
55 * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail
\r
56 provider * (markwalters1009[at]gmail.com)
\r
57 * 0.5 AWL AWL: From: address is in the auto white-list
\r
58 X-QM-Scan-Virus: ClamAV says the message is clean
\r
59 X-BeenThere: notmuch@notmuchmail.org
\r
60 X-Mailman-Version: 2.1.13
\r
62 List-Id: "Use and development of the notmuch mail system."
\r
63 <notmuch.notmuchmail.org>
\r
64 List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,
\r
65 <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>
\r
66 List-Archive: <http://notmuchmail.org/pipermail/notmuch>
\r
67 List-Post: <mailto:notmuch@notmuchmail.org>
\r
68 List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>
\r
69 List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,
\r
70 <mailto:notmuch-request@notmuchmail.org?subject=subscribe>
\r
71 X-List-Received-Date: Tue, 15 Jan 2013 23:43:45 -0000
\r
74 On Tue, 15 Jan 2013, Jani Nikula <jani@nikula.org> wrote:
\r
77 > Notmuch remote usage [1] is a pretty handy way of accessing a notmuch
\r
78 > database on a remote server. However, the more you have saved searches
\r
79 > and tags, the slower notmuch-hello becomes, and it ends up being by and
\r
80 > far the biggest usability issue with remote notmuch. This is because
\r
81 > notmuch-hello issues a separate 'notmuch count' for each saved search
\r
84 > One could argue that notmuch-hello should be fixed somehow, but I chose
\r
85 > to try another route: batch support for notmuch count. This enables
\r
86 > notmuch-hello to get the counts for all the saved searches or tags in a
\r
87 > single call. The performance improvement is huge in remote usage, but
\r
88 > it's not limited to that. Regular local usage benefits from it too, but
\r
89 > it's not as obviously noticeable.
\r
91 This series looks good to me (that is the code looks fine).
\r
95 Do we want this functionality? I think it is useful even on local setups
\r
96 particularly if people have lots of tags (the section that shows all
\r
97 tags can be quite noticeably sped up). It is a substantial improvement
\r
98 on remote setups but I am not sure if that is sufficiently common to
\r
99 warrant the change. At least the code path is the same so it will get
\r
102 Secondly, if we do the functionality should it be more general so that
\r
103 it can do searches etc too. I think this is less clear. Count is likely
\r
104 to be the most useful one since running several (simultaneous) counts is
\r
105 probably more common than running several simultaneous searches.
\r
113 > Here's a script that demonstrates one-by-one count vs. batch count,
\r
114 > locally and over ssh (assuming ssh key authentication is set up), over
\r
119 > echo "tag count:"
\r
120 > notmuch search --output=tags "*" | wc -l
\r
122 > for remote in "" "ssh example.com"; do
\r
124 > echo "one-by-one count:"
\r
125 > time sh -c 'for i in `seq 10`; do notmuch search --format=text0 --output=tags "*" | xargs -0 -n 1 -I "{}" $remote notmuch count tag:"{}" > /dev/null; done'
\r
127 > echo "batch count:"
\r
128 > time sh -c 'for i in `seq 10`; do notmuch search --format=text --output=tags "*" | sed "s/.*/tag:\"\0\"/" | $remote notmuch count --batch > /dev/null; done'
\r
131 > And here's the output of it in my setup:
\r
135 > one-by-one count:
\r
145 > one-by-one count:
\r
156 > As can be seen, in local usage (the first pair of results) the speedup
\r
157 > is more than 10x, although one-by-one notmuch count is usually
\r
158 > sufficiently fast. The difference is more noticeable in remote use (the
\r
159 > second pair of results), where the speedup is 20x here, and any
\r
160 > additional, occasional network latency is multiplied by tag count. (That
\r
161 > result is actually faster than usual for me, but it's still 5+ seconds
\r
162 > to display or refresh notmuch-hello.)
\r
164 > Mark has written a patch that I've been using to switch notmuch-hello to
\r
165 > use batch count. That has made me switch from running notmuch in ssh to
\r
166 > using remote notmuch. The great thing is that we could switch to using
\r
167 > that in Emacs with no special casing for remote usage, and it would
\r
168 > speed things up also in local use. I'm expecting Mark to post his patch
\r
169 > in reply to this series.
\r
171 > Mark actually wrote the elisp part based on the rough idea prior to any
\r
172 > of this cli plumbing, so I felt obliged to follow up. So thanks Mark!
\r
179 > [1] http://notmuchmail.org/remoteusage/ (the page could use some
\r
180 > cleanup; it's really not nearly as complicated as the page suggests)
\r
184 > cli: remove useless strdup
\r
185 > cli: extract count printing to a separate function in notmuch count
\r
186 > cli: add --batch option to notmuch count
\r
187 > man: document notmuch count --batch and --input options
\r
188 > test: notmuch count --batch and --input options
\r
190 > man/man1/notmuch-count.1 | 20 +++++++++
\r
191 > notmuch-count.c | 111 +++++++++++++++++++++++++++++++++++-----------
\r
192 > test/count | 46 +++++++++++++++++++
\r
193 > 3 files changed, 150 insertions(+), 27 deletions(-)
\r