From 9007dbe7b04e152a633083d5c3780c1f7f79a1df Mon Sep 17 00:00:00 2001 From: Mark Walters Date: Tue, 15 Jan 2013 23:43:41 +0000 Subject: [PATCH] Re: [PATCH 0/5] notmuch batch count --- bb/d3be5cbb8af3937986a951bbc88464c94823d7 | 196 ++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 bb/d3be5cbb8af3937986a951bbc88464c94823d7 diff --git a/bb/d3be5cbb8af3937986a951bbc88464c94823d7 b/bb/d3be5cbb8af3937986a951bbc88464c94823d7 new file mode 100644 index 000000000..d23030485 --- /dev/null +++ b/bb/d3be5cbb8af3937986a951bbc88464c94823d7 @@ -0,0 +1,196 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id CEDC2431FBC + for ; Tue, 15 Jan 2013 15:43:45 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -1.098 +X-Spam-Level: +X-Spam-Status: No, score=-1.098 tagged_above=-999 required=5 + tests=[DKIM_ADSP_CUSTOM_MED=0.001, FREEMAIL_FROM=0.001, + NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_DNSWL_MED=-2.3] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id G6h2ZyYz4Ygw for ; + Tue, 15 Jan 2013 15:43:45 -0800 (PST) +Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) + (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id DEF3B431FAE + for ; Tue, 15 Jan 2013 15:43:44 -0800 (PST) +Received: from smtp.qmul.ac.uk ([138.37.6.40]) + by mail2.qmul.ac.uk with esmtp (Exim 4.71) + (envelope-from ) + id 1TvGAM-0003nV-Ra; Tue, 15 Jan 2013 23:43:41 +0000 +Received: from 93-97-24-31.zone5.bethere.co.uk ([93.97.24.31] helo=localhost) + by smtp.qmul.ac.uk with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) + (envelope-from ) + id 1TvGAM-0001ET-Ev; Tue, 15 Jan 2013 23:43:38 +0000 +From: Mark Walters +To: Jani Nikula , notmuch@notmuchmail.org +Subject: Re: [PATCH 0/5] notmuch batch count +In-Reply-To: +References: +User-Agent: Notmuch/0.14+255~gff3cc55 (http://notmuchmail.org) Emacs/23.4.1 + (x86_64-pc-linux-gnu) +Date: Tue, 15 Jan 2013 23:43:41 +0000 +Message-ID: <8738y2ui4y.fsf@qmul.ac.uk> +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-Sender-Host-Address: 93.97.24.31 +X-QM-SPAM-Info: Sender has good ham record. :) +X-QM-Body-MD5: c33bdbdeee0d570020f8373a98493d2c (of first 20000 bytes) +X-SpamAssassin-Score: -1.8 +X-SpamAssassin-SpamBar: - +X-SpamAssassin-Report: The QM spam filters have analysed this message to + determine if it is + spam. We require at least 5.0 points to mark a message as spam. + This message scored -1.8 points. + Summary of the scoring: + * -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at http://www.dnswl.org/, + * medium trust + * [138.37.6.40 listed in list.dnswl.org] + * 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail + provider * (markwalters1009[at]gmail.com) + * 0.5 AWL AWL: From: address is in the auto white-list +X-QM-Scan-Virus: ClamAV says the message is clean +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 15 Jan 2013 23:43:45 -0000 + + +On Tue, 15 Jan 2013, Jani Nikula wrote: +> Hi all - +> +> Notmuch remote usage [1] is a pretty handy way of accessing a notmuch +> database on a remote server. However, the more you have saved searches +> and tags, the slower notmuch-hello becomes, and it ends up being by and +> far the biggest usability issue with remote notmuch. This is because +> notmuch-hello issues a separate 'notmuch count' for each saved search +> and tag. +> +> One could argue that notmuch-hello should be fixed somehow, but I chose +> to try another route: batch support for notmuch count. This enables +> notmuch-hello to get the counts for all the saved searches or tags in a +> single call. The performance improvement is huge in remote usage, but +> it's not limited to that. Regular local usage benefits from it too, but +> it's not as obviously noticeable. + +This series looks good to me (that is the code looks fine). + +Two questions are: + +Do we want this functionality? I think it is useful even on local setups +particularly if people have lots of tags (the section that shows all +tags can be quite noticeably sped up). It is a substantial improvement +on remote setups but I am not sure if that is sufficiently common to +warrant the change. At least the code path is the same so it will get +enough testing. + +Secondly, if we do the functionality should it be more general so that +it can do searches etc too. I think this is less clear. Count is likely +to be the most useful one since running several (simultaneous) counts is +probably more common than running several simultaneous searches. + +Best wishes + +Mark + + +> +> Here's a script that demonstrates one-by-one count vs. batch count, +> locally and over ssh (assuming ssh key authentication is set up), over +> 10 iterations: +> +> #!/bin/bash +> +> echo "tag count:" +> notmuch search --output=tags "*" | wc -l +> +> for remote in "" "ssh example.com"; do +> export remote +> echo "one-by-one count:" +> time sh -c 'for i in `seq 10`; do notmuch search --format=text0 --output=tags "*" | xargs -0 -n 1 -I "{}" $remote notmuch count tag:"{}" > /dev/null; done' +> +> echo "batch count:" +> time sh -c 'for i in `seq 10`; do notmuch search --format=text --output=tags "*" | sed "s/.*/tag:\"\0\"/" | $remote notmuch count --batch > /dev/null; done' +> done +> +> And here's the output of it in my setup: +> +> tag count: +> 36 +> one-by-one count: +> +> real 0m2.349s +> user 0m0.552s +> sys 0m0.868s +> batch count: +> +> real 0m0.179s +> user 0m0.120s +> sys 0m0.064s +> one-by-one count: +> +> real 0m56.527s +> user 0m1.424s +> sys 0m1.164s +> batch count: +> +> real 0m2.407s +> user 0m0.068s +> sys 0m0.040s +> +> As can be seen, in local usage (the first pair of results) the speedup +> is more than 10x, although one-by-one notmuch count is usually +> sufficiently fast. The difference is more noticeable in remote use (the +> second pair of results), where the speedup is 20x here, and any +> additional, occasional network latency is multiplied by tag count. (That +> result is actually faster than usual for me, but it's still 5+ seconds +> to display or refresh notmuch-hello.) +> +> Mark has written a patch that I've been using to switch notmuch-hello to +> use batch count. That has made me switch from running notmuch in ssh to +> using remote notmuch. The great thing is that we could switch to using +> that in Emacs with no special casing for remote usage, and it would +> speed things up also in local use. I'm expecting Mark to post his patch +> in reply to this series. +> +> Mark actually wrote the elisp part based on the rough idea prior to any +> of this cli plumbing, so I felt obliged to follow up. So thanks Mark! +> +> +> BR, +> Jani. +> +> +> [1] http://notmuchmail.org/remoteusage/ (the page could use some +> cleanup; it's really not nearly as complicated as the page suggests) +> +> +> Jani Nikula (5): +> cli: remove useless strdup +> cli: extract count printing to a separate function in notmuch count +> cli: add --batch option to notmuch count +> man: document notmuch count --batch and --input options +> test: notmuch count --batch and --input options +> +> man/man1/notmuch-count.1 | 20 +++++++++ +> notmuch-count.c | 111 +++++++++++++++++++++++++++++++++++----------- +> test/count | 46 +++++++++++++++++++ +> 3 files changed, 150 insertions(+), 27 deletions(-) +> +> -- +> 1.7.10.4 -- 2.26.2