--- /dev/null
+Return-Path: <tomi.ollila@iki.fi>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id C703D431FBC\r
+ for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 21:02:18 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: 0\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none]\r
+ autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id iZwx+1BNfBWU for <notmuch@notmuchmail.org>;\r
+ Tue, 15 Jan 2013 21:02:17 -0800 (PST)\r
+Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34])\r
+ by olra.theworths.org (Postfix) with ESMTP id 45B73431FAE\r
+ for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 21:02:17 -0800 (PST)\r
+Received: from guru.guru-group.fi (localhost [IPv6:::1])\r
+ by guru.guru-group.fi (Postfix) with ESMTP id D2553100045;\r
+ Wed, 16 Jan 2013 07:02:05 +0200 (EET)\r
+From: Tomi Ollila <tomi.ollila@iki.fi>\r
+To: Mark Walters <markwalters1009@gmail.com>, Jani Nikula <jani@nikula.org>,\r
+ notmuch@notmuchmail.org\r
+Subject: Re: [PATCH 0/5] notmuch batch count\r
+In-Reply-To: <8738y2ui4y.fsf@qmul.ac.uk>\r
+References: <cover.1358273133.git.jani@nikula.org> <8738y2ui4y.fsf@qmul.ac.uk>\r
+User-Agent: Notmuch/0.14+260~gd7a61d6 (http://notmuchmail.org) Emacs/24.2.1\r
+ (x86_64-unknown-linux-gnu)\r
+X-Face: HhBM'cA~<r"^Xv\KRN0P{vn'Y"Kd;zg_y3S[4)KSN~s?O\"QPoL\r
+ $[Xv_BD:i/F$WiEWax}R(MPS`^UaptOGD`*/=@\1lKoVa9tnrg0TW?"r7aRtgk[F\r
+ !)g;OY^,BjTbr)Np:%c_o'jj,Z\r
+Date: Wed, 16 Jan 2013 07:02:05 +0200\r
+Message-ID: <m27gndsotu.fsf@guru.guru-group.fi>\r
+MIME-Version: 1.0\r
+Content-Type: text/plain\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Wed, 16 Jan 2013 05:02:19 -0000\r
+\r
+On Wed, Jan 16 2013, Mark Walters <markwalters1009@gmail.com> wrote:\r
+\r
+> On Tue, 15 Jan 2013, Jani Nikula <jani@nikula.org> wrote:\r
+>> Hi all -\r
+>>\r
+>> Notmuch remote usage [1] is a pretty handy way of accessing a notmuch\r
+>> database on a remote server. However, the more you have saved searches\r
+>> and tags, the slower notmuch-hello becomes, and it ends up being by and\r
+>> far the biggest usability issue with remote notmuch. This is because\r
+>> notmuch-hello issues a separate 'notmuch count' for each saved search\r
+>> and tag.\r
+>>\r
+>> One could argue that notmuch-hello should be fixed somehow, but I chose\r
+>> to try another route: batch support for notmuch count. This enables\r
+>> notmuch-hello to get the counts for all the saved searches or tags in a\r
+>> single call. The performance improvement is huge in remote usage, but\r
+>> it's not limited to that. Regular local usage benefits from it too, but\r
+>> it's not as obviously noticeable.\r
+>\r
+> This series looks good to me (that is the code looks fine).\r
+>\r
+> Two questions are:\r
+>\r
+> Do we want this functionality? I think it is useful even on local setups\r
+> particularly if people have lots of tags (the section that shows all\r
+> tags can be quite noticeably sped up). It is a substantial improvement\r
+> on remote setups but I am not sure if that is sufficiently common to\r
+> warrant the change. At least the code path is the same so it will get\r
+> enough testing.\r
+\r
+I do want the functionality. Especialy where I am now it takes about\r
+0.4 sec for 'ssh remote echo foo' to get executed (using connection sharing).\r
+pipelining the count requests could make all the count requests emacs\r
+does (in my current set) to complete in less than 1 sec. \r
+\r
+> Secondly, if we do the functionality should it be more general so that\r
+> it can do searches etc too. I think this is less clear. Count is likely\r
+> to be the most useful one since running several (simultaneous) counts is\r
+> probably more common than running several simultaneous searches.\r
+\r
+One could argue that we'd should send json "documents" to notmuch in\r
+stdin and notmuch would output json(/sexp) "documents". That is just\r
+SMOP. I bet Austin would like this solution, especially the part\r
+that involves writing or integrating json parser >;). \r
+I'd be happy with this 'batch' approach. \r
+\r
+I'll be testing this soon, but refrain from reviewing the code\r
+until 0.15 is out.\r
+\r
+>\r
+> Best wishes\r
+>\r
+> Mark\r
+\r
+\r
+Tomi\r
+\r
+\r
+>\r
+>\r
+>>\r
+>> Here's a script that demonstrates one-by-one count vs. batch count,\r
+>> locally and over ssh (assuming ssh key authentication is set up), over\r
+>> 10 iterations:\r
+>>\r
+>> #!/bin/bash\r
+>>\r
+>> echo "tag count:"\r
+>> notmuch search --output=tags "*" | wc -l\r
+>>\r
+>> for remote in "" "ssh example.com"; do\r
+>> export remote\r
+>> echo "one-by-one count:"\r
+>> time sh -c 'for i in `seq 10`; do notmuch search --format=text0 --output=tags "*" | xargs -0 -n 1 -I "{}" $remote notmuch count tag:"{}" > /dev/null; done'\r
+>>\r
+>> echo "batch count:"\r
+>> time sh -c 'for i in `seq 10`; do notmuch search --format=text --output=tags "*" | sed "s/.*/tag:\"\0\"/" | $remote notmuch count --batch > /dev/null; done'\r
+>> done\r
+>>\r
+>> And here's the output of it in my setup:\r
+>>\r
+>> tag count:\r
+>> 36\r
+>> one-by-one count:\r
+>>\r
+>> real 0m2.349s\r
+>> user 0m0.552s\r
+>> sys 0m0.868s\r
+>> batch count:\r
+>>\r
+>> real 0m0.179s\r
+>> user 0m0.120s\r
+>> sys 0m0.064s\r
+>> one-by-one count:\r
+>>\r
+>> real 0m56.527s\r
+>> user 0m1.424s\r
+>> sys 0m1.164s\r
+>> batch count:\r
+>>\r
+>> real 0m2.407s\r
+>> user 0m0.068s\r
+>> sys 0m0.040s\r
+>>\r
+>> As can be seen, in local usage (the first pair of results) the speedup\r
+>> is more than 10x, although one-by-one notmuch count is usually\r
+>> sufficiently fast. The difference is more noticeable in remote use (the\r
+>> second pair of results), where the speedup is 20x here, and any\r
+>> additional, occasional network latency is multiplied by tag count. (That\r
+>> result is actually faster than usual for me, but it's still 5+ seconds\r
+>> to display or refresh notmuch-hello.)\r
+>>\r
+>> Mark has written a patch that I've been using to switch notmuch-hello to\r
+>> use batch count. That has made me switch from running notmuch in ssh to\r
+>> using remote notmuch. The great thing is that we could switch to using\r
+>> that in Emacs with no special casing for remote usage, and it would\r
+>> speed things up also in local use. I'm expecting Mark to post his patch\r
+>> in reply to this series.\r
+>>\r
+>> Mark actually wrote the elisp part based on the rough idea prior to any\r
+>> of this cli plumbing, so I felt obliged to follow up. So thanks Mark!\r
+>>\r
+>>\r
+>> BR,\r
+>> Jani.\r
+>>\r
+>>\r
+>> [1] http://notmuchmail.org/remoteusage/ (the page could use some\r
+>> cleanup; it's really not nearly as complicated as the page suggests)\r
+>>\r
+>>\r
+>> Jani Nikula (5):\r
+>> cli: remove useless strdup\r
+>> cli: extract count printing to a separate function in notmuch count\r
+>> cli: add --batch option to notmuch count\r
+>> man: document notmuch count --batch and --input options\r
+>> test: notmuch count --batch and --input options\r
+>>\r
+>> man/man1/notmuch-count.1 | 20 +++++++++\r
+>> notmuch-count.c | 111 +++++++++++++++++++++++++++++++++++-----------\r
+>> test/count | 46 +++++++++++++++++++\r
+>> 3 files changed, 150 insertions(+), 27 deletions(-)\r
+>>\r
+>> -- \r
+>> 1.7.10.4\r
+> _______________________________________________\r
+> notmuch mailing list\r
+> notmuch@notmuchmail.org\r
+> http://notmuchmail.org/mailman/listinfo/notmuch\r
projects / notmuch-archives.git / commitdiff