--- /dev/null
+Return-Path: <jani@nikula.org>\r
+X-Original-To: notmuch@notmuchmail.org\r
+Delivered-To: notmuch@notmuchmail.org\r
+Received: from localhost (localhost [127.0.0.1])\r
+ by olra.theworths.org (Postfix) with ESMTP id CA6AF431FC2\r
+ for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 13:54:56 -0800 (PST)\r
+X-Virus-Scanned: Debian amavisd-new at olra.theworths.org\r
+X-Spam-Flag: NO\r
+X-Spam-Score: -0.7\r
+X-Spam-Level: \r
+X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5\r
+ tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled\r
+Received: from olra.theworths.org ([127.0.0.1])\r
+ by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024)\r
+ with ESMTP id AhpxEAPAqw-c for <notmuch@notmuchmail.org>;\r
+ Tue, 15 Jan 2013 13:54:52 -0800 (PST)\r
+Received: from mail-la0-f52.google.com (mail-la0-f52.google.com\r
+ [209.85.215.52]) (using TLSv1 with cipher RC4-SHA (128/128 bits))\r
+ (No client certificate requested)\r
+ by olra.theworths.org (Postfix) with ESMTPS id 6C0B4431FAE\r
+ for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 13:54:52 -0800 (PST)\r
+Received: by mail-la0-f52.google.com with SMTP id fq12so688239lab.39\r
+ for <notmuch@notmuchmail.org>; Tue, 15 Jan 2013 13:54:49 -0800 (PST)\r
+X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r
+ d=google.com; s=20120113;\r
+ h=x-received:from:to:cc:subject:date:message-id:x-mailer\r
+ :x-gm-message-state;\r
+ bh=h4tOJC5uO+mtODGWUGEAiW5WzWRAGB6/XT2eJG6Jogk=;\r
+ b=QGZR4mShSoEeFslQzXddO8xvMH0/oxQGY+ovdFuEHudoj+IJDlIJFqWYOEjkqtQCJ4\r
+ np2pA1irxAwy3rvfJh/ftPLYr5Joo1++tF8wwVmz5wJHjSwjv0HhlzhTXpiVQLLG6N5d\r
+ SipW4myTvwyWOEwUT3XYsHcXd8uAt/gPErsXG57TaTGmtUMiH7BOvNSwVanQnLQ7cYE2\r
+ czXhCY0kMB8XKycachRmfNjyFSh/Iy04AjeiiyBOEo0bqJ2CMfeoesDYNM3mvj3/7de5\r
+ c4I9w3bo18S+oa7q3Ev10aX863hi7qYxpM/7v3FXj85b5uNFPnY5O6eQWa6ZqTnp3wpB\r
+ sE4g==\r
+X-Received: by 10.112.82.202 with SMTP id k10mr37740175lby.22.1358286889385;\r
+ Tue, 15 Jan 2013 13:54:49 -0800 (PST)\r
+Received: from localhost (dsl-hkibrasgw4-50df51-27.dhcp.inet.fi.\r
+ [80.223.81.27])\r
+ by mx.google.com with ESMTPS id hc20sm7206757lab.11.2013.01.15.13.54.47\r
+ (version=TLSv1.2 cipher=RC4-SHA bits=128/128);\r
+ Tue, 15 Jan 2013 13:54:48 -0800 (PST)\r
+From: Jani Nikula <jani@nikula.org>\r
+To: notmuch@notmuchmail.org\r
+Subject: [PATCH 0/5] notmuch batch count\r
+Date: Tue, 15 Jan 2013 23:54:42 +0200\r
+Message-Id: <cover.1358273133.git.jani@nikula.org>\r
+X-Mailer: git-send-email 1.7.10.4\r
+X-Gm-Message-State:\r
+ ALoCoQmGsPOSh7SFJq7RpsCQKHrCi7gtN1V9bLuwpxA/STR/pzK+MoIMbWTNp6WTwDcX7H3V1Yw7\r
+X-BeenThere: notmuch@notmuchmail.org\r
+X-Mailman-Version: 2.1.13\r
+Precedence: list\r
+List-Id: "Use and development of the notmuch mail system."\r
+ <notmuch.notmuchmail.org>\r
+List-Unsubscribe: <http://notmuchmail.org/mailman/options/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=unsubscribe>\r
+List-Archive: <http://notmuchmail.org/pipermail/notmuch>\r
+List-Post: <mailto:notmuch@notmuchmail.org>\r
+List-Help: <mailto:notmuch-request@notmuchmail.org?subject=help>\r
+List-Subscribe: <http://notmuchmail.org/mailman/listinfo/notmuch>,\r
+ <mailto:notmuch-request@notmuchmail.org?subject=subscribe>\r
+X-List-Received-Date: Tue, 15 Jan 2013 21:54:57 -0000\r
+\r
+Hi all -\r
+\r
+Notmuch remote usage [1] is a pretty handy way of accessing a notmuch\r
+database on a remote server. However, the more you have saved searches\r
+and tags, the slower notmuch-hello becomes, and it ends up being by and\r
+far the biggest usability issue with remote notmuch. This is because\r
+notmuch-hello issues a separate 'notmuch count' for each saved search\r
+and tag.\r
+\r
+One could argue that notmuch-hello should be fixed somehow, but I chose\r
+to try another route: batch support for notmuch count. This enables\r
+notmuch-hello to get the counts for all the saved searches or tags in a\r
+single call. The performance improvement is huge in remote usage, but\r
+it's not limited to that. Regular local usage benefits from it too, but\r
+it's not as obviously noticeable.\r
+\r
+Here's a script that demonstrates one-by-one count vs. batch count,\r
+locally and over ssh (assuming ssh key authentication is set up), over\r
+10 iterations:\r
+\r
+#!/bin/bash\r
+\r
+echo "tag count:"\r
+notmuch search --output=tags "*" | wc -l\r
+\r
+for remote in "" "ssh example.com"; do\r
+ export remote\r
+ echo "one-by-one count:"\r
+ time sh -c 'for i in `seq 10`; do notmuch search --format=text0 --output=tags "*" | xargs -0 -n 1 -I "{}" $remote notmuch count tag:"{}" > /dev/null; done'\r
+\r
+ echo "batch count:"\r
+ time sh -c 'for i in `seq 10`; do notmuch search --format=text --output=tags "*" | sed "s/.*/tag:\"\0\"/" | $remote notmuch count --batch > /dev/null; done'\r
+done\r
+\r
+And here's the output of it in my setup:\r
+\r
+tag count:\r
+36\r
+one-by-one count:\r
+\r
+real 0m2.349s\r
+user 0m0.552s\r
+sys 0m0.868s\r
+batch count:\r
+\r
+real 0m0.179s\r
+user 0m0.120s\r
+sys 0m0.064s\r
+one-by-one count:\r
+\r
+real 0m56.527s\r
+user 0m1.424s\r
+sys 0m1.164s\r
+batch count:\r
+\r
+real 0m2.407s\r
+user 0m0.068s\r
+sys 0m0.040s\r
+\r
+As can be seen, in local usage (the first pair of results) the speedup\r
+is more than 10x, although one-by-one notmuch count is usually\r
+sufficiently fast. The difference is more noticeable in remote use (the\r
+second pair of results), where the speedup is 20x here, and any\r
+additional, occasional network latency is multiplied by tag count. (That\r
+result is actually faster than usual for me, but it's still 5+ seconds\r
+to display or refresh notmuch-hello.)\r
+\r
+Mark has written a patch that I've been using to switch notmuch-hello to\r
+use batch count. That has made me switch from running notmuch in ssh to\r
+using remote notmuch. The great thing is that we could switch to using\r
+that in Emacs with no special casing for remote usage, and it would\r
+speed things up also in local use. I'm expecting Mark to post his patch\r
+in reply to this series.\r
+\r
+Mark actually wrote the elisp part based on the rough idea prior to any\r
+of this cli plumbing, so I felt obliged to follow up. So thanks Mark!\r
+\r
+\r
+BR,\r
+Jani.\r
+\r
+\r
+[1] http://notmuchmail.org/remoteusage/ (the page could use some\r
+cleanup; it's really not nearly as complicated as the page suggests)\r
+\r
+\r
+Jani Nikula (5):\r
+ cli: remove useless strdup\r
+ cli: extract count printing to a separate function in notmuch count\r
+ cli: add --batch option to notmuch count\r
+ man: document notmuch count --batch and --input options\r
+ test: notmuch count --batch and --input options\r
+\r
+ man/man1/notmuch-count.1 | 20 +++++++++\r
+ notmuch-count.c | 111 +++++++++++++++++++++++++++++++++++-----------\r
+ test/count | 46 +++++++++++++++++++\r
+ 3 files changed, 150 insertions(+), 27 deletions(-)\r
+\r
+-- \r
+1.7.10.4\r
+\r
projects / notmuch-archives.git / commitdiff