From 7637bcd7396d3ff99ebb8e6621705bf4d4114217 Mon Sep 17 00:00:00 2001 From: Jani Nikula Date: Tue, 15 Jan 2013 23:54:42 +0200 Subject: [PATCH] [PATCH 0/5] notmuch batch count --- 50/8f6f0bd9630f7939bf6ddc13630242318cba18 | 164 ++++++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 50/8f6f0bd9630f7939bf6ddc13630242318cba18 diff --git a/50/8f6f0bd9630f7939bf6ddc13630242318cba18 b/50/8f6f0bd9630f7939bf6ddc13630242318cba18 new file mode 100644 index 000000000..40646b8af --- /dev/null +++ b/50/8f6f0bd9630f7939bf6ddc13630242318cba18 @@ -0,0 +1,164 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id CA6AF431FC2 + for ; Tue, 15 Jan 2013 13:54:56 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: -0.7 +X-Spam-Level: +X-Spam-Status: No, score=-0.7 tagged_above=-999 required=5 + tests=[RCVD_IN_DNSWL_LOW=-0.7] autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id AhpxEAPAqw-c for ; + Tue, 15 Jan 2013 13:54:52 -0800 (PST) +Received: from mail-la0-f52.google.com (mail-la0-f52.google.com + [209.85.215.52]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) + (No client certificate requested) + by olra.theworths.org (Postfix) with ESMTPS id 6C0B4431FAE + for ; Tue, 15 Jan 2013 13:54:52 -0800 (PST) +Received: by mail-la0-f52.google.com with SMTP id fq12so688239lab.39 + for ; Tue, 15 Jan 2013 13:54:49 -0800 (PST) +X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=google.com; s=20120113; + h=x-received:from:to:cc:subject:date:message-id:x-mailer + :x-gm-message-state; + bh=h4tOJC5uO+mtODGWUGEAiW5WzWRAGB6/XT2eJG6Jogk=; + b=QGZR4mShSoEeFslQzXddO8xvMH0/oxQGY+ovdFuEHudoj+IJDlIJFqWYOEjkqtQCJ4 + np2pA1irxAwy3rvfJh/ftPLYr5Joo1++tF8wwVmz5wJHjSwjv0HhlzhTXpiVQLLG6N5d + SipW4myTvwyWOEwUT3XYsHcXd8uAt/gPErsXG57TaTGmtUMiH7BOvNSwVanQnLQ7cYE2 + czXhCY0kMB8XKycachRmfNjyFSh/Iy04AjeiiyBOEo0bqJ2CMfeoesDYNM3mvj3/7de5 + c4I9w3bo18S+oa7q3Ev10aX863hi7qYxpM/7v3FXj85b5uNFPnY5O6eQWa6ZqTnp3wpB + sE4g== +X-Received: by 10.112.82.202 with SMTP id k10mr37740175lby.22.1358286889385; + Tue, 15 Jan 2013 13:54:49 -0800 (PST) +Received: from localhost (dsl-hkibrasgw4-50df51-27.dhcp.inet.fi. + [80.223.81.27]) + by mx.google.com with ESMTPS id hc20sm7206757lab.11.2013.01.15.13.54.47 + (version=TLSv1.2 cipher=RC4-SHA bits=128/128); + Tue, 15 Jan 2013 13:54:48 -0800 (PST) +From: Jani Nikula +To: notmuch@notmuchmail.org +Subject: [PATCH 0/5] notmuch batch count +Date: Tue, 15 Jan 2013 23:54:42 +0200 +Message-Id: +X-Mailer: git-send-email 1.7.10.4 +X-Gm-Message-State: + ALoCoQmGsPOSh7SFJq7RpsCQKHrCi7gtN1V9bLuwpxA/STR/pzK+MoIMbWTNp6WTwDcX7H3V1Yw7 +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 15 Jan 2013 21:54:57 -0000 + +Hi all - + +Notmuch remote usage [1] is a pretty handy way of accessing a notmuch +database on a remote server. However, the more you have saved searches +and tags, the slower notmuch-hello becomes, and it ends up being by and +far the biggest usability issue with remote notmuch. This is because +notmuch-hello issues a separate 'notmuch count' for each saved search +and tag. + +One could argue that notmuch-hello should be fixed somehow, but I chose +to try another route: batch support for notmuch count. This enables +notmuch-hello to get the counts for all the saved searches or tags in a +single call. The performance improvement is huge in remote usage, but +it's not limited to that. Regular local usage benefits from it too, but +it's not as obviously noticeable. + +Here's a script that demonstrates one-by-one count vs. batch count, +locally and over ssh (assuming ssh key authentication is set up), over +10 iterations: + +#!/bin/bash + +echo "tag count:" +notmuch search --output=tags "*" | wc -l + +for remote in "" "ssh example.com"; do + export remote + echo "one-by-one count:" + time sh -c 'for i in `seq 10`; do notmuch search --format=text0 --output=tags "*" | xargs -0 -n 1 -I "{}" $remote notmuch count tag:"{}" > /dev/null; done' + + echo "batch count:" + time sh -c 'for i in `seq 10`; do notmuch search --format=text --output=tags "*" | sed "s/.*/tag:\"\0\"/" | $remote notmuch count --batch > /dev/null; done' +done + +And here's the output of it in my setup: + +tag count: +36 +one-by-one count: + +real 0m2.349s +user 0m0.552s +sys 0m0.868s +batch count: + +real 0m0.179s +user 0m0.120s +sys 0m0.064s +one-by-one count: + +real 0m56.527s +user 0m1.424s +sys 0m1.164s +batch count: + +real 0m2.407s +user 0m0.068s +sys 0m0.040s + +As can be seen, in local usage (the first pair of results) the speedup +is more than 10x, although one-by-one notmuch count is usually +sufficiently fast. The difference is more noticeable in remote use (the +second pair of results), where the speedup is 20x here, and any +additional, occasional network latency is multiplied by tag count. (That +result is actually faster than usual for me, but it's still 5+ seconds +to display or refresh notmuch-hello.) + +Mark has written a patch that I've been using to switch notmuch-hello to +use batch count. That has made me switch from running notmuch in ssh to +using remote notmuch. The great thing is that we could switch to using +that in Emacs with no special casing for remote usage, and it would +speed things up also in local use. I'm expecting Mark to post his patch +in reply to this series. + +Mark actually wrote the elisp part based on the rough idea prior to any +of this cli plumbing, so I felt obliged to follow up. So thanks Mark! + + +BR, +Jani. + + +[1] http://notmuchmail.org/remoteusage/ (the page could use some +cleanup; it's really not nearly as complicated as the page suggests) + + +Jani Nikula (5): + cli: remove useless strdup + cli: extract count printing to a separate function in notmuch count + cli: add --batch option to notmuch count + man: document notmuch count --batch and --input options + test: notmuch count --batch and --input options + + man/man1/notmuch-count.1 | 20 +++++++++ + notmuch-count.c | 111 +++++++++++++++++++++++++++++++++++----------- + test/count | 46 +++++++++++++++++++ + 3 files changed, 150 insertions(+), 27 deletions(-) + +-- +1.7.10.4 + -- 2.26.2