From 33be35b396da6e70be5b5ddab54d400b84a42680 Mon Sep 17 00:00:00 2001 From: Tomi Ollila Date: Tue, 22 Jan 2013 15:43:07 +0200 Subject: [PATCH] Re: [PATCH 0/5] notmuch batch count --- 8b/aa33affc8104607de690a1d010dd62edfea6c1 | 201 ++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100644 8b/aa33affc8104607de690a1d010dd62edfea6c1 diff --git a/8b/aa33affc8104607de690a1d010dd62edfea6c1 b/8b/aa33affc8104607de690a1d010dd62edfea6c1 new file mode 100644 index 000000000..7a70446c4 --- /dev/null +++ b/8b/aa33affc8104607de690a1d010dd62edfea6c1 @@ -0,0 +1,201 @@ +Return-Path: +X-Original-To: notmuch@notmuchmail.org +Delivered-To: notmuch@notmuchmail.org +Received: from localhost (localhost [127.0.0.1]) + by olra.theworths.org (Postfix) with ESMTP id 4830A431FAF + for ; Tue, 22 Jan 2013 05:43:21 -0800 (PST) +X-Virus-Scanned: Debian amavisd-new at olra.theworths.org +X-Spam-Flag: NO +X-Spam-Score: 0 +X-Spam-Level: +X-Spam-Status: No, score=0 tagged_above=-999 required=5 tests=[none] + autolearn=disabled +Received: from olra.theworths.org ([127.0.0.1]) + by localhost (olra.theworths.org [127.0.0.1]) (amavisd-new, port 10024) + with ESMTP id Ek585k-Kxx7q for ; + Tue, 22 Jan 2013 05:43:19 -0800 (PST) +Received: from guru.guru-group.fi (guru.guru-group.fi [46.183.73.34]) + by olra.theworths.org (Postfix) with ESMTP id 6A13D431FAE + for ; Tue, 22 Jan 2013 05:43:19 -0800 (PST) +Received: from guru.guru-group.fi (localhost [IPv6:::1]) + by guru.guru-group.fi (Postfix) with ESMTP id 66578100086; + Tue, 22 Jan 2013 15:43:07 +0200 (EET) +From: Tomi Ollila +To: Jani Nikula , Mark Walters , + notmuch@notmuchmail.org +Subject: Re: [PATCH 0/5] notmuch batch count +In-Reply-To: <87fw1u30zz.fsf@nikula.org> +References: <8738y2ui4y.fsf@qmul.ac.uk> + <87fw1u30zz.fsf@nikula.org> +User-Agent: Notmuch/0.15+11~ge1e719d (http://notmuchmail.org) Emacs/24.2.1 + (x86_64-unknown-linux-gnu) +X-Face: HhBM'cA~ +MIME-Version: 1.0 +Content-Type: text/plain +X-BeenThere: notmuch@notmuchmail.org +X-Mailman-Version: 2.1.13 +Precedence: list +List-Id: "Use and development of the notmuch mail system." + +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Tue, 22 Jan 2013 13:43:21 -0000 + +On Mon, Jan 21 2013, Jani Nikula wrote: + +> On Wed, 16 Jan 2013, Tomi Ollila wrote: +>> One could argue that we'd should send json "documents" to notmuch in +>> stdin and notmuch would output json(/sexp) "documents". That is just +>> SMOP. I bet Austin would like this solution, especially the part +>> that involves writing or integrating json parser >;). +>> I'd be happy with this 'batch' approach. +>> +>> I'll be testing this soon, but refrain from reviewing the code +>> until 0.15 is out. +> +> id:87a9s5cp38.fsf@zancas.localnet ;) + +OK :D + +I started testing: + +First a remote script to prepare: + +cat > remote-notmuch-test.sh << EOF +#!/bin/bash +printf -v ARGS '%q ' "$@" # bash feature +ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS +EOF +chmod 755 remote-notmuch-test.sh + +Note 2 things about the script: + + 1) '~' is in quotes so shell doesn't expand it -- ssh uses /etc/passwd + instead of environment variable to get HOME (security feature). + This can be easly tested by prefixing the command with HOME=/tmp. + 2) The remote address '0.1' is ignored when control socket is active + if control socket is down, connection will fail (0.0 connects to + localhost (I'm not sure which interface in that case)). + +Another "script", for emacs: + +cat > remote-notmuch-test.el << EOF +(load "notmuch") +(setq notmuch-command "/home/too/remote-notmuch-test.sh") + +(setq notmuch-saved-searches '(("inbox" . "tag:inbox") + ("unread" . "tag:unread") + ("contrib" . "tag:notmuch::contrib") + ("pushed" . "tag:notmuch::pushed") + ("needs-review" . "tag:notmuch::needs-review") + ("moreinfo" . "tag:notmuch::moreinfo") + ("stale" . "tag:notmuch::stale") + ("wip" . "tag:notmuch::wip") + ("bug" . "tag:notmuch::bug") + ("obsolete" . "tag:notmuch::obsolete"))) + +(notmuch) +EOF + +First ssh connection with ControlMaster needs to be started +from one terminal: + +ssh -S '~'/.ssh/master-notmuch-remote1 -M remote.host.tld + +Then, from another terminal, emacs: + +emacs -l remote-notmuch-test.el + +works, takes quite a long time -- opening 'all tags' even longer. + +Let's edit wrapper script: + +cat > remote-notmuch-test.sh << EOF +#!/bin/bash +printf -v ARGS '%q ' "$@" # bash feature +echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log +ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS +EOF +chmod 755 remote-notmuch-test.sh + +Now, after emacs -l remote-notmuch-test.el run and all tags opened, +exit emacs and run the replay script + +env TIME='grepme %Us user %Ss system %P cpu %E total' /usr/bin/time sh ../notmuch-replay.1744.log +last line outputs: +grepme 0.38s user 0.40s system 1% cpu 0:48.21 total + +and wc notmuch-replay.1744.log + 61 240 3787 notmuch-replay.1744.log + +and each of the individual command in script printed either + +grepme 0.00s user 0.00s system 1% cpu 0:00.78 total or +grepme 0.00s user 0.00s system 1% cpu 0:00.79 total. + +( 0.785 * 61 = 47.9 so notmuch was wasted elsewhere :) + +--- + +Ok, now patching 2 notmuch trees, one in local and one in +remote machine and building and installing those. + +Running emacs -l remote-notmuch-test.el +was fast (not in one second, but just a few). + +Unsuprisingly, the "replay" script cannot be run anymore +as it requires input from stdin, therefore tuning it as follows + +cat > remote-notmuch-test.sh << EOF +#!/bin/bash +printf -v ARGS '%q ' "$@" # bash feature +echo `date`: $ARGS >> notmuch-dates.$PPID.log +echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log +ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS +date >> notmuch-dates.$PPID.log +EOF +chmod 755 remote-notmuch-test.sh + +now, again: emacs -l remote-notmuch-test.el +then 'echo output of echo : >> notmuch-dates.2678.log' +and finally opening 'all tags' + +cat notmuch-dates.2678.log + +Tue 22 Jan 15:04:34 EET 2013: count +Tue 22 Jan 15:04:35 EET 2013 +Tue 22 Jan 15:04:35 EET 2013: count --batch +Tue 22 Jan 15:04:36 EET 2013 +Tue 22 Jan 15:04:36 EET 2013: search-tags +Tue 22 Jan 15:04:37 EET 2013 +output of echo : +Tue 22 Jan 15:04:45 EET 2013: count +Tue 22 Jan 15:04:45 EET 2013 +Tue 22 Jan 15:04:45 EET 2013: count --batch +Tue 22 Jan 15:04:47 EET 2013 +Tue 22 Jan 15:04:47 EET 2013: search-tags +Tue 22 Jan 15:04:47 EET 2013 +Tue 22 Jan 15:04:47 EET 2013: count --batch +Tue 22 Jan 15:04:49 EET 2013 + +3 seconds for first tags and 4 seconds for second tags (approximately :) +-- quite an improvement 48 seconds. + +Also, changes work fine. Now that I forgot my laptop charger to office +I'll have to wait until tomorrow to do the core review. + +(maybe the 2 'count's could be combined in elisp code ;) + +> +> J. + +Tomi -- 2.26.2