From: Tomi Ollila <tomi.ollila@iki.fi>
To: Jani Nikula <jani@nikula.org>, Mark Walters <markwalters1009@gmail.com>,
	notmuch@notmuchmail.org
Subject: Re: [PATCH 0/5] notmuch batch count
In-Reply-To: <87fw1u30zz.fsf@nikula.org>
References: <cover.1358273133.git.jani@nikula.org> <8738y2ui4y.fsf@qmul.ac.uk>
	<m27gndsotu.fsf@guru.guru-group.fi> <87fw1u30zz.fsf@nikula.org>
User-Agent: Notmuch/0.15+11~ge1e719d (http://notmuchmail.org) Emacs/24.2.1
	(x86_64-unknown-linux-gnu)
Date: Tue, 22 Jan 2013 15:43:07 +0200
Message-ID: <m2r4ldjpuc.fsf@guru.guru-group.fi>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: list

On Mon, Jan 21 2013, Jani Nikula <jani@nikula.org> wrote:

> On Wed, 16 Jan 2013, Tomi Ollila <tomi.ollila@iki.fi> wrote:
>> One could argue that we'd should send json "documents" to notmuch in
>> stdin and notmuch would output json(/sexp) "documents". That is just
>> SMOP. I bet Austin would like this solution, especially the part
>> that involves writing or integrating json parser >;). 
>> I'd be happy with this 'batch' approach. 
>>
>> I'll be testing this soon, but refrain from reviewing the code
>> until 0.15 is out.
>
> id:87a9s5cp38.fsf@zancas.localnet ;)

OK :D

I started testing:

First a remote script to prepare:

cat > remote-notmuch-test.sh << EOF
#!/bin/bash
printf -v ARGS '%q ' "$@" # bash feature
ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS
EOF
chmod 755 remote-notmuch-test.sh

Note 2 things about the script:

    1)	'~' is in quotes so shell doesn't expand it -- ssh uses /etc/passwd
        instead of environment variable to get HOME (security feature).
        This can be easly tested by prefixing the command with HOME=/tmp.
    2)	The remote address '0.1' is ignored when control socket is active
 	if control socket is down, connection will fail (0.0 connects to
        localhost (I'm not sure which interface in that case)).

Another "script", for emacs:

cat > remote-notmuch-test.el << EOF
(load "notmuch")
(setq notmuch-command "/home/too/remote-notmuch-test.sh")

(setq notmuch-saved-searches '(("inbox" . "tag:inbox")
                               ("unread" . "tag:unread")
                               ("contrib" . "tag:notmuch::contrib")
                               ("pushed" . "tag:notmuch::pushed")
                               ("needs-review" . "tag:notmuch::needs-review")
                               ("moreinfo" . "tag:notmuch::moreinfo")
                               ("stale" . "tag:notmuch::stale")
                               ("wip" . "tag:notmuch::wip")
                               ("bug" . "tag:notmuch::bug")
                               ("obsolete" . "tag:notmuch::obsolete")))

(notmuch)
EOF

First ssh connection with ControlMaster needs to be started
from one terminal:

ssh -S '~'/.ssh/master-notmuch-remote1 -M remote.host.tld

Then, from another terminal, emacs:

emacs -l remote-notmuch-test.el

works, takes quite a long time -- opening 'all tags' even longer.

Let's edit wrapper script:

cat > remote-notmuch-test.sh << EOF
#!/bin/bash
printf -v ARGS '%q ' "$@" # bash feature
echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log
ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS
EOF
chmod 755 remote-notmuch-test.sh

Now, after emacs -l remote-notmuch-test.el run and all tags opened,
exit emacs and run the replay script

env TIME='grepme %Us user %Ss system %P cpu %E total' /usr/bin/time sh ../notmuch-replay.1744.log
last line outputs:
grepme 0.38s user 0.40s system 1% cpu 0:48.21 total

and wc notmuch-replay.1744.log
  61  240 3787 notmuch-replay.1744.log

and each of the individual command in script printed either

grepme 0.00s user 0.00s system 1% cpu 0:00.78 total or
grepme 0.00s user 0.00s system 1% cpu 0:00.79 total.

( 0.785 * 61 = 47.9 so notmuch was wasted elsewhere :)

---

Ok, now patching 2 notmuch trees, one in local and one in
remote machine and building and installing those.

Running emacs -l remote-notmuch-test.el 
was fast (not in one second, but just a few).

Unsuprisingly, the "replay" script cannot be run anymore
as it requires input from stdin, therefore tuning it as follows

cat > remote-notmuch-test.sh << EOF
#!/bin/bash
printf -v ARGS '%q ' "$@" # bash feature
echo `date`: $ARGS >> notmuch-dates.$PPID.log
echo /usr/bin/time ./remote-notmuch-test.sh $ARGS >> notmuch-replay.$PPID.log
ssh -S '~'/.ssh/master-notmuch-remote1 -oControlMaster=no 0.1 notmuch $ARGS
date >> notmuch-dates.$PPID.log
EOF
chmod 755 remote-notmuch-test.sh

now, again: emacs -l remote-notmuch-test.el 
then 'echo output of echo : >> notmuch-dates.2678.log'
and finally opening 'all tags' 

cat notmuch-dates.2678.log

Tue 22 Jan 15:04:34 EET 2013: count
Tue 22 Jan 15:04:35 EET 2013
Tue 22 Jan 15:04:35 EET 2013: count --batch
Tue 22 Jan 15:04:36 EET 2013
Tue 22 Jan 15:04:36 EET 2013: search-tags
Tue 22 Jan 15:04:37 EET 2013
output of echo :
Tue 22 Jan 15:04:45 EET 2013: count
Tue 22 Jan 15:04:45 EET 2013
Tue 22 Jan 15:04:45 EET 2013: count --batch
Tue 22 Jan 15:04:47 EET 2013
Tue 22 Jan 15:04:47 EET 2013: search-tags
Tue 22 Jan 15:04:47 EET 2013
Tue 22 Jan 15:04:47 EET 2013: count --batch
Tue 22 Jan 15:04:49 EET 2013

3 seconds for first tags and 4 seconds for second tags (approximately :)
-- quite an improvement 48 seconds.

Also, changes work fine. Now that I forgot my laptop charger to office
I'll have to wait until tomorrow to do the core review.

(maybe the 2 'count's could be combined in elisp code ;)

>
> J.

Tomi